<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">How long does it take to your devices (using mode 5 or 6, ALB is prefered for GlusterFS) to take-over the MAC? This can result in your error -&nbsp;&nbsp;"transport endpoint is not connected” - there are some timeouts within gluster set by default.<div class="">I am using LACP and it works without any problem. Can you share your mode 5 / 6 configuration ?</div><div class=""><br class=""></div><div class="">Thanks.</div><div class="">Martin<br class=""><div class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 25 Feb 2019, at 13:44, Jorick Astrego &lt;<a href="mailto:jorick@netbulae.eu" class="">jorick@netbulae.eu</a>&gt; wrote:</div><br class="Apple-interchange-newline"><div class="">
  
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
  
  <div text="#000000" bgcolor="#FFFFFF" class=""><p class="">Hi,</p><p class="">Well no, mode 5 and mode 6 also have fault tollerance and don't
      need any special switch config.</p><p class="">Quick google search:</p><p class=""><a class="moz-txt-link-freetext" href="https://serverfault.com/questions/734246/does-balance-alb-and-balance-tlb-support-fault-tolerance">https://serverfault.com/questions/734246/does-balance-alb-and-balance-tlb-support-fault-tolerance</a><br class="">
    </p>
    <blockquote class=""><p class="">Bonding Mode 5 (balance-tlb) works by looking at all the
        devices in the bond, and sending out the slave with the least
        current traffic load. Traffic is only received by one slave (the
        "primary slave"). If a slave is lost, that slave is not
        considered for transmission, so this mode is fault-tolerant.</p><p class="">Bonding Mode 6 (balance-alb) works as above, except incoming
        ARP requests are intercepted by the bonding driver, and the
        bonding driver generates ARP replies so that external hosts are
        tricked into sending their traffic into one of the other bonding
        slaves instead of the primary slave. If many hosts in the same
        broadcast domain contact the bond, then traffic should balance
        roughly evenly into all slaves.</p><p class="">If a slave is lost in Mode 6, then it may take some time for a
        remote host to time out its ARP table entry and send a new ARP
        request. A TCP or SCTP retransmission tents to lead into ARP
        request fairly quickly, but a UDP datagram does not, and will
        rely on the usual ARP table refresh. So Mode 6 <em class="">is</em>
        fault tolerant, but convergence on slave loss may take some time
        depending on the Layer 4 protocol used.</p><p class="">If you are worried about fast fault tolerance, then consider
        using Mode 4 (802.3ad aka LACP) which negotiates link
        aggregation between the bond and the switch, and constantly
        updates the link status between the aggregation partners. Mode 4
        also has configurable load balance hashing so is better for
        in-order delivery of TCP streams compared to Mode 5 or Mode 6.</p>
    </blockquote><p class=""><a class="moz-txt-link-freetext" href="https://wiki.linuxfoundation.org/networking/bonding">https://wiki.linuxfoundation.org/networking/bonding</a><br class="">
    </p>
    <ul class="">
      <li class="level2">
        <div class="li"> <strong class=""> balance-tlb or 5</strong><br class="">
          Adaptive transmit load balancing: channel bonding that does
          not require any special switch support. The outgoing traffic
          is distributed according to the current load (computed
          relative to the speed) on each slave. Incoming traffic is
          received by the current slave. <b class="">If the receiving slave
            fails, another slave takes over the MAC address of the
            failed receiving slave.</b></div>
        <ul class="">
          <li class="level3">
            <div class="li"> Prerequisite:</div>
            <ol class="">
              <li class="level4">
                <div class="li"> Ethtool support in the base drivers for
                  retrieving the speed of each slave.</div>
              </li>
            </ol>
          </li>
        </ul>
      </li>
      <li class="level2">
        <div class="li"> <strong class=""> balance-alb or 6&nbsp;</strong><br class="">
          Adaptive load balancing: <b class="">includes balance-tlb plus receive
            load balancing</b> (rlb) for IPV4 traffic, and does not
          require any special switch support. The receive load balancing
          is achieved by ARP negotiation.</div>
        <ul class="">
          <li class="level3">
            <div class="li"> The bonding driver intercepts the ARP
              Replies sent by the local system on their way out and
              overwrites the source hardware address with the unique
              hardware address of one of the slaves in the bond such
              that different peers use different hardware addresses for
              the server.</div>
          </li>
          <li class="level3">
            <div class="li"> Receive traffic from connections created by
              the server is also balanced. When the local system sends
              an ARP Request the bonding driver copies and saves the
              peer's IP information from the ARP packet.</div>
          </li>
          <li class="level3">
            <div class="li"> When the ARP Reply arrives from the peer,
              its hardware address is retrieved and the bonding driver
              initiates an ARP reply to this peer assigning it to one of
              the slaves in the bond.</div>
          </li>
          <li class="level3">
            <div class="li"> A problematic outcome of using ARP
              negotiation for balancing is that each time that an ARP
              request is broadcast it uses the hardware address of the
              bond. Hence, peers learn the hardware address of the bond
              and the balancing of receive traffic collapses to the
              current slave. This is handled by sending updates (ARP
              Replies) to all the peers with their individually assigned
              hardware address such that the traffic is redistributed.
              Receive traffic is also redistributed when a new slave is
              added to the bond and when an inactive slave is
              re-activated. The receive load is distributed sequentially
              (round robin) among the group of highest speed slaves in
              the bond.</div>
          </li>
          <li class="level3">
            <div class="li"> When a link is reconnected or a new slave
              joins the bond the receive traffic is redistributed among
              all active slaves in the bond by initiating ARP Replies
              with the selected mac address to each of the clients. The
              updelay parameter (detailed below) must be set to a value
              equal or greater than the switch's forwarding delay so
              that the ARP Replies sent to the peers will not be blocked
              by the switch.</div>
          </li>
        </ul>
      </li>
    </ul>
    <div class="moz-cite-prefix">On 2/25/19 1:16 PM, Martin Toth wrote:<br class="">
    </div>
    <blockquote type="cite" cite="mid:ED57C951-1D9B-47F2-967E-A2F1FB0382AE@gmail.com" class="">
      <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
      Hi Alex,
      <div class=""><br class="">
      </div>
      <div class="">you have to use bond mode 4 (LACP - 802.3ad) in
        order to achieve redundancy of cables/ports/switches. I suppose
        this is what you want.</div>
      <div class=""><br class="">
      </div>
      <div class="">BR,</div>
      <div class="">Martin</div>
      <div class="">
        <div class="">
          <div class=""><br class="">
            <blockquote type="cite" class="">
              <div class="">On 25 Feb 2019, at 11:43, Alex K &lt;<a href="mailto:rightkicktech@gmail.com" class="" moz-do-not-send="true">rightkicktech@gmail.com</a>&gt;
                wrote:</div>
              <br class="Apple-interchange-newline">
              <div class="">
                <div dir="ltr" class="">
                  <div class="">Hi All, <br class="">
                  </div>
                  <div class=""><br class="">
                  </div>
                  <div class="">I was asking if it is possible to have
                    the two separate cables connected to two different
                    physical switched. When trying mode6 or mode1 in
                    this setup gluster was refusing to start the
                    volumes, giving me "transport endpoint is not
                    connected". <br class="">
                  </div>
                  <div class=""><br class="">
                  </div>
                  <div class="">server1: cable1 ---------------- switch1
                    --------------------- server2: cable1<br class="">
                  </div>
                  <div class="">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
                    &nbsp;&nbsp; |<br class="">
                  </div>
                  <div class="">server1: cable2 ---------------- switch2
                    --------------------- server2: cable2<br class="">
                  </div>
                  <div class=""><br class="">
                  </div>
                  <div class="">Both switches are connected with each
                    other also. This is done to achieve redundancy for
                    the switches. <br class="">
                  </div>
                  <div class="">When disconnecting cable2 from both
                    servers, then gluster was happy. <br class="">
                  </div>
                  <div class="">What could be the&nbsp;problem?<br class="">
                  </div>
                  <div class=""><br class="">
                  </div>
                  <div class="">Thanx,</div>
                  <div class="">Alex<br class="">
                  </div>
                  <div class=""><br class="">
                  </div>
                </div>
                <br class="">
                <div class="gmail_quote">
                  <div dir="ltr" class="gmail_attr">On Mon, Feb 25, 2019
                    at 11:32 AM Jorick Astrego &lt;<a href="mailto:jorick@netbulae.eu" class="" moz-do-not-send="true">jorick@netbulae.eu</a>&gt;
                    wrote:<br class="">
                  </div>
                  <blockquote class="gmail_quote" style="margin:0px 0px
                    0px 0.8ex;border-left:1px solid
                    rgb(204,204,204);padding-left:1ex">
                    <div bgcolor="#FFFFFF" class=""><p class="">Hi,</p><p class="">We use bonding mode 6 (balance-alb)
                        for GlusterFS traffic</p><p class=""><a class="gmail-m_-151486401419277343moz-txt-link-freetext" href="https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html/administration_guide/network4" target="_blank" moz-do-not-send="true">https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html/administration_guide/network4</a><br class="">
                      </p>
                      <blockquote class=""><p class="">Preferred bonding mode for Red Hat
                          Gluster Storage client is mode 6
                          (balance-alb), this allows client to transmit
                          writes in parallel on separate NICs much of
                          the time. <br class="">
                        </p>
                      </blockquote><p class="">Regards,</p><p class="">Jorick Astrego<br class="">
                      </p>
                      <div class="gmail-m_-151486401419277343moz-cite-prefix">On
                        2/25/19 5:41 AM, Dmitry Melekhov wrote:<br class="">
                      </div>
                      <blockquote type="cite" class="">
                        <div class="gmail-m_-151486401419277343moz-cite-prefix">23.02.2019
                          19:54, Alex K пишет:<br class="">
                        </div>
                        <blockquote type="cite" class="">
                          <div dir="ltr" class="">
                            <div class="">Hi all, <br class="">
                            </div>
                            <div class=""><br class="">
                            </div>
                            <div class="">I have a replica 3 setup where
                              each server was configured with a dual
                              interfaces in mode 6 bonding. All cables
                              were connected to one common network
                              switch. <br class="">
                            </div>
                            <div class=""><br class="">
                            </div>
                            <div class="">To add redundancy to the
                              switch, and avoid being a single point of
                              failure, I connected each second cable of
                              each server to a second switch. This
                              turned out to not function as gluster was
                              refusing to start the volume logging
                              "transport endpoint is disconnected"
                              although all nodes were able to reach each
                              other (ping) in the storage network. I
                              switched the mode to mode 1
                              (active/passive) and initially it worked
                              but following a reboot of all cluster same
                              issue appeared. Gluster is not starting
                              the volumes. <br class="">
                            </div>
                            <div class=""><br class="">
                            </div>
                            <div class="">Isn't active/passive supposed
                              to work like that? Can one have such
                              redundant network setup or are there any
                              other recommended approaches?</div>
                            <div class=""><br class="">
                            </div>
                          </div>
                        </blockquote><p class=""><br class="">
                        </p><p class="">Yes, we use lacp, I guess this is
                          mode 4 ( we use teamd ), it is, no doubt, best
                          way.</p><p class=""><br class="">
                        </p>
                        <blockquote type="cite" class="">
                          <div dir="ltr" class="">
                            <div class="">Thanx, <br class="">
                            </div>
                            <div class="">Alex<br class="">
                            </div>
                          </div>
                          <br class="">
                          <fieldset class="gmail-m_-151486401419277343mimeAttachmentHeader"></fieldset>
                          <pre class="gmail-m_-151486401419277343moz-quote-pre">_______________________________________________
Gluster-users mailing list
<a class="gmail-m_-151486401419277343moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org" target="_blank" moz-do-not-send="true">Gluster-users@gluster.org</a>
<a class="gmail-m_-151486401419277343moz-txt-link-freetext" href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank" moz-do-not-send="true">https://lists.gluster.org/mailman/listinfo/gluster-users</a></pre>
                        </blockquote><p class=""><br class="">
                        </p>
                        <br class="">
                        <fieldset class="gmail-m_-151486401419277343mimeAttachmentHeader"></fieldset>
                        <pre class="gmail-m_-151486401419277343moz-quote-pre">_______________________________________________
Gluster-users mailing list
<a class="gmail-m_-151486401419277343moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org" target="_blank" moz-do-not-send="true">Gluster-users@gluster.org</a>
<a class="gmail-m_-151486401419277343moz-txt-link-freetext" href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank" moz-do-not-send="true">https://lists.gluster.org/mailman/listinfo/gluster-users</a></pre>
                      </blockquote>
                      <br class="">
                      <br class="">
                      <br class="">
                      <br class="">
                      <span style="color:rgb(96,76,120)" class=""><font class=""><span class="" lang="NL">Met
                            vriendelijke groet, With kind regards,<br class="">
                            <br class="">
                            Jorick Astrego<br class="">
                          </span></font></span><b style="color:rgb(96,76,120)" class=""><br class="">
                        Netbulae Virtualization Experts </b><br class="">
                      <hr style="border-color:rgb(204,204,204)
                        currentcolor currentcolor;border-style:solid
                        none none;border-width:1px medium medium" class="">
                      <table style="width:522px" class="">
                        <tbody class="">
                          <tr class="">
                            <td style="width:130px;font-size:10px" class="">Tel: 053 20 30 270</td>
                            <td style="width:130px;font-size:10px" class=""><a href="mailto:info@netbulae.eu" target="_blank" class="" moz-do-not-send="true">info@netbulae.eu</a></td>
                            <td style="width:130px;font-size:10px" class="">Staalsteden 4-3A</td>
                            <td style="width:130px;font-size:10px" class="">KvK 08198180</td>
                          </tr>
                          <tr class="">
                            <td style="width:130px;font-size:10px" class="">Fax: 053 20 30 271</td>
                            <td style="width:130px;font-size:10px" class=""><a href="http://www.netbulae.eu/" target="_blank" class="" moz-do-not-send="true">www.netbulae.eu</a></td>
                            <td style="width:130px;font-size:10px" class="">7547 TA Enschede</td>
                            <td style="width:130px;font-size:10px" class="">BTW NL821234584B01</td>
                          </tr>
                        </tbody>
                      </table>
                      <br class="">
                      <hr style="border-color:rgb(204,204,204)
                        currentcolor currentcolor;border-style:solid
                        none none;border-width:1px medium medium" class=""><br class="">
                    </div>
                    _______________________________________________<br class="">
                    Gluster-users mailing list<br class="">
                    <a href="mailto:Gluster-users@gluster.org" target="_blank" class="" moz-do-not-send="true">Gluster-users@gluster.org</a><br class="">
                    <a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank" class="" moz-do-not-send="true">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote>
                </div>
                _______________________________________________<br class="">
                Gluster-users mailing list<br class="">
                <a href="mailto:Gluster-users@gluster.org" class="" moz-do-not-send="true">Gluster-users@gluster.org</a><br class="">
                <a class="moz-txt-link-freetext" href="https://lists.gluster.org/mailman/listinfo/gluster-users">https://lists.gluster.org/mailman/listinfo/gluster-users</a></div>
            </blockquote>
          </div>
          <br class="">
        </div>
      </div>
      <br class="">
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <pre class="moz-quote-pre" wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="https://lists.gluster.org/mailman/listinfo/gluster-users">https://lists.gluster.org/mailman/listinfo/gluster-users</a></pre>
    </blockquote>
  <br class="">
<br class="">
<b style="color:#604c78" class=""></b><br class=""><br class=""><span style="color:#604c78;" class=""><font class=""><span style="mso-fareast-language:en-gb;" lang="NL" class="">Met vriendelijke groet, With kind regards,<br class=""><br class="">Jorick Astrego<br class=""></span></font></span><b style="color:#604c78" class=""><br class="">Netbulae Virtualization Experts </b><br class=""><hr style="border:none;border-top:1px solid #ccc;" class=""><table style="width: 522px" class=""><tbody class=""><tr class=""><td style="width: 130px;font-size: 10px" class="">Tel:  053 20 30 270</td>    <td style="width: 130px;font-size: 10px" class=""><a href="mailto:info@netbulae.eu" class="">info@netbulae.eu</a></td>    <td style="width: 130px;font-size: 10px" class="">Staalsteden 4-3A</td>    <td style="width: 130px;font-size: 10px" class="">KvK 08198180</td></tr><tr class="">    <td style="width: 130px;font-size: 10px" class="">Fax: 053 20 30 271</td>    <td style="width: 130px;font-size: 10px" class=""><a href="http://www.netbulae.eu" class="">www.netbulae.eu</a></td>    <td style="width: 130px;font-size: 10px" class="">7547 TA Enschede</td>    <td style="width: 130px;font-size: 10px" class="">BTW NL821234584B01</td></tr></tbody></table><br class=""><hr style="border:none;border-top:1px solid #ccc;" class=""><br class="">
</div>


_______________________________________________<br class="">Gluster-users mailing list<br class=""><a href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class="">https://lists.gluster.org/mailman/listinfo/gluster-users</div></blockquote></div><br class=""></div></div></body></html>