<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">How long does it take to your devices (using mode 5 or 6, ALB is prefered for GlusterFS) to take-over the MAC? This can result in your error - "transport endpoint is not connected” - there are some timeouts within gluster set by default.<div class="">I am using LACP and it works without any problem. Can you share your mode 5 / 6 configuration ?</div><div class=""><br class=""></div><div class="">Thanks.</div><div class="">Martin<br class=""><div class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 25 Feb 2019, at 13:44, Jorick Astrego <<a href="mailto:jorick@netbulae.eu" class="">jorick@netbulae.eu</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
<div text="#000000" bgcolor="#FFFFFF" class=""><p class="">Hi,</p><p class="">Well no, mode 5 and mode 6 also have fault tollerance and don't
need any special switch config.</p><p class="">Quick google search:</p><p class=""><a class="moz-txt-link-freetext" href="https://serverfault.com/questions/734246/does-balance-alb-and-balance-tlb-support-fault-tolerance">https://serverfault.com/questions/734246/does-balance-alb-and-balance-tlb-support-fault-tolerance</a><br class="">
</p>
<blockquote class=""><p class="">Bonding Mode 5 (balance-tlb) works by looking at all the
devices in the bond, and sending out the slave with the least
current traffic load. Traffic is only received by one slave (the
"primary slave"). If a slave is lost, that slave is not
considered for transmission, so this mode is fault-tolerant.</p><p class="">Bonding Mode 6 (balance-alb) works as above, except incoming
ARP requests are intercepted by the bonding driver, and the
bonding driver generates ARP replies so that external hosts are
tricked into sending their traffic into one of the other bonding
slaves instead of the primary slave. If many hosts in the same
broadcast domain contact the bond, then traffic should balance
roughly evenly into all slaves.</p><p class="">If a slave is lost in Mode 6, then it may take some time for a
remote host to time out its ARP table entry and send a new ARP
request. A TCP or SCTP retransmission tents to lead into ARP
request fairly quickly, but a UDP datagram does not, and will
rely on the usual ARP table refresh. So Mode 6 <em class="">is</em>
fault tolerant, but convergence on slave loss may take some time
depending on the Layer 4 protocol used.</p><p class="">If you are worried about fast fault tolerance, then consider
using Mode 4 (802.3ad aka LACP) which negotiates link
aggregation between the bond and the switch, and constantly
updates the link status between the aggregation partners. Mode 4
also has configurable load balance hashing so is better for
in-order delivery of TCP streams compared to Mode 5 or Mode 6.</p>
</blockquote><p class=""><a class="moz-txt-link-freetext" href="https://wiki.linuxfoundation.org/networking/bonding">https://wiki.linuxfoundation.org/networking/bonding</a><br class="">
</p>
<ul class="">
<li class="level2">
<div class="li"> <strong class=""> balance-tlb or 5</strong><br class="">
Adaptive transmit load balancing: channel bonding that does
not require any special switch support. The outgoing traffic
is distributed according to the current load (computed
relative to the speed) on each slave. Incoming traffic is
received by the current slave. <b class="">If the receiving slave
fails, another slave takes over the MAC address of the
failed receiving slave.</b></div>
<ul class="">
<li class="level3">
<div class="li"> Prerequisite:</div>
<ol class="">
<li class="level4">
<div class="li"> Ethtool support in the base drivers for
retrieving the speed of each slave.</div>
</li>
</ol>
</li>
</ul>
</li>
<li class="level2">
<div class="li"> <strong class=""> balance-alb or 6 </strong><br class="">
Adaptive load balancing: <b class="">includes balance-tlb plus receive
load balancing</b> (rlb) for IPV4 traffic, and does not
require any special switch support. The receive load balancing
is achieved by ARP negotiation.</div>
<ul class="">
<li class="level3">
<div class="li"> The bonding driver intercepts the ARP
Replies sent by the local system on their way out and
overwrites the source hardware address with the unique
hardware address of one of the slaves in the bond such
that different peers use different hardware addresses for
the server.</div>
</li>
<li class="level3">
<div class="li"> Receive traffic from connections created by
the server is also balanced. When the local system sends
an ARP Request the bonding driver copies and saves the
peer's IP information from the ARP packet.</div>
</li>
<li class="level3">
<div class="li"> When the ARP Reply arrives from the peer,
its hardware address is retrieved and the bonding driver
initiates an ARP reply to this peer assigning it to one of
the slaves in the bond.</div>
</li>
<li class="level3">
<div class="li"> A problematic outcome of using ARP
negotiation for balancing is that each time that an ARP
request is broadcast it uses the hardware address of the
bond. Hence, peers learn the hardware address of the bond
and the balancing of receive traffic collapses to the
current slave. This is handled by sending updates (ARP
Replies) to all the peers with their individually assigned
hardware address such that the traffic is redistributed.
Receive traffic is also redistributed when a new slave is
added to the bond and when an inactive slave is
re-activated. The receive load is distributed sequentially
(round robin) among the group of highest speed slaves in
the bond.</div>
</li>
<li class="level3">
<div class="li"> When a link is reconnected or a new slave
joins the bond the receive traffic is redistributed among
all active slaves in the bond by initiating ARP Replies
with the selected mac address to each of the clients. The
updelay parameter (detailed below) must be set to a value
equal or greater than the switch's forwarding delay so
that the ARP Replies sent to the peers will not be blocked
by the switch.</div>
</li>
</ul>
</li>
</ul>
<div class="moz-cite-prefix">On 2/25/19 1:16 PM, Martin Toth wrote:<br class="">
</div>
<blockquote type="cite" cite="mid:ED57C951-1D9B-47F2-967E-A2F1FB0382AE@gmail.com" class="">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" class="">
Hi Alex,
<div class=""><br class="">
</div>
<div class="">you have to use bond mode 4 (LACP - 802.3ad) in
order to achieve redundancy of cables/ports/switches. I suppose
this is what you want.</div>
<div class=""><br class="">
</div>
<div class="">BR,</div>
<div class="">Martin</div>
<div class="">
<div class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 25 Feb 2019, at 11:43, Alex K <<a href="mailto:rightkicktech@gmail.com" class="" moz-do-not-send="true">rightkicktech@gmail.com</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">
<div class="">Hi All, <br class="">
</div>
<div class=""><br class="">
</div>
<div class="">I was asking if it is possible to have
the two separate cables connected to two different
physical switched. When trying mode6 or mode1 in
this setup gluster was refusing to start the
volumes, giving me "transport endpoint is not
connected". <br class="">
</div>
<div class=""><br class="">
</div>
<div class="">server1: cable1 ---------------- switch1
--------------------- server2: cable1<br class="">
</div>
<div class="">
|<br class="">
</div>
<div class="">server1: cable2 ---------------- switch2
--------------------- server2: cable2<br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Both switches are connected with each
other also. This is done to achieve redundancy for
the switches. <br class="">
</div>
<div class="">When disconnecting cable2 from both
servers, then gluster was happy. <br class="">
</div>
<div class="">What could be the problem?<br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Thanx,</div>
<div class="">Alex<br class="">
</div>
<div class=""><br class="">
</div>
</div>
<br class="">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Mon, Feb 25, 2019
at 11:32 AM Jorick Astrego <<a href="mailto:jorick@netbulae.eu" class="" moz-do-not-send="true">jorick@netbulae.eu</a>>
wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0px 0px
0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF" class=""><p class="">Hi,</p><p class="">We use bonding mode 6 (balance-alb)
for GlusterFS traffic</p><p class=""><a class="gmail-m_-151486401419277343moz-txt-link-freetext" href="https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html/administration_guide/network4" target="_blank" moz-do-not-send="true">https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html/administration_guide/network4</a><br class="">
</p>
<blockquote class=""><p class="">Preferred bonding mode for Red Hat
Gluster Storage client is mode 6
(balance-alb), this allows client to transmit
writes in parallel on separate NICs much of
the time. <br class="">
</p>
</blockquote><p class="">Regards,</p><p class="">Jorick Astrego<br class="">
</p>
<div class="gmail-m_-151486401419277343moz-cite-prefix">On
2/25/19 5:41 AM, Dmitry Melekhov wrote:<br class="">
</div>
<blockquote type="cite" class="">
<div class="gmail-m_-151486401419277343moz-cite-prefix">23.02.2019
19:54, Alex K пишет:<br class="">
</div>
<blockquote type="cite" class="">
<div dir="ltr" class="">
<div class="">Hi all, <br class="">
</div>
<div class=""><br class="">
</div>
<div class="">I have a replica 3 setup where
each server was configured with a dual
interfaces in mode 6 bonding. All cables
were connected to one common network
switch. <br class="">
</div>
<div class=""><br class="">
</div>
<div class="">To add redundancy to the
switch, and avoid being a single point of
failure, I connected each second cable of
each server to a second switch. This
turned out to not function as gluster was
refusing to start the volume logging
"transport endpoint is disconnected"
although all nodes were able to reach each
other (ping) in the storage network. I
switched the mode to mode 1
(active/passive) and initially it worked
but following a reboot of all cluster same
issue appeared. Gluster is not starting
the volumes. <br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Isn't active/passive supposed
to work like that? Can one have such
redundant network setup or are there any
other recommended approaches?</div>
<div class=""><br class="">
</div>
</div>
</blockquote><p class=""><br class="">
</p><p class="">Yes, we use lacp, I guess this is
mode 4 ( we use teamd ), it is, no doubt, best
way.</p><p class=""><br class="">
</p>
<blockquote type="cite" class="">
<div dir="ltr" class="">
<div class="">Thanx, <br class="">
</div>
<div class="">Alex<br class="">
</div>
</div>
<br class="">
<fieldset class="gmail-m_-151486401419277343mimeAttachmentHeader"></fieldset>
<pre class="gmail-m_-151486401419277343moz-quote-pre">_______________________________________________
Gluster-users mailing list
<a class="gmail-m_-151486401419277343moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org" target="_blank" moz-do-not-send="true">Gluster-users@gluster.org</a>
<a class="gmail-m_-151486401419277343moz-txt-link-freetext" href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank" moz-do-not-send="true">https://lists.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote><p class=""><br class="">
</p>
<br class="">
<fieldset class="gmail-m_-151486401419277343mimeAttachmentHeader"></fieldset>
<pre class="gmail-m_-151486401419277343moz-quote-pre">_______________________________________________
Gluster-users mailing list
<a class="gmail-m_-151486401419277343moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org" target="_blank" moz-do-not-send="true">Gluster-users@gluster.org</a>
<a class="gmail-m_-151486401419277343moz-txt-link-freetext" href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank" moz-do-not-send="true">https://lists.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br class="">
<br class="">
<br class="">
<br class="">
<span style="color:rgb(96,76,120)" class=""><font class=""><span class="" lang="NL">Met
vriendelijke groet, With kind regards,<br class="">
<br class="">
Jorick Astrego<br class="">
</span></font></span><b style="color:rgb(96,76,120)" class=""><br class="">
Netbulae Virtualization Experts </b><br class="">
<hr style="border-color:rgb(204,204,204)
currentcolor currentcolor;border-style:solid
none none;border-width:1px medium medium" class="">
<table style="width:522px" class="">
<tbody class="">
<tr class="">
<td style="width:130px;font-size:10px" class="">Tel: 053 20 30 270</td>
<td style="width:130px;font-size:10px" class=""><a href="mailto:info@netbulae.eu" target="_blank" class="" moz-do-not-send="true">info@netbulae.eu</a></td>
<td style="width:130px;font-size:10px" class="">Staalsteden 4-3A</td>
<td style="width:130px;font-size:10px" class="">KvK 08198180</td>
</tr>
<tr class="">
<td style="width:130px;font-size:10px" class="">Fax: 053 20 30 271</td>
<td style="width:130px;font-size:10px" class=""><a href="http://www.netbulae.eu/" target="_blank" class="" moz-do-not-send="true">www.netbulae.eu</a></td>
<td style="width:130px;font-size:10px" class="">7547 TA Enschede</td>
<td style="width:130px;font-size:10px" class="">BTW NL821234584B01</td>
</tr>
</tbody>
</table>
<br class="">
<hr style="border-color:rgb(204,204,204)
currentcolor currentcolor;border-style:solid
none none;border-width:1px medium medium" class=""><br class="">
</div>
_______________________________________________<br class="">
Gluster-users mailing list<br class="">
<a href="mailto:Gluster-users@gluster.org" target="_blank" class="" moz-do-not-send="true">Gluster-users@gluster.org</a><br class="">
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank" class="" moz-do-not-send="true">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote>
</div>
_______________________________________________<br class="">
Gluster-users mailing list<br class="">
<a href="mailto:Gluster-users@gluster.org" class="" moz-do-not-send="true">Gluster-users@gluster.org</a><br class="">
<a class="moz-txt-link-freetext" href="https://lists.gluster.org/mailman/listinfo/gluster-users">https://lists.gluster.org/mailman/listinfo/gluster-users</a></div>
</blockquote>
</div>
<br class="">
</div>
</div>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="https://lists.gluster.org/mailman/listinfo/gluster-users">https://lists.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br class="">
<br class="">
<b style="color:#604c78" class=""></b><br class=""><br class=""><span style="color:#604c78;" class=""><font class=""><span style="mso-fareast-language:en-gb;" lang="NL" class="">Met vriendelijke groet, With kind regards,<br class=""><br class="">Jorick Astrego<br class=""></span></font></span><b style="color:#604c78" class=""><br class="">Netbulae Virtualization Experts </b><br class=""><hr style="border:none;border-top:1px solid #ccc;" class=""><table style="width: 522px" class=""><tbody class=""><tr class=""><td style="width: 130px;font-size: 10px" class="">Tel: 053 20 30 270</td> <td style="width: 130px;font-size: 10px" class=""><a href="mailto:info@netbulae.eu" class="">info@netbulae.eu</a></td> <td style="width: 130px;font-size: 10px" class="">Staalsteden 4-3A</td> <td style="width: 130px;font-size: 10px" class="">KvK 08198180</td></tr><tr class=""> <td style="width: 130px;font-size: 10px" class="">Fax: 053 20 30 271</td> <td style="width: 130px;font-size: 10px" class=""><a href="http://www.netbulae.eu" class="">www.netbulae.eu</a></td> <td style="width: 130px;font-size: 10px" class="">7547 TA Enschede</td> <td style="width: 130px;font-size: 10px" class="">BTW NL821234584B01</td></tr></tbody></table><br class=""><hr style="border:none;border-top:1px solid #ccc;" class=""><br class="">
</div>
_______________________________________________<br class="">Gluster-users mailing list<br class=""><a href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class="">https://lists.gluster.org/mailman/listinfo/gluster-users</div></blockquote></div><br class=""></div></div></body></html>