<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi,</p>
<p>Have not measured it as we have been running this way for years
now and haven't experienced any problems with "transport endpoint
is not connected” with this setup.</p>
<p>We used the default options "BONDING_OPTS='mode=6 miimon=100'"</p>
<blockquote>
<dl class="variablelist compact">
<dt><span class="term"> <code class="literal">miimon=<em
class="replaceable">time_in_milliseconds</em> </code> </span></dt>
<dd>
<div class="para"> Specifies (in milliseconds) how often MII
link monitoring occurs. This is useful if high availability
is required because MII is used to verify that the NIC is
active. </div>
</dd>
</dl>
</blockquote>
<p><br>
</p>
<div class="moz-cite-prefix">On 2/25/19 2:22 PM, Martin Toth wrote:<br>
</div>
<blockquote type="cite"
cite="mid:9B82193A-15EF-43F6-882B-BA8E7862770A@gmail.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
How long does it take to your devices (using mode 5 or 6, ALB is
prefered for GlusterFS) to take-over the MAC? This can result in
your error - "transport endpoint is not connected” - there are
some timeouts within gluster set by default.
<div class="">I am using LACP and it works without any problem.
Can you share your mode 5 / 6 configuration ?</div>
<div class=""><br class="">
</div>
<div class="">Thanks.</div>
<div class="">Martin<br class="">
<div class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On 25 Feb 2019, at 13:44, Jorick Astrego
<<a href="mailto:jorick@netbulae.eu" class=""
moz-do-not-send="true">jorick@netbulae.eu</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8" class="">
<div text="#000000" bgcolor="#FFFFFF" class="">
<p class="">Hi,</p>
<p class="">Well no, mode 5 and mode 6 also have fault
tollerance and don't need any special switch config.</p>
<p class="">Quick google search:</p>
<p class=""><a class="moz-txt-link-freetext"
href="https://serverfault.com/questions/734246/does-balance-alb-and-balance-tlb-support-fault-tolerance"
moz-do-not-send="true">https://serverfault.com/questions/734246/does-balance-alb-and-balance-tlb-support-fault-tolerance</a><br
class="">
</p>
<blockquote class="">
<p class="">Bonding Mode 5 (balance-tlb) works by
looking at all the devices in the bond, and
sending out the slave with the least current
traffic load. Traffic is only received by one
slave (the "primary slave"). If a slave is lost,
that slave is not considered for transmission, so
this mode is fault-tolerant.</p>
<p class="">Bonding Mode 6 (balance-alb) works as
above, except incoming ARP requests are
intercepted by the bonding driver, and the bonding
driver generates ARP replies so that external
hosts are tricked into sending their traffic into
one of the other bonding slaves instead of the
primary slave. If many hosts in the same broadcast
domain contact the bond, then traffic should
balance roughly evenly into all slaves.</p>
<p class="">If a slave is lost in Mode 6, then it
may take some time for a remote host to time out
its ARP table entry and send a new ARP request. A
TCP or SCTP retransmission tents to lead into ARP
request fairly quickly, but a UDP datagram does
not, and will rely on the usual ARP table refresh.
So Mode 6 <em class="">is</em> fault tolerant,
but convergence on slave loss may take some time
depending on the Layer 4 protocol used.</p>
<p class="">If you are worried about fast fault
tolerance, then consider using Mode 4 (802.3ad aka
LACP) which negotiates link aggregation between
the bond and the switch, and constantly updates
the link status between the aggregation partners.
Mode 4 also has configurable load balance hashing
so is better for in-order delivery of TCP streams
compared to Mode 5 or Mode 6.</p>
</blockquote>
<p class=""><a class="moz-txt-link-freetext"
href="https://wiki.linuxfoundation.org/networking/bonding"
moz-do-not-send="true">https://wiki.linuxfoundation.org/networking/bonding</a><br
class="">
</p>
<ul class="">
<li class="level2">
<div class="li"> <strong class=""> balance-tlb or
5</strong><br class="">
Adaptive transmit load balancing: channel
bonding that does not require any special switch
support. The outgoing traffic is distributed
according to the current load (computed relative
to the speed) on each slave. Incoming traffic is
received by the current slave. <b class="">If
the receiving slave fails, another slave takes
over the MAC address of the failed receiving
slave.</b></div>
<ul class="">
<li class="level3">
<div class="li"> Prerequisite:</div>
<ol class="">
<li class="level4">
<div class="li"> Ethtool support in the
base drivers for retrieving the speed of
each slave.</div>
</li>
</ol>
</li>
</ul>
</li>
<li class="level2">
<div class="li"> <strong class=""> balance-alb or
6 </strong><br class="">
Adaptive load balancing: <b class="">includes
balance-tlb plus receive load balancing</b>
(rlb) for IPV4 traffic, and does not require any
special switch support. The receive load
balancing is achieved by ARP negotiation.</div>
<ul class="">
<li class="level3">
<div class="li"> The bonding driver intercepts
the ARP Replies sent by the local system on
their way out and overwrites the source
hardware address with the unique hardware
address of one of the slaves in the bond
such that different peers use different
hardware addresses for the server.</div>
</li>
<li class="level3">
<div class="li"> Receive traffic from
connections created by the server is also
balanced. When the local system sends an ARP
Request the bonding driver copies and saves
the peer's IP information from the ARP
packet.</div>
</li>
<li class="level3">
<div class="li"> When the ARP Reply arrives
from the peer, its hardware address is
retrieved and the bonding driver initiates
an ARP reply to this peer assigning it to
one of the slaves in the bond.</div>
</li>
<li class="level3">
<div class="li"> A problematic outcome of
using ARP negotiation for balancing is that
each time that an ARP request is broadcast
it uses the hardware address of the bond.
Hence, peers learn the hardware address of
the bond and the balancing of receive
traffic collapses to the current slave. This
is handled by sending updates (ARP Replies)
to all the peers with their individually
assigned hardware address such that the
traffic is redistributed. Receive traffic is
also redistributed when a new slave is added
to the bond and when an inactive slave is
re-activated. The receive load is
distributed sequentially (round robin) among
the group of highest speed slaves in the
bond.</div>
</li>
<li class="level3">
<div class="li"> When a link is reconnected or
a new slave joins the bond the receive
traffic is redistributed among all active
slaves in the bond by initiating ARP Replies
with the selected mac address to each of the
clients. The updelay parameter (detailed
below) must be set to a value equal or
greater than the switch's forwarding delay
so that the ARP Replies sent to the peers
will not be blocked by the switch.</div>
</li>
</ul>
</li>
</ul>
<div class="moz-cite-prefix">On 2/25/19 1:16 PM,
Martin Toth wrote:<br class="">
</div>
<blockquote type="cite"
cite="mid:ED57C951-1D9B-47F2-967E-A2F1FB0382AE@gmail.com"
class="">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8" class="">
Hi Alex,
<div class=""><br class="">
</div>
<div class="">you have to use bond mode 4 (LACP -
802.3ad) in order to achieve redundancy of
cables/ports/switches. I suppose this is what you
want.</div>
<div class=""><br class="">
</div>
<div class="">BR,</div>
<div class="">Martin</div>
<div class="">
<div class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On 25 Feb 2019, at 11:43, Alex
K <<a
href="mailto:rightkicktech@gmail.com"
class="" moz-do-not-send="true">rightkicktech@gmail.com</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">
<div class="">Hi All, <br class="">
</div>
<div class=""><br class="">
</div>
<div class="">I was asking if it is
possible to have the two separate
cables connected to two different
physical switched. When trying mode6
or mode1 in this setup gluster was
refusing to start the volumes, giving
me "transport endpoint is not
connected". <br class="">
</div>
<div class=""><br class="">
</div>
<div class="">server1: cable1
---------------- switch1
--------------------- server2: cable1<br
class="">
</div>
<div class="">
|<br class="">
</div>
<div class="">server1: cable2
---------------- switch2
--------------------- server2: cable2<br
class="">
</div>
<div class=""><br class="">
</div>
<div class="">Both switches are
connected with each other also. This
is done to achieve redundancy for the
switches. <br class="">
</div>
<div class="">When disconnecting cable2
from both servers, then gluster was
happy. <br class="">
</div>
<div class="">What could be the problem?<br
class="">
</div>
<div class=""><br class="">
</div>
<div class="">Thanx,</div>
<div class="">Alex<br class="">
</div>
<div class=""><br class="">
</div>
</div>
<br class="">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On
Mon, Feb 25, 2019 at 11:32 AM Jorick
Astrego <<a
href="mailto:jorick@netbulae.eu"
class="" moz-do-not-send="true">jorick@netbulae.eu</a>>
wrote:<br class="">
</div>
<blockquote class="gmail_quote"
style="margin:0px 0px
 0px
0.8ex;border-left:1px solid

rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF" class="">
<p class="">Hi,</p>
<p class="">We use bonding mode 6
(balance-alb) for GlusterFS
traffic</p>
<p class=""><a
class="gmail-m_-151486401419277343moz-txt-link-freetext"
href="https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html/administration_guide/network4"
target="_blank"
moz-do-not-send="true">https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html/administration_guide/network4</a><br
class="">
</p>
<blockquote class="">
<p class="">Preferred bonding mode
for Red Hat Gluster Storage
client is mode 6 (balance-alb),
this allows client to transmit
writes in parallel on separate
NICs much of the time. <br
class="">
</p>
</blockquote>
<p class="">Regards,</p>
<p class="">Jorick Astrego<br
class="">
</p>
<div
class="gmail-m_-151486401419277343moz-cite-prefix">On
2/25/19 5:41 AM, Dmitry Melekhov
wrote:<br class="">
</div>
<blockquote type="cite" class="">
<div
class="gmail-m_-151486401419277343moz-cite-prefix">23.02.2019
19:54, Alex K пишет:<br class="">
</div>
<blockquote type="cite" class="">
<div dir="ltr" class="">
<div class="">Hi all, <br
class="">
</div>
<div class=""><br class="">
</div>
<div class="">I have a replica
3 setup where each server
was configured with a dual
interfaces in mode 6
bonding. All cables were
connected to one common
network switch. <br
class="">
</div>
<div class=""><br class="">
</div>
<div class="">To add
redundancy to the switch,
and avoid being a single
point of failure, I
connected each second cable
of each server to a second
switch. This turned out to
not function as gluster was
refusing to start the volume
logging "transport endpoint
is disconnected" although
all nodes were able to reach
each other (ping) in the
storage network. I switched
the mode to mode 1
(active/passive) and
initially it worked but
following a reboot of all
cluster same issue appeared.
Gluster is not starting the
volumes. <br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Isn't
active/passive supposed to
work like that? Can one have
such redundant network setup
or are there any other
recommended approaches?</div>
<div class=""><br class="">
</div>
</div>
</blockquote>
<p class=""><br class="">
</p>
<p class="">Yes, we use lacp, I
guess this is mode 4 ( we use
teamd ), it is, no doubt, best
way.</p>
<p class=""><br class="">
</p>
<blockquote type="cite" class="">
<div dir="ltr" class="">
<div class="">Thanx, <br
class="">
</div>
<div class="">Alex<br class="">
</div>
</div>
<br class="">
<fieldset
class="gmail-m_-151486401419277343mimeAttachmentHeader"></fieldset>
<pre class="gmail-m_-151486401419277343moz-quote-pre">_______________________________________________
Gluster-users mailing list
<a class="gmail-m_-151486401419277343moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org" target="_blank" moz-do-not-send="true">Gluster-users@gluster.org</a>
<a class="gmail-m_-151486401419277343moz-txt-link-freetext" href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank" moz-do-not-send="true">https://lists.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<p class=""><br class="">
</p>
<br class="">
<fieldset
class="gmail-m_-151486401419277343mimeAttachmentHeader"></fieldset>
<pre class="gmail-m_-151486401419277343moz-quote-pre">_______________________________________________
Gluster-users mailing list
<a class="gmail-m_-151486401419277343moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org" target="_blank" moz-do-not-send="true">Gluster-users@gluster.org</a>
<a class="gmail-m_-151486401419277343moz-txt-link-freetext" href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank" moz-do-not-send="true">https://lists.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br class="">
<br class="">
<br class="">
<br class="">
<span style="color:rgb(96,76,120)"
class=""><font class=""><span
class="" lang="NL">Met
vriendelijke groet, With kind
regards,<br class="">
<br class="">
Jorick Astrego<br class="">
</span></font></span><b
style="color:rgb(96,76,120)"
class=""><br class="">
Netbulae Virtualization Experts </b><br
class="">
<hr
style="border-color:rgb(204,204,204)

currentcolor
currentcolor;border-style:solid

none none;border-width:1px medium
medium" class="">
<table style="width:522px" class="">
<tbody class="">
<tr class="">
<td
style="width:130px;font-size:10px"
class="">Tel: 053 20 30 270</td>
<td
style="width:130px;font-size:10px"
class=""><a
href="mailto:info@netbulae.eu"
target="_blank" class=""
moz-do-not-send="true">info@netbulae.eu</a></td>
<td
style="width:130px;font-size:10px"
class="">Staalsteden 4-3A</td>
<td
style="width:130px;font-size:10px"
class="">KvK 08198180</td>
</tr>
<tr class="">
<td
style="width:130px;font-size:10px"
class="">Fax: 053 20 30 271</td>
<td
style="width:130px;font-size:10px"
class=""><a
href="http://www.netbulae.eu/"
target="_blank" class=""
moz-do-not-send="true">www.netbulae.eu</a></td>
<td
style="width:130px;font-size:10px"
class="">7547 TA Enschede</td>
<td
style="width:130px;font-size:10px"
class="">BTW NL821234584B01</td>
</tr>
</tbody>
</table>
<br class="">
<hr
style="border-color:rgb(204,204,204)

currentcolor
currentcolor;border-style:solid

none none;border-width:1px medium
medium" class=""><br class="">
</div>
_______________________________________________<br class="">
Gluster-users mailing list<br class="">
<a
href="mailto:Gluster-users@gluster.org"
target="_blank" class=""
moz-do-not-send="true">Gluster-users@gluster.org</a><br
class="">
<a
href="https://lists.gluster.org/mailman/listinfo/gluster-users"
rel="noreferrer" target="_blank"
class="" moz-do-not-send="true">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote>
</div>
_______________________________________________<br class="">
Gluster-users mailing list<br class="">
<a href="mailto:Gluster-users@gluster.org"
class="" moz-do-not-send="true">Gluster-users@gluster.org</a><br
class="">
<a class="moz-txt-link-freetext"
href="https://lists.gluster.org/mailman/listinfo/gluster-users"
moz-do-not-send="true">https://lists.gluster.org/mailman/listinfo/gluster-users</a></div>
</blockquote>
</div>
<br class="">
</div>
</div>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org" moz-do-not-send="true">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="https://lists.gluster.org/mailman/listinfo/gluster-users" moz-do-not-send="true">https://lists.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br class="">
<br class="">
<br class="">
<br class="">
<span style="color:#604c78;" class=""><font class=""><span
style="mso-fareast-language:en-gb;" class=""
lang="NL">Met vriendelijke groet, With kind
regards,<br class="">
<br class="">
Jorick Astrego<br class="">
</span></font></span><b style="color:#604c78"
class=""><br class="">
Netbulae Virtualization Experts </b><br class="">
<hr style="border:none;border-top:1px solid #ccc;"
class="">
<table style="width: 522px" class="">
<tbody class="">
<tr class="">
<td style="width: 130px;font-size: 10px"
class="">Tel: 053 20 30 270</td>
<td style="width: 130px;font-size: 10px"
class=""><a href="mailto:info@netbulae.eu"
class="" moz-do-not-send="true">info@netbulae.eu</a></td>
<td style="width: 130px;font-size: 10px"
class="">Staalsteden 4-3A</td>
<td style="width: 130px;font-size: 10px"
class="">KvK 08198180</td>
</tr>
<tr class="">
<td style="width: 130px;font-size: 10px"
class="">Fax: 053 20 30 271</td>
<td style="width: 130px;font-size: 10px"
class=""><a href="http://www.netbulae.eu"
class="" moz-do-not-send="true">www.netbulae.eu</a></td>
<td style="width: 130px;font-size: 10px"
class="">7547 TA Enschede</td>
<td style="width: 130px;font-size: 10px"
class="">BTW NL821234584B01</td>
</tr>
</tbody>
</table>
<br class="">
<hr style="border:none;border-top:1px solid #ccc;"
class=""><br class="">
</div>
_______________________________________________<br
class="">
Gluster-users mailing list<br class="">
<a href="mailto:Gluster-users@gluster.org" class=""
moz-do-not-send="true">Gluster-users@gluster.org</a><br
class="">
<a class="moz-txt-link-freetext" href="https://lists.gluster.org/mailman/listinfo/gluster-users">https://lists.gluster.org/mailman/listinfo/gluster-users</a></div>
</blockquote>
</div>
<br class="">
</div>
</div>
</blockquote>
<BR />
<BR />
<b style="color:#604c78"></b><br><br><span style="color:#604c78;"><font color="000000"><span style="mso-fareast-language:en-gb;" lang="NL">Met vriendelijke groet, With kind regards,<br><br>Jorick Astrego<br></span></font></span><b style="color:#604c78"><br>Netbulae Virtualization Experts </b><br><hr style="border:none;border-top:1px solid #ccc;"><table style="width: 522px"><tbody><tr><td style="width: 130px;font-size: 10px">Tel: 053 20 30 270</td> <td style="width: 130px;font-size: 10px">info@netbulae.eu</td> <td style="width: 130px;font-size: 10px">Staalsteden 4-3A</td> <td style="width: 130px;font-size: 10px">KvK 08198180</td></tr><tr> <td style="width: 130px;font-size: 10px">Fax: 053 20 30 271</td> <td style="width: 130px;font-size: 10px">www.netbulae.eu</td> <td style="width: 130px;font-size: 10px">7547 TA Enschede</td> <td style="width: 130px;font-size: 10px">BTW NL821234584B01</td></tr></tbody></table><br><hr style="border:none;border-top:1px solid #ccc;"><BR />
</body>
</html>