<div dir="ltr"><div>Hi Ravi,</div><div><br></div><div>I think it probably is a firewall issue with the network provider. I was hoping to see a specific connection failure message we could send to them, but will take it up with them anyway. <br></div><div><br></div><div>Thanks for your help.</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 29 May 2019 at 23:10, Ravishankar N <<a href="mailto:ravishankar@redhat.com">ravishankar@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>I don't see a "Connected to gvol0-client-1" in the log. Perhaps
a firewall issue like the last time? Even in the earlier add-brick
log from the other email thread, connection to the 2nd brick was
not established.</p>
<p>-Ravi<br>
</p>
<div class="gmail-m_-6927197928405069712moz-cite-prefix">On 29/05/19 2:26 PM, David Cunningham
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>Hi Ravi and Joe,</div>
<div><br>
</div>
<div>The command "gluster volume status gvol0" shows all 3
nodes as being online, even on gfs3 as below. I've
attached the glfsheal-gvol0.log, in which I can't see
anything like a connection error. Would you have any
further suggestions? Thank you.</div>
<div><br>
</div>
<div>[root@gfs3 glusterfs]# gluster volume status gvol0<br>
Status of volume: gvol0<br>
Gluster process TCP Port
RDMA Port Online Pid<br>
------------------------------------------------------------------------------<br>
Brick gfs1:/nodirectwritedata/gluster/gvol0 49152
0 Y 7706 <br>
Brick gfs2:/nodirectwritedata/gluster/gvol0 49152
0 Y 7625 <br>
Brick gfs3:/nodirectwritedata/gluster/gvol0 49152
0 Y 7307 <br>
Self-heal Daemon on localhost N/A
N/A Y 7316 <br>
Self-heal Daemon on gfs1 N/A
N/A Y 40591<br>
Self-heal Daemon on gfs2 N/A
N/A Y 7634 <br>
<br>
Task Status of Volume gvol0<br>
------------------------------------------------------------------------------<br>
There are no active volume tasks<br>
<br>
</div>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, 29 May 2019 at 16:26,
Ravishankar N <<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p><br>
</p>
<div class="gmail-m_-6927197928405069712gmail-m_356646914622959683moz-cite-prefix">On
29/05/19 6:21 AM, David Cunningham wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>Hello all,</div>
<div><br>
</div>
<div>We are seeing a strange issue where a new
node gfs3 shows another node gfs2 as not
connected on the "gluster volume heal" info:</div>
<div><br>
</div>
<div>[root@gfs3 bricks]# gluster volume heal
gvol0 info<br>
Brick gfs1:/nodirectwritedata/gluster/gvol0<br>
Status: Connected<br>
Number of entries: 0<br>
<br>
Brick gfs2:/nodirectwritedata/gluster/gvol0<br>
Status: Transport endpoint is not connected<br>
Number of entries: -<br>
<br>
Brick gfs3:/nodirectwritedata/gluster/gvol0<br>
Status: Connected<br>
Number of entries: 0<br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>However it does show the same node
connected on "gluster peer status". Does
anyone know why this would be?</div>
<div><br>
</div>
<div>[root@gfs3 bricks]# gluster peer status<br>
Number of Peers: 2<br>
<br>
Hostname: gfs2<br>
Uuid: 91863102-23a8-43e1-b3d3-f0a1bd57f350<br>
State: Peer in Cluster (Connected)<br>
<br>
Hostname: gfs1<br>
Uuid: 32c99e7d-71f2-421c-86fc-b87c0f68ad1b<br>
State: Peer in Cluster (Connected)<br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>In nodirectwritedata-gluster-gvol0.log on
gfs3 we see this logged with regards to
gfs2:</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<p>You need to check glfsheal-$volname.log on the node where
you ran the command and check for any connection related
errors.</p>
<p>-Ravi<br>
</p>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div><br>
</div>
<div>[2019-05-29 00:17:50.646360] I [MSGID:
115029]
[server-handshake.c:537:server_setvolume]
0-gvol0-server: accepted client from
CTX_ID:30d74196-fece-4380-adc0-338760188b81-GRAPH_ID:0-PID:7718-HOST:gfs2.xxx.com-PC_NAME:gvol0-client-2-RECON_NO:-0
(version: 5.6)<br>
[2019-05-29 00:17:50.761120] I [MSGID:
115036] [server.c:469:server_rpc_notify]
0-gvol0-server: disconnecting connection
from
CTX_ID:30d74196-fece-4380-adc0-338760188b81-GRAPH_ID:0-PID:7718-HOST:gfs2.xxx.com-PC_NAME:gvol0-client-2-RECON_NO:-0<br>
[2019-05-29 00:17:50.761352] I [MSGID:
101055] [client_t.c:435:gf_client_unref]
0-gvol0-server: Shutting down connection
CTX_ID:30d74196-fece-4380-adc0-338760188b81-GRAPH_ID:0-PID:7718-HOST:gfs2.xxx.com-PC_NAME:gvol0-client-2-RECON_NO:-0<br>
</div>
<div><br>
</div>
<div>Thanks in advance for any assistance.<br>
</div>
<div><br>
-- <br>
<div dir="ltr" class="gmail-m_-6927197928405069712gmail-m_356646914622959683gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>David Cunningham,
Voisonics Limited<br>
<a href="http://voisonics.com/" target="_blank">http://voisonics.com/</a><br>
USA: +1 213 221 1092<br>
New Zealand: +64 (0)28
2558 3782</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset class="gmail-m_-6927197928405069712gmail-m_356646914622959683mimeAttachmentHeader"></fieldset>
<pre class="gmail-m_-6927197928405069712gmail-m_356646914622959683moz-quote-pre">_______________________________________________
Gluster-users mailing list
<a class="gmail-m_-6927197928405069712gmail-m_356646914622959683moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a class="gmail-m_-6927197928405069712gmail-m_356646914622959683moz-txt-link-freetext" href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<br>
-- <br>
<div dir="ltr" class="gmail-m_-6927197928405069712gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>David Cunningham, Voisonics Limited<br>
<a href="http://voisonics.com/" target="_blank">http://voisonics.com/</a><br>
USA: +1 213 221 1092<br>
New Zealand: +64 (0)28 2558 3782</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>David Cunningham, Voisonics Limited<br><a href="http://voisonics.com/" target="_blank">http://voisonics.com/</a><br>USA: +1 213 221 1092<br>New Zealand: +64 (0)28 2558 3782</div></div></div></div></div></div></div></div></div></div></div>