<div dir="ltr"><div dir="ltr"><div dir="ltr">Hi Ahemad,<div><br></div><div>The logs don't seem to indicate anything specific.</div><div>Except this message in the glusterd logs, which I am not sure whether it might cause any problems</div><div>[2020-06-16 07:19:27.418884] E [MSGID: 101097] [xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing: /usr/lib64/glusterfs/7.5/rpc-transport/socket.so: undefined symbol: xlator_api<br></div><div><a class="gmail_plusreply" id="plusReplyChip-0" href="mailto:srakonde@redhat.com" tabindex="-1">@Sanju Rakonde</a> could you please let us know what this error message means and whether this could be leading to this specific problem?<br></div><div><br></div><div><a class="gmail_plusreply" id="plusReplyChip-1" href="mailto:ahemad_shaik@yahoo.com" tabindex="-1">@ahemad shaik</a> replica 3 volumes will be available when 1 of 3 bicks is down. Could you also paste the exact error that you are getting while accessing or doing any operations on the mount? Did you try rebooting any other node and facing the same problem?<br></div><div>To check whether there is any issue with the client connection can you give the output of this command from node4?</div><div><div dir="ltr" style="margin:12pt 0px 0pt;outline:none;padding:0px;line-height:1.2;text-align:justify"><span style="background-color:transparent;color:blue;font-family:arial;font-size:11pt;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre">cat <mount-path>/.meta/graphs/active/<vol-name>-client-*/private | egrep -i 'connected'</span></div>In your case the actual command might look like this</div><div><span style="color:rgb(0,0,255);font-family:arial;font-size:14.6667px;text-align:justify;white-space:pre">cat /mnt/.meta/graphs/active/glustervol</span><span style="color:rgb(0,0,255);font-family:arial;font-size:14.6667px;text-align:justify;white-space:pre">-client-*/private | egrep -i 'connected'</span></div><div><br></div><div>Regards,</div><div>Karthik</div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jun 16, 2020 at 1:31 PM ahemad shaik <<a href="mailto:ahemad_shaik@yahoo.com" target="_blank">ahemad_shaik@yahoo.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:16px"><div></div>
<div dir="ltr">Hi Karthik,</div><div dir="ltr"><br></div><div dir="ltr">Please find attached logs.</div><div dir="ltr"><br></div><div dir="ltr">kindly suggest on how to make the volume high available. </div><div dir="ltr"><br></div><div dir="ltr">Thanks,</div><div dir="ltr">Ahemad</div><div dir="ltr"><br></div><div dir="ltr"><br></div><div><br></div>
</div><div id="gmail-m_-3964018097011916620gmail-m_-6570966698334022408yahoo_quoted_2388246086">
<div style="font-family:"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:13px;color:rgb(38,40,42)">
<div>
On Tuesday, 16 June, 2020, 12:09:10 pm IST, Karthik Subrahmanya <<a href="mailto:ksubrahm@redhat.com" target="_blank">ksubrahm@redhat.com</a>> wrote:
</div>
<div><br></div>
<div><br></div>
<div><div id="gmail-m_-3964018097011916620gmail-m_-6570966698334022408yiv7997530779"><div><div dir="ltr"><div>Hi,</div><div><br clear="none"></div>Thanks for the clarification.<br clear="none"><div>In that case can you attach complete glusterd, bricks and mount logs from all the nodes when this happened.</div><div>Also paste the output that you are seeing when you try to access or do operations on the mount point.</div><div><br clear="none"></div><div>Regards,</div><div>Karthik</div></div><br clear="none"><div id="gmail-m_-3964018097011916620gmail-m_-6570966698334022408yiv7997530779yqt70165"><div><div dir="ltr">On Tue, Jun 16, 2020 at 11:55 AM ahemad shaik <<a rel="nofollow" shape="rect" href="mailto:ahemad_shaik@yahoo.com" target="_blank">ahemad_shaik@yahoo.com</a>> wrote:<br clear="none"></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:Helvetica,Arial,sans-serif;font-size:16px"><div></div>
<div dir="ltr">Sorry, It was a typo.</div><div dir="ltr"><br clear="none"></div><div dir="ltr">The command i exact command i have used is below.</div><div dir="ltr"><br clear="none"></div><div dir="ltr"><div><div dir="ltr" style="color:rgb(0,0,0);font-family:Helvetica,Arial,sans-serif;font-size:16px">The volume is mounted on node4.</div><div><br clear="none"></div></div></div><div dir="ltr"><span>""mount -t glusterfs node1:/glustervol /mnt/ ""</span><br clear="none"></div><div dir="ltr"><span><br clear="none"></span></div><div dir="ltr"><br clear="none"></div><div dir="ltr"><span>gluster Volume is created from node1,node2 and node3. </span></div><div dir="ltr"><span><br clear="none"></span></div><div dir="ltr"><span>"<span><span style="color:rgb(38,40,42);font-family:Helvetica,Arial,sans-serif;font-size:16px">"gluster volume create glustervol replica 3 transport tcp node1:/data node2:/data node3:/data force""</span></span></span></div><div dir="ltr"><span><br clear="none"></span></div><div dir="ltr"><span>I have tried rebooting node3 to test high availability. </span></div><div dir="ltr"><span><br clear="none"></span></div><div dir="ltr"><span>I hope it is clear now.</span></div><div dir="ltr"><span><br clear="none"></span></div><div dir="ltr">Please let me know if any questions.</div><div dir="ltr"><br clear="none"></div><div dir="ltr">Thanks,</div><div dir="ltr">Ahemad </div><div dir="ltr"><span><br clear="none"></span></div><div dir="ltr"><span><br clear="none"></span></div><div><br clear="none"></div>
</div><div id="gmail-m_-3964018097011916620gmail-m_-6570966698334022408yiv7997530779gmail-m_-3893404307628396150ydp9e117283yahoo_quoted_2427867594">
<div style="font-family:Helvetica,Arial,sans-serif;font-size:13px;color:rgb(38,40,42)">
<div>
On Tuesday, 16 June, 2020, 11:45:48 am IST, Karthik Subrahmanya <<a rel="nofollow" shape="rect" href="mailto:ksubrahm@redhat.com" target="_blank">ksubrahm@redhat.com</a>> wrote:
</div>
<div><br clear="none"></div>
<div><br clear="none"></div>
<div><div id="gmail-m_-3964018097011916620gmail-m_-6570966698334022408yiv7997530779gmail-m_-3893404307628396150ydp9e117283yiv8125704669"><div><div dir="ltr">Hi Ahemad,<div><br clear="none"></div><div>A quick question on the mount command that you have used</div>"mount -t glusterfs node4:/glustervol /mnt/"<br clear="none">Here you are specifying the hostname as node4 instead of node{1,2,3} which actually host the volume that you intend to mount. Is this a typo or did you paste the same command that you used for mounting?<br clear="none">If it is the actual command that you have used, then node4 seems to have some old volume details which are not cleared properly and it is being used while mounting. According to the peer info that you provided, only node1, 2 & 3 are part of the list, so node4 is unaware of the volume that you want to mount and this command is mounting a volume which is only visible to node4.<div><br clear="none"></div><div>Regards,</div><div>Karthik</div></div><br clear="none"><div id="gmail-m_-3964018097011916620gmail-m_-6570966698334022408yiv7997530779gmail-m_-3893404307628396150ydp9e117283yiv8125704669yqt55135"><div><div dir="ltr">On Tue, Jun 16, 2020 at 11:11 AM ahemad shaik <<a rel="nofollow" shape="rect" href="mailto:ahemad_shaik@yahoo.com" target="_blank">ahemad_shaik@yahoo.com</a>> wrote:<br clear="none"></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:Helvetica,Arial,sans-serif;font-size:16px"><div></div>
<div dir="ltr"><div><div dir="ltr">Hi Karthik,</div><div dir="ltr"><br clear="none"></div><div dir="ltr"><br clear="none"></div><div dir="ltr">Please provide the following info, I see there are errors unable to connect to port and warning that transport point end connected. Please find the complete logs below.</div><div dir="ltr"><br clear="none"></div><div dir="ltr">kindly suggest.</div><div dir="ltr"><br clear="none"></div><div>1. gluster peer status</div><div><br clear="none"></div><div>gluster peer status</div><div>Number of Peers: 2</div><div><br clear="none"></div><div>Hostname: node1</div><div>Uuid: 0e679115-15ad-4a85-9d0a-9178471ef90</div><div>State: Peer in Cluster (Connected)</div><div><br clear="none"></div><div>Hostname: node2</div><div>Uuid: 785a7c5b-86d3-45b9-b371-7e66e7fa88e0</div><div>State: Peer in Cluster (Connected)</div><div><br clear="none"></div><div><br clear="none"></div><div>gluster pool list</div><div>UUID Hostname State</div><div>0e679115-15ad-4a85-9d0a-9178471ef90 node1<span style="white-space:pre-wrap"> </span> Connected</div><div>785a7c5b-86d3-45b9-b371-7e66e7fa88e0 node2 Connected</div><div>ec137af6-4845-4ebb-955a-fac1df9b7b6c localhost(node3) Connected</div><div><br clear="none"></div><div>2. gluster volume info glustervol</div><div><br clear="none"></div><div>Volume Name: glustervol</div><div>Type: Replicate</div><div>Volume ID: 5422bb27-1863-47d5-b216-61751a01b759</div><div>Status: Started</div><div>Snapshot Count: 0</div><div>Number of Bricks: 1 x 3 = 3</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: node1:/data</div><div>Brick2: node2:/data</div><div>Brick3: node3:/data</div><div>Options Reconfigured:</div><div>performance.client-io-threads: off</div><div>nfs.disable: on</div><div>storage.fips-mode-rchecksum: on</div><div>transport.address-family: inet</div><div><br clear="none"></div><div>3. gluster volume status glustervol</div><div><br clear="none"></div><div>gluster volume status glustervol</div><div>Status of volume: glustervol</div><div>Gluster process TCP Port RDMA Port Online Pid</div><div>------------------------------------------------------------------------------</div><div>Brick node1:/data 49152 0 Y 59739</div><div>Brick node2:/data 49153 0 Y 3498</div><div>Brick node3:/data 49152 0 Y 1880</div><div>Self-heal Daemon on localhost N/A N/A Y 1905</div><div>Self-heal Daemon on node1 N/A N/A Y 3519</div><div>Self-heal Daemon on node2 N/A N/A Y 59760</div><div><br clear="none"></div><div>Task Status of Volume glustervol</div><div>------------------------------------------------------------------------------</div><div>There are no active volume tasks</div><div><br clear="none"></div><div>4. client log from node4 when you saw unavailability-</div><div><br clear="none"></div><div>Below are the logs when i reboot server node3, we can see in logs that "0-glustervol-client-2: disconnected from glustervol-client-2".</div><div><br clear="none"></div><div>Please find the complete logs below from the reboot to until the server available. I am testing high availability by just rebooting server. In real case scenario chances are there that server may not available for some hours so i just dont want to have the long down time.</div><div><br clear="none"></div><div><br clear="none"></div><div>[2020-06-16 05:14:25.256136] I [MSGID: 114046] [client-handshake.c:1105:client_setvolume_cbk] 0-glustervol-client-0: Connected to glustervol-client-0, attached to remote volume '/data'.</div><div>[2020-06-16 05:14:25.256179] I [MSGID: 108005] [afr-common.c:5247:__afr_handle_child_up_event] 0-glustervol-replicate-0: Subvolume 'glustervol-client-0' came back up; going online.</div><div>[2020-06-16 05:14:25.257972] I [MSGID: 114046] [client-handshake.c:1105:client_setvolume_cbk] 0-glustervol-client-1: Connected to glustervol-client-1, attached to remote volume '/data'.</div><div>[2020-06-16 05:14:25.258014] I [MSGID: 108002] [afr-common.c:5609:afr_notify] 0-glustervol-replicate-0: Client-quorum is met</div><div>[2020-06-16 05:14:25.260312] I [MSGID: 114046] [client-handshake.c:1105:client_setvolume_cbk] 0-glustervol-client-2: Connected to glustervol-client-2, attached to remote volume '/data'.</div><div>[2020-06-16 05:14:25.261935] I [fuse-bridge.c:5145:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.23</div><div>[2020-06-16 05:14:25.261957] I [fuse-bridge.c:5756:fuse_graph_sync] 0-fuse: switched to graph 0</div><div>[2020-06-16 05:16:59.729400] I [MSGID: 114018] [client.c:2331:client_rpc_notify] 0-glustervol-client-2: disconnected from glustervol-client-2. Client process will keep trying to connect to glusterd until brick's port is available</div><div>[2020-06-16 05:16:59.730053] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a6a41badb] (--> /lib64/libgfrpc.so.0(+0xd7e4)[0x7f4a6a1c27e4] (--> /lib64/libgfrpc.so.0(+0xd8fe)[0x7f4a6a1c28fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f4a6a1c3987] (--> /lib64/libgfrpc.so.0(+0xf518)[0x7f4a6a1c4518] ))))) 0-glustervol-client-2: forced unwinding frame type(GlusterFS 4.x v1) op(LOOKUP(27)) called at 2020-06-16 05:16:08.175698 (xid=0xae)</div><div>[2020-06-16 05:16:59.730089] W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-glustervol-client-2: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]</div><div>[2020-06-16 05:16:59.730336] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a6a41badb] (--> /lib64/libgfrpc.so.0(+0xd7e4)[0x7f4a6a1c27e4] (--> /lib64/libgfrpc.so.0(+0xd8fe)[0x7f4a6a1c28fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f4a6a1c3987] (--> /lib64/libgfrpc.so.0(+0xf518)[0x7f4a6a1c4518] ))))) 0-glustervol-client-2: forced unwinding frame type(GlusterFS 4.x v1) op(LOOKUP(27)) called at 2020-06-16 05:16:10.237849 (xid=0xaf)</div><div>[2020-06-16 05:16:59.730540] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a6a41badb] (--> /lib64/libgfrpc.so.0(+0xd7e4)[0x7f4a6a1c27e4] (--> /lib64/libgfrpc.so.0(+0xd8fe)[0x7f4a6a1c28fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f4a6a1c3987] (--> /lib64/libgfrpc.so.0(+0xf518)[0x7f4a6a1c4518] ))))) 0-glustervol-client-2: forced unwinding frame type(GlusterFS 4.x v1) op(LOOKUP(27)) called at 2020-06-16 05:16:22.694419 (xid=0xb0)</div><div>[2020-06-16 05:16:59.731132] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a6a41badb] (--> /lib64/libgfrpc.so.0(+0xd7e4)[0x7f4a6a1c27e4] (--> /lib64/libgfrpc.so.0(+0xd8fe)[0x7f4a6a1c28fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f4a6a1c3987] (--> /lib64/libgfrpc.so.0(+0xf518)[0x7f4a6a1c4518] ))))) 0-glustervol-client-2: forced unwinding frame type(GlusterFS 4.x v1) op(LOOKUP(27)) called at 2020-06-16 05:16:27.574139 (xid=0xb1)</div><div>[2020-06-16 05:16:59.731319] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a6a41badb] (--> /lib64/libgfrpc.so.0(+0xd7e4)[0x7f4a6a1c27e4] (--> /lib64/libgfrpc.so.0(+0xd8fe)[0x7f4a6a1c28fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f4a6a1c3987] (--> /lib64/libgfrpc.so.0(+0xf518)[0x7f4a6a1c4518] ))))) 0-glustervol-client-2: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at 2020-06-16 05:16:34.231433 (xid=0xb2)</div><div>[2020-06-16 05:16:59.731352] W [rpc-clnt-ping.c:210:rpc_clnt_ping_cbk] 0-glustervol-client-2: socket disconnected</div><div>[2020-06-16 05:16:59.731464] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a6a41badb] (--> /lib64/libgfrpc.so.0(+0xd7e4)[0x7f4a6a1c27e4] (--> /lib64/libgfrpc.so.0(+0xd8fe)[0x7f4a6a1c28fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f4a6a1c3987] (--> /lib64/libgfrpc.so.0(+0xf518)[0x7f4a6a1c4518] ))))) 0-glustervol-client-2: forced unwinding frame type(GlusterFS 4.x v1) op(LOOKUP(27)) called at 2020-06-16 05:16:41.213884 (xid=0xb3)</div><div>[2020-06-16 05:16:59.731650] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a6a41badb] (--> /lib64/libgfrpc.so.0(+0xd7e4)[0x7f4a6a1c27e4] (--> /lib64/libgfrpc.so.0(+0xd8fe)[0x7f4a6a1c28fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f4a6a1c3987] (--> /lib64/libgfrpc.so.0(+0xf518)[0x7f4a6a1c4518] ))))) 0-glustervol-client-2: forced unwinding frame type(GlusterFS 4.x v1) op(LOOKUP(27)) called at 2020-06-16 05:16:48.756212 (xid=0xb4)</div><div>[2020-06-16 05:16:59.731876] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a6a41badb] (--> /lib64/libgfrpc.so.0(+0xd7e4)[0x7f4a6a1c27e4] (--> /lib64/libgfrpc.so.0(+0xd8fe)[0x7f4a6a1c28fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f4a6a1c3987] (--> /lib64/libgfrpc.so.0(+0xf518)[0x7f4a6a1c4518] ))))) 0-glustervol-client-2: forced unwinding frame type(GlusterFS 4.x v1) op(LOOKUP(27)) called at 2020-06-16 05:16:52.258940 (xid=0xb5)</div><div>[2020-06-16 05:16:59.732060] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a6a41badb] (--> /lib64/libgfrpc.so.0(+0xd7e4)[0x7f4a6a1c27e4] (--> /lib64/libgfrpc.so.0(+0xd8fe)[0x7f4a6a1c28fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f4a6a1c3987] (--> /lib64/libgfrpc.so.0(+0xf518)[0x7f4a6a1c4518] ))))) 0-glustervol-client-2: forced unwinding frame type(GlusterFS 4.x v1) op(LOOKUP(27)) called at 2020-06-16 05:16:54.618301 (xid=0xb6)</div><div>[2020-06-16 05:16:59.732246] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a6a41badb] (--> /lib64/libgfrpc.so.0(+0xd7e4)[0x7f4a6a1c27e4] (--> /lib64/libgfrpc.so.0(+0xd8fe)[0x7f4a6a1c28fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f4a6a1c3987] (--> /lib64/libgfrpc.so.0(+0xf518)[0x7f4a6a1c4518] ))))) 0-glustervol-client-2: forced unwinding frame type(GlusterFS 4.x v1) op(LOOKUP(27)) called at 2020-06-16 05:16:58.288790 (xid=0xb7)</div><div>[2020-06-16 05:17:10.245302] I [rpc-clnt.c:2028:rpc_clnt_reconfig] 0-glustervol-client-2: changing port to 49152 (from 0)</div><div>[2020-06-16 05:17:10.249896] I [MSGID: 114046] [client-handshake.c:1105:client_setvolume_cbk] 0-glustervol-client-2: Connected to glustervol-client-2, attached to remote volume '/data'</div><div>The message "W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-glustervol-client-2: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]" repeated 8 times between [2020-06-16 05:16:59.730089] and [2020-06-16 05:16:59.732278]</div></div><br clear="none"></div><div dir="ltr">Thanks,</div><div dir="ltr">Ahemad</div><div><br clear="none"></div>
</div><div id="gmail-m_-3964018097011916620gmail-m_-6570966698334022408yiv7997530779gmail-m_-3893404307628396150ydp9e117283yiv8125704669gmail-m_-3754008474340859303yahoo_quoted_3186537145">
<div style="font-family:Helvetica,Arial,sans-serif;font-size:13px;color:rgb(38,40,42)">
<div>
On Tuesday, 16 June, 2020, 10:58:42 am IST, ahemad shaik <<a rel="nofollow" shape="rect" href="mailto:ahemad_shaik@yahoo.com" target="_blank">ahemad_shaik@yahoo.com</a>> wrote:
</div>
<div><br clear="none"></div>
<div><br clear="none"></div>
<div><div id="gmail-m_-3964018097011916620gmail-m_-6570966698334022408yiv7997530779gmail-m_-3893404307628396150ydp9e117283yiv8125704669gmail-m_-3754008474340859303yiv0738301498"><div><div style="font-family:Helvetica,Arial,sans-serif;font-size:16px"><div></div>
<div dir="ltr">Hi Karthik,</div><div dir="ltr"><br clear="none"></div><div dir="ltr">Please find the details below.</div><div dir="ltr"><br clear="none"></div><div dir="ltr"><div><div>Please provide the following info:</div><div>1. gluster peer status</div><div><br clear="none"></div><div>gluster peer status</div><div>Number of Peers: 2</div><div><br clear="none"></div><div>Hostname: node1</div><div>Uuid: 0e679115-15ad-4a85-9d0a-9178471ef90</div><div>State: Peer in Cluster (Connected)</div><div><br clear="none"></div><div>Hostname: node2</div><div>Uuid: 785a7c5b-86d3-45b9-b371-7e66e7fa88e0</div><div>State: Peer in Cluster (Connected)</div><div><br clear="none"></div><div><br clear="none"></div><div>gluster pool list</div><div>UUID Hostname State</div><div>0e679115-15ad-4a85-9d0a-9178471ef90 node1<span style="white-space:pre-wrap"> </span>Connected</div><div>785a7c5b-86d3-45b9-b371-7e66e7fa88e0 node2 Connected</div><div>ec137af6-4845-4ebb-955a-fac1df9b7b6c localhost(node3) Connected</div><div><br clear="none"></div><div>2. gluster volume info glustervol</div><div><br clear="none"></div><div>Volume Name: glustervol</div><div>Type: Replicate</div><div>Volume ID: 5422bb27-1863-47d5-b216-61751a01b759</div><div>Status: Started</div><div>Snapshot Count: 0</div><div>Number of Bricks: 1 x 3 = 3</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: node1:/data</div><div>Brick2: node2:/data</div><div>Brick3: node3:/data</div><div>Options Reconfigured:</div><div>performance.client-io-threads: off</div><div>nfs.disable: on</div><div>storage.fips-mode-rchecksum: on</div><div>transport.address-family: inet</div><div><br clear="none"></div><div>3. gluster volume status glustervol</div><div><br clear="none"></div><div>gluster volume status glustervol</div><div>Status of volume: glustervol</div><div>Gluster process TCP Port RDMA Port Online Pid</div><div>------------------------------------------------------------------------------</div><div>Brick node1:/data 49152 0 Y 59739</div><div>Brick node2:/data 49153 0 Y 3498</div><div>Brick node3:/data 49152 0 Y 1880</div><div>Self-heal Daemon on localhost N/A N/A Y 1905</div><div>Self-heal Daemon on node1 N/A N/A Y 3519</div><div>Self-heal Daemon on node2 N/A N/A Y 59760</div><div><br clear="none"></div><div>Task Status of Volume glustervol</div><div>------------------------------------------------------------------------------</div><div>There are no active volume tasks</div><div><br clear="none"></div><div>4. client log from node4 when you saw unavailability-</div><div><br clear="none"></div><div>Below are the logs when i reboot server node3, we can see in logs that "0-glustervol-client-2: disconnected from glustervol-client-2".</div><div><br clear="none"></div><div>Please find the complete logs below from the reboot to until the server available. I am testing high availability by just rebooting server. In real case scenario chances are there that server may not available for some hours so we just dont want to have the long down time.</div><div><br clear="none"></div><div><br clear="none"></div><div>[2020-06-16 05:14:25.256136] I [MSGID: 114046] [client-handshake.c:1105:client_setvolume_cbk] 0-glustervol-client-0: Connected to glustervol-client-0, attached to remote volume '/data'.</div><div>[2020-06-16 05:14:25.256179] I [MSGID: 108005] [afr-common.c:5247:__afr_handle_child_up_event] 0-glustervol-replicate-0: Subvolume 'glustervol-client-0' came back up; going online.</div><div>[2020-06-16 05:14:25.257972] I [MSGID: 114046] [client-handshake.c:1105:client_setvolume_cbk] 0-glustervol-client-1: Connected to glustervol-client-1, attached to remote volume '/data'.</div><div>[2020-06-16 05:14:25.258014] I [MSGID: 108002] [afr-common.c:5609:afr_notify] 0-glustervol-replicate-0: Client-quorum is met</div><div>[2020-06-16 05:14:25.260312] I [MSGID: 114046] [client-handshake.c:1105:client_setvolume_cbk] 0-glustervol-client-2: Connected to glustervol-client-2, attached to remote volume '/data'.</div><div>[2020-06-16 05:14:25.261935] I [fuse-bridge.c:5145:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.23</div><div>[2020-06-16 05:14:25.261957] I [fuse-bridge.c:5756:fuse_graph_sync] 0-fuse: switched to graph 0</div><div>[2020-06-16 05:16:59.729400] I [MSGID: 114018] [client.c:2331:client_rpc_notify] 0-glustervol-client-2: disconnected from glustervol-client-2. Client process will keep trying to connect to glusterd until brick's port is available</div><div>[2020-06-16 05:16:59.730053] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a6a41badb] (--> /lib64/libgfrpc.so.0(+0xd7e4)[0x7f4a6a1c27e4] (--> /lib64/libgfrpc.so.0(+0xd8fe)[0x7f4a6a1c28fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f4a6a1c3987] (--> /lib64/libgfrpc.so.0(+0xf518)[0x7f4a6a1c4518] ))))) 0-glustervol-client-2: forced unwinding frame type(GlusterFS 4.x v1) op(LOOKUP(27)) called at 2020-06-16 05:16:08.175698 (xid=0xae)</div><div>[2020-06-16 05:16:59.730089] W [MSGID: 114031] [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-glustervol-client-2: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]</div><div>[2020-06-16 05:16:59.730336] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a6a41badb] (--> /lib64/libgfrpc.so.0(+0xd7e4)[0x7f4a6a1c27e4] (--> /lib64/libgfrpc.so.0(+0xd8fe)[0x7f4a6a1c28fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f4a6a1c3987] (--> /lib64/libgfrpc.so.0(+0xf518)[0x7f4a6a1c4518] ))))) 0-glustervol-client-2: forced unwinding frame type(GlusterFS 4.x v1) op(LOOKUP(27)) called at 2020-06-16 05:16:10.237849 (xid=0xaf)</div><div>[2020-06-16 05:16:59.730540] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a6a41badb] (--> /lib64/libgfrpc.so.0(+0xd7e4)[0x7f4a6a1c27e4] (--> /lib64/libgfrpc.so.0(+0xd8fe)[0x7f4a6a1c28fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f4a6a1c3987] (--> /lib64/libgfrpc.so.0(+0xf518)[0x7f4a6a1c4518] ))))) 0-glustervol-client-2: forced unwinding frame type(GlusterFS 4.x v1) op(LOOKUP(27)) called at 2020-06-16 05:16:22.694419 (xid=0xb0)</div><div>[2020-06-16 05:16:59.731132] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a6a41badb] (--> /lib64/libgfrpc.so.0(+0xd7e4)[0x7f4a6a1c27e4] (--> /lib64/libgfrpc.so.0(+0xd8fe)[0x7f4a6a1c28fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f4a6a1c3987] (--> /lib64/libgfrpc.so.0(+0xf518)[0x7f4a6a1c4518] ))))) 0-glustervol-client-2: forced unwinding frame type(GlusterFS 4.x v1) op(LOOKUP(27)) called at 2020-06-16 05:16:27.574139 (xid=0xb1)</div><div>[2020-06-16 05:16:59.731319] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a6a41badb] (--> /lib64/libgfrpc.so.0(+0xd7e4)[0x7f4a6a1c27e4] (--> /lib64/libgfrpc.so.0(+0xd8fe)[0x7f4a6a1c28fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f4a6a1c3987] (--> /lib64/libgfrpc.so.0(+0xf518)[0x7f4a6a1c4518] ))))) 0-glustervol-client-2: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at 2020-06-16 05:16:34.231433 (xid=0xb2)</div><div>[2020-06-16 05:16:59.731352] W [rpc-clnt-ping.c:210:rpc_clnt_ping_cbk] 0-glustervol-client-2: socket disconnected</div><div>[2020-06-16 05:16:59.731464] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a6a41badb] (--> /lib64/libgfrpc.so.0(+0xd7e4)[0x7f4a6a1c27e4] (--> /lib64/libgfrpc.so.0(+0xd8fe)[0x7f4a6a1c28fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f4a6a1c3987] (--> /lib64/libgfrpc.so.0(+0xf518)[0x7f4a6a1c4518] ))))) 0-glustervol-client-2: forced unwinding frame type(GlusterFS 4.x v1) op(LOOKUP(27)) called at 2020-06-16 05:16:41.213884 (xid=0xb3)</div><div>[2020-06-16 05:16:59.731650] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a6a41badb] (--> /lib64/libgfrpc.so.0(+0xd7e4)[0x7f4a6a1c27e4] (--> /lib64/libgfrpc.so.0(+0xd8fe)[0x7f4a6a1c28fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f4a6a1c3987] (--> /lib64/libgfrpc.so.0(+0xf518)[0x7f4a6a1c4518] ))))) 0-glustervol-client-2: forced unwinding frame type(GlusterFS 4.x v1) op(LOOKUP(27)) called at 2020-06-16 05:16:48.756212 (xid=0xb4)</div><div>[2020-06-16 05:16:59.731876] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a6a41badb] (--> /lib64/libgfrpc.so.0(+0xd7e4)[0x7f4a6a1c27e4] (--> /lib64/libgfrpc.so.0(+0xd8fe)[0x7f4a6a1c28fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f4a6a1c3987] (--> /lib64/libgfrpc.so.0(+0xf518)[0x7f4a6a1c4518] ))))) 0-glustervol-client-2: forced unwinding frame type(GlusterFS 4.x v1) op(LOOKUP(27)) called at 2020-06-16 05:16:52.258940 (xid=0xb5)</div><div>[2020-06-16 05:16:59.732060] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a6a41badb] (--> /lib64/libgfrpc.so.0(+0xd7e4)[0x7f4a6a1c27e4] (--> /lib64/libgfrpc.so.0(+0xd8fe)[0x7f4a6a1c28fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f4a6a1c3987] (--> /lib64/libgfrpc.so.0(+0xf518)[0x7f4a6a1c4518] ))))) 0-glustervol-client-2: forced unwinding frame type(GlusterFS 4.x v1) op(LOOKUP(27)) called at 2020-06-16 05:16:54.618301 (xid=0xb6)</div><div>[2020-06-16 05:16:59.732246] E [rpc-clnt.c:346:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a6a41badb] (--> /lib64/libgfrpc.so.0(+0xd7e4)[0x7f4a6a1c27e4] (--> /lib64/libgfrpc.so.0(+0xd8fe)[0x7f4a6a1c28fe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f4a6a1c3987] (--> /lib64/libgfrpc.so.0(+0xf518)[0x7f4a6a1c4518] ))))) 0-glustervol-client-2: forced unwinding frame type(GlusterFS 4.x v1) op(LOOKUP(27)) called at 2020-06-16 05:16:58.288790 (xid=0xb7)</div><div>[2020-06-16 05:17:10.245302] I [rpc-clnt.c:2028:rpc_clnt_reconfig] 0-glustervol-client-2: changing port to 49152 (from 0)</div><div>[2020-06-16 05:17:10.249896] I [MSGID: 114046] [client-handshake.c:1105:client_setvolume_cbk] 0-glustervol-client-2: Connected to glustervol-client-2, attached to remote volume '/data'.</div></div><br clear="none"></div><div dir="ltr">Thanks,</div><div dir="ltr">Ahemad</div><div><br clear="none"></div>
</div><div id="gmail-m_-3964018097011916620gmail-m_-6570966698334022408yiv7997530779gmail-m_-3893404307628396150ydp9e117283yiv8125704669gmail-m_-3754008474340859303yiv0738301498yqt14957"><div id="gmail-m_-3964018097011916620gmail-m_-6570966698334022408yiv7997530779gmail-m_-3893404307628396150ydp9e117283yiv8125704669gmail-m_-3754008474340859303yiv0738301498yahoo_quoted_2615652543">
<div style="font-family:Helvetica,Arial,sans-serif;font-size:13px;color:rgb(38,40,42)">
<div>
On Tuesday, 16 June, 2020, 10:10:16 am IST, Karthik Subrahmanya <<a rel="nofollow" shape="rect" href="mailto:ksubrahm@redhat.com" target="_blank">ksubrahm@redhat.com</a>> wrote:
</div>
<div><br clear="none"></div>
<div><br clear="none"></div>
<div><div id="gmail-m_-3964018097011916620gmail-m_-6570966698334022408yiv7997530779gmail-m_-3893404307628396150ydp9e117283yiv8125704669gmail-m_-3754008474340859303yiv0738301498"><div><div dir="ltr">Hi Ahemad,<div><br clear="none"></div><div>Please provide the following info:</div><div>1. gluster peer status</div><div>2. gluster volume info glustervol</div><div>3. gluster volume status glustervol</div><div>4. client log from node4 when you saw unavailability</div><div><br clear="none"></div><div>Regards,</div><div>Karthik</div></div><br clear="none"><div><div id="gmail-m_-3964018097011916620gmail-m_-6570966698334022408yiv7997530779gmail-m_-3893404307628396150ydp9e117283yiv8125704669gmail-m_-3754008474340859303yiv0738301498yqt31047"><div dir="ltr">On Mon, Jun 15, 2020 at 11:07 PM ahemad shaik <<a rel="nofollow" shape="rect" href="mailto:ahemad_shaik@yahoo.com" target="_blank">ahemad_shaik@yahoo.com</a>> wrote:<br clear="none"></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:Helvetica,Arial,sans-serif;font-size:16px"><div dir="ltr">Hi There,</div><div dir="ltr"><br clear="none"></div><div dir="ltr">I have created 3 replica gluster volume with 3 bricks from 3 nodes.</div><div dir="ltr"><br clear="none"></div><div dir="ltr"><span>"gluster volume create glustervol replica 3 transport tcp node1:/data node2:/data node3:/data force"</span></div><div dir="ltr"><br clear="none"></div><div dir="ltr">mounted on client node using below command.</div><div dir="ltr"><br clear="none"></div><div dir="ltr"><div><div>"mount -t glusterfs node4:/glustervol /mnt/"</div><div><br clear="none"></div><div dir="ltr">when any of the node (either node1,node2 or node3) goes down, gluster mount/volume (/mnt) not accessible at client (node4).</div><div dir="ltr"><br clear="none"></div><div dir="ltr">purpose of replicated volume is high availability but not able to achieve it.</div><div dir="ltr"><br clear="none"></div><div dir="ltr">Is it a bug or i am missing anything.</div><div dir="ltr"><br clear="none"></div><div dir="ltr"><br clear="none"></div><div dir="ltr">Any suggestions will be great help!!!</div><div dir="ltr"><br clear="none"></div><div dir="ltr">kindly suggest.</div><div dir="ltr"><br clear="none"></div><div dir="ltr">Thanks,</div><div dir="ltr">Ahemad </div></div></div><div dir="ltr"> </div></div></div>________<br clear="none">
<br clear="none">
<br clear="none">
<br clear="none">
Community Meeting Calendar:<br clear="none">
<br clear="none">
Schedule -<br clear="none">
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br clear="none">
Bridge: <a rel="nofollow" shape="rect" href="https://bluejeans.com/441850968" target="_blank">https://bluejeans.com/441850968</a><br clear="none">
<br clear="none">
Gluster-users mailing list<br clear="none">
<a rel="nofollow" shape="rect" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br clear="none">
<a rel="nofollow" shape="rect" href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br clear="none">
</blockquote></div></div></div></div></div>
</div>
</div></div></div></div></div>
</div>
</div></div></blockquote></div></div></div></div></div>
</div>
</div></div></blockquote></div></div></div></div></div>
</div>
</div></div></blockquote></div>