<div dir="ltr">Could you attach glusterd.log and cmd_history.log files from all the nodes?<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jun 21, 2017 at 11:40 PM, Victor Nomura <span dir="ltr"><<a href="mailto:victor@mezine.com" target="_blank">victor@mezine.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div link="blue" vlink="purple" lang="EN-CA"><div class="m_-1470604523331958467WordSection1"><p class="MsoNormal">Hi All,<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">I’m fairly new to Gluster (3.10.3) and got it going for a couple of months now but suddenly after a power failure in our building it all came crashing down. No client is able to connect after powering back the 3 nodes I have setup.<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">Looking at the logs, it looks like there’s some sort of “Lock” placed on the volume which prevents all the clients from connecting to the Gluster endpoint.<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">I can’t even do a #gluster volume status all command IF more than 1 node is powered up. I have to shutdown node2-3 and then I am able to issue the command<span style="color:#1f497d"> on node1</span> to see volume status. When all nodes are powered up and I check the peer status, it says that all peers are connected. Trying to connect to the Gluster volume from all clients says gluster endpoint is not available and times<span style="color:#1f497d"> </span>out. There are no network issues and each node can ping each other and there are no firewalls or any other device between the nodes and clients.<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">Please help if you think you know how to fix this. I have a feeling it’s this “lock” that’s not “released” due to the whole setup losing power all of a sudden. I’ve tried restarting all the nodes, restarting glusterfs-server etc. I’m out of ideas.<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">Thanks in advance!<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">Victor<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">Volume Name: teravolume<u></u><u></u></p><p class="MsoNormal">Type: Distributed-Replicate<u></u><u></u></p><p class="MsoNormal">Volume ID: 85af74d0-f1bc-4b0d-8901-<wbr>4dea6e4efae5<u></u><u></u></p><p class="MsoNormal">Status: Started<u></u><u></u></p><p class="MsoNormal">Snapshot Count: 0<u></u><u></u></p><p class="MsoNormal">Number of Bricks: 3 x 2 = 6<u></u><u></u></p><p class="MsoNormal">Transport-type: tcp<u></u><u></u></p><p class="MsoNormal">Bricks:<u></u><u></u></p><p class="MsoNormal">Brick1: gfsnode1:/media/brick1<u></u><u></u></p><p class="MsoNormal">Brick2: gfsnode2:/media/brick1<u></u><u></u></p><p class="MsoNormal">Brick3: gfsnode3:/media/brick1<u></u><u></u></p><p class="MsoNormal">Brick4: gfsnode1:/media/brick2<u></u><u></u></p><p class="MsoNormal">Brick5: gfsnode2:/media/brick2<u></u><u></u></p><p class="MsoNormal">Brick6: gfsnode3:/media/brick2<u></u><u></u></p><p class="MsoNormal">Options Reconfigured:<u></u><u></u></p><p class="MsoNormal">nfs.disable: on<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal"><u></u> <u></u></p><p class="MsoNormal">[2017-06-21 16:02:52.376709] W [MSGID: 106118] [glusterd-handler.c:5913:__<wbr>glusterd_peer_rpc_notify] 0-management: Lock not released for teravolume<u></u><u></u></p><p class="MsoNormal">[2017-06-21 16:03:03.429032] I [MSGID: 106163] [glusterd-handshake.c:1309:__<wbr>glusterd_mgmt_hndsk_versions_<wbr>ack] 0-management: using the op-version 31000<u></u><u></u></p><p class="MsoNormal">[2017-06-21 16:13:13.326478] E [rpc-clnt.c:200:call_bail] 0-management: bailing out frame type(Peer mgmt) op(--(2)) xid = 0x105 sent = 2017-06-21 16:03:03.202284. timeout = 600 for 192.168.150.52:$<u></u><u></u></p><p class="MsoNormal">[2017-06-21 16:13:13.326519] E [rpc-clnt.c:200:call_bail] 0-management: bailing out frame type(Peer mgmt) op(--(2)) xid = 0x105 sent = 2017-06-21 16:03:03.204555. timeout = 600 for 192.168.150.53:$<u></u><u></u></p><p class="MsoNormal">[2017-06-21 16:18:34.456522] I [MSGID: 106004] [glusterd-handler.c:5888:__<wbr>glusterd_peer_rpc_notify] 0-management: Peer <gfsnode2> (<e1e1caa5-9842-40d8-8492-<wbr>a82b079879a3>), in state <Peer in Cluste$<u></u><u></u></p><p class="MsoNormal">[2017-06-21 16:18:34.456619] W [glusterd-locks.c:675:<wbr>glusterd_mgmt_v3_unlock] (-->/usr/lib/x86_64-linux-gnu/<wbr>glusterfs/3.10.3/xlator/mgmt/<wbr>glusterd.so(+0x1f879) [0x7fee6bc22879] -->/usr/lib/x86_64-l$<u></u><u></u></p><p class="MsoNormal">[2017-06-21 16:18:34.456638] W [MSGID: 106118] [glusterd-handler.c:5913:__<wbr>glusterd_peer_rpc_notify] 0-management: Lock not released for teravolume<u></u><u></u></p><p class="MsoNormal">[2017-06-21 16:18:34.456661] I [MSGID: 106004] [glusterd-handler.c:5888:__<wbr>glusterd_peer_rpc_notify] 0-management: Peer <gfsnode3> (<59b9effa-2b88-4764-9130-<wbr>4f31c14c362e>), in state <Peer in Cluste$<u></u><u></u></p><p class="MsoNormal">[2017-06-21 16:18:34.456692] W [glusterd-locks.c:675:<wbr>glusterd_mgmt_v3_unlock] (-->/usr/lib/x86_64-linux-gnu/<wbr>glusterfs/3.10.3/xlator/mgmt/<wbr>glusterd.so(+0x1f879) [0x7fee6bc22879] -->/usr/lib/x86_64-l$<u></u><u></u></p><p class="MsoNormal">[2017-06-21 16:18:43.323944] I [MSGID: 106163] [glusterd-handshake.c:1309:__<wbr>glusterd_mgmt_hndsk_versions_<wbr>ack] 0-management: using the op-version 31000<u></u><u></u></p><p class="MsoNormal">[2017-06-21 16:18:34.456699] W [MSGID: 106118] [glusterd-handler.c:5913:__<wbr>glusterd_peer_rpc_notify] 0-management: Lock not released for teravolume<u></u><u></u></p><p class="MsoNormal">[2017-06-21 16:18:45.628552] I [MSGID: 106163] [glusterd-handshake.c:1309:__<wbr>glusterd_mgmt_hndsk_versions_<wbr>ack] 0-management: using the op-version 31000<u></u><u></u></p><p class="MsoNormal">[2017-06-21 16:23:40.607173] I [MSGID: 106499] [glusterd-handler.c:4363:__<wbr>glusterd_handle_status_volume] 0-management: Received status volume req for volume teravolume<u></u><u></u></p></div></div><br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>