<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-CA link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>Hi All,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I’m fairly new to Gluster (3.10.3) and got it going for a couple of months now but suddenly after a power failure in our building it all came crashing down. No client is able to connect after powering back the 3 nodes I have setup.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Looking at the logs, it looks like there’s some sort of “Lock” placed on the volume which prevents all the clients from connecting to the Gluster endpoint.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I can’t even do a #gluster volume status all command IF more than 1 node is powered up. I have to shutdown node2-3 and then I am able to issue the command<span style='color:#1F497D'> on node1</span> to see volume status. When all nodes are powered up and I check the peer status, it says that all peers are connected. Trying to connect to the Gluster volume from all clients says gluster endpoint is not available and times<span style='color:#1F497D'> </span>out. There are no network issues and each node can ping each other and there are no firewalls or any other device between the nodes and clients.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Please help if you think you know how to fix this. I have a feeling it’s this “lock” that’s not “released” due to the whole setup losing power all of a sudden. I’ve tried restarting all the nodes, restarting glusterfs-server etc. I’m out of ideas.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thanks in advance!<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Victor<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Volume Name: teravolume<o:p></o:p></p><p class=MsoNormal>Type: Distributed-Replicate<o:p></o:p></p><p class=MsoNormal>Volume ID: 85af74d0-f1bc-4b0d-8901-4dea6e4efae5<o:p></o:p></p><p class=MsoNormal>Status: Started<o:p></o:p></p><p class=MsoNormal>Snapshot Count: 0<o:p></o:p></p><p class=MsoNormal>Number of Bricks: 3 x 2 = 6<o:p></o:p></p><p class=MsoNormal>Transport-type: tcp<o:p></o:p></p><p class=MsoNormal>Bricks:<o:p></o:p></p><p class=MsoNormal>Brick1: gfsnode1:/media/brick1<o:p></o:p></p><p class=MsoNormal>Brick2: gfsnode2:/media/brick1<o:p></o:p></p><p class=MsoNormal>Brick3: gfsnode3:/media/brick1<o:p></o:p></p><p class=MsoNormal>Brick4: gfsnode1:/media/brick2<o:p></o:p></p><p class=MsoNormal>Brick5: gfsnode2:/media/brick2<o:p></o:p></p><p class=MsoNormal>Brick6: gfsnode3:/media/brick2<o:p></o:p></p><p class=MsoNormal>Options Reconfigured:<o:p></o:p></p><p class=MsoNormal>nfs.disable: on<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>[2017-06-21 16:02:52.376709] W [MSGID: 106118] [glusterd-handler.c:5913:__glusterd_peer_rpc_notify] 0-management: Lock not released for teravolume<o:p></o:p></p><p class=MsoNormal>[2017-06-21 16:03:03.429032] I [MSGID: 106163] [glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31000<o:p></o:p></p><p class=MsoNormal>[2017-06-21 16:13:13.326478] E [rpc-clnt.c:200:call_bail] 0-management: bailing out frame type(Peer mgmt) op(--(2)) xid = 0x105 sent = 2017-06-21 16:03:03.202284. timeout = 600 for 192.168.150.52:$<o:p></o:p></p><p class=MsoNormal>[2017-06-21 16:13:13.326519] E [rpc-clnt.c:200:call_bail] 0-management: bailing out frame type(Peer mgmt) op(--(2)) xid = 0x105 sent = 2017-06-21 16:03:03.204555. timeout = 600 for 192.168.150.53:$<o:p></o:p></p><p class=MsoNormal>[2017-06-21 16:18:34.456522] I [MSGID: 106004] [glusterd-handler.c:5888:__glusterd_peer_rpc_notify] 0-management: Peer <gfsnode2> (<e1e1caa5-9842-40d8-8492-a82b079879a3>), in state <Peer in Cluste$<o:p></o:p></p><p class=MsoNormal>[2017-06-21 16:18:34.456619] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.3/xlator/mgmt/glusterd.so(+0x1f879) [0x7fee6bc22879] -->/usr/lib/x86_64-l$<o:p></o:p></p><p class=MsoNormal>[2017-06-21 16:18:34.456638] W [MSGID: 106118] [glusterd-handler.c:5913:__glusterd_peer_rpc_notify] 0-management: Lock not released for teravolume<o:p></o:p></p><p class=MsoNormal>[2017-06-21 16:18:34.456661] I [MSGID: 106004] [glusterd-handler.c:5888:__glusterd_peer_rpc_notify] 0-management: Peer <gfsnode3> (<59b9effa-2b88-4764-9130-4f31c14c362e>), in state <Peer in Cluste$<o:p></o:p></p><p class=MsoNormal>[2017-06-21 16:18:34.456692] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.3/xlator/mgmt/glusterd.so(+0x1f879) [0x7fee6bc22879] -->/usr/lib/x86_64-l$<o:p></o:p></p><p class=MsoNormal>[2017-06-21 16:18:43.323944] I [MSGID: 106163] [glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31000<o:p></o:p></p><p class=MsoNormal>[2017-06-21 16:18:34.456699] W [MSGID: 106118] [glusterd-handler.c:5913:__glusterd_peer_rpc_notify] 0-management: Lock not released for teravolume<o:p></o:p></p><p class=MsoNormal>[2017-06-21 16:18:45.628552] I [MSGID: 106163] [glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31000<o:p></o:p></p><p class=MsoNormal>[2017-06-21 16:23:40.607173] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume teravolume<o:p></o:p></p></div></body></html>