<div dir="ltr">Pawan - I couldn't reach to any conclusive analysis so far. But, looking at the client (nfs) & glusterd log files, it does look like that there is an issue w.r.t peer connections. Does restarting all the glusterd one by one solve this?<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, May 29, 2017 at 4:50 PM, Pawan Alwandi <span dir="ltr"><<a href="mailto:pawan@platform.sh" target="_blank">pawan@platform.sh</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Sorry for big attachment in previous mail...last 1000 lines of those logs attached now.<br></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, May 29, 2017 at 4:44 PM, Pawan Alwandi <span dir="ltr"><<a href="mailto:pawan@platform.sh" target="_blank">pawan@platform.sh</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Thu, May 25, 2017 at 9:54 PM, Atin Mukherjee <span dir="ltr"><<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><br><div class="gmail_quote"><span class="m_5371961948397340435m_7653404875185003785gmail-"><div>On Thu, 25 May 2017 at 19:11, Pawan Alwandi <pawan@platform.sh> wrote:<br></div></span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><div><div><div>Hello Atin,<br><br></div><span class="m_5371961948397340435m_7653404875185003785gmail-"><div>Yes, glusterd on other instances are up and running. Below is the requested output on all the three hosts.<br></div><div><br></div>Host 1<br><br><span style="font-family:monospace,monospace"># gluster peer status<br>Number of Peers: 2<br><br>Hostname: 192.168.0.7<br>Uuid: 5ec54b4f-f60c-48c6-9e55-95f2bb<wbr>58f633<br>State: Peer in Cluster (Disconnected)</span></span></div></div></div></div></blockquote><div><br></div><div>Glusterd is disconnected here.</div><span class="m_5371961948397340435m_7653404875185003785gmail-"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><div><div><span style="font-family:monospace,monospace"><br><br>Hostname: 192.168.0.6<br>Uuid: 83e9a0b9-6bd5-483b-8516-d89288<wbr>05ed95<br>State: Peer in Cluster (Disconnected)</span></div></div></div></div></blockquote><div><br></div></span><div>Same as above</div><div><br></div><div>Can you please check what does glusterd log have to say here about these disconnects?</div></div></div></blockquote><div><br></div></span><div>glusterd keeps logging this every 3s<br></div><div><br><span style="font-family:monospace,monospace">[2017-05-29 11:04:52.182782] W [socket.c:852:__socket_keepali<wbr>ve] 0-socket: failed to set keep idle -1 on socket 5, Invalid argument<br>[2017-05-29 11:04:52.182808] E [socket.c:2966:socket_connect] 0-management: Failed to set keep-alive: Invalid argument<br>[2017-05-29 11:04:52.183032] W [socket.c:852:__socket_keepali<wbr>ve] 0-socket: failed to set keep idle -1 on socket 20, Invalid argument<br>[2017-05-29 11:04:52.183052] E [socket.c:2966:socket_connect] 0-management: Failed to set keep-alive: Invalid argument<br>[2017-05-29 11:04:52.183622] E [rpc-clnt.c:362:saved_frames_u<wbr>nwind] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>lusterfs.so.0(_gf_log_callingf<wbr>n+0x1a3)[0x7f767c46d483] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>frpc.so.0(saved_frames_unwind+<wbr>0x1cf)[0x7f767c2383af] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>frpc.so.0(saved_frames_destroy<wbr>+0xe)[0x7f767c2384ce] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>frpc.so.0(rpc_clnt_connection_<wbr>cleanup+0x7e)[0x7f767c239c8e] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>frpc.so.0(rpc_clnt_notify+0x88<wbr>)[0x7f767c23a4a8] ))))) 0-management: forced unwinding frame type(GLUSTERD-DUMP) op(DUMP(1)) called at 2017-05-29 11:04:52.183210 (xid=0x23419)<br>[2017-05-29 11:04:52.183735] W [glusterd-locks.c:681:glusterd<wbr>_mgmt_v3_unlock] (-->/usr/lib/x86_64-linux-gnu/<wbr>glusterfs/3.7.9/xlator/mgmt/gl<wbr>usterd.so(glusterd_big_locked_<wbr>notify+0x4b) [0x7f767734dffb] -->/usr/lib/x86_64-linux-gnu/g<wbr>lusterfs/3.7.9/xlator/mgmt/glu<wbr>sterd.so(__glusterd_peer_rpc_<wbr>notify+0x14a) [0x7f7677357c6a] -->/usr/lib/x86_64-linux-gnu/g<wbr>lusterfs/3.7.9/xlator/mgmt/glu<wbr>sterd.so(glusterd_mgmt_v3_unlo<wbr>ck+0x4c3) [0x7f76773f0ef3] ) 0-management: Lock for vol shared not held<br>[2017-05-29 11:04:52.183928] E [rpc-clnt.c:362:saved_frames_u<wbr>nwind] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>lusterfs.so.0(_gf_log_callingf<wbr>n+0x1a3)[0x7f767c46d483] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>frpc.so.0(saved_frames_unwind+<wbr>0x1cf)[0x7f767c2383af] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>frpc.so.0(saved_frames_destroy<wbr>+0xe)[0x7f767c2384ce] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>frpc.so.0(rpc_clnt_connection_<wbr>cleanup+0x7e)[0x7f767c239c8e] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>frpc.so.0(rpc_clnt_notify+0x88<wbr>)[0x7f767c23a4a8] ))))) 0-management: forced unwinding frame type(GLUSTERD-DUMP) op(DUMP(1)) called at 2017-05-29 11:04:52.183422 (xid=0x23419)<br>[2017-05-29 11:04:52.184027] W [glusterd-locks.c:681:glusterd<wbr>_mgmt_v3_unlock] (-->/usr/lib/x86_64-linux-gnu/<wbr>glusterfs/3.7.9/xlator/mgmt/gl<wbr>usterd.so(glusterd_big_locked_<wbr>notify+0x4b) [0x7f767734dffb] -->/usr/lib/x86_64-linux-gnu/g<wbr>lusterfs/3.7.9/xlator/mgmt/glu<wbr>sterd.so(__glusterd_peer_rpc_<wbr>notify+0x14a) [0x7f7677357c6a] -->/usr/lib/x86_64-linux-gnu/g<wbr>lusterfs/3.7.9/xlator/mgmt/glu<wbr>sterd.so(glusterd_mgmt_v3_unlo<wbr>ck+0x4c3) [0x7f76773f0ef3] ) 0-management: Lock for vol shared not held</span><br><br> </div><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail_quote"><span class="m_5371961948397340435m_7653404875185003785gmail-"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><div><div><span style="font-family:monospace,monospace"><br><br># gluster volume status<br>Status of volume: shared<br>Gluster process <wbr> TCP Port RDMA Port Online Pid<br>------------------------------<wbr>------------------------------<wbr>------------------<br>Brick 192.168.0.5:/data/exports/shar<wbr>ed 49152 0 Y 2105 <br>NFS Server on localhost 2049 0 Y 2089 <br>Self-heal Daemon on localhost N/A N/A Y 2097 </span></div></div></div></div></blockquote><div><br></div></span><div>Volume status output does show all the bricks are up. So I'm not sure why are you seeing the volume as read only. Can you please provide the mount log?</div></div></div></blockquote><div><br></div></span><div>The attached tar has nfs.log, etc-glusterfs-glusterd.vol.log<wbr>, glustershd.log from host1.<br></div><div><div class="m_5371961948397340435h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail_quote"><div><div class="m_5371961948397340435m_7653404875185003785gmail-h5"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><div><div><span style="font-family:monospace,monospace"><br> <br>Task Status of Volume shared<br>------------------------------<wbr>------------------------------<wbr>------------------<br>There are no active volume tasks</span><br><br></div>Host 2<br><br><span style="font-family:monospace,monospace"># gluster peer status<br>Number of Peers: 2<br><br>Hostname: 192.168.0.7<br>Uuid: 5ec54b4f-f60c-48c6-9e55-95f2bb<wbr>58f633<br>State: Peer in Cluster (Connected)<br><br>Hostname: 192.168.0.5<br>Uuid: 7f2a6e11-2a53-4ab4-9ceb-8be6a9<wbr>f2d073<br>State: Peer in Cluster (Connected)</span><br><br><br><span style="font-family:monospace,monospace"># gluster volume status<br>Status of volume: shared<br>Gluster process Port Online Pid<br>------------------------------<wbr>------------------------------<wbr>------------------<br>Brick 192.168.0.5:/data/exports/shar<wbr>ed 49152 Y 2105<br>Brick 192.168.0.6:/data/exports/shar<wbr>ed 49152 Y 2188<br>Brick 192.168.0.7:/data/exports/shar<wbr>ed 49152 Y 2453<br>NFS Server on localhost 2049 Y 2194<br>Self-heal Daemon on localhost N/A Y 2199<br>NFS Server on 192.168.0.5 2049 Y 2089<br>Self-heal Daemon on 192.168.0.5 N/A Y 2097<br>NFS Server on 192.168.0.7 2049 Y 2458<br>Self-heal Daemon on 192.168.0.7 N/A Y 2463<br> <br>Task Status of Volume shared<br>------------------------------<wbr>------------------------------<wbr>------------------<br>There are no active volume tasks</span><br><br></div>Host 3<br><br><span style="font-family:monospace,monospace"># gluster peer status<br>Number of Peers: 2<br><br>Hostname: 192.168.0.5<br>Uuid: 7f2a6e11-2a53-4ab4-9ceb-8be6a9<wbr>f2d073<br>State: Peer in Cluster (Connected)<br><br>Hostname: 192.168.0.6<br>Uuid: 83e9a0b9-6bd5-483b-8516-d89288<wbr>05ed95<br>State: Peer in Cluster (Connected)<br><br></span></div><span style="font-family:monospace,monospace"># gluster volume status<br>Status of volume: shared<br>Gluster process Port Online Pid<br>------------------------------<wbr>------------------------------<wbr>------------------<br>Brick 192.168.0.5:/data/exports/shar<wbr>ed 49152 Y 2105<br>Brick 192.168.0.6:/data/exports/shar<wbr>ed 49152 Y 2188<br>Brick 192.168.0.7:/data/exports/shar<wbr>ed 49152 Y 2453<br>NFS Server on localhost 2049 Y 2458<br>Self-heal Daemon on localhost N/A Y 2463<br>NFS Server on 192.168.0.6 2049 Y 2194<br>Self-heal Daemon on 192.168.0.6 N/A Y 2199<br>NFS Server on 192.168.0.5 2049 Y 2089<br>Self-heal Daemon on 192.168.0.5 N/A Y 2097<br> <br>Task Status of Volume shared<br>------------------------------<wbr>------------------------------<wbr>------------------<br>There are no active volume tasks</span><br><div><div><div><div><div><div><br><br><br><br><br></div></div></div></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 24, 2017 at 8:32 PM, Atin Mukherjee <span><<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>Are the other glusterd instances are up? output of gluster peer status & gluster volume status please?<br></div><div class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929HOEnZb"><div class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 24, 2017 at 4:20 PM, Pawan Alwandi <span><<a href="mailto:pawan@platform.sh" target="_blank">pawan@platform.sh</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><div><div>Thanks Atin,<br><br></div>So I got gluster downgraded to 3.7.9 on host 1 and now have the glusterfs and glusterfsd processes come up. But I see the volume is mounted read only.<br><br></div>I see these being logged every 3s:<br><br><span style="font-family:monospace,monospace">[2017-05-24 10:45:44.440435] W [socket.c:852:__socket_keepali<wbr>ve] 0-socket: failed to set keep idle -1 on socket 17, Invalid argument<br>[2017-05-24 10:45:44.440475] E [socket.c:2966:socket_connect] 0-management: Failed to set keep-alive: Invalid argument<br>[2017-05-24 10:45:44.440734] W [socket.c:852:__socket_keepali<wbr>ve] 0-socket: failed to set keep idle -1 on socket 20, Invalid argument<br>[2017-05-24 10:45:44.440754] E [socket.c:2966:socket_connect] 0-management: Failed to set keep-alive: Invalid argument<br>[2017-05-24 10:45:44.441354] E [rpc-clnt.c:362:saved_frames_u<wbr>nwind] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>lusterfs.so.0(_gf_log_callingf<wbr>n+0x1a3)[0x7f767c46d483] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>frpc.so.0(saved_frames_unwind+<wbr>0x1cf)[0x7f767c2383af] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>frpc.so.0(saved_frames_destroy<wbr>+0xe)[0x7f767c2384ce] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>frpc.so.0(rpc_clnt_connection_<wbr>cleanup+0x7e)[0x7f767c239c8e] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>frpc.so.0(rpc_clnt_notify+0x88<wbr>)[0x7f767c23a4a8] ))))) 0-management: forced unwinding frame type(GLUSTERD-DUMP) op(DUMP(1)) called at 2017-05-24 10:45:44.440945 (xid=0xbf)<br>[2017-05-24 10:45:44.441505] W [glusterd-locks.c:681:glusterd<wbr>_mgmt_v3_unlock] (-->/usr/lib/x86_64-linux-gnu/<wbr>glusterfs/3.7.9/xlator/mgmt/gl<wbr>usterd.so(glusterd_big_locked_<wbr>notify+0x4b) [0x7f767734dffb] -->/usr/lib/x86_64-linux-gnu/g<wbr>lusterfs/3.7.9/xlator/mgmt/glu<wbr>sterd.so(__glusterd_peer_rpc_n<wbr>otify+0x14a) [0x7f7677357c6a] -->/usr/lib/x86_64-linux-gnu/g<wbr>lusterfs/3.7.9/xlator/mgmt/glu<wbr>sterd.so(glusterd_mgmt_v3_unlo<wbr>ck+0x4c3) [0x7f76773f0ef3] ) 0-management: Lock for vol shared not held<br>[2017-05-24 10:45:44.441660] E [rpc-clnt.c:362:saved_frames_u<wbr>nwind] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>lusterfs.so.0(_gf_log_callingf<wbr>n+0x1a3)[0x7f767c46d483] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>frpc.so.0(saved_frames_unwind+<wbr>0x1cf)[0x7f767c2383af] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>frpc.so.0(saved_frames_destroy<wbr>+0xe)[0x7f767c2384ce] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>frpc.so.0(rpc_clnt_connection_<wbr>cleanup+0x7e)[0x7f767c239c8e] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>frpc.so.0(rpc_clnt_notify+0x88<wbr>)[0x7f767c23a4a8] ))))) 0-management: forced unwinding frame type(GLUSTERD-DUMP) op(DUMP(1)) called at 2017-05-24 10:45:44.441086 (xid=0xbf)<br>[2017-05-24 10:45:44.441790] W [glusterd-locks.c:681:glusterd<wbr>_mgmt_v3_unlock] (-->/usr/lib/x86_64-linux-gnu/<wbr>glusterfs/3.7.9/xlator/mgmt/gl<wbr>usterd.so(glusterd_big_locked_<wbr>notify+0x4b) [0x7f767734dffb] -->/usr/lib/x86_64-linux-gnu/g<wbr>lusterfs/3.7.9/xlator/mgmt/glu<wbr>sterd.so(__glusterd_peer_rpc_n<wbr>otify+0x14a) [0x7f7677357c6a] -->/usr/lib/x86_64-linux-gnu/g<wbr>lusterfs/3.7.9/xlator/mgmt/glu<wbr>sterd.so(glusterd_mgmt_v3_unlo<wbr>ck+0x4c3) [0x7f76773f0ef3] ) 0-management: Lock for vol shared not held</span><br><br></div>The heal info says this:<br><div><br><span style="font-family:monospace,monospace"># gluster volume heal shared info<br>Brick 192.168.0.5:/data/exports/shar<wbr>ed<br>Number of entries: 0<br><br>Brick 192.168.0.6:/data/exports/shar<wbr>ed<br>Status: Transport endpoint is not connected<br><br>Brick 192.168.0.7:/data/exports/shar<wbr>ed<br>Status: Transport endpoint is not connected<br></span><br></div><div>Any idea whats up here?<span class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019HOEnZb"><font color="#888888"><br></font></span></div><span class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019HOEnZb"><font color="#888888"><div><br></div><div>Pawan<br></div></font></span></div><div class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019HOEnZb"><div class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, May 22, 2017 at 9:42 PM, Atin Mukherjee <span><<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Mon, May 22, 2017 at 9:05 PM, Pawan Alwandi <span><<a href="mailto:pawan@platform.sh" target="_blank">pawan@platform.sh</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail_extra"><br><div class="gmail_quote"><span class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019m_-4654240943671477651m_-5453945879301133293gmail-">On Mon, May 22, 2017 at 8:36 PM, Atin Mukherjee <span><<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019m_-4654240943671477651m_-5453945879301133293gmail-m_-3457795168368374547gmail-">On Mon, May 22, 2017 at 7:51 PM, Atin Mukherjee <span><<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div>Sorry Pawan, I did miss the other part of the attachments. So looking from the <a href="http://glusterd.info" target="_blank">glusterd.info</a> file from all the hosts, it looks like host2 and host3 do not have the correct op-version. Can you please set the op-version as "operating-version=30702" in host2 and host3 and restart glusterd instance one by one on all the nodes?<br></div></div></blockquote><div><br></div></span><div>Please ensure that all the hosts are upgraded to the same bits before doing this change.<br></div></div></div></div></blockquote></span><div class="gmail_quote"><br>Having to upgrade all 3 hosts to newer version before gluster could work successfully on any of them means application downtime. The applications running on these hosts
are expected to be highly available. So with the way the things are right now, is an online upgrade possible? My upgrade steps are: (1) stop the applications (2) umount the gluster volume, and then (3) upgrade gluster one host at a time.<br></div></div></div></div></blockquote></span><div><br>One of the way to mitigate this is to first do an online upgrade to
glusterfs-3.7.9 (op-version:30707) given this bug was introduced in
3.7.10 and then come to 3.11.<br><br> </div><div><div class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019m_-4654240943671477651h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail_extra"><div class="gmail_quote"><div class="gmail_quote"><br><div>Our goal is to get gluster upgraded to 3.11 from 3.6.9, and to make this an online upgrade we are okay to take two steps 3.6.9 -> 3.7 and then 3.7 to 3.11.<br></div><div><div class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019m_-4654240943671477651m_-5453945879301133293gmail-h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail_extra"><div class="gmail_quote"><div> <br></div><span class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019m_-4654240943671477651m_-5453945879301133293gmail-m_-3457795168368374547gmail-"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><br></div>Apparently it looks like there is a bug which you have uncovered, during peer handshaking if one of the glusterd instance is running with old bits then during validating the handshake request there is a possibility that uuid received will be blank and the same was ignored however there was a patch <a href="http://review.gluster.org/13519" target="_blank">http://review.gluster.org/1351<wbr>9</a> which had some additional changes which was always looking at this field and doing some extra checks which was causing the handshake to fail. For now, the above workaround should suffice. I'll be sending a patch pretty soon.<br></div></blockquote><div><br></div></span><div>Posted a patch <a href="https://review.gluster.org/#/c/17358" target="_blank">https://review.gluster.org/#/c<wbr>/17358</a> . <br><br></div><div><div class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019m_-4654240943671477651m_-5453945879301133293gmail-m_-3457795168368374547gmail-h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><br></div></blockquote></div></div></div></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail_extra"><div class="gmail_quote"><div><div class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019m_-4654240943671477651m_-5453945879301133293gmail-m_-3457795168368374547gmail-h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><br><br></div><div class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019m_-4654240943671477651m_-5453945879301133293gmail-m_-3457795168368374547gmail-m_-6125136540958523807gmail-HOEnZb"><div class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019m_-4654240943671477651m_-5453945879301133293gmail-m_-3457795168368374547gmail-m_-6125136540958523807gmail-h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, May 22, 2017 at 11:35 AM, Pawan Alwandi <span><<a href="mailto:pawan@platform.sh" target="_blank">pawan@platform.sh</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><div>Hello Atin,<br><br></div>The tar's have the content of `/var/lib/glusterd` too for all 3 nodes, please check again.<br><br></div>Thanks<br></div><div class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019m_-4654240943671477651m_-5453945879301133293gmail-m_-3457795168368374547gmail-m_-6125136540958523807gmail-m_-3407916076621243690HOEnZb"><div class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019m_-4654240943671477651m_-5453945879301133293gmail-m_-3457795168368374547gmail-m_-6125136540958523807gmail-m_-3407916076621243690h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, May 22, 2017 at 11:32 AM, Atin Mukherjee <span><<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div>Pawan,<br><br></div>I see you have provided the log files from the nodes, however it'd be really helpful if you can provide me the content of /var/lib/glusterd from all the nodes to get to the root cause of this issue.<br></div><div class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019m_-4654240943671477651m_-5453945879301133293gmail-m_-3457795168368374547gmail-m_-6125136540958523807gmail-m_-3407916076621243690m_-8487362792773169404HOEnZb"><div class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019m_-4654240943671477651m_-5453945879301133293gmail-m_-3457795168368374547gmail-m_-6125136540958523807gmail-m_-3407916076621243690m_-8487362792773169404h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, May 19, 2017 at 12:09 PM, Pawan Alwandi <span><<a href="mailto:pawan@platform.sh" target="_blank">pawan@platform.sh</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><div>Hello Atin,<br><br></div>Thanks for continued support. I've attached requested files from all 3 nodes.<br><br></div><div>(I think we already verified the UUIDs to be correct, anyway let us know if you find any more info in the logs)<span class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019m_-4654240943671477651m_-5453945879301133293gmail-m_-3457795168368374547gmail-m_-6125136540958523807gmail-m_-3407916076621243690m_-8487362792773169404m_2275467663456357232HOEnZb"><font color="#888888"><br></font></span></div><span class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019m_-4654240943671477651m_-5453945879301133293gmail-m_-3457795168368374547gmail-m_-6125136540958523807gmail-m_-3407916076621243690m_-8487362792773169404m_2275467663456357232HOEnZb"><font color="#888888"><div><br></div>Pawan<br></font></span></div><div class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019m_-4654240943671477651m_-5453945879301133293gmail-m_-3457795168368374547gmail-m_-6125136540958523807gmail-m_-3407916076621243690m_-8487362792773169404m_2275467663456357232HOEnZb"><div class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019m_-4654240943671477651m_-5453945879301133293gmail-m_-3457795168368374547gmail-m_-6125136540958523807gmail-m_-3407916076621243690m_-8487362792773169404m_2275467663456357232h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, May 18, 2017 at 11:45 PM, Atin Mukherjee <span><<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><br><div class="gmail_quote"><div><div class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019m_-4654240943671477651m_-5453945879301133293gmail-m_-3457795168368374547gmail-m_-6125136540958523807gmail-m_-3407916076621243690m_-8487362792773169404m_2275467663456357232m_-6419501248012758644h5"><div>On Thu, 18 May 2017 at 23:40, Atin Mukherjee <<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail_quote"><div>On Wed, 17 May 2017 at 12:47, Pawan Alwandi <pawan@platform.sh> wrote:<br></div></div></div><div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><div><div>Hello Atin,<br><br></div></div></div></div></blockquote></div></div><div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><div>I realized that these <a href="http://gluster.readthedocs.io/en/latest/Upgrade-Guide/upgrade_to_3.10/" target="_blank">http://gluster.readthedocs.io/<wbr>en/latest/Upgrade-Guide/upgrad<wbr>e_to_3.10/</a> instructions only work for upgrades from 3.7, while we are running 3.6.2. Are there instructions/suggestion you have for us to upgrade from 3.6 version?<br><br></div></div></div></blockquote></div></div><div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div>I believe upgrade from 3.6 to 3.7 and then to 3.10 would work, but I see similar errors reported when I upgraded to 3.7 too.<br><br></div></div></blockquote></div></div><div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>For what its worth, I was able to set the op-version (gluster v set all cluster.op-version 30702) but that doesn't seem to help.<br><div><br><span style="font-family:monospace,monospace">[2017-05-17 06:48:33.700014] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.7.20 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid)<br>[2017-05-17 06:48:33.703808] I [MSGID: 106478] [glusterd.c:1383:init] 0-management: Maximum allowed open file descriptors set to 65536<br>[2017-05-17 06:48:33.703836] I [MSGID: 106479] [glusterd.c:1432:init] 0-management: Using /var/lib/glusterd as working directory<br>[2017-05-17 06:48:33.708866] W [MSGID: 103071] [rdma.c:4594:__gf_rdma_ctx_cre<wbr>ate] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device]<br>[2017-05-17 06:48:33.709011] W [MSGID: 103055] [rdma.c:4901:init] 0-rdma.management: Failed to initialize IB Device<br>[2017-05-17 06:48:33.709033] W [rpc-transport.c:359:rpc_trans<wbr>port_load] 0-rpc-transport: 'rdma' initialization failed<br>[2017-05-17 06:48:33.709088] W [rpcsvc.c:1642:rpcsvc_create_l<wbr>istener] 0-rpc-service: cannot create listener, initing the transport failed<br>[2017-05-17 06:48:33.709105] E [MSGID: 106243] [glusterd.c:1656:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport<br>[2017-05-17 06:48:35.480043] I [MSGID: 106513] [glusterd-store.c:2068:gluster<wbr>d_restore_op_version] 0-glusterd: retrieved op-version: 30600<br>[2017-05-17 06:48:35.605779] I [MSGID: 106498] [glusterd-handler.c:3640:glust<wbr>erd_friend_add_from_peerinfo] 0-management: connect returned 0<br>[2017-05-17 06:48:35.607059] I [rpc-clnt.c:1046:rpc_clnt_conn<wbr>ection_init] 0-management: setting frame-timeout to 600<br>[2017-05-17 06:48:35.607670] I [rpc-clnt.c:1046:rpc_clnt_conn<wbr>ection_init] 0-management: setting frame-timeout to 600<br>[2017-05-17 06:48:35.607025] I [MSGID: 106498] [glusterd-handler.c:3640:glust<wbr>erd_friend_add_from_peerinfo] 0-management: connect returned 0<br>[2017-05-17 06:48:35.608125] I [MSGID: 106544] [glusterd.c:159:glusterd_uuid_<wbr>init] 0-management: retrieved UUID: 7f2a6e11-2a53-4ab4-9ceb-8be6a9<wbr>f2d073</span></div></div></blockquote></div></div><div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><span style="font-family:monospace,monospace"><br>Final graph:<br>+-----------------------------<wbr>------------------------------<wbr>-------------------+<br> 1: volume management<br> 2: type mgmt/glusterd<br> 3: option rpc-auth.auth-glusterfs on<br> 4: option rpc-auth.auth-unix on<br> 5: option rpc-auth.auth-null on<br> 6: option rpc-auth-allow-insecure on<br> 7: option transport.socket.listen-backlo<wbr>g 128<br> 8: option event-threads 1<br> 9: option ping-timeout 0<br> 10: option transport.socket.read-fail-log off<br> 11: option transport.socket.keepalive-int<wbr>erval 2<br> 12: option transport.socket.keepalive-tim<wbr>e 10<br> 13: option transport-type rdma<br> 14: option working-directory /var/lib/glusterd<br> 15: end-volume<br> 16: <br>+-----------------------------<wbr>------------------------------<wbr>-------------------+<br></span></div></div><div><div><span style="font-family:monospace,monospace">[2017-05-17 06:48:35.609868] I [MSGID: 101190] [event-epoll.c:632:event_dispa<wbr>tch_epoll_worker] 0-epoll: Started thread with index 1<br>[2017-05-17 06:48:35.610839] W [socket.c:596:__socket_rwv] 0-management: readv on <a href="http://192.168.0.7:24007" target="_blank">192.168.0.7:24007</a> failed (No data available)<br>[2017-05-17 06:48:35.611907] E [rpc-clnt.c:370:saved_frames_u<wbr>nwind] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>lusterfs.so.0(_gf_log_callingf<wbr>n+0x1a3)[0x7fd6c2d70bb3] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>frpc.so.0(saved_frames_unwind+<wbr>0x1cf)[0x7fd6c2b3a2df] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>frpc.so.0(saved_frames_destroy<wbr>+0xe)[0x7fd6c2b3a3fe] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>frpc.so.0(rpc_clnt_connection_<wbr>cleanup+0x89)[0x7fd6c2b3ba39] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>frpc.so.0(rpc_clnt_notify+0x16<wbr>0)[0x7fd6c2b3c380] ))))) 0-management: forced unwinding frame type(GLUSTERD-DUMP) op(DUMP(1)) called at 2017-05-17 06:48:35.609965 (xid=0x1)<br>[2017-05-17 06:48:35.611928] E [MSGID: 106167] [glusterd-handshake.c:2091:__g<wbr>lusterd_peer_dump_version_cbk] 0-management: Error through RPC layer, retry again later<br>[2017-05-17 06:48:35.611944] I [MSGID: 106004] [glusterd-handler.c:5201:__glu<wbr>sterd_peer_rpc_notify] 0-management: Peer <192.168.0.7> (<5ec54b4f-f60c-48c6-9e55-95f2<wbr>bb58f633>), in state <Peer in Cluster>, has disconnected from glusterd.<br>[2017-05-17 06:48:35.612024] W [glusterd-locks.c:681:glusterd<wbr>_mgmt_v3_unlock] (-->/usr/lib/x86_64-linux-gnu/<wbr>glusterfs/3.7.20/xlator/mgmt/g<wbr>lusterd.so(glusterd_big_locked<wbr>_notify+0x4b) [0x7fd6bdc4912b] -->/usr/lib/x86_64-linux-gnu/g<wbr>lusterfs/3.7.20/xlator/mgmt/gl<wbr>usterd.so(__glusterd_peer_rpc_<wbr>notify+0x160) [0x7fd6bdc52dd0] -->/usr/lib/x86_64-linux-gnu/g<wbr>lusterfs/3.7.20/xlator/mgmt/gl<wbr>usterd.so(glusterd_mgmt_v3_unl<wbr>ock+0x4c3) [0x7fd6bdcef1b3] ) 0-management: Lock for vol shared not held<br>[2017-05-17 06:48:35.612039] W [MSGID: 106118] [glusterd-handler.c:5223:__glu<wbr>sterd_peer_rpc_notify] 0-management: Lock not released for shared<br>[2017-05-17 06:48:35.612079] W [socket.c:596:__socket_rwv] 0-management: readv on <a href="http://192.168.0.6:24007" target="_blank">192.168.0.6:24007</a> failed (No data available)<br>[2017-05-17 06:48:35.612179] E [rpc-clnt.c:370:saved_frames_u<wbr>nwind] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>lusterfs.so.0(_gf_log_callingf<wbr>n+0x1a3)[0x7fd6c2d70bb3] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>frpc.so.0(saved_frames_unwind+<wbr>0x1cf)[0x7fd6c2b3a2df] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>frpc.so.0(saved_frames_destroy<wbr>+0xe)[0x7fd6c2b3a3fe] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>frpc.so.0(rpc_clnt_connection_<wbr>cleanup+0x89)[0x7fd6c2b3ba39] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>frpc.so.0(rpc_clnt_notify+0x16<wbr>0)[0x7fd6c2b3c380] ))))) 0-management: forced unwinding frame type(GLUSTERD-DUMP) op(DUMP(1)) called at 2017-05-17 06:48:35.610007 (xid=0x1)<br>[2017-05-17 06:48:35.612197] E [MSGID: 106167] [glusterd-handshake.c:2091:__g<wbr>lusterd_peer_dump_version_cbk] 0-management: Error through RPC layer, retry again later<br>[2017-05-17 06:48:35.612211] I [MSGID: 106004] [glusterd-handler.c:5201:__glu<wbr>sterd_peer_rpc_notify] 0-management: Peer <192.168.0.6> (<83e9a0b9-6bd5-483b-8516-d892<wbr>8805ed95>), in state <Peer in Cluster>, has disconnected from glusterd.<br>[2017-05-17 06:48:35.612292] W [glusterd-locks.c:681:glusterd<wbr>_mgmt_v3_unlock] (-->/usr/lib/x86_64-linux-gnu/<wbr>glusterfs/3.7.20/xlator/mgmt/g<wbr>lusterd.so(glusterd_big_locked<wbr>_notify+0x4b) [0x7fd6bdc4912b] -->/usr/lib/x86_64-linux-gnu/g<wbr>lusterfs/3.7.20/xlator/mgmt/gl<wbr>usterd.so(__glusterd_peer_rpc_<wbr>notify+0x160) [0x7fd6bdc52dd0] -->/usr/lib/x86_64-linux-gnu/g<wbr>lusterfs/3.7.20/xlator/mgmt/gl<wbr>usterd.so(glusterd_mgmt_v3_unl<wbr>ock+0x4c3) [0x7fd6bdcef1b3] ) 0-management: Lock for vol shared not held<br>[2017-05-17 06:48:35.613432] W [MSGID: 106118] [glusterd-handler.c:5223:__glu<wbr>sterd_peer_rpc_notify] 0-management: Lock not released for shared<br>[2017-05-17 06:48:35.614317] E [MSGID: 106170] [glusterd-handshake.c:1051:gd_<wbr>validate_mgmt_hndsk_req] 0-management: Request from peer <a href="http://192.168.0.6:991" target="_blank">192.168.0.6:991</a> has an entry in peerinfo, but uuid does not match</span></div></div></blockquote></div></div><div><div class="gmail_quote"><div><br></div><div>Apologies for delay. My initial suspect was correct. You have an incorrect UUID in the peer file which is causing this. Can you please provide me the </div></div></div></blockquote><div><br></div></div></div><div>Clicked the send button accidentally!</div><div><br></div><div>Can you please send me the content of /var/lib/glusterd & glusterd log from all the nodes?</div><div><div class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019m_-4654240943671477651m_-5453945879301133293gmail-m_-3457795168368374547gmail-m_-6125136540958523807gmail-m_-3407916076621243690m_-8487362792773169404m_2275467663456357232m_-6419501248012758644h5"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><span style="font-family:monospace,monospace"></span><br><br><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, May 15, 2017 at 10:31 PM, Atin Mukherjee <span><<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><br><div class="gmail_quote"><span><div>On Mon, 15 May 2017 at 11:58, Pawan Alwandi <pawan@platform.sh> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div>Hi Atin,<br><br></div>I see below error. Do I require gluster to be upgraded on all 3 hosts for this to work? Right now I have host 1 running 3.10.1 and host 2 & 3 running 3.6.2<br><div><br><span style="font-family:monospace,monospace"># gluster v set all cluster.op-version 31001<br>volume set: failed: Required op_version (31001) is not supported</span></div></div></blockquote><div><br></div></span><div>Yes you should given 3.6 version is EOLed.</div><div><div class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019m_-4654240943671477651m_-5453945879301133293gmail-m_-3457795168368374547gmail-m_-6125136540958523807gmail-m_-3407916076621243690m_-8487362792773169404m_2275467663456357232m_-6419501248012758644m_11629714894935959m_-3519483590900227861m_6025082608089837941h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><span style="font-family:monospace,monospace"></span><br><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, May 15, 2017 at 3:32 AM, Atin Mukherjee <span><<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span><div><div><div class="gmail_quote"><div>On Sun, 14 May 2017 at 21:43, Atin Mukherjee <<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div>Allright, I see that you haven't bumped up the op-version. Can you please execute:<br><br></div>gluster v set all cluster.op-version 30101 and then restart glusterd on all the nodes and check the brick status?</div></blockquote><div><br></div></div></div></div></span><div><div><div class="gmail_quote"><div>s/30101/31001</div><div><br></div></div></div></div><div class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019m_-4654240943671477651m_-5453945879301133293gmail-m_-3457795168368374547gmail-m_-6125136540958523807gmail-m_-3407916076621243690m_-8487362792773169404m_2275467663456357232m_-6419501248012758644m_11629714894935959m_-3519483590900227861m_6025082608089837941m_-6808072551270144839m_6351166450766159261HOEnZb"><div class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019m_-4654240943671477651m_-5453945879301133293gmail-m_-3457795168368374547gmail-m_-6125136540958523807gmail-m_-3407916076621243690m_-8487362792773169404m_2275467663456357232m_-6419501248012758644m_11629714894935959m_-3519483590900227861m_6025082608089837941m_-6808072551270144839m_6351166450766159261h5"><div><div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, May 14, 2017 at 8:55 PM, Pawan Alwandi <span><<a href="mailto:pawan@platform.sh" target="_blank">pawan@platform.sh</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><div>Hello Atin,<br><br></div><div>Thanks for looking at this. Below is the output you requested for.<br><br></div><div>Again, I'm seeing those errors after upgrading gluster on host 1.<br></div><div><br></div>Host 1<br><br><span style="font-family:monospace,monospace"># cat /var/lib/glusterd/<a href="http://glusterd.info" target="_blank">glusterd.inf<wbr>o</a><br>UUID=7f2a6e11-2a53-4ab4-9ceb-8<wbr>be6a9f2d073<br>operating-version=30600<br><br># cat /var/lib/glusterd/peers/*<br>uuid=5ec54b4f-f60c-48c6-9e55-9<wbr>5f2bb58f633<br>state=3<br>hostname1=192.168.0.7<br>uuid=83e9a0b9-6bd5-483b-8516-d<wbr>8928805ed95<br>state=3<br>hostname1=192.168.0.6<br><br># gluster --version<br>glusterfs 3.10.1<br><br></span></div>Host 2<br><br><span style="font-family:monospace,monospace"># cat /var/lib/glusterd/<a href="http://glusterd.info" target="_blank">glusterd.inf<wbr>o</a><br>UUID=83e9a0b9-6bd5-483b-8516-d<wbr>8928805ed95<br>operating-version=30600<br><br># cat /var/lib/glusterd/peers/*<br>uuid=5ec54b4f-f60c-48c6-9e55-9<wbr>5f2bb58f633<br>state=3<br>hostname1=192.168.0.7<br>uuid=7f2a6e11-2a53-4ab4-9ceb-8<wbr>be6a9f2d073<br>state=3<br>hostname1=192.168.0.5<br><br># gluster --version<br>glusterfs 3.6.2 built on Jan 21 2015 14:23:44<br></span><br><div>Host 3<br><span style="font-family:monospace,monospace"><br># cat /var/lib/glusterd/<a href="http://glusterd.info" target="_blank">glusterd.inf<wbr>o</a><br>UUID=5ec54b4f-f60c-48c6-9e55-9<wbr>5f2bb58f633<br>operating-version=30600<br><br># cat /var/lib/glusterd/peers/*<br>uuid=7f2a6e11-2a53-4ab4-9ceb-8<wbr>be6a9f2d073<br>state=3<br>hostname1=192.168.0.5<br>uuid=83e9a0b9-6bd5-483b-8516-d<wbr>8928805ed95<br>state=3<br>hostname1=192.168.0.6<br><br># gluster --version<br>glusterfs 3.6.2 built on Jan 21 2015 14:23:44<br><br></span><span style="font-family:monospace,monospace"><br></span></div></div><div class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019m_-4654240943671477651m_-5453945879301133293gmail-m_-3457795168368374547gmail-m_-6125136540958523807gmail-m_-3407916076621243690m_-8487362792773169404m_2275467663456357232m_-6419501248012758644m_11629714894935959m_-3519483590900227861m_6025082608089837941m_-6808072551270144839m_6351166450766159261m_217048032756755719m_-7911327992255523967m_8557086518318074861HOEnZb"><div class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019m_-4654240943671477651m_-5453945879301133293gmail-m_-3457795168368374547gmail-m_-6125136540958523807gmail-m_-3407916076621243690m_-8487362792773169404m_2275467663456357232m_-6419501248012758644m_11629714894935959m_-3519483590900227861m_6025082608089837941m_-6808072551270144839m_6351166450766159261m_217048032756755719m_-7911327992255523967m_8557086518318074861h5"><div class="gmail_extra"><br><div class="gmail_quote">On Sat, May 13, 2017 at 6:28 PM, Atin Mukherjee <span><<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>I have already asked for the following earlier:</div><span><div><br></div><div><div style="color:rgb(49,49,49);word-spacing:1px"><div>Can you please provide output of following from all the nodes:<br><br></div>cat /var/lib/glusterd/<a href="http://glusterd.info/" target="_blank">glusterd.inf<wbr>o</a><br></div><span style="color:rgb(49,49,49);word-spacing:1px;background-color:rgb(255,255,255)">cat /var/lib/glusterd/peers/*</span></div></span><div><font color="#313131"><span style="word-spacing:1px;background-color:rgb(255,255,255)"><br></span></font><div class="gmail_quote"><div><div class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019m_-4654240943671477651m_-5453945879301133293gmail-m_-3457795168368374547gmail-m_-6125136540958523807gmail-m_-3407916076621243690m_-8487362792773169404m_2275467663456357232m_-6419501248012758644m_11629714894935959m_-3519483590900227861m_6025082608089837941m_-6808072551270144839m_6351166450766159261m_217048032756755719m_-7911327992255523967m_8557086518318074861m_7562369390283405024h5"><div>On Sat, 13 May 2017 at 12:22, Pawan Alwandi <pawan@platform.sh> wrote:<br></div></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019m_-4654240943671477651m_-5453945879301133293gmail-m_-3457795168368374547gmail-m_-6125136540958523807gmail-m_-3407916076621243690m_-8487362792773169404m_2275467663456357232m_-6419501248012758644m_11629714894935959m_-3519483590900227861m_6025082608089837941m_-6808072551270144839m_6351166450766159261m_217048032756755719m_-7911327992255523967m_8557086518318074861m_7562369390283405024h5"><div><div><div>Hello folks,<br><br></div></div></div><div><div>Does anyone have any idea whats going on here?<br><br></div></div><div><div>Thanks,<br></div>Pawan<br></div><div class="gmail_extra"><br></div><div class="gmail_extra"><div class="gmail_quote">On Wed, May 10, 2017 at 5:02 PM, Pawan Alwandi <span><<a href="mailto:pawan@platform.sh" target="_blank">pawan@platform.sh</a>></span> wrote:<br></div></div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><div><div>Hello,<br><br></div>I'm trying to upgrade gluster from 3.6.2 to 3.10.1 but don't see the glusterfsd and glusterfs processes coming up. <a href="http://gluster.readthedocs.io/en/latest/Upgrade-Guide/upgrade_to_3.10/" target="_blank">http://gluster.readthedocs.io/<wbr>en/latest/Upgrade-Guide/upgrad<wbr>e_to_3.10/</a> is the process that I'm trying to follow.<br><br></div>This is a 3 node server setup with a replicated volume having replica count of 3.<br><br></div>Logs below:<br><div><div><div><div><br><span style="font-family:monospace,monospace">[2017-05-10 09:07:03.507959] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.1 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid)<br>[2017-05-10 09:07:03.512827] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536<br>[2017-05-10 09:07:03.512855] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory<br>[2017-05-10 09:07:03.520426] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_cre<wbr>ate] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device]<br>[2017-05-10 09:07:03.520452] W [MSGID: 103055] [rdma.c:4897:init] 0-rdma.management: Failed to initialize IB Device<br>[2017-05-10 09:07:03.520465] W [rpc-transport.c:350:rpc_trans<wbr>port_load] 0-rpc-transport: 'rdma' initialization failed<br>[2017-05-10 09:07:03.520518] W [rpcsvc.c:1661:rpcsvc_create_l<wbr>istener] 0-rpc-service: cannot create listener, initing the transport failed<br>[2017-05-10 09:07:03.520534] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport<br>[2017-05-10 09:07:04.931764] I [MSGID: 106513] [glusterd-store.c:2197:gluster<wbr>d_restore_op_version] 0-glusterd: retrieved op-version: 30600<br>[2017-05-10 09:07:04.964354] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_<wbr>init] 0-management: retrieved UUID: 7f2a6e11-2a53-4ab4-9ceb-8be6a9<wbr>f2d073<br>[2017-05-10 09:07:04.993944] I [MSGID: 106498] [glusterd-handler.c:3669:glust<wbr>erd_friend_add_from_peerinfo] 0-management: connect returned 0<br>[2017-05-10 09:07:04.995864] I [MSGID: 106498] [glusterd-handler.c:3669:glust<wbr>erd_friend_add_from_peerinfo] 0-management: connect returned 0<br>[2017-05-10 09:07:04.995879] W [MSGID: 106062] [glusterd-handler.c:3466:glust<wbr>erd_transport_inet_options_bui<wbr>ld] 0-glusterd: Failed to get tcp-user-timeout<br>[2017-05-10 09:07:04.995903] I [rpc-clnt.c:1059:rpc_clnt_conn<wbr>ection_init] 0-management: setting frame-timeout to 600<br>[2017-05-10 09:07:04.996325] I [rpc-clnt.c:1059:rpc_clnt_conn<wbr>ection_init] 0-management: setting frame-timeout to 600<br>Final graph:<br>+-----------------------------<wbr>------------------------------<wbr>-------------------+<br> 1: volume management<br> 2: type mgmt/glusterd<br> 3: option rpc-auth.auth-glusterfs on<br> 4: option rpc-auth.auth-unix on<br> 5: option rpc-auth.auth-null on<br> 6: option rpc-auth-allow-insecure on<br> 7: option transport.socket.listen-backlo<wbr>g 128<br> 8: option event-threads 1<br> 9: option ping-timeout 0<br> 10: option transport.socket.read-fail-log off<br> 11: option transport.socket.keepalive-int<wbr>erval 2<br> 12: option transport.socket.keepalive-tim<wbr>e 10<br> 13: option transport-type rdma<br> 14: option working-directory /var/lib/glusterd<br> 15: end-volume<br> 16: <br>+-----------------------------<wbr>------------------------------<wbr>-------------------+<br>[2017-05-10 09:07:04.996310] W [MSGID: 106062] [glusterd-handler.c:3466:glust<wbr>erd_transport_inet_options_bui<wbr>ld] 0-glusterd: Failed to get tcp-user-timeout<br>[2017-05-10 09:07:05.000461] I [MSGID: 101190] [event-epoll.c:629:event_dispa<wbr>tch_epoll_worker] 0-epoll: Started thread with index 1<br>[2017-05-10 09:07:05.001493] W [socket.c:593:__socket_rwv] 0-management: readv on <a href="http://192.168.0.7:24007" target="_blank">192.168.0.7:24007</a> failed (No data available)<br>[2017-05-10 09:07:05.001513] I [MSGID: 106004] [glusterd-handler.c:5882:__glu<wbr>sterd_peer_rpc_notify] 0-management: Peer <192.168.0.7> (<5ec54b4f-f60c-48c6-9e55-95f2<wbr>bb58f633>), in state <Peer in Cluster>, h<br>as disconnected from glusterd.<br>[2017-05-10 09:07:05.001677] W [glusterd-locks.c:675:glusterd<wbr>_mgmt_v3_unlock] (-->/usr/lib/x86_64-linux-gnu/<wbr>glusterfs/3.10.1/xlator/mgmt/g<wbr>lusterd.so(+0x20559) [0x7f0bf9d74559] -->/usr/lib/x86_64-linux-gnu<br>/glusterfs/3.10.1/xlator/mgmt/<wbr>glusterd.so(+0x29cf0) [0x7f0bf9d7dcf0] -->/usr/lib/x86_64-linux-gnu/g<wbr>lusterfs/3.10.1/xlator/mgmt/gl<wbr>usterd.so(+0xd5ba3) [0x7f0bf9e29ba3] ) 0-management: Lock for vol shared no<br>t held<br>[2017-05-10 09:07:05.001696] W [MSGID: 106118] [glusterd-handler.c:5907:__glu<wbr>sterd_peer_rpc_notify] 0-management: Lock not released for shared<br>[2017-05-10 09:07:05.003099] E [rpc-clnt.c:365:saved_frames_u<wbr>nwind] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>lusterfs.so.0(_gf_log_callingf<wbr>n+0x13c)[0x7f0bfeeca73c] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>frpc.so.0(s<br>aved_frames_unwind+0x1cf)[0x7f<wbr>0bfec904bf] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>frpc.so.0(saved_frames_destroy<wbr>+0xe)[0x7f0bfec905de] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>frpc.so.0(rpc_clnt_connection_<wbr>cleanup+0x<br>91)[0x7f0bfec91c21] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>frpc.so.0(rpc_clnt_notify+0x29<wbr>0)[0x7f0bfec92710] ))))) 0-management: forced unwinding frame type(GLUSTERD-DUMP) op(DUMP(1)) called at 2017-05-10 09:0<br>7:05.000627 (xid=0x1)<br>[2017-05-10 09:07:05.003129] E [MSGID: 106167] [glusterd-handshake.c:2181:__g<wbr>lusterd_peer_dump_version_cbk] 0-management: Error through RPC layer, retry again later<br>[2017-05-10 09:07:05.003251] W [socket.c:593:__socket_rwv] 0-management: readv on <a href="http://192.168.0.6:24007" target="_blank">192.168.0.6:24007</a> failed (No data available)<br>[2017-05-10 09:07:05.003267] I [MSGID: 106004] [glusterd-handler.c:5882:__glu<wbr>sterd_peer_rpc_notify] 0-management: Peer <192.168.0.6> (<83e9a0b9-6bd5-483b-8516-d892<wbr>8805ed95>), in state <Peer in Cluster>, h<br>as disconnected from glusterd.<br>[2017-05-10 09:07:05.003318] W [glusterd-locks.c:675:glusterd<wbr>_mgmt_v3_unlock] (-->/usr/lib/x86_64-linux-gnu/<wbr>glusterfs/3.10.1/xlator/mgmt/g<wbr>lusterd.so(+0x20559) [0x7f0bf9d74559] -->/usr/lib/x86_64-linux-gnu<br>/glusterfs/3.10.1/xlator/mgmt/<wbr>glusterd.so(+0x29cf0) [0x7f0bf9d7dcf0] -->/usr/lib/x86_64-linux-gnu/g<wbr>lusterfs/3.10.1/xlator/mgmt/gl<wbr>usterd.so(+0xd5ba3) [0x7f0bf9e29ba3] ) 0-management: Lock for vol shared no<br>t held<br>[2017-05-10 09:07:05.003329] W [MSGID: 106118] [glusterd-handler.c:5907:__glu<wbr>sterd_peer_rpc_notify] 0-management: Lock not released for shared<br>[2017-05-10 09:07:05.003457] E [rpc-clnt.c:365:saved_frames_u<wbr>nwind] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>lusterfs.so.0(_gf_log_callingf<wbr>n+0x13c)[0x7f0bfeeca73c] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>frpc.so.0(s<br>aved_frames_unwind+0x1cf)[0x7f<wbr>0bfec904bf] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>frpc.so.0(saved_frames_destroy<wbr>+0xe)[0x7f0bfec905de] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>frpc.so.0(rpc_clnt_connection_<wbr>cleanup+0x<br>91)[0x7f0bfec91c21] (--> /usr/lib/x86_64-linux-gnu/libg<wbr>frpc.so.0(rpc_clnt_notify+0x29<wbr>0)[0x7f0bfec92710] ))))) 0-management: forced unwinding frame type(GLUSTERD-DUMP) op(DUMP(1)) called at 2017-05-10 09:0<br>7:05.001407 (xid=0x1)</span><br><br></div><div>There are a bunch of errors reported but I'm not sure which is signal and which ones are noise. Does anyone have any idea whats going on here?<br><br></div><div>Thanks,<br></div><div>Pawan<br></div><div><br></div></div></div></div></div>
</blockquote></div></div></div></div><span>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a></span></blockquote></div></div><span class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019m_-4654240943671477651m_-5453945879301133293gmail-m_-3457795168368374547gmail-m_-6125136540958523807gmail-m_-3407916076621243690m_-8487362792773169404m_2275467663456357232m_-6419501248012758644m_11629714894935959m_-3519483590900227861m_6025082608089837941m_-6808072551270144839m_6351166450766159261m_217048032756755719m_-7911327992255523967m_8557086518318074861m_7562369390283405024HOEnZb"><font color="#888888"><div>-- <br></div><div>- Atin (atinm)</div>
</font></span></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</blockquote></div></div></div></div></div><span class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019m_-4654240943671477651m_-5453945879301133293gmail-m_-3457795168368374547gmail-m_-6125136540958523807gmail-m_-3407916076621243690m_-8487362792773169404m_2275467663456357232m_-6419501248012758644m_11629714894935959m_-3519483590900227861m_6025082608089837941m_-6808072551270144839m_6351166450766159261HOEnZb"><font color="#888888"><div>-- <br></div><div>- Atin (atinm)</div>
</font></span></blockquote></div><br></div>
</blockquote></div></div></div></div><span class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019m_-4654240943671477651m_-5453945879301133293gmail-m_-3457795168368374547gmail-m_-6125136540958523807gmail-m_-3407916076621243690m_-8487362792773169404m_2275467663456357232m_-6419501248012758644m_11629714894935959m_-3519483590900227861m_6025082608089837941HOEnZb"><font color="#888888"><div>-- <br></div><div>- Atin (atinm)</div>
</font></span></blockquote></div><br></div>
</blockquote></div></div><div>-- <br></div><div>- Atin (atinm)</div></blockquote></div></div></div></div><span class="m_5371961948397340435m_7653404875185003785gmail-m_-7258279994633888264m_-8751329659860719929m_7260066143239081019m_-4654240943671477651m_-5453945879301133293gmail-m_-3457795168368374547gmail-m_-6125136540958523807gmail-m_-3407916076621243690m_-8487362792773169404m_2275467663456357232m_-6419501248012758644HOEnZb"><font color="#888888"><div>-- <br></div><div>- Atin (atinm)</div>
</font></span></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div></div></div><br></div></div>
</blockquote></div></div></div></div></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail_extra"><br></div></div>
</blockquote></div></div></div><br></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</blockquote></div></div></div></div><span class="m_5371961948397340435m_7653404875185003785gmail-HOEnZb"><font color="#888888"><div dir="ltr">-- <br></div><div>- Atin (atinm)</div>
</font></span></blockquote></div></div></div><br></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>