<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr">On Fri, 4 Jan 2019 at 17:10, mohammad kashif <<a href="mailto:kashif.alig@gmail.com">kashif.alig@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hi Nithya</div><div><br></div><div>rebalance logs has only these warnings</div><div>2019-01-04 09:59:20.826261] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-atlasglust-client-5: error returned while attempting to connect to host:(null), port:0
[2019-01-04 09:59:20.828113] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-atlasglust-client-6: error returned while attempting to connect to host:(null), port:0
[2019-01-04 09:59:20.832017] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-atlasglust-client-4: error returned while attempting to connect to host:(null), port:0 <br></div></div></div></div></div></div></div></div></div></blockquote><div><br></div><div>Please send me the rebalance logs if possible. Are 08 and 09 the newly added nodes? Are no directories being created on those ?</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div></div><div><br></div><div>gluster volume rebalance atlasglust status<br> Node status run time in h:m:s<br> --------- ----------- ------------<br> localhost fix-layout in progress 1:0:59<br> <a href="http://pplxgluster02.physics.ox.ac.uk" target="_blank">pplxgluster02.physics.ox.ac.uk</a> fix-layout in progress 1:0:59<br> <a href="http://pplxgluster03.physics.ox.ac.uk" target="_blank">pplxgluster03.physics.ox.ac.uk</a> fix-layout in progress 1:0:59<br> <a href="http://pplxgluster04.physics.ox.ac.uk" target="_blank">pplxgluster04.physics.ox.ac.uk</a> fix-layout in progress 1:0:59<br> <a href="http://pplxgluster05.physics.ox.ac.uk" target="_blank">pplxgluster05.physics.ox.ac.uk</a> fix-layout in progress 1:0:59<br> <a href="http://pplxgluster06.physics.ox.ac.uk" target="_blank">pplxgluster06.physics.ox.ac.uk</a> fix-layout in progress 1:0:59<br> <a href="http://pplxgluster07.physics.ox.ac.uk" target="_blank">pplxgluster07.physics.ox.ac.uk</a> fix-layout in progress 1:0:59<br> <a href="http://pplxgluster08.physics.ox.ac.uk" target="_blank">pplxgluster08.physics.ox.ac.uk</a> fix-layout in progress 1:0:59<br> <a href="http://pplxgluster09.physics.ox.ac.uk" target="_blank">pplxgluster09.physics.ox.ac.uk</a> fix-layout in progress 1:0:59<br></div><div><br></div><div>But there is no new entry in logs for last one hour and I can't see any new directories being created.</div><div><br></div><div>Thanks</div><div><br></div><div>Kashif<br></div><br></div></div></div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Jan 4, 2019 at 10:42 AM Nithya Balachandran <<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr">On Fri, 4 Jan 2019 at 15:48, mohammad kashif <<a href="mailto:kashif.alig@gmail.com" target="_blank">kashif.alig@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hi <br></div><div><br></div><div>I have updated our distributed gluster storage from 3.12.9-1 to 4.1.6-1. The existing cluster had seven servers totalling in around 450 TB. OS is Centos7. The update went OK and I could access files.</div><div>Then I added two more servers of 90TB each to cluster and started fix-layout</div><div><br></div><div>gluster volume rebalance atlasglust fix-layout start <br></div><div><br></div><div>Some directories were created at new servers and then stopped although rebalance status was showing that it is still running. I think it stopped creating new directories after this error</div><div><br></div><div>E [MSGID: 106061] [glusterd-utils.c:10697:glusterd_volume_rebalance_use_rsp_dict] 0-glusterd: failed to get index
The message "E [MSGID: 106061] [glusterd-utils.c:10697:glusterd_volume_rebalance_use_rsp_dict] 0-glusterd: failed to get index" repeated 7 times between [2019-01-03 13:16:31.146779] and [2019-01-03 13:16:31.158612]</div><div> <br></div></div></div></div></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div></div><div>There are also many warning like this</div><div>[2019-01-03 16:04:34.120777] I [MSGID: 106499] [glusterd-handler.c:4314:__glusterd_handle_status_volume] 0-management: Received status volume req for volume atlasglust
[2019-01-03 17:04:28.541805] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-management: error returned while attempting to connect to host:(null), port:0</div><div><br></div></div></div></div></div></div></blockquote><div>These are the glusterd logs. Do you see any errors in the rebalance logs for this volume?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div></div><div>I waited for around 12 hours and then stopped fix-layout and started again</div><div>I can see the same error again</div><div><br></div><div>[2019-01-04 09:59:20.825930] E [MSGID: 106061] [glusterd-utils.c:10697:glusterd_volume_rebalance_use_rsp_dict] 0-glusterd: failed to get index
The message "E [MSGID: 106061] [glusterd-utils.c:10697:glusterd_volume_rebalance_use_rsp_dict] 0-glusterd: failed to get index" repeated 7 times between [2019-01-04 09:59:20.825930] and [2019-01-04 09:59:20.837068] <br></div><div><br></div><div>Please suggest as it is our production service. <br></div><div><br></div><div>At the moment, I have stopped clients from using file system. Would it be OK if I allow clients to access file system while fix-layout is still going.</div><div><br></div><div>Thanks</div><div><br></div><div>Kashif<br></div><div><br></div><div><br></div><div><br></div></div></div></div></div></div>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div></div>
</blockquote></div>
</blockquote></div></div>