<div dir="ltr"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div><br></div><div>Hi,</div><div><br></div><div>Rebalance will abort itself if it cannot reach any of the nodes. Are all the bricks still up and reachable?</div><div><br></div><div>Regards,</div><div>Nithya</div></div></div></blockquote><div><br></div><div>Yes the bricks appear to be fine. I restarted the rebalance and the process is moving along again:</div><div><br></div><div># gluster vol rebalance tank status<br>                  Node Rebalanced-files      size    scanned    failures    skipped        status  run time in h:m:s<br>                ---------    -----------  -----------  -----------  -----------  -----------     ------------   --------------<br>                localhost      226973     14.9TB    1572952       0       0      in progress    44:26:48<br>                serverB            0     0Bytes     631667       0       0       completed    37:2:14<br>volume rebalance: tank: success<br></div><div><br></div><div># df -hP |grep data<br>/dev/mapper/gluster_vg-gluster_lv1_data  60T  24T  36T  40% /gluster_bricks/data1<br>/dev/mapper/gluster_vg-gluster_lv2_data  60T  24T  36T  40% /gluster_bricks/data2<br>/dev/mapper/gluster_vg-gluster_lv3_data  60T  17T  43T  29% /gluster_bricks/data3<br>/dev/mapper/gluster_vg-gluster_lv4_data  60T  17T  43T  29% /gluster_bricks/data4<br>/dev/mapper/gluster_vg-gluster_lv5_data  60T  19T  41T  31% /gluster_bricks/data5<br>/dev/mapper/gluster_vg-gluster_lv6_data  60T  19T  41T  31% /gluster_bricks/data6<br></div><div><br></div><div>Thanks,</div><div><br></div><div>HB</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div># gluster vol rebalance tank status<br>                  Node Rebalanced-files      size    scanned    failures    skipped        status  run time in h:m:s<br>                ---------    -----------  -----------  -----------  -----------  -----------     ------------   --------------<br>                localhost      1348706     57.8TB    2234439       9       6        failed    190:24:3<br>                serverB             0     0Bytes       7       0       0       completed    63:47:55<br>volume rebalance: tank: success<br></div><div><br></div><div># gluster vol status tank<br>Status of volume: tank<br>Gluster process               TCP Port  RDMA Port  Online  Pid<br>------------------------------------------------------------------------------<br>Brick serverA:/gluster_bricks/data1    49162   0      Y    20318<br>Brick serverB:/gluster_bricks/data1    49166   0      Y    3432 <br>Brick serverA:/gluster_bricks/data2    49163   0      Y    20323<br>Brick serverB:/gluster_bricks/data2    49167   0      Y    3435 <br>Brick serverA:/gluster_bricks/data3    49164   0      Y    4625 <br>Brick serverA:/gluster_bricks/data4    49165   0      Y    4644 <br>Brick serverA:/gluster_bricks/data5    49166   0      Y    5088 <br>Brick serverA:/gluster_bricks/data6    49167   0      Y    5128 <br>Brick serverB:/gluster_bricks/data3    49168   0      Y    22314<br>Brick serverB:/gluster_bricks/data4    49169   0      Y    22345<br>Brick serverB:/gluster_bricks/data5    49170   0      Y    22889<br>Brick serverB:/gluster_bricks/data6    49171   0      Y    22932<br>Self-heal Daemon on localhost        N/A    N/A     Y    6202 <br>Self-heal Daemon on serverB        N/A    N/A     Y    22981<br> <br>Task Status of Volume tank<br>------------------------------------------------------------------------------<br>Task         : Rebalance      <br>ID          : eec64343-8e0d-4523-ad05-5678f9eb9eb2<br>Status        : failed<br></div><div><br></div><div># df -hP |grep data<br>/dev/mapper/gluster_vg-gluster_lv1_data  60T  31T  29T  52% /gluster_bricks/data1<br>/dev/mapper/gluster_vg-gluster_lv2_data  60T  31T  29T  51% /gluster_bricks/data2<br>/dev/mapper/gluster_vg-gluster_lv3_data  60T  15T  46T  24% /gluster_bricks/data3<br>/dev/mapper/gluster_vg-gluster_lv4_data  60T  15T  46T  24% /gluster_bricks/data4<br>/dev/mapper/gluster_vg-gluster_lv5_data  60T  15T  45T  25% /gluster_bricks/data5<br>/dev/mapper/gluster_vg-gluster_lv6_data  60T  15T  45T  25% /gluster_bricks/data6<br></div><div><br></div><div><br></div><div>The rebalance log on serverA shows a disconnect from serverB<br><br></div><div>[2019-09-08 15:41:44.285591] C [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] 0-tank-client-10: server <serverB>:49170 has not responded in the last 42 seconds, disconnecting.<br>[2019-09-08 15:41:44.285739] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-tank-client-10: disconnected from tank-client-10. Client process will keep trying to connect to glusterd until brick's port is available<br>[2019-09-08 15:41:44.286023] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7ff986e8b132] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7ff986c5299e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7ff986c52aae] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7ff986c54220] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2b0)[0x7ff986c54ce0] ))))) 0-tank-client-10: forced unwinding frame type(GlusterFS 3.3) op(FXATTROP(34)) called at 2019-09-08 15:40:44.040333 (xid=0x7f8cfac)<br></div><div><br></div><div>Does this type of failure cause data corruption? What is the best course of action at this point?</div><div><br></div><div>Thanks,</div><div><br></div><div>HB</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Sep 11, 2019 at 11:58 PM Strahil <<a href="mailto:hunter86_bg@yahoo.com" target="_blank">hunter86_bg@yahoo.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><p dir="ltr">Hi Nithya,</p>
<p dir="ltr">Thanks for the detailed explanation.<br>
It makes sense.</p>
<p dir="ltr">Best Regards,<br>
Strahil Nikolov</p>
<div class="gmail-m_4837919376999641059gmail-m_-8315880090101177330gmail-m_-8066655633455489597quote">On Sep 12, 2019 08:18, Nithya Balachandran <<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a>> wrote:<br type="attribution"><blockquote class="gmail-m_4837919376999641059gmail-m_-8315880090101177330gmail-m_-8066655633455489597quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><img src="https://ssl.gstatic.com/ui/v1/icons/mail/images/cleardot.gif"><br></div><br><div class="gmail-m_4837919376999641059gmail-m_-8315880090101177330gmail-m_-8066655633455489597elided-text"><div dir="ltr">On Wed, 11 Sep 2019 at 09:47, Strahil <<a href="mailto:hunter86_bg@yahoo.com" target="_blank">hunter86_bg@yahoo.com</a>> wrote:<br></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><p dir="ltr">Hi Nithya,<br></p>
<p dir="ltr">I just reminded about your previous e-mail which left me with the impression that old volumes need that.<br>
This is the one 1 mean:</p>
<div align="left"><p dir="ltr">>It looks like this is a replicate volume. If >that is the case then yes, you are >running an old version of Gluster for >which this was the default </p></div></blockquote><div><br></div><div>Hi Strahil,</div><div><br></div><div>I'm providing a little more detail here which I hope will explain things.</div><div>Rebalance was always a volume wide operation - a <font size="1" face="times new roman, serif"><b>rebalance start</b></font> operation will start rebalance processes on all nodes of the volume. However, different processes would behave differently. In earlier releases, all nodes would crawl the bricks and update the directory layouts. However, only one node in each replica/disperse set would actually migrate files,so the rebalance status would only show one node doing any "work" (scanning, rebalancing etc). However, this one node will process all the files in its replica sets. Rerunning rebalance on other nodes would make no difference as it will always be the same node that ends up migrating files.</div><div><div>So for instance, for a replicate volume with server1:/brick1, server2:/brick2 and server3:/brick3 in that order, only the rebalance process on server1 would migrate files. In newer releases, all 3 nodes would migrate files.</div><div><br></div></div><div>The rebalance status does not capture the directory operations of fixing layouts which is why it looks like the other nodes are not doing anything.<br></div><div><br></div><div>Hope this helps.<br></div><div><br></div><div>Regards,</div><div>Nithya</div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div align="left"><p dir="ltr">behaviour. <br>
</p>
</div><p dir="ltr">><br>
><br>
</p>
<div align="left"><p dir="ltr">>Regards,<br>
</p>
</div><p dir="ltr">><br>
</p>
<div align="left"><p dir="ltr">>Nithya<br>
</p>
</div><p dir="ltr"><br>
Best Regards,<br>
Strahil Nikolov</p>
<div>On Sep 9, 2019 06:36, Nithya Balachandran <<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a>> wrote:<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div><div dir="ltr">On Sat, 7 Sep 2019 at 00:03, Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com" target="_blank">hunter86_bg@yahoo.com</a>> wrote:<br></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:"courier new",courier,monaco,monospace,sans-serif;font-size:16px"><div></div>
<div dir="ltr">As it was mentioned, you might have to run rebalance on the other node - but it is better to wait this node is over.</div><div dir="ltr"><br></div></div></div></blockquote><div><br></div><div>Hi Strahil,</div><div><br></div><div>Rebalance does not need to be run on the other node - the operation is a volume wide one . Only a single node per replica set would migrate files in the version used in this case .</div><div><br></div><div>Regards,</div><div>Nithya</div><div><br></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div style="font-family:"courier new",courier,monaco,monospace,sans-serif;font-size:16px"><div dir="ltr"></div><div dir="ltr">Best Regards,</div><div dir="ltr">Strahil Nikolov<br></div><div><br></div>
</div><div>
<div style="font-family:"helvetica neue",helvetica,arial,sans-serif;font-size:13px;color:rgb(38,40,42)">
<div>
Ð’ петък, 6 Ñептември 2019 г., 15:29:20 ч. Гринуич+3, Herb Burnswell <<a href="mailto:herbert.burnswell@gmail.com" target="_blank">herbert.burnswell@gmail.com</a>> </div></div></div></div></blockquote></div></div></blockquote></div></blockquote></div></div></blockquote></div></blockquote></div>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div></div>
</blockquote></div></div>