Hi Dev list,<div><br></div><div>How can I find the details about the rebalance_status/status ids ? Is it actually normal that some systems are in '4' , others in '3' ?</div><div><br></div><div>Is it safe to forcefully start a new rebalance ?</div><div><br></div><div>Best Regards,</div><div>Strahil Nikolov </div><div> <br> <blockquote style="margin: 0 0 20px 0;"> <div style="font-family:Roboto, sans-serif; color:#6D00F6;"> <div>On Mon, Nov 7, 2022 at 9:15, Shreyansh Shah</div><div><shreyansh.shah@alpha-grep.com> wrote:</div> </div> <div style="padding: 10px 0 0 20px; margin: 10px 0 0 0; border-left: 1px solid #6D00F6;"> <div id="yiv2066393322"><div><div dir="ltr">Hi Strahil,<br clear="none">Adding the info below:<br clear="none"><br clear="none">--------------------------------------<br clear="none">Node IP = 10.132.0.19<br clear="none">rebalance_status=1<br clear="none">status=4<br clear="none">rebalance_op=19<br clear="none">rebalance-id=39a89b51-2549-4348-aa47-0db321c3a32f<br clear="none">rebalanced-files=27054<br clear="none">size=7104425578505<br clear="none">scanned=72141<br clear="none">failures=10<br clear="none">skipped=19611<br clear="none">run-time=92805.000000<br clear="none">--------------------------------------<br clear="none">Node IP = 10.132.0.20<br clear="none">rebalance_status=1<br clear="none">status=4<br clear="none">rebalance_op=19<br clear="none">rebalance-id=39a89b51-2549-4348-aa47-0db321c3a32f<br clear="none">rebalanced-files=23945<br clear="none">size=7126809216060<br clear="none">scanned=71208<br clear="none">failures=7<br clear="none">skipped=18834<br clear="none">run-time=94029.000000<br clear="none">--------------------------------------<br clear="none">Node IP = 10.132.1.12<br clear="none">rebalance_status=1<br clear="none">status=4<br clear="none">rebalance_op=19<br clear="none">rebalance-id=39a89b51-2549-4348-aa47-0db321c3a32f<br clear="none">rebalanced-files=12533<br clear="none">size=12945021256<br clear="none">scanned=40398<br clear="none">failures=14<br clear="none">skipped=1194<br clear="none">run-time=92201.000000<br clear="none">--------------------------------------<br clear="none">Node IP = 10.132.1.13<br clear="none">rebalance_status=1<br clear="none">status=3<br clear="none">rebalance_op=19<br clear="none">rebalance-id=39a89b51-2549-4348-aa47-0db321c3a32f<br clear="none">rebalanced-files=41483<br clear="none">size=8845076025598<br clear="none">scanned=179920<br clear="none">failures=25<br clear="none">skipped=62373<br clear="none">run-time=130017.000000<br clear="none">--------------------------------------<br clear="none">Node IP = 10.132.1.14<br clear="none">rebalance_status=1<br clear="none">status=3<br clear="none">rebalance_op=19<br clear="none">rebalance-id=39a89b51-2549-4348-aa47-0db321c3a32f<br clear="none">rebalanced-files=43603<br clear="none">size=7834691799355<br clear="none">scanned=204140<br clear="none">failures=2878<br clear="none">skipped=87761<br clear="none">run-time=130016.000000<br clear="none">--------------------------------------<br clear="none">Node IP = 10.132.1.15<br clear="none">rebalance_status=1<br clear="none">status=4<br clear="none">rebalance_op=19<br clear="none">rebalance-id=39a89b51-2549-4348-aa47-0db321c3a32f<br clear="none">rebalanced-files=29968<br clear="none">size=6389568855140<br clear="none">scanned=69320<br clear="none">failures=7<br clear="none">skipped=17999<br clear="none">run-time=93654.000000<br clear="none">--------------------------------------<br clear="none">Node IP = 10.132.1.16<br clear="none">rebalance_status=1<br clear="none">status=4<br clear="none">rebalance_op=19<br clear="none">rebalance-id=39a89b51-2549-4348-aa47-0db321c3a32f<br clear="none">rebalanced-files=23226<br clear="none">size=5899338197718<br clear="none">scanned=56169<br clear="none">failures=7<br clear="none">skipped=12659<br clear="none">run-time=94030.000000<br clear="none">--------------------------------------<br clear="none">Node IP = 10.132.1.17<br clear="none">rebalance_status=1<br clear="none">status=4<br clear="none">rebalance_op=19<br clear="none">rebalance-id=39a89b51-2549-4348-aa47-0db321c3a32f<br clear="none">rebalanced-files=17538<br clear="none">size=6247281008602<br clear="none">scanned=50038<br clear="none">failures=8<br clear="none">skipped=11335<br clear="none">run-time=92203.000000<br clear="none">--------------------------------------<br clear="none">Node IP = 10.132.1.18<br clear="none">rebalance_status=1<br clear="none">status=4<br clear="none">rebalance_op=19<br clear="none">rebalance-id=39a89b51-2549-4348-aa47-0db321c3a32f<br clear="none">rebalanced-files=20394<br clear="none">size=6395008466977<br clear="none">scanned=50060<br clear="none">failures=7<br clear="none">skipped=13784<br clear="none">run-time=92103.000000<br clear="none">--------------------------------------<br clear="none">Node IP = 10.132.1.19<br clear="none">rebalance_status=1<br clear="none">status=1<br clear="none">rebalance_op=19<br clear="none">rebalance-id=39a89b51-2549-4348-aa47-0db321c3a32f<br clear="none">rebalanced-files=0<br clear="none">size=0<br clear="none">scanned=0<br clear="none">failures=0<br clear="none">skipped=0<br clear="none">run-time=0.000000<br clear="none">--------------------------------------<br clear="none">Node IP = 10.132.1.20<br clear="none">rebalance_status=1<br clear="none">status=3<br clear="none">rebalance_op=19<br clear="none">rebalance-id=39a89b51-2549-4348-aa47-0db321c3a32f<br clear="none">rebalanced-files=0<br clear="none">size=0<br clear="none">scanned=24<br clear="none">failures=0<br clear="none">skipped=2<br clear="none">run-time=1514.000000<br clear="none"></div><br clear="none"><div id="yiv2066393322yqt85214" class="yiv2066393322yqt7393817086"><div class="yiv2066393322gmail_quote"><div dir="ltr" class="yiv2066393322gmail_attr">On Thu, Nov 3, 2022 at 10:10 PM Strahil Nikolov <<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:hunter86_bg@yahoo.com" target="_blank" href="mailto:hunter86_bg@yahoo.com">hunter86_bg@yahoo.com</a>> wrote:<br clear="none"></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex;" class="yiv2066393322gmail_quote">And the other servers ?<br clear="none"> <br clear="none"> <blockquote style="margin:0px 0px 20px;"> <div style="font-family:Roboto, sans-serif;color:rgb(109,0,246);"> <div>On Thu, Nov 3, 2022 at 16:21, Shreyansh Shah</div><div><<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:shreyansh.shah@alpha-grep.com" target="_blank" href="mailto:shreyansh.shah@alpha-grep.com">shreyansh.shah@alpha-grep.com</a>> wrote:</div> </div> <div style="padding:10px 0px 0px 20px;margin:10px 0px 0px;border-left:1px solid rgb(109,0,246);"> <div id="yiv2066393322m_498652272516546339yiv8596481873"><div><div dir="ltr">Hi Strahil,<br clear="none">Thank you for your reply. <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="http://node_state.info">node_state.info</a> has the below data<br clear="none"><br clear="none"><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex;">root@gluster-11:/usr/var/lib/glusterd/vols/data# cat <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="http://node_state.info">node_state.info</a><br clear="none"> rebalance_status=1<br clear="none">status=3<br clear="none">rebalance_op=19<br clear="none">rebalance-id=39a89b51-2549-4348-aa47-0db321c3a32f<br clear="none">rebalanced-files=0<br clear="none">size=0<br clear="none">scanned=24<br clear="none">failures=0<br clear="none">skipped=2<br clear="none">run-time=1514.000000</blockquote><br clear="none"><br clear="none"></div><br clear="none"><div id="yiv2066393322m_498652272516546339yiv8596481873yqt62607"><div><div dir="ltr">On Thu, Nov 3, 2022 at 4:00 PM Strahil Nikolov <<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:hunter86_bg@yahoo.com" target="_blank" href="mailto:hunter86_bg@yahoo.com">hunter86_bg@yahoo.com</a>> wrote:<br clear="none"></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex;">I would check the details in /var/lib/glusterd/vols/<VOLUME_NAME>/<a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="http://node_state.info">node_state.info</a><div><br clear="none"></div><div>Best Regards,</div><div>Strahil Nikolov <br clear="none"> <br clear="none"> <blockquote style="margin:0px 0px 20px;"> <div style="font-family:Roboto, sans-serif;color:rgb(109,0,246);"> <div>On Wed, Nov 2, 2022 at 9:06, Shreyansh Shah</div><div><<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:shreyansh.shah@alpha-grep.com" target="_blank" href="mailto:shreyansh.shah@alpha-grep.com">shreyansh.shah@alpha-grep.com</a>> wrote:</div> </div> <div style="padding:10px 0px 0px 20px;margin:10px 0px 0px;border-left:1px solid rgb(109,0,246);"> <div id="yiv2066393322m_498652272516546339yiv8596481873m_6581624436217853905yiv9715499931"><div><div dir="ltr">Hi,<br clear="none">I Would really appreciate it if someone would be able to help on the above issue. We are stuck as we cannot run rebalance due to this and thus are not able to extract peak performance from the setup due to unbalanced data.<br clear="none">Adding gluster info (without the bricks) below. Please let me know if any other details/logs are needed.<br clear="none"><br clear="none"><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex;">Volume Name: data<br clear="none">Type: Distribute<br clear="none">Volume ID: 75410231-bb25-4f14-bcde-caf18fce1d31<br clear="none">Status: Started<br clear="none">Snapshot Count: 0<br clear="none">Number of Bricks: 41<br clear="none">Transport-type: tcp<br clear="none">Options Reconfigured:<br clear="none">server.event-threads: 4<br clear="none">network.ping-timeout: 90<br clear="none">client.keepalive-time: 60<br clear="none">server.keepalive-time: 60<br clear="none">storage.health-check-interval: 60<br clear="none">performance.client-io-threads: on<br clear="none">nfs.disable: on<br clear="none">transport.address-family: inet<br clear="none">performance.cache-size: 8GB<br clear="none">performance.cache-refresh-timeout: 60<br clear="none">cluster.min-free-disk: 3%<br clear="none">client.event-threads: 4<br clear="none">performance.io-thread-count: 16</blockquote><div><br clear="none"></div></div><br clear="none"><div id="yiv2066393322m_498652272516546339yiv8596481873m_6581624436217853905yiv9715499931yqt71186"><div><div dir="ltr">On Fri, Oct 28, 2022 at 11:40 AM Shreyansh Shah <<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:shreyansh.shah@alpha-grep.com" target="_blank" href="mailto:shreyansh.shah@alpha-grep.com">shreyansh.shah@alpha-grep.com</a>> wrote:<br clear="none"></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex;"><div dir="ltr">Hi,<br clear="none">We are running glusterfs 5.10 server volume. Recently we added a few new bricks and started a rebalance operation. After a couple of days the rebalance operation was just stuck, with one of the peers showing In-Progress with no file being read/transferred and the rest showing Failed/Completed, so we stopped it using "gluster volume rebalance data stop". Now when we are trying to start it again, we get the below error. Any assistance would be appreciated<br clear="none"><br clear="none"><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px;"></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex;">root@gluster-11:~# gluster volume rebalance data status<br clear="none">volume rebalance: data: failed: Rebalance not started for volume data.<br clear="none">root@gluster-11:~# gluster volume rebalance data start<br clear="none">volume rebalance: data: failed: Rebalance on data is already started<br clear="none">root@gluster-11:~# gluster volume rebalance data stop<br clear="none">volume rebalance: data: failed: Rebalance not started for volume data.</blockquote><div> </div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px;"></blockquote>-- <br clear="none"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><font face="arial, helvetica, sans-serif">Regards,<br clear="none">Shreyansh Shah</font><br clear="none"><div style="margin:0in 0in 0.0001pt;"><font><font color="#0b5394" style="font-weight:bold;">Alpha</font><font color="#666666" style="font-weight:bold;">Grep</font><font color="#000000"><b> Securities Pvt. Ltd.</b><br clear="none"></font></font></div></div></div></div></div></div>
</blockquote></div></div><br clear="all"><div><br clear="none"></div>-- <br clear="none"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><font face="arial, helvetica, sans-serif">Regards,<br clear="none">Shreyansh Shah</font><br clear="none"><div style="margin:0in 0in 0.0001pt;"><font><font color="#0b5394" style="font-weight:bold;">Alpha</font><font color="#666666" style="font-weight:bold;">Grep</font><font color="#000000"><b> Securities Pvt. Ltd.</b><br clear="none"></font></font></div></div></div></div></div>
</div></div> </div> </blockquote></div></blockquote></div></div><br clear="all"><div><br clear="none"></div>-- <br clear="none"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><font face="arial, helvetica, sans-serif">Regards,<br clear="none">Shreyansh Shah</font><br clear="none"><div style="margin:0in 0in 0.0001pt;"><font><font color="#0b5394" style="font-weight:bold;">Alpha</font><font color="#666666" style="font-weight:bold;">Grep</font><font color="#000000"><b> Securities Pvt. Ltd.</b><br clear="none"></font></font></div></div></div></div></div>
</div></div> </div> </blockquote></blockquote></div></div><br clear="all"><div><br clear="none"></div>-- <br clear="none"><div dir="ltr" class="yiv2066393322gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><font face="arial, helvetica, sans-serif">Regards,<br clear="none">Shreyansh Shah</font><br clear="none"><div style="margin:0in 0in 0.0001pt;"><font><font color="#0b5394" style="font-weight:bold;">Alpha</font><font color="#666666" style="font-weight:bold;">Grep</font><font color="#000000"><b> Securities Pvt. Ltd.</b><br clear="none"></font></font></div></div></div></div></div>
</div></div> </div> </blockquote></div>