<div dir="ltr"><div dir="ltr">Dear list,<br><br>After looking at my rebalance log more I saw another message that helped me solve the problem:<br><br><div>[2019-05-06 22:46:01.074035] W [MSGID: 0] [dht-rebalance.c:1075:__dht_check_free_space] 0-data-dht: Write will cross min-free-disk for file - /ilri/miseq/MiSeq1/MiSeq1Output_2014/140624_M01601_0035_000000000-A6L82/Data/TileStatus/TileStatusL1T1114.tpl on subvol - data-replicate-1. Looking for new subvol</div><div><br></div><div>I have not test cluster.min-free-disk, but it appears that the default value for is 10%, and my bricks are at about 97% capacity so the "No space error" in my previous message makes sense. I reduced the cluster.min-free-disk to 2% and restarted the data rebalance and now I see that it is already rebalancing files. The issue is solved.<br></div><div><br></div><div>Sorry about that! Thank you,<br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, May 7, 2019 at 4:12 PM Alan Orth <<a href="mailto:alan.orth@gmail.com">alan.orth@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">Dear list,<br><br><div>We are using a Distributed-Replicate volume with replica 2 on Gluster 4.1.7 on CentOS 7. One of our nodes died recently and we will add new nodes and bricks to replace it soon. In preparation for the maintenance I wanted to rebalance the volume to make the disk thrashing less intense when we add/remove bricks, but after eight hours of scanning I see millions of "failures" in the rebalance status. The volume rebalance log shows many errors like:<br></div><br>[2019-05-07 06:06:02.310843] E [MSGID: 109023] [dht-rebalance.c:2907:gf_defrag_migrate_single_file] 0-data-dht: migrate-data failed for /ilri/miseq/MiSeq2/MiSeq2Output2018/180912_M03021_0002_000000000-BVM95/Thumbnail_Images/L001/C174.1/s_1_2103_c.jpg [No space left on device]<br><div><br></div><div>The bricks on the healthy nodes all have 1.5TB of free space so I'm not sure what this error means. Could it be because one of the replicas is unavailable? I saw a similar bug report¹ about that. I've started a simple fix-layout without data migration and it is working fine.<br></div><div><br></div><div>Thank you,<br></div><div><br></div><div>¹ <a href="https://access.redhat.com/solutions/456333" target="_blank">https://access.redhat.com/solutions/456333</a><br></div>--<br>Alan Orth<br><a href="mailto:alan.orth@gmail.com" target="_blank">alan.orth@gmail.com</a><br><a href="https://picturingjordan.com" target="_blank">https://picturingjordan.com</a><br><a href="https://englishbulgaria.net" target="_blank">https://englishbulgaria.net</a><br><a href="https://mjanja.ch" target="_blank">https://mjanja.ch</a><br>"In heaven all the interesting people are missing." ―Friedrich Nietzsche</div></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature">Alan Orth<br><a href="mailto:alan.orth@gmail.com" target="_blank">alan.orth@gmail.com</a><br><a href="https://picturingjordan.com" target="_blank">https://picturingjordan.com</a><br><a href="https://englishbulgaria.net" target="_blank">https://englishbulgaria.net</a><br><a href="https://mjanja.ch" target="_blank">https://mjanja.ch</a><br>"In heaven all the interesting people are missing." ―Friedrich Nietzsche</div>