<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Good morning Ashish,</div><div class=""><br class=""></div><div class="">your explanations are always very useful, thank you very much: I will remember these suggestions for any future needs.</div><div class="">Anyway, during the week-end, the remove-brick procedures ended successfully and we were able to free up all bricks defined on server s04, s05 and 6 bricks of 12 on server s06.</div><div class="">So, we can say that, thanks to your suggestions, we are about to complete this first phase (removing of all bricks defined on s04, s05 and s06 servers).</div><div class=""><br class=""></div><div class="">I really appreciated your support. </div><div class="">Now I have a last question (I hope): after remove-brick commit I noticed that some data remain on each brick (about 1.2GB of data).</div><div class="">Please, take a look to the “df-h_on_s04_s05_s06.txt”.</div><div class="">The situation is almost the same on all 3 servers mentioned above: a long list of directories names and some files that are still on the brick, but respective size is 0.</div><div class=""><br class=""></div><div class="">Examples:</div><div class=""><br class=""></div><div class="">a lot of empty directories on /gluster/mnt*/brick/.glusterfs</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">8<span class="Apple-tab-span" style="white-space:pre"> </span>/gluster/mnt2/brick/.glusterfs/b7/1b</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">0<span class="Apple-tab-span" style="white-space:pre"> </span>/gluster/mnt2/brick/.glusterfs/b7/ee/b7ee94a5-a77c-4c02-85a5-085992840c83</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">0<span class="Apple-tab-span" style="white-space:pre"> </span>/gluster/mnt2/brick/.glusterfs/b7/ee/b7ee85d4-ce48-43a7-a89a-69c728ee8273</span></div></div><div class=""><br class=""></div><div class="">some empty files in directories in /gluster/mnt*/brick/*</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[root@s04 ~]# cd /gluster/mnt1/brick/</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[root@s04 brick]# ls -l</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">totale 32</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">drwxr-xr-x 7 root root 100 11 set 22.14 </span><span style="font-variant-ligatures: no-common-ligatures; color: #5330e1" class=""><b class="">archive_calypso</b></span></div></div><div class=""><br class=""></div><div class=""><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[root@s04 brick]# cd archive_calypso/</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[root@s04 archive_calypso]# ll</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">totale 0</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">drwxr-x--- 3 root 5200 29 11 set 22.13 </span><span style="font-variant-ligatures: no-common-ligatures; color: #5330e1" class=""><b class="">ans002</b></span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">drwxr-x--- 3 5104 5100 32 11 set 22.14 </span><span style="font-variant-ligatures: no-common-ligatures; color: #5330e1" class=""><b class="">ans004</b></span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">drwxr-x--- 3 4506 4500 31 11 set 22.14 </span><span style="font-variant-ligatures: no-common-ligatures; color: #5330e1" class=""><b class="">ans006</b></span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">drwxr-x--- 3 4515 4500 28 11 set 22.14 </span><span style="font-variant-ligatures: no-common-ligatures; color: #5330e1" class=""><b class="">ans015</b></span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">drwxr-x--- 4 4321 4300 54 11 set 22.14 </span><span style="font-variant-ligatures: no-common-ligatures; color: #5330e1" class=""><b class="">ans021</b></span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[root@s04 archive_calypso]# du -a *</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">0<span class="Apple-tab-span" style="white-space:pre"> </span>ans002/archive/ans002/HINDCASTS/RUN_ATMWANG_LANSENS/19810501.0/echam5/echam_sf006_198110.01.gz</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">0<span class="Apple-tab-span" style="white-space:pre"> </span>ans002/archive/ans002/HINDCASTS/RUN_ATMWANG_LANSENS/19810501.0/echam5</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">0<span class="Apple-tab-span" style="white-space:pre"> </span>ans002/archive/ans002/HINDCASTS/RUN_ATMWANG_LANSENS/19810501.0</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">0<span class="Apple-tab-span" style="white-space:pre"> </span>ans002/archive/ans002/HINDCASTS/RUN_ATMWANG_LANSENS/19810501.1/echam5/echam_sf006_198105.01.gz</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">0<span class="Apple-tab-span" style="white-space:pre"> </span>ans002/archive/ans002/HINDCASTS/RUN_ATMWANG_LANSENS/19810501.1/echam5/echam_sf006_198109.01.gz</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">8<span class="Apple-tab-span" style="white-space:pre"> </span>ans002/archive/ans002/HINDCASTS/RUN_ATMWANG_LANSENS/19810501.1/echam5</span></div></div><div class=""><br class=""></div><div class="">What we have to do with this data? Should I backup this “empty” dirs and files on a different storage before deleting them?</div><div class=""><br class=""></div><div class="">As soon as all the bricks will be empty, I plan to re-add the new bricks using the following commands:</div><div class=""><br class=""></div><div class=""><i class="">gluster peer detach s04</i></div><div class=""><i class="">gluster peer detach s05</i></div><div class=""><i class="">gluster peer detach s06</i></div><div class=""><i class=""><br class=""></i></div><div class=""><i class="">gluster peer probe s04</i></div><div class=""><i class="">gluster peer probe s05</i></div><div class=""><i class="">gluster peer probe s06</i></div><div class=""><i class=""><br class=""></i></div><div class=""><i class="">gluster volume add-brick tier2 s04-stg:/gluster/mnt1/brick s05-stg:/gluster/mnt1/brick s06-stg:/gluster/mnt1/brick s04-stg:/gluster/mnt2/brick s05-stg:/gluster/mnt2/brick s06-stg:/gluster/mnt2/brick s04-stg:/gluster/mnt3/brick s05-stg:/gluster/mnt3/brick s06-stg:/gluster/mnt3/brick s04-stg:/gluster/mnt4/brick s05-stg:/gluster/mnt4/brick s06-stg:/gluster/mnt4/brick s04-stg:/gluster/mnt5/brick s05-stg:/gluster/mnt5/brick s06-stg:/gluster/mnt5/brick s04-stg:/gluster/mnt6/brick s05-stg:/gluster/mnt6/brick s06-stg:/gluster/mnt6/brick s04-stg:/gluster/mnt7/brick s05-stg:/gluster/mnt7/brick s06-stg:/gluster/mnt7/brick s04-stg:/gluster/mnt8/brick s05-stg:/gluster/mnt8/brick s06-stg:/gluster/mnt8/brick s04-stg:/gluster/mnt9/brick s05-stg:/gluster/mnt9/brick s06-stg:/gluster/mnt9/brick s04-stg:/gluster/mnt10/brick s05-stg:/gluster/mnt10/brick s06-stg:/gluster/mnt10/brick s04-stg:/gluster/mnt11/brick s05-stg:/gluster/mnt11/brick s06-stg:/gluster/mnt11/brick s04-stg:/gluster/mnt12/brick s05-stg:/gluster/mnt12/brick s06-stg:/gluster/mnt12/brick force</i></div><div class=""><i class=""><br class=""></i></div><div class=""><i class="">gluster volume rebalance tier2 fix-layout start</i></div><div class=""><i class=""><br class=""></i></div><div class=""><i class="">gluster volume rebalance tier2 start</i></div><div class=""><br class=""></div><div class="">From your point of view, are they the right commands to close this repairing task?</div><div class=""><br class=""></div><div class="">Thank you very much for your help.</div><div class="">Regards,</div><div class="">Mauro</div><div class=""><br class=""></div><div class=""></div></body></html>