<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><br class=""></div><div class="">Hi Nithya,</div><div class=""><br class=""></div><div class="">thank you for involving EC group.</div><div class="">I will wait for your suggestions.</div><div class=""><br class=""></div><div class="">Regards,</div><div class="">Mauro</div><br class=""><div><blockquote type="cite" class=""><div class="">Il giorno 13 set 2018, alle ore 13:38, Nithya Balachandran <<a href="mailto:nbalacha@redhat.com" class="">nbalacha@redhat.com</a>> ha scritto:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">This looks like an issue because rebalance switched to using fallocate which EC did not have implemented at that point.<div class=""><br class=""></div><div class="">@Pranith, @Ashish, which version of gluster had support for fallocate in EC?</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Regards,</div><div class="">Nithya</div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On 12 September 2018 at 19:24, Mauro Tridici <span dir="ltr" class=""><<a href="mailto:mauro.tridici@cmcc.it" target="_blank" class="">mauro.tridici@cmcc.it</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class=""><div class=""><div class="">Dear All,</div><div class=""><br class=""></div><div class="">I recently added 3 servers (each one with 12 bricks) to an existing Gluster Distributed Disperse Volume.</div><div class="">Volume extension has been completed without error and I already executed the rebalance procedure with fix-layout option with no problem.</div><div class="">I just launched the rebalance procedure without fix-layout option, but, as you can see in the output below, I noticed that some failures have been detected.</div><div class=""><br class=""></div><div class=""><font face="Courier" size="2" class="">[root@s01 glusterfs]# gluster v rebalance tier2 status</font></div><div class=""><font face="Courier" size="2" class=""> Node Rebalanced-files size scanned failures skipped status run time in h:m:s</font></div><div class=""><font face="Courier" size="2" class=""> --------- ----------- ----------- ----------- ----------- ----------- ------------ --------------</font></div><div class=""><font face="Courier" size="2" class=""> localhost 71176 3.2MB 2137557 1530391 8128 in progress 13:59:05</font></div><div class=""><font face="Courier" size="2" class=""> s02-stg 0 0Bytes 0 0 0 completed 11:53:28</font></div><div class=""><font face="Courier" size="2" class=""> s03-stg 0 0Bytes 0 0 0 completed 11:53:32</font></div><div class=""><font face="Courier" size="2" class=""> s04-stg 0 0Bytes 0 0 0 completed 0:00:06</font></div><div class=""><font face="Courier" size="2" class=""> s05-stg 15 0Bytes 17055 0 18 completed 10:48:01</font></div><div class=""><font face="Courier" size="2" class=""> s06-stg 0 0Bytes 0 0 0 completed 0:00:06</font></div><div class=""><font face="Courier" size="2" class="">Estimated time left for rebalance to complete : 0:46:53</font></div><div class=""><font face="Courier" size="2" class="">volume rebalance: tier2: success</font></div><div class=""><br class=""></div><div class="">In the volume rebalance log file, I detected a lot of error messages similar to the following ones:</div><div class=""><br class=""></div><div class=""><font face="Courier" size="2" class="">[2018-09-12 13:15:50.756703] E [MSGID: 0] [dht-rebalance.c:1696:dht_<wbr class="">migrate_file] 0-tier2-dht: Create dst failed on - tier2-disperse-6 for file - /CSP/sp1/CESM/archive/sps_<wbr class="">200508_003/atm/hist/postproc/<a href="http://sps_200508_003.cam.h0.2005-12_grid.nc/" target="_blank" class="">s<wbr class="">ps_200508_003.cam.h0.2005-12_<wbr class="">grid.nc</a></font></div><div class=""><font face="Courier" size="2" class="">[2018-09-12 13:15:50.757025] E [MSGID: 109023] [dht-rebalance.c:2733:gf_<wbr class="">defrag_migrate_single_file] 0-tier2-dht: migrate-data failed for /CSP/sp1/CESM/archive/sps_<wbr class="">200508_003/atm/hist/postproc/<a href="http://sps_200508_003.cam.h0.2005-12_grid.nc/" target="_blank" class="">s<wbr class="">ps_200508_003.cam.h0.2005-12_<wbr class="">grid.nc</a></font></div><div class=""><font face="Courier" size="2" class="">[2018-09-12 13:15:50.759183] E [MSGID: 109023] [dht-rebalance.c:844:__dht_<wbr class="">rebalance_create_dst_file] 0-tier2-dht: fallocate failed for /CSP/sp1/CESM/archive/sps_<wbr class="">200508_003/atm/hist/postproc/<a href="http://sps_200508_003.cam.h0.2005-09_grid.nc/" target="_blank" class="">s<wbr class="">ps_200508_003.cam.h0.2005-09_<wbr class="">grid.nc</a> on tier2-disperse-9 (Operation not supported)</font></div><div class=""><font face="Courier" size="2" class="">[2018-09-12 13:15:50.759206] E [MSGID: 0] [dht-rebalance.c:1696:dht_<wbr class="">migrate_file] 0-tier2-dht: Create dst failed on - tier2-disperse-9 for file - /CSP/sp1/CESM/archive/sps_<wbr class="">200508_003/atm/hist/postproc/<a href="http://sps_200508_003.cam.h0.2005-09_grid.nc/" target="_blank" class="">s<wbr class="">ps_200508_003.cam.h0.2005-09_<wbr class="">grid.nc</a></font></div><div class=""><font face="Courier" size="2" class="">[2018-09-12 13:15:50.759536] E [MSGID: 109023] [dht-rebalance.c:2733:gf_<wbr class="">defrag_migrate_single_file] 0-tier2-dht: migrate-data failed for /CSP/sp1/CESM/archive/sps_<wbr class="">200508_003/atm/hist/postproc/<a href="http://sps_200508_003.cam.h0.2005-09_grid.nc/" target="_blank" class="">s<wbr class="">ps_200508_003.cam.h0.2005-09_<wbr class="">grid.nc</a></font></div><div class=""><font face="Courier" size="2" class="">[2018-09-12 13:15:50.777219] E [MSGID: 109023] [dht-rebalance.c:844:__dht_<wbr class="">rebalance_create_dst_file] 0-tier2-dht: fallocate failed for /CSP/sp1/CESM/archive/sps_<wbr class="">200508_003/atm/hist/postproc/<a href="http://sps_200508_003.cam.h0.2006-01_grid.nc/" target="_blank" class="">s<wbr class="">ps_200508_003.cam.h0.2006-01_<wbr class="">grid.nc</a> on tier2-disperse-10 (Operation not supported)</font></div><div class=""><font face="Courier" size="2" class="">[2018-09-12 13:15:50.777241] E [MSGID: 0] [dht-rebalance.c:1696:dht_<wbr class="">migrate_file] 0-tier2-dht: Create dst failed on - tier2-disperse-10 for file - /CSP/sp1/CESM/archive/sps_<wbr class="">200508_003/atm/hist/postproc/<a href="http://sps_200508_003.cam.h0.2006-01_grid.nc/" target="_blank" class="">s<wbr class="">ps_200508_003.cam.h0.2006-01_<wbr class="">grid.nc</a></font></div><div class=""><font face="Courier" size="2" class="">[2018-09-12 13:15:50.777676] E [MSGID: 109023] [dht-rebalance.c:2733:gf_<wbr class="">defrag_migrate_single_file] 0-tier2-dht: migrate-data failed for /CSP/sp1/CESM/archive/sps_<wbr class="">200508_003/atm/hist/postproc/<a href="http://sps_200508_003.cam.h0.2006-01_grid.nc/" target="_blank" class="">s<wbr class="">ps_200508_003.cam.h0.2006-01_<wbr class="">grid.nc</a></font></div><div class=""><br class=""></div><div class="">Could you please help me to understand what is happening and how to solve it?</div><div class=""><br class=""></div><div class="">Our Gluster implementation is based on Gluster v.3.10.5</div><div class=""><br class=""></div><div class="">Thank you in advance,</div><div class="">Mauro</div>
</div>
<br class=""></div><br class="">______________________________<wbr class="">_________________<br class="">
Gluster-users mailing list<br class="">
<a href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class="">
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank" class="">https://lists.gluster.org/<wbr class="">mailman/listinfo/gluster-users</a><br class=""></blockquote></div><br class=""></div>
</div></blockquote></div><br class=""><div class="">
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div class=""><br class="Apple-interchange-newline">-------------------------</div><div class="">Mauro Tridici</div><div class=""><br class=""></div><div class="">Fondazione CMCC</div><div class="">CMCC Supercomputing Center</div><div class="">presso Complesso Ecotekne - Università del Salento -</div><div class="">Strada Prov.le Lecce - Monteroni sn</div><div class="">73100 Lecce IT</div><div class=""><a href="http://www.cmcc.it" class="">http://www.cmcc.it</a></div><div class=""><br class=""></div><div class="">mobile: (+39) 327 5630841</div><div class="">email: <a href="mailto:mauro.tridici@cmcc.it" class="">mauro.tridici@cmcc.it</a></div></span></span>
</div>
<br class=""></body></html>