<div dir="ltr">This looks like an issue because rebalance switched to using fallocate which EC did not have implemented at that point.<div><br></div><div>@Pranith, @Ashish, which version of gluster had support for fallocate in EC?</div><div><br></div><div><br></div><div>Regards,</div><div>Nithya</div></div><div class="gmail_extra"><br><div class="gmail_quote">On 12 September 2018 at 19:24, Mauro Tridici <span dir="ltr"><<a href="mailto:mauro.tridici@cmcc.it" target="_blank">mauro.tridici@cmcc.it</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><div>Dear All,</div><div><br></div><div>I recently added 3 servers (each one with 12 bricks) to an existing Gluster Distributed Disperse Volume.</div><div>Volume extension has been completed without error and I already executed the rebalance procedure with fix-layout option with no problem.</div><div>I just launched the rebalance procedure without fix-layout option, but, as you can see in the output below, I noticed that some failures have been detected.</div><div><br></div><div><font face="Courier" size="2">[root@s01 glusterfs]# gluster v rebalance tier2 status</font></div><div><font face="Courier" size="2"> Node Rebalanced-files size scanned failures skipped status run time in h:m:s</font></div><div><font face="Courier" size="2"> --------- ----------- ----------- ----------- ----------- ----------- ------------ --------------</font></div><div><font face="Courier" size="2"> localhost 71176 3.2MB 2137557 1530391 8128 in progress 13:59:05</font></div><div><font face="Courier" size="2"> s02-stg 0 0Bytes 0 0 0 completed 11:53:28</font></div><div><font face="Courier" size="2"> s03-stg 0 0Bytes 0 0 0 completed 11:53:32</font></div><div><font face="Courier" size="2"> s04-stg 0 0Bytes 0 0 0 completed 0:00:06</font></div><div><font face="Courier" size="2"> s05-stg 15 0Bytes 17055 0 18 completed 10:48:01</font></div><div><font face="Courier" size="2"> s06-stg 0 0Bytes 0 0 0 completed 0:00:06</font></div><div><font face="Courier" size="2">Estimated time left for rebalance to complete : 0:46:53</font></div><div><font face="Courier" size="2">volume rebalance: tier2: success</font></div><div><br></div><div>In the volume rebalance log file, I detected a lot of error messages similar to the following ones:</div><div><br></div><div><font face="Courier" size="2">[2018-09-12 13:15:50.756703] E [MSGID: 0] [dht-rebalance.c:1696:dht_<wbr>migrate_file] 0-tier2-dht: Create dst failed on - tier2-disperse-6 for file - /CSP/sp1/CESM/archive/sps_<wbr>200508_003/atm/hist/postproc/<a href="http://sps_200508_003.cam.h0.2005-12_grid.nc" target="_blank">s<wbr>ps_200508_003.cam.h0.2005-12_<wbr>grid.nc</a></font></div><div><font face="Courier" size="2">[2018-09-12 13:15:50.757025] E [MSGID: 109023] [dht-rebalance.c:2733:gf_<wbr>defrag_migrate_single_file] 0-tier2-dht: migrate-data failed for /CSP/sp1/CESM/archive/sps_<wbr>200508_003/atm/hist/postproc/<a href="http://sps_200508_003.cam.h0.2005-12_grid.nc" target="_blank">s<wbr>ps_200508_003.cam.h0.2005-12_<wbr>grid.nc</a></font></div><div><font face="Courier" size="2">[2018-09-12 13:15:50.759183] E [MSGID: 109023] [dht-rebalance.c:844:__dht_<wbr>rebalance_create_dst_file] 0-tier2-dht: fallocate failed for /CSP/sp1/CESM/archive/sps_<wbr>200508_003/atm/hist/postproc/<a href="http://sps_200508_003.cam.h0.2005-09_grid.nc" target="_blank">s<wbr>ps_200508_003.cam.h0.2005-09_<wbr>grid.nc</a> on tier2-disperse-9 (Operation not supported)</font></div><div><font face="Courier" size="2">[2018-09-12 13:15:50.759206] E [MSGID: 0] [dht-rebalance.c:1696:dht_<wbr>migrate_file] 0-tier2-dht: Create dst failed on - tier2-disperse-9 for file - /CSP/sp1/CESM/archive/sps_<wbr>200508_003/atm/hist/postproc/<a href="http://sps_200508_003.cam.h0.2005-09_grid.nc" target="_blank">s<wbr>ps_200508_003.cam.h0.2005-09_<wbr>grid.nc</a></font></div><div><font face="Courier" size="2">[2018-09-12 13:15:50.759536] E [MSGID: 109023] [dht-rebalance.c:2733:gf_<wbr>defrag_migrate_single_file] 0-tier2-dht: migrate-data failed for /CSP/sp1/CESM/archive/sps_<wbr>200508_003/atm/hist/postproc/<a href="http://sps_200508_003.cam.h0.2005-09_grid.nc" target="_blank">s<wbr>ps_200508_003.cam.h0.2005-09_<wbr>grid.nc</a></font></div><div><font face="Courier" size="2">[2018-09-12 13:15:50.777219] E [MSGID: 109023] [dht-rebalance.c:844:__dht_<wbr>rebalance_create_dst_file] 0-tier2-dht: fallocate failed for /CSP/sp1/CESM/archive/sps_<wbr>200508_003/atm/hist/postproc/<a href="http://sps_200508_003.cam.h0.2006-01_grid.nc" target="_blank">s<wbr>ps_200508_003.cam.h0.2006-01_<wbr>grid.nc</a> on tier2-disperse-10 (Operation not supported)</font></div><div><font face="Courier" size="2">[2018-09-12 13:15:50.777241] E [MSGID: 0] [dht-rebalance.c:1696:dht_<wbr>migrate_file] 0-tier2-dht: Create dst failed on - tier2-disperse-10 for file - /CSP/sp1/CESM/archive/sps_<wbr>200508_003/atm/hist/postproc/<a href="http://sps_200508_003.cam.h0.2006-01_grid.nc" target="_blank">s<wbr>ps_200508_003.cam.h0.2006-01_<wbr>grid.nc</a></font></div><div><font face="Courier" size="2">[2018-09-12 13:15:50.777676] E [MSGID: 109023] [dht-rebalance.c:2733:gf_<wbr>defrag_migrate_single_file] 0-tier2-dht: migrate-data failed for /CSP/sp1/CESM/archive/sps_<wbr>200508_003/atm/hist/postproc/<a href="http://sps_200508_003.cam.h0.2006-01_grid.nc" target="_blank">s<wbr>ps_200508_003.cam.h0.2006-01_<wbr>grid.nc</a></font></div><div><br></div><div>Could you please help me to understand what is happening and how to solve it?</div><div><br></div><div>Our Gluster implementation is based on Gluster v.3.10.5</div><div><br></div><div>Thank you in advance,</div><div>Mauro</div>
</div>
<br></div><br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>