<div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hi Mauro,<br><br></div>As Nithya highlighted FALLOCATE support for EC volumes went in 3.11 as part of <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1454686">https://bugzilla.redhat.com/show_bug.cgi?id=1454686</a>. Hence, upgrading to 3.12 as suggested before would be a right move.<br><br></div><div>Here is the documentation for upgrading to 3.12: <a href="https://docs.gluster.org/en/latest/Upgrade-Guide/upgrade_to_3.12/">https://docs.gluster.org/en/latest/Upgrade-Guide/upgrade_to_3.12/</a><br></div></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Regards,<br><p style="font-weight:bold;margin:0;padding:0;font-size:14px;text-transform:uppercase;margin-bottom:0"><span>Sunil kumar</span> <span>Acharya</span></p>
<p style="font-weight:normal;font-size:10px;margin:0px 0px 4px;text-transform:uppercase"><span>Senior Software Engineer</span></p>
<p style="font-weight:normal;margin:0;font-size:10px;color:#999"><a style="color:#0088ce;font-size:10px;margin:0;text-decoration:none;font-family:'overpass',sans-serif" href="https://www.redhat.com" target="_blank">Red Hat <span><br><br></span></a></p>
<p style="font-weight:normal;margin:0px 0px 6px;font-size:10px;color:rgb(153,153,153)">
<span href="tel:+91-8067935170">T: <a href="http://redhatemailsignature-marketing.itos.redhat.com/" style="color:#0088ce;font-size:10px;margin:0;text-decoration:none;font-family:'overpass',sans-serif" target="_blank">+91-8067935170</a> </span>
</p>
<a href="https://red.ht/sig" target="_blank"> <img src="https://www.redhat.com/profiles/rh/themes/redhatdotcom/img/logo-red-hat-black.png" width="90" height="auto"></a>
<div><a href="https://redhat.com/trusted" style="text-decoration:none;color:#c00;font-weight:bold" target="_blank">TRIED. TESTED. TRUSTED.</a></div><br></div></div></div>
<br><div class="gmail_quote">On Sat, Sep 15, 2018 at 3:42 AM, Mauro Tridici <span dir="ltr"><<a href="mailto:mauro.tridici@cmcc.it" target="_blank">mauro.tridici@cmcc.it</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><br></div><div>Hi Nithya,</div><div><br></div><div>thank you very much for your answer.</div><div>I will wait for @Sunil opinion too before starting the upgrade procedure.</div><div><br></div><div>Since it will be the first upgrade of our Gluster cluster, I would like to know if it could be a “virtually dangerous" procedure and if it will be the risk of losing data :-) </div><div>Unfortunately, I can’t do a preventive copy of the volume data in another location.</div><div>If it is possible, could you please illustrate the right steps needed to complete the upgrade procedure from the 3.10.5 to the 3.12 version?</div><div><br></div><div>Thank you again, Nithya.</div><div>Thank you to all of you for the help!</div><div><br></div><div>Regards,</div><div>Mauro</div><div><div class="h5"><br><div><blockquote type="cite"><div>Il giorno 14 set 2018, alle ore 16:59, Nithya Balachandran <<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a>> ha scritto:</div><br class="m_-7509393663119327190Apple-interchange-newline"><div><div dir="ltr"><div dir="ltr">Hi Mauro,<div><br></div><div><br></div><div>The rebalance code started using fallocate in 3.10.5 (<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1473132" target="_blank">https://bugzilla.redhat.com/<wbr>show_bug.cgi?id=1473132</a>) which works fine on replicated volumes. However, we neglected to test this with EC volumes on 3.10. Once we discovered the issue, the EC fallocate implementation was made available in 3.11.</div><div><br></div><div>At this point, I'm afraid the only option I see is to upgrade to at least 3.12.</div><div><br></div><div>@Sunil, do you have anything to add?</div><div><br></div><div>Regards,</div><div>Nithya</div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 13 September 2018 at 18:34, Mauro Tridici <span dir="ltr"><<a href="mailto:mauro.tridici@cmcc.it" target="_blank">mauro.tridici@cmcc.it</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><br></div><div>Hi Nithya,</div><div><br></div><div>thank you for involving EC group.</div><div>I will wait for your suggestions.</div><div><br></div><div>Regards,</div><div>Mauro</div><div><div class="m_-7509393663119327190h5"><br><div><blockquote type="cite"><div>Il giorno 13 set 2018, alle ore 13:38, Nithya Balachandran <<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a>> ha scritto:</div><br class="m_-7509393663119327190m_821703117890918883Apple-interchange-newline"><div><div dir="ltr">This looks like an issue because rebalance switched to using fallocate which EC did not have implemented at that point.<div><br></div><div>@Pranith, @Ashish, which version of gluster had support for fallocate in EC?</div><div><br></div><div><br></div><div>Regards,</div><div>Nithya</div></div><div class="gmail_extra"><br><div class="gmail_quote">On 12 September 2018 at 19:24, Mauro Tridici <span dir="ltr"><<a href="mailto:mauro.tridici@cmcc.it" target="_blank">mauro.tridici@cmcc.it</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><div>Dear All,</div><div><br></div><div>I recently added 3 servers (each one with 12 bricks) to an existing Gluster Distributed Disperse Volume.</div><div>Volume extension has been completed without error and I already executed the rebalance procedure with fix-layout option with no problem.</div><div>I just launched the rebalance procedure without fix-layout option, but, as you can see in the output below, I noticed that some failures have been detected.</div><div><br></div><div><font size="2" face="Courier">[root@s01 glusterfs]# gluster v rebalance tier2 status</font></div><div><font size="2" face="Courier"> Node Rebalanced-files size scanned failures skipped status run time in h:m:s</font></div><div><font size="2" face="Courier"> --------- ----------- ----------- ----------- ----------- ----------- ------------ --------------</font></div><div><font size="2" face="Courier"> localhost 71176 3.2MB 2137557 1530391 8128 in progress 13:59:05</font></div><div><font size="2" face="Courier"> s02-stg 0 0Bytes 0 0 0 completed 11:53:28</font></div><div><font size="2" face="Courier"> s03-stg 0 0Bytes 0 0 0 completed 11:53:32</font></div><div><font size="2" face="Courier"> s04-stg 0 0Bytes 0 0 0 completed 0:00:06</font></div><div><font size="2" face="Courier"> s05-stg 15 0Bytes 17055 0 18 completed 10:48:01</font></div><div><font size="2" face="Courier"> s06-stg 0 0Bytes 0 0 0 completed 0:00:06</font></div><div><font size="2" face="Courier">Estimated time left for rebalance to complete : 0:46:53</font></div><div><font size="2" face="Courier">volume rebalance: tier2: success</font></div><div><br></div><div>In the volume rebalance log file, I detected a lot of error messages similar to the following ones:</div><div><br></div><div><font size="2" face="Courier">[2018-09-12 13:15:50.756703] E [MSGID: 0] [dht-rebalance.c:1696:dht_migr<wbr>ate_file] 0-tier2-dht: Create dst failed on - tier2-disperse-6 for file - /CSP/sp1/CESM/archive/sps_2005<wbr>08_003/atm/hist/postproc/<a href="http://sps_200508_003.cam.h0.2005-12_grid.nc/" target="_blank">sps_2<wbr>00508_003.cam.h0.2005-12_grid.<wbr>nc</a></font></div><div><font size="2" face="Courier">[2018-09-12 13:15:50.757025] E [MSGID: 109023] [dht-rebalance.c:2733:gf_defra<wbr>g_migrate_single_file] 0-tier2-dht: migrate-data failed for /CSP/sp1/CESM/archive/sps_2005<wbr>08_003/atm/hist/postproc/<a href="http://sps_200508_003.cam.h0.2005-12_grid.nc/" target="_blank">sps_2<wbr>00508_003.cam.h0.2005-12_grid.<wbr>nc</a></font></div><div><font size="2" face="Courier">[2018-09-12 13:15:50.759183] E [MSGID: 109023] [dht-rebalance.c:844:__dht_reb<wbr>alance_create_dst_file] 0-tier2-dht: fallocate failed for /CSP/sp1/CESM/archive/sps_2005<wbr>08_003/atm/hist/postproc/<a href="http://sps_200508_003.cam.h0.2005-09_grid.nc/" target="_blank">sps_2<wbr>00508_003.cam.h0.2005-09_grid.<wbr>nc</a> on tier2-disperse-9 (Operation not supported)</font></div><div><font size="2" face="Courier">[2018-09-12 13:15:50.759206] E [MSGID: 0] [dht-rebalance.c:1696:dht_migr<wbr>ate_file] 0-tier2-dht: Create dst failed on - tier2-disperse-9 for file - /CSP/sp1/CESM/archive/sps_2005<wbr>08_003/atm/hist/postproc/<a href="http://sps_200508_003.cam.h0.2005-09_grid.nc/" target="_blank">sps_2<wbr>00508_003.cam.h0.2005-09_grid.<wbr>nc</a></font></div><div><font size="2" face="Courier">[2018-09-12 13:15:50.759536] E [MSGID: 109023] [dht-rebalance.c:2733:gf_defra<wbr>g_migrate_single_file] 0-tier2-dht: migrate-data failed for /CSP/sp1/CESM/archive/sps_2005<wbr>08_003/atm/hist/postproc/<a href="http://sps_200508_003.cam.h0.2005-09_grid.nc/" target="_blank">sps_2<wbr>00508_003.cam.h0.2005-09_grid.<wbr>nc</a></font></div><div><font size="2" face="Courier">[2018-09-12 13:15:50.777219] E [MSGID: 109023] [dht-rebalance.c:844:__dht_reb<wbr>alance_create_dst_file] 0-tier2-dht: fallocate failed for /CSP/sp1/CESM/archive/sps_2005<wbr>08_003/atm/hist/postproc/<a href="http://sps_200508_003.cam.h0.2006-01_grid.nc/" target="_blank">sps_2<wbr>00508_003.cam.h0.2006-01_grid.<wbr>nc</a> on tier2-disperse-10 (Operation not supported)</font></div><div><font size="2" face="Courier">[2018-09-12 13:15:50.777241] E [MSGID: 0] [dht-rebalance.c:1696:dht_migr<wbr>ate_file] 0-tier2-dht: Create dst failed on - tier2-disperse-10 for file - /CSP/sp1/CESM/archive/sps_2005<wbr>08_003/atm/hist/postproc/<a href="http://sps_200508_003.cam.h0.2006-01_grid.nc/" target="_blank">sps_2<wbr>00508_003.cam.h0.2006-01_grid.<wbr>nc</a></font></div><div><font size="2" face="Courier">[2018-09-12 13:15:50.777676] E [MSGID: 109023] [dht-rebalance.c:2733:gf_defra<wbr>g_migrate_single_file] 0-tier2-dht: migrate-data failed for /CSP/sp1/CESM/archive/sps_2005<wbr>08_003/atm/hist/postproc/<a href="http://sps_200508_003.cam.h0.2006-01_grid.nc/" target="_blank">sps_2<wbr>00508_003.cam.h0.2006-01_grid.<wbr>nc</a></font></div><div><br></div><div>Could you please help me to understand what is happening and how to solve it?</div><div><br></div><div>Our Gluster implementation is based on Gluster v.3.10.5</div><div><br></div><div>Thank you in advance,</div><div>Mauro</div>
</div>
<br></div><br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mail<wbr>man/listinfo/gluster-users</a><br></blockquote></div><br></div>
</div></blockquote></div><br></div></div><div>
<span class="m_-7509393663119327190m_821703117890918883Apple-style-span" style="border-collapse:separate;font-family:Helvetica;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-variant-alternates:normal;font-variant-east-asian:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span class="m_-7509393663119327190m_821703117890918883Apple-style-span" style="border-collapse:separate;font-family:Helvetica;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-variant-alternates:normal;font-variant-east-asian:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div><br class="m_-7509393663119327190m_821703117890918883Apple-interchange-newline">-------------------------</div><div>Mauro Tridici</div><div><br></div><div>Fondazione CMCC</div><div>CMCC Supercomputing Center</div><div>presso Complesso Ecotekne - Università del Salento -</div><div>Strada Prov.le Lecce - Monteroni sn</div><div>73100 Lecce IT</div><div><a href="http://www.cmcc.it/" target="_blank">http://www.cmcc.it</a></div><div><br></div><div>mobile: (+39) 327 5630841</div><div>email: <a href="mailto:mauro.tridici@cmcc.it" target="_blank">mauro.tridici@cmcc.it</a></div></span></span>
</div>
<br></div></blockquote></div><br></div>
</div></blockquote></div><br><div>
<span class="m_-7509393663119327190Apple-style-span" style="border-collapse:separate;color:rgb(0,0,0);font-family:Helvetica;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span class="m_-7509393663119327190Apple-style-span" style="border-collapse:separate;color:rgb(0,0,0);font-family:Helvetica;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><div><br class="m_-7509393663119327190Apple-interchange-newline">-------------------------</div><div>Mauro Tridici</div><div><br></div><div>Fondazione CMCC</div><div>CMCC Supercomputing Center</div><div>presso Complesso Ecotekne - Università del Salento -</div><div>Strada Prov.le Lecce - Monteroni sn</div><div>73100 Lecce IT</div><div><a href="http://www.cmcc.it" target="_blank">http://www.cmcc.it</a></div><div><br></div><div>mobile: (+39) 327 5630841</div><div>email: <a href="mailto:mauro.tridici@cmcc.it" target="_blank">mauro.tridici@cmcc.it</a></div></span></span>
</div>
<br></div></div></div></blockquote></div><br></div>