<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><br class=""></div><div class="">Hi Hari,</div><div class=""><br class=""></div><div class="">thank you very much for the explanation and for your important support.</div><div class=""><br class=""></div><div class="">Best regards,</div><div class="">Mauro</div><br class=""><div><blockquote type="cite" class=""><div class="">Il giorno 11 set 2018, alle ore 10:49, Hari Gowtham <<a href="mailto:hgowtham@redhat.com" class="">hgowtham@redhat.com</a>> ha scritto:</div><br class="Apple-interchange-newline"><div class=""><div class="">Hi Mauro,<br class=""><br class="">It was because the quota crawl takes some time and it was working on it.<br class="">When we ran the fix-issues it makes changes to the backend and does a lookup.<br class="">It takes time for the whole thing to reflect in the quota list command.<br class="">Earlier, it didnt reflect as it was still crawling. So this is the<br class="">same as the first reason I<br class="">have mentioned above in the 3 situations that could have happened.<br class="">This is the expected behavior.<br class=""><br class="">Regards,<br class="">Hari.<br class="">On Mon, Sep 10, 2018 at 8:03 PM Mauro Tridici <<a href="mailto:mauro.tridici@cmcc.it" class="">mauro.tridici@cmcc.it</a>> wrote:<br class=""><blockquote type="cite" class=""><br class=""><br class="">Hi Hari,<br class=""><br class="">good news for us!<br class=""><br class="">A few seconds ago, I submitted the gluster quota list command in order to save the current quota status.<br class=""><br class="">[root@s01 auto]# gluster volume quota tier2 list /ASC<br class=""> Path Hard-limit Soft-limit Used Available Soft-limit exceeded? Hard-limit exceeded?<br class="">-------------------------------------------------------------------------------------------------------------------------------<br class="">/ASC 10.0TB 99%(9.9TB) 2.6TB 7.4TB No No<br class=""><br class="">In the same time, I was asking myself how I can stimulate a sort of directory “scan” in order to refresh the quota value without waiting for the automatic scan.<br class="">So, I decided to start a “du -hs /tier2/ASC” session (without specify each single brick path as I usually do after quota-fsck script execution).<br class=""><br class="">[root@s01 auto]# du -hs /tier2/ASC<br class="">22G /tier2/ASC<br class=""><br class="">Now, magically, the quota value reflects the real disk space usage info provided by the “du” command.<br class=""><br class="">[root@s01 auto]# gluster volume quota tier2 list /ASC<br class=""> Path Hard-limit Soft-limit Used Available Soft-limit exceeded? Hard-limit exceeded?<br class="">-------------------------------------------------------------------------------------------------------------------------------<br class="">/ASC 10.0TB 99%(9.9TB) 21.8GB 10.0TB No No<br class=""><br class="">Do you think that I was only lucky or is there a particular reason why everything is now working?<br class=""><br class="">Thank you,<br class="">Mauro<br class=""><br class=""><br class="">Il giorno 10 set 2018, alle ore 16:08, Mauro Tridici <<a href="mailto:mauro.tridici@cmcc.it" class="">mauro.tridici@cmcc.it</a>> ha scritto:<br class=""><br class=""><br class="">Hi Hari,<br class=""><br class="">thank you very much for your support.<br class="">I will do everything you suggested and I will contact you as soon as all the steps will be completed.<br class=""><br class="">Thank you,<br class="">Mauro<br class=""><br class="">Il giorno 10 set 2018, alle ore 16:02, Hari Gowtham <<a href="mailto:hgowtham@redhat.com" class="">hgowtham@redhat.com</a>> ha scritto:<br class=""><br class="">Hi Mauro,<br class=""><br class="">I went through the log file you have shared.<br class="">I don't find any mismatch.<br class=""><br class="">This can be because of various reasons:<br class="">1) the accounting which was wrong is now fine. but as per your comment<br class="">above if this is the case,<br class="">then the crawl should still be happening which is why the its not yet<br class="">reflected. (will reflect after a while)<br class="">2) the fix-issue part of the script might be wrong.<br class="">3) or the final script that we use might be wrong.<br class=""><br class="">You can wait for a while (based on the number of files the time will<br class="">vary) and then see if the accounting is fine.<br class="">If its not fine even after a while, then we will have to run the<br class="">script (6th patch set has worked so can be reused) without "fix-issue"<br class="">This will give us the mismatch in log file, which i can read and let<br class="">you know where the lookup has to be done.<br class="">On Mon, Sep 10, 2018 at 4:58 PM Mauro Tridici <<a href="mailto:mauro.tridici@cmcc.it" class="">mauro.tridici@cmcc.it</a>> wrote:<br class=""><br class=""><br class=""><br class="">Dear Hari,<br class=""><br class="">the log files that I attached to my last mail have been generated running quota-fsck script after deleting the files.<br class="">The quota-fsck script version that I used is the one in the following link <a href="https://review.gluster.org/#/c/19179/9..9/extras/quota/quota_fsck.py" class="">https://review.gluster.org/#/c/19179/9..9/extras/quota/quota_fsck.py</a><br class="">I didn’t edit the log files, but during the execution I forgot to redirect the stderr and stdout to the same log file, sorry, mea culpa!<br class=""><br class="">Anyway, as you suggested, I executed again the quota-fsck script with option —fix-issues.<br class="">At the end of script execution, I launched the du command, but the problem is still there.<br class=""><br class="">[root@s02 auto]# df -hT /tier2/ASC/<br class="">File system Tipo Dim. Usati Dispon. Uso% Montato su<br class="">s02-stg:tier2 fuse.glusterfs 10T 2,6T 7,5T 26% /tier2<br class=""><br class="">I’m sorry to bother you so much.<br class="">Last time I used the script everything went smoothly, but this time it seems to be more difficult.<br class=""><br class="">In attachment you can find the new log files.<br class=""><br class="">Thank you,<br class="">Mauro<br class=""><br class=""><br class="">Il giorno 10 set 2018, alle ore 12:27, Hari Gowtham <<a href="mailto:hgowtham@redhat.com" class="">hgowtham@redhat.com</a>> ha scritto:<br class=""><br class="">On Mon, Sep 10, 2018 at 3:13 PM Mauro Tridici <<a href="mailto:mauro.tridici@cmcc.it" class="">mauro.tridici@cmcc.it</a>> wrote:<br class=""><br class=""><br class=""><br class="">Dear Hari,<br class=""><br class="">I followed you suggestions, but, unfortunately, nothing is changed.<br class="">I tried to execute both the quota-fsck script with —fix-issues options both the "setfattr -n trusted.glusterfs.quota.dirty -v 0x3100” command against the files and directory mentioned by you (on each available brick).<br class=""><br class=""><br class="">There can be an issue with fix-issue in the script. As the directories<br class="">with accounting mismatch awre found its better to set the dirty xattr<br class="">and then do a du(this way its wasy and has to resolve the issue). The<br class="">script can be used when we dont know where the issue is.<br class=""><br class="">Disk quota assigned to /tier2/ASC directory seems to be partially used (about 2,6 TB used), but the “real and current” situation is the following one (I deleted all files in primavera directory):<br class=""><br class=""><br class="">If the files are deleted, then state of the log file from the script<br class="">is outdated. The folders I suggested are as per the old log file, So<br class="">setting the dirty xattr and then doing a lookup (du on that dir) might<br class="">not help.<br class=""><br class=""><br class="">[root@s03 qc]# du -hsc /tier2/ASC/*<br class="">22G /tier2/ASC/orientgate<br class="">26K /tier2/ASC/primavera<br class="">22G totale<br class=""><br class="">So, I think that the problem should be only in "orientgate” or in “primavera” directory, right!?<br class="">For this reason, in order to collect some fresh logs, I executed again the check script starting from the top level directory “ASC” using the following bash script (named hari-20180910) based on the new version of quota_fsck (rel. 9):<br class=""><br class="">hari-20180910 script:<br class=""><br class="">#!/bin/bash<br class=""><br class="">#set -xv<br class=""><br class="">host=$(hostname)<br class=""><br class="">for i in {1..12}<br class="">do<br class="">./quota_fsck_r9.py --full-logs --sub-dir ASC /gluster/mnt$i/brick >> $host.log<br class="">done<br class="">~<br class=""><br class="">In attachment, you can find the log files generated by the script.<br class=""><br class="">SOME IMPORTANT NOTES:<br class=""><br class="">- in the new log files, “primavera” directory is no more present<br class=""><br class="">Is there something more that I can do?<br class=""><br class="">As there were files that were deleted, the accounting would have changed again.<br class=""><br class="">Need to look from the beginning, as the above suggestions may not be<br class="">true anymore.<br class=""><br class="">I find that the log files are edited. A few lines are missing. Can you<br class="">send the actual log file from running the script<br class="">And i would recommend you to run the script after all the files are<br class="">deleted (or other major modifications are done).<br class="">So that we can fix once at the end.<br class=""><br class="">If the fix-issue argument on script doesn't work on the directory/<br class="">subdirectory where you find mismatch, then you can send the whole<br class="">file.<br class="">Will check the log and let you know where you need to do the lookup.<br class=""><br class=""><br class="">Thank you very much for your patience.<br class="">Regards,<br class="">Mauro<br class=""><br class=""><br class="">Il giorno 10 set 2018, alle ore 10:51, Hari Gowtham <<a href="mailto:hgowtham@redhat.com" class="">hgowtham@redhat.com</a>> ha scritto:<br class=""><br class="">Hi,<br class=""><br class="">Looking at the logs, I can see that the file:<br class=""><br class="">/orientgate/ftp/climate/3_urban_adaptation_health/6_budapest_veszprem_hungary/RHMSS_CMCC-CM_NMMB_Balkan_8km_1971-2005<br class="">/orientgate/ftp/climate/3_urban_adaptation_health/6_budapest_veszprem_hungary/RHMSS_ERA40_NMMB_Balkan_8km_1971-2000<br class="">/orientgate/ftp/climate/3_urban_adaptation_health/6_budapest_veszprem_hungary/RHMSS_CMCC-CM_NMMB-RCP8.5_Balkan_8km_2010-2100<br class="">/primavera/cam<br class=""><br class="">has mismatch.<br class=""><br class="">You can try setting dirty for this and then do a du on it.<br class=""><br class="">A few corrections for my above comments.<br class="">The contri size in the xattr and the aggregated size have to be checked.<br class=""><br class="">On Mon, Sep 10, 2018 at 1:16 PM Mauro Tridici <<a href="mailto:mauro.tridici@cmcc.it" class="">mauro.tridici@cmcc.it</a>> wrote:<br class=""><br class=""><br class=""><br class="">Hi Hari,<br class=""><br class="">thank you very much for your help.<br class="">I will try to use the latest available version of quota_fsck script and I will provide you a feedback as soon as possible.<br class=""><br class="">Thank you again for the detailed explanation.<br class="">Regards,<br class="">Mauro<br class=""><br class="">Il giorno 10 set 2018, alle ore 09:17, Hari Gowtham <<a href="mailto:hgowtham@redhat.com" class="">hgowtham@redhat.com</a>> ha scritto:<br class=""><br class="">Hi Mauro,<br class=""><br class="">The problem might be at some other place, So setting the xattr and<br class="">doing the lookup might not have fixed the issue.<br class=""><br class="">To resolve this we need to read the log file reported by the fsck<br class="">script. In this log file we need to look for the size reported by the<br class="">xattr (the value "SIZE:" in the log file) and the size reported by the<br class="">stat on the file (the value after "st_size=" ).<br class=""><br class=""><br class="">The contri size in the xattr and the aggregated size have to be checked<br class=""><br class="">These two should be the same. If they mismatch, then we have to find<br class="">the top most dir which has the mismatch.<br class=""><br class=""><br class="">Bottom most dir/file has to be found. Replace top with bottom in the<br class="">following places as well.<br class=""><br class="">On this top most directory you have to do a set dirty xattr and then<br class="">do a lookup.<br class=""><br class="">If there are two different directories without a common top directory,<br class="">then both these have to undergo the above process.<br class=""><br class="">The fsck script should work fine. can you try the "--fix-issue" with<br class="">the latest script instead of the 6th patch used above?<br class=""><br class=""><br class=""><br class=""><br class=""><br class=""><br class="">--<br class="">Regards,<br class="">Hari Gowtham.<br class=""><br class=""><br class=""><br class="">-------------------------<br class="">Mauro Tridici<br class=""><br class="">Fondazione CMCC<br class="">CMCC Supercomputing Center<br class="">presso Complesso Ecotekne - Università del Salento -<br class="">Strada Prov.le Lecce - Monteroni sn<br class="">73100 Lecce IT<br class=""><a href="http://www.cmcc.it" class="">http://www.cmcc.it</a><br class=""><br class="">mobile: (+39) 327 5630841<br class="">email: mauro.tridici@cmcc.it<br class=""><br class=""><br class=""><br class="">--<br class="">Regards,<br class="">Hari Gowtham.<br class=""><br class=""><br class=""><br class="">-------------------------<br class="">Mauro Tridici<br class=""><br class="">Fondazione CMCC<br class="">CMCC Supercomputing Center<br class="">presso Complesso Ecotekne - Università del Salento -<br class="">Strada Prov.le Lecce - Monteroni sn<br class="">73100 Lecce IT<br class="">http://www.cmcc.it<br class=""><br class="">mobile: (+39) 327 5630841<br class="">email: mauro.tridici@cmcc.it<br class=""><br class=""><br class=""><br class="">--<br class="">Regards,<br class="">Hari Gowtham.<br class=""><br class=""><br class=""><br class="">-------------------------<br class="">Mauro Tridici<br class=""><br class="">Fondazione CMCC<br class="">CMCC Supercomputing Center<br class="">presso Complesso Ecotekne - Università del Salento -<br class="">Strada Prov.le Lecce - Monteroni sn<br class="">73100 Lecce IT<br class="">http://www.cmcc.it<br class=""><br class="">mobile: (+39) 327 5630841<br class="">email: mauro.tridici@cmcc.it<br class=""><br class=""><br class=""><br class="">-------------------------<br class="">Mauro Tridici<br class=""><br class="">Fondazione CMCC<br class="">CMCC Supercomputing Center<br class="">presso Complesso Ecotekne - Università del Salento -<br class="">Strada Prov.le Lecce - Monteroni sn<br class="">73100 Lecce IT<br class="">http://www.cmcc.it<br class=""><br class="">mobile: (+39) 327 5630841<br class="">email: mauro.tridici@cmcc.it<br class=""><br class=""></blockquote><br class=""><br class="">-- <br class="">Regards,<br class="">Hari Gowtham.<br class=""></div></div></blockquote></div><br class=""><div class="">
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div class=""><br class="Apple-interchange-newline">-------------------------</div><div class="">Mauro Tridici</div><div class=""><br class=""></div><div class="">Fondazione CMCC</div><div class="">CMCC Supercomputing Center</div><div class="">presso Complesso Ecotekne - Università del Salento -</div><div class="">Strada Prov.le Lecce - Monteroni sn</div><div class="">73100 Lecce IT</div><div class=""><a href="http://www.cmcc.it" class="">http://www.cmcc.it</a></div><div class=""><br class=""></div><div class="">mobile: (+39) 327 5630841</div><div class="">email: <a href="mailto:mauro.tridici@cmcc.it" class="">mauro.tridici@cmcc.it</a></div></span></span>
</div>
<br class=""></body></html>