[Gluster-users] Wrong directory quota usage
João Baúto
joao.bauto at neuro.fchampalimaud.org
Sat Aug 15 23:27:56 UTC 2020
Hi Srijan & Strahil,
I ran the quota_fsck script mentioned in Hari's blog post in all bricks and
it detected a lot of size mismatch.
The script was executed as,
- python quota_fsck.py --sub-dir projectB --fix-issues /mnt/tank
/tank/volume2/brick (in all nodes and bricks)
Here is a snippet from the script,
Size Mismatch /tank/volume2/brick/projectB {'parents':
{'00000000-0000-0000-0000-000000000001': {'contri_file_count':
18446744073035296610L, 'contri_size': 18446645297413872640L,
'contri_dir_count': 18446744073709527653L}}, 'version': '1', 'file_count':
18446744073035296610L, 'dirty': False, 'dir_count': 18446744073709527653L,
'size': 18446645297413872640L} 15204281691754
MARKING DIRTY: /tank/volume2/brick/projectB
stat on /mnt/tank/projectB
Files verified : 683223
Directories verified : 46823
Objects Fixed : 705230
Checking the xattr in the bricks I can see the directory in question marked
as dirty,
# getfattr -d -m. -e hex /tank/volume2/brick/projectB
getfattr: Removing leading '/' from absolute path names
# file: tank/volume2/brick/projectB
trusted.gfid=0x3ca2bce0455945efa6662813ce20fc0c
trusted.glusterfs.9582685f-07fa-41fd-b9fc-ebab3a6989cf.xtime=0x5f372478000a7705
trusted.glusterfs.dht=0xe1a4060c000000003ffffffe5ffffffc
trusted.glusterfs.mdata=0x010000000000000000000000005f3724750000000013ddf679000000005ce2aff90000000007fdacb0000000005ce2aff90000000007fdacb0
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x00000ca6ccf7a80000000000000790a1000000000000b6ea
trusted.glusterfs.quota.dirty=0x3100
trusted.glusterfs.quota.limit-set.1=0x0000640000000000ffffffffffffffff
trusted.glusterfs.quota.size.1=0x00000ca6ccf7a80000000000000790a1000000000000b6ea
Now, my question is how do I trigger Gluster to recalculate the quota for
this directory? Is it automatic but it takes a while? Because the quota
list did change but not to a good "result".
Path Hard-limit Soft-limit Used
Available Soft-limit exceeded? Hard-limit exceeded?
/projectB 100.0TB 80%(80.0TB) 16383.9PB 190.1TB
No No
I would like to avoid a disable/enable quota in the volume as it removes
the configs.
Thank you for all the help!
*João Baúto*
---------------
*Scientific Computing and Software Platform*
Champalimaud Research
Champalimaud Center for the Unknown
Av. Brasília, Doca de Pedrouços
1400-038 Lisbon, Portugal
fchampalimaud.org <https://www.fchampalimaud.org/>
Srijan Sivakumar <ssivakum at redhat.com> escreveu no dia sábado, 15/08/2020
à(s) 11:57:
> Hi João,
>
> The quota accounting error is what we're looking at here. I think you've
> already looked into the blog post by Hari and are using the script to fix
> the accounting issue.
> That should help you out in fixing this issue.
>
> Let me know if you face any issues while using it.
>
> Regards,
> Srijan Sivakumar
>
>
> On Fri 14 Aug, 2020, 17:10 João Baúto, <joao.bauto at neuro.fchampalimaud.org>
> wrote:
>
>> Hi Strahil,
>>
>> I have tried removing the quota for that specific directory and setting
>> it again but it didn't work (maybe it has to be a quota disable and enable
>> in the volume options). Currently testing a solution
>> by Hari with the quota_fsck.py script (https://medium.com/@harigowtham/
>> glusterfs-quota-fix-accounting-840df33fcd3a) and its detecting a lot of
>> size mismatch in files.
>>
>> Thank you,
>> *João Baúto*
>> ---------------
>>
>> *Scientific Computing and Software Platform*
>> Champalimaud Research
>> Champalimaud Center for the Unknown
>> Av. Brasília, Doca de Pedrouços
>> 1400-038 Lisbon, Portugal
>> fchampalimaud.org <https://www.fchampalimaud.org/>
>>
>>
>> Strahil Nikolov <hunter86_bg at yahoo.com> escreveu no dia sexta,
>> 14/08/2020 à(s) 10:16:
>>
>>> Hi João,
>>>
>>> Based on your output it seems that the quota size is different on the 2
>>> bricks.
>>>
>>> Have you tried to remove the quota and then recreate it ? Maybe it will
>>> be the easiest way to fix it.
>>>
>>> Best Regards,
>>> Strahil Nikolov
>>>
>>>
>>> На 14 август 2020 г. 4:35:14 GMT+03:00, "João Baúto" <
>>> joao.bauto at neuro.fchampalimaud.org> написа:
>>> >Hi all,
>>> >
>>> >We have a 4-node distributed cluster with 2 bricks per node running
>>> >Gluster
>>> >7.7 + ZFS. We use directory quota to limit the space used by our
>>> >members on
>>> >each project. Two days ago we noticed inconsistent space used reported
>>> >by
>>> >Gluster in the quota list.
>>> >
>>> >A small snippet of gluster volume quota vol list,
>>> >
>>> > Path Hard-limit Soft-limit Used
>>> >Available Soft-limit exceeded? Hard-limit exceeded?
>>> >/projectA 5.0TB 80%(4.0TB) 3.1TB 1.9TB
>>> > No No
>>> >*/projectB 100.0TB 80%(80.0TB) 16383.4PB 740.9TB
>>> > No No*
>>> >/projectC 70.0TB 80%(56.0TB) 50.0TB 20.0TB
>>> > No No
>>> >
>>> >The total space available in the cluster is 360TB, the quota for
>>> >projectB
>>> >is 100TB and, as you can see, its reporting 16383.4PB used and 740TB
>>> >available (already decreased from 750TB).
>>> >
>>> >There was an issue in Gluster 3.x related to the wrong directory quota
>>> >(
>>> >
>>> https://lists.gluster.org/pipermail/gluster-users/2016-February/025305.html
>>> > and
>>> >
>>> https://lists.gluster.org/pipermail/gluster-users/2018-November/035374.html
>>> )
>>> >but it's marked as solved (not sure if the solution still applies).
>>> >
>>> >*On projectB*
>>> ># getfattr -d -m . -e hex projectB
>>> ># file: projectB
>>> >trusted.gfid=0x3ca2bce0455945efa6662813ce20fc0c
>>>
>>> >trusted.glusterfs.9582685f-07fa-41fd-b9fc-ebab3a6989cf.xtime=0x5f35e69800098ed9
>>> >trusted.glusterfs.dht=0xe1a4060c000000003ffffffe5ffffffc
>>>
>>> >trusted.glusterfs.mdata=0x010000000000000000000000005f355c59000000000939079f000000005ce2aff90000000007fdacb0000000005ce2aff90000000007fdacb0
>>>
>>> >trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x0000ab0f227a860000000000478e33acffffffffffffc112
>>> >trusted.glusterfs.quota.dirty=0x3000
>>> >trusted.glusterfs.quota.limit-set.1=0x0000640000000000ffffffffffffffff
>>>
>>> >trusted.glusterfs.quota.size.1=0x0000ab0f227a860000000000478e33acffffffffffffc112
>>> >
>>> >*On projectA*
>>> ># getfattr -d -m . -e hex projectA
>>> ># file: projectA
>>> >trusted.gfid=0x05b09ded19354c0eb544d22d4659582e
>>>
>>> >trusted.glusterfs.9582685f-07fa-41fd-b9fc-ebab3a6989cf.xtime=0x5f1aeb9f00044c64
>>> >trusted.glusterfs.dht=0xe1a4060c000000001fffffff3ffffffd
>>>
>>> >trusted.glusterfs.mdata=0x010000000000000000000000005f1ac6a10000000018f30a4e000000005c338fab0000000017a3135a000000005b0694fb000000001584a21b
>>>
>>> >trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x0000067de3bbe20000000000000128610000000000033498
>>> >trusted.glusterfs.quota.dirty=0x3000
>>> >trusted.glusterfs.quota.limit-set.1=0x0000460000000000ffffffffffffffff
>>>
>>> >trusted.glusterfs.quota.size.1=0x0000067de3bbe20000000000000128610000000000033498
>>> >
>>> >Any idea on what's happening and how to fix it?
>>> >
>>> >Thanks!
>>> >*João Baúto*
>>> >---------------
>>> >
>>> >*Scientific Computing and Software Platform*
>>> >Champalimaud Research
>>> >Champalimaud Center for the Unknown
>>> >Av. Brasília, Doca de Pedrouços
>>> >1400-038 Lisbon, Portugal
>>> >fchampalimaud.org <https://www.fchampalimaud.org/>
>>>
>> ________
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968
>>
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200816/8f78ad3a/attachment.html>
More information about the Gluster-users
mailing list