[Gluster-users] Quota list not reflecting disk usage
Steve Dainard
sdainard at spd1.com
Thu Feb 11 19:58:29 UTC 2016
What would happen if I:
- Did not disable quotas
- Did not stop the volume (140T volume takes at least 3-4 days to do
any find operations, which is too much downtime)
- Find and remove all xattrs:
trusted.glusterfs.quota.242dcfd9-6aea-4cb8-beb2-c0ed91ad70d3.contri on
the /brick/volumename/modules
- set the dirty bit on /brick/volumename/modules
As far as an upgrade to 3.7, I'm not comfortable with running the
newest release - which version is RHGS based on? I typically like to
follow supported product version if I can, so I know most of the kinks
are worked out :)
On Wed, Feb 10, 2016 at 11:02 PM, Manikandan Selvaganesh
<mselvaga at redhat.com> wrote:
> Hi Steve,
>
> We suspect the mismatching in accounting is probably because of the
> xattr's being not cleaned up properly. Please ensure you do the following
> steps and make sure the xattr's are cleaned up properly before quota
> is enabled for the next time.
>
> 1) stop the volume
> 2) on each brick in the backend do
> Find and remove all the xattrs and make sure they are not present
> # find <brickpath>/module | xargs getfattr -d -m . -e hex | grep quota | grep -E 'contri|size'
> # setxattr -x xattrname <path>
>
> 3) set dirty on <brickpath>/
> # setxattr -n trusted.glusterfs.quota.dirty -v 0x3100
> By setting dirty value on root as 1(0x3100), the contri will be calculated again
> and the proper contri will be crawled and updated again.
>
> 4) Start volume and from a fuse mount
> # stat /mountpath
>
> If you have ever performed a rename, then there is a possibility of two contributions
> getting created for a single entry.
>
> We have fixed quite some rename issues and have refactored the marker approach. Also
> as I have mentioned already we have also done Versioning of xattr's which solves the
> issue you are facing in 3.7. It would be really helpful in a production environment if
> you could upgrade to 3.7
>
> --
> Thanks & Regards,
> Manikandan Selvaganesh.
>
> ----- Original Message -----
> From: "Steve Dainard" <sdainard at spd1.com>
> To: "Manikandan Selvaganesh" <mselvaga at redhat.com>
> Cc: "Vijaikumar Mallikarjuna" <vmallika at redhat.com>, "Gluster Devel" <gluster-devel at gluster.org>, "gluster-users at gluster.org List" <gluster-users at gluster.org>
> Sent: Thursday, February 11, 2016 1:48:19 AM
> Subject: Re: [Gluster-users] Quota list not reflecting disk usage
>
> So after waiting out the process of disabling quotas, waiting for the
> xattrs to be cleaned up, re-enabling quotas and waiting for the
> xattr's to be created, then applying quotas I'm running into the same
> issue.
>
> Yesterday at ~2pm one of the quotas was listed as:
> /modules|100.0GB|18.3GB|81.7GB
>
> I initiated a copy from that glusterfs fuse mount to another fuse
> mount for a different volume, and now I'm seeing:
> /modules|100.0GB|27.4GB|72.6GB
>
> So an increase of 9GB usage.
>
> There were no writes at all to this directory during or after the cp.
>
> I did a bit of digging through the /modules directory on one of the
> gluster nodes and created this spreadsheet:
> https://docs.google.com/spreadsheets/d/1l_6ze68TCOcx6LEh9MFwmqPZ9bM-70CUlSM_8tpQ654/edit?usp=sharing
>
> The /modules/R/3.2.2 directory quota value doesn't come close to
> matching the du value.
>
> Funny bit, there are TWO quota contribution attributes:
> # getfattr -d -m quota -e hex 3.2.2
> # file: 3.2.2
> trusted.glusterfs.quota.242dcfd9-6aea-4cb8-beb2-c0ed91ad70d3.contri=0x0000000009af6000
> trusted.glusterfs.quota.c890be20-1bb9-4aec-a8d0-eacab0446f16.contri=0x0000000013fda800
> trusted.glusterfs.quota.dirty=0x3000
> trusted.glusterfs.quota.size=0x0000000013fda800
>
> For reference, another directory /modules/R/2.14.2 has only one
> contribution attribute:
> # getfattr -d -m quota -e hex 2.14.2
> # file: 2.14.2
> trusted.glusterfs.quota.c890be20-1bb9-4aec-a8d0-eacab0446f16.contri=0x0000000000692800
> trusted.glusterfs.quota.dirty=0x3000
> trusted.glusterfs.quota.size=0x0000000000692800
>
> Questions:
> 1. Why wasn't the
> trusted.glusterfs.quota.242dcfd9-6aea-4cb8-beb2-c0ed91ad70d3.contri=0x0000000009af6000
> cleaned up?
> 2A. How can I remove old attributes from the fs, and then force a
> re-calculation of contributions for the quota path /modules once I've
> done this on all gluster nodes?
> 2B. Or am I stuck yet again removing quotas completely, waiting for
> the automated setfattr to remove the quotas for
> c890be20-1bb9-4aec-a8d0-eacab0446f16 ID, manually removing attrs for
> 242dcfd9-6aea-4cb8-beb2-c0ed91ad70d3, re-enabling quotas, waiting for
> xattrs to be generated, then enabling limits?
> 3. Shouldn't there be a command to re-trigger quota accounting on a
> directory that confirms the attrs are set correctly and checks that
> the contribution attr actually match disk usage?
>
> On Tue, Feb 2, 2016 at 3:00 AM, Manikandan Selvaganesh
> <mselvaga at redhat.com> wrote:
>> Hi Steve,
>>
>> As you have mentioned, if you are using a glusterfs version lesser than 3.7,
>> then you are doing it right. We are sorry to say but unfortunately that's the only
>> way(manually going and cleaning up the xattr's before enabling quota or wait for
>> the process to complete itself, which would take quite some time depending upon the
>> files) that can be done so as not to mess up quota enforcing/accounting. Also, we could
>> not find anything that could help us with the logs too. Thanks for the
>> point. We are in the process of writing blogs and documenting clearly about quota and
>> it's internal working. There is an initial blog[1] which we have written. More blogs will
>> follow.
>>
>> With glusterfs-3.7, we have introduced something called "Quota versioning".
>> So whenever you enable quota, we are suffixing a number(1..N) with the quota xattr's,
>> say you enable quota for the first time and the xattr will be like,
>> "trusted.glusterfs.quota.size.<suffix number from 1..N>". So all the quota related xattr's
>> will have the number suffixed to the xattr. With the versioning patch[2], when you disable and
>> enable quota again for the next time, it will be "trusted.glusterfs.quota.size.2"(Similarly
>> for other quota related xattr's). So quota accounting can happen independently depending on
>> the suffix and the cleanup process can go on independently which solves the issue that you
>> have.
>>
>> [1] https://manikandanselvaganesh.wordpress.com/
>>
>> [2] http://review.gluster.org/12386
>>
>> --
>> Thanks & Regards,
>> Manikandan Selvaganesh.
>>
>> ----- Original Message -----
>> From: "Vijaikumar Mallikarjuna" <vmallika at redhat.com>
>> To: "Steve Dainard" <sdainard at spd1.com>
>> Cc: "Manikandan Selvaganesh" <mselvaga at redhat.com>
>> Sent: Tuesday, February 2, 2016 10:12:51 AM
>> Subject: Re: [Gluster-users] Quota list not reflecting disk usage
>>
>> Hi Steve,
>>
>> Sorry for the delay. Mani and myself was busy with something else at work,
>> we will update you on this by eod.
>>
>> Many quota issues has been fixed in 3.7, also version numbers are added to
>> quota xattrs, so when quota is disabled we don't need to cleanup the xattrs.
>>
>> Thanks,
>> Vijay
>>
>>
>>
>>
>>
>> On Tue, Feb 2, 2016 at 12:26 AM, Steve Dainard <sdainard at spd1.com> wrote:
>>
>>> I haven't heard anything back on this thread so here's where I've landed:
>>>
>>> It appears that the quota xattr's are not being cleared when quota's
>>> are disabled, so when they are disabled and re-enabled the value for
>>> size is added to the previous size, making it appear that the 'Used'
>>> space is significantly greater than it should be. This seems like a
>>> bug, but I don't know what to file it against, or if the logs I
>>> attached prove this.
>>>
>>> Also; the documentation doesn't make mention of how the quota system
>>> works, and what happens when quotas are enabled/disabled. There seems
>>> to be a background task for both settings:
>>> On enable: "/usr/bin/find . -exec /usr/bin/stat {} \ ;"
>>> On disable: setfattr is removing quota xattrs
>>>
>>> The thing is neither of these tasks are listed in 'gluster volume
>>> status <volume>' ie:
>>>
>>> Status of volume: storage
>>> Gluster process Port Online Pid
>>>
>>> ------------------------------------------------------------------------------
>>> Brick 10.0.231.50:/mnt/raid6-storage/storage 49156 Y 24899
>>> Brick 10.0.231.51:/mnt/raid6-storage/storage 49156 Y 2991
>>> Brick 10.0.231.52:/mnt/raid6-storage/storage 49156 Y 28853
>>> Brick 10.0.231.53:/mnt/raid6-storage/storage 49153 Y 2705
>>> NFS Server on localhost N/A N N/A
>>> Quota Daemon on localhost N/A Y 30066
>>> NFS Server on 10.0.231.52 N/A N N/A
>>> Quota Daemon on 10.0.231.52 N/A Y 24976
>>> NFS Server on 10.0.231.53 N/A N N/A
>>> Quota Daemon on 10.0.231.53 N/A Y 30334
>>> NFS Server on 10.0.231.51 N/A N N/A
>>> Quota Daemon on 10.0.231.51 N/A Y 15781
>>>
>>> Task Status of Volume storage
>>>
>>> ------------------------------------------------------------------------------
>>> ******There are no active volume tasks*******
>>>
>>> (I added the asterisks above)
>>> So without any visibility into these running tasks, or knowing of
>>> their existence (not documented) it becomes very difficult to know
>>> what's going on. On any reasonably large storage system these tasks
>>> take days to complete and there should be some indication of this.
>>>
>>> Where I'm at right now:
>>> - I disabled the quota's on volume 'storage'
>>> - I started to manually remove xattrs until I realized there is an
>>> automated task to do this.
>>> - After waiting for 'ps aux | grep setfattr' to return nothing, I
>>> re-enabled quotas
>>> - I'm currently waiting for the stat tasks to complete
>>> - Once the entire filesystem has been stat'ed, I'm going to set limits
>>> again.
>>>
>>> As a note, this is a pretty brutal process on a system with 140T of
>>> storage, and I can't imagine how much worse this would be if my nodes
>>> had more than 12 disks per, or if I was at PB scale.
>>>
>>> On Mon, Jan 25, 2016 at 12:31 PM, Steve Dainard <sdainard at spd1.com> wrote:
>>> > Here's a l link to a tarball of one of the gluster hosts logs:
>>> > https://dl.dropboxusercontent.com/u/21916057/gluster01.tar.gz
>>> >
>>> > I wanted to include past logs in case they were useful.
>>> >
>>> > Also, the volume I'm trying to get quota's working on is 'storage'
>>> > you'll notice I have a brick issue on a different volume 'vm-storage'.
>>> >
>>> > In regards to the 3.7 upgrade. I'm a bit hesitant to move to the
>>> > current release, I prefer to stay on a stable release with maintenance
>>> > updates if possible.
>>> >
>>> > On Mon, Jan 25, 2016 at 12:09 PM, Manikandan Selvaganesh
>>> > <mselvaga at redhat.com> wrote:
>>> >> Hi Steve,
>>> >>
>>> >> Also, do you have any plans to upgrade to the latest version. With 3.7,
>>> >> we have re factored some approaches used in quota and marker and that
>>> have
>>> >> fixed quite some issues.
>>> >>
>>> >> --
>>> >> Thanks & Regards,
>>> >> Manikandan Selvaganesh.
>>> >>
>>> >> ----- Original Message -----
>>> >> From: "Manikandan Selvaganesh" <mselvaga at redhat.com>
>>> >> To: "Steve Dainard" <sdainard at spd1.com>
>>> >> Cc: "gluster-users at gluster.org List" <gluster-users at gluster.org>
>>> >> Sent: Tuesday, January 26, 2016 1:31:10 AM
>>> >> Subject: Re: [Gluster-users] Quota list not reflecting disk usage
>>> >>
>>> >> Hi Steve,
>>> >>
>>> >> Could you send us the glusterfs logs, it could help us debug the issue!!
>>> >>
>>> >> --
>>> >> Thanks & Regards,
>>> >> Manikandan Selvaganesh.
>>> >>
>>> >> ----- Original Message -----
>>> >> From: "Steve Dainard" <sdainard at spd1.com>
>>> >> To: "Manikandan Selvaganesh" <mselvaga at redhat.com>
>>> >> Cc: "gluster-users at gluster.org List" <gluster-users at gluster.org>
>>> >> Sent: Tuesday, January 26, 2016 12:56:22 AM
>>> >> Subject: Re: [Gluster-users] Quota list not reflecting disk usage
>>> >>
>>> >> Something is seriously wrong with the quota output:
>>> >>
>>> >> # gluster volume quota storage list
>>> >> Path Hard-limit Soft-limit Used
>>> >> Available Soft-limit exceeded? Hard-limit exceeded?
>>> >>
>>> ---------------------------------------------------------------------------------------------------------------------------
>>> >> /projects-CanSISE 10.0TB 80% 27.8TB
>>> >> 0Bytes Yes Yes
>>> >> /data4/climate 105.0TB 80% 307.1TB
>>> >> 0Bytes Yes Yes
>>> >> /data4/forestry 50.0GB 80% 61.9GB
>>> >> 0Bytes Yes Yes
>>> >> /data4/projects 800.0GB 80% 2.0TB
>>> >> 0Bytes Yes Yes
>>> >> /data4/strays 85.0GB 80% 230.5GB
>>> >> 0Bytes Yes Yes
>>> >> /data4/gis 2.2TB 80% 6.3TB
>>> >> 0Bytes Yes Yes
>>> >> /data4/modperl 1.0TB 80% 953.2GB
>>> >> 70.8GB Yes No
>>> >> /data4/dem 1.0GB 80% 0Bytes
>>> >> 1.0GB No No
>>> >> /projects-hydrology-archive0 5.0TB 80% 14.4TB
>>> >> 0Bytes Yes Yes
>>> >> /climate-downscale-idf-ec 7.5TB 80% 5.1TB
>>> >> 2.4TB No No
>>> >> /climate-downscale-idf 5.0TB 80% 6.1TB
>>> >> 0Bytes Yes Yes
>>> >> /home 5.0TB 80% 11.8TB
>>> >> 0Bytes Yes Yes
>>> >> /projects-hydrology-scratch0 7.0TB 80% 169.1GB
>>> >> 6.8TB No No
>>> >> /projects-rci-scratch 10.0TB 80% 1.9TB
>>> >> 8.1TB No No
>>> >> /projects-dataportal 1.0TB 80% 775.4GB
>>> >> 248.6GB No No
>>> >> /modules 1.0TB 80% 36.1GB
>>> >> 987.9GB No No
>>> >> /data4/climate/downscale/CMIP5 65.0TB 80% 56.4TB
>>> >> 8.6TB Yes No
>>> >>
>>> >> Gluster is listing 'Used' space of over 307TB on /data4/climate, but
>>> >> the volume capacity is only 146T.
>>> >>
>>> >> This has happened after disabling quotas on the volume, re-enabling
>>> >> quotas, and then setting quotas again. There was a lot of glusterfsd
>>> >> CPU usage afterwards, and now 3 days later the quota's I set were all
>>> >> missing except
>>> >>
>>> >> /data4/projects|800.0GB|2.0TB|0Bytes
>>> >>
>>> >> So I re-set the quotas and the output above is what I have.
>>> >>
>>> >> Previous to disabling quota's this was the output:
>>> >> # gluster volume quota storage list
>>> >> Path Hard-limit Soft-limit Used
>>> >> Available Soft-limit exceeded? Hard-limit exceeded?
>>> >>
>>> ---------------------------------------------------------------------------------------------------------------------------
>>> >> /data4/climate 105.0TB 80% 151.6TB
>>> >> 0Bytes Yes Yes
>>> >> /data4/forestry 50.0GB 80% 45.4GB
>>> >> 4.6GB Yes No
>>> >> /data4/projects 800.0GB 80% 753.1GB
>>> >> 46.9GB Yes No
>>> >> /data4/strays 85.0GB 80% 80.8GB
>>> >> 4.2GB Yes No
>>> >> /data4/gis 2.2TB 80% 2.1TB
>>> >> 91.8GB Yes No
>>> >> /data4/modperl 1.0TB 80% 948.1GB
>>> >> 75.9GB Yes No
>>> >> /data4/dem 1.0GB 80% 0Bytes
>>> >> 1.0GB No No
>>> >> /projects-CanSISE 10.0TB 80% 11.9TB
>>> >> 0Bytes Yes Yes
>>> >> /projects-hydrology-archive0 5.0TB 80% 4.8TB
>>> >> 174.0GB Yes No
>>> >> /climate-downscale-idf-ec 7.5TB 80% 5.0TB
>>> >> 2.5TB No No
>>> >> /climate-downscale-idf 5.0TB 80% 3.8TB
>>> >> 1.2TB No No
>>> >> /home 5.0TB 80% 4.7TB
>>> >> 283.8GB Yes No
>>> >> /projects-hydrology-scratch0 7.0TB 80% 95.9GB
>>> >> 6.9TB No No
>>> >> /projects-rci-scratch 10.0TB 80% 1.7TB
>>> >> 8.3TB No No
>>> >> /projects-dataportal 1.0TB 80% 775.4GB
>>> >> 248.6GB No No
>>> >> /modules 1.0TB 80% 14.6GB
>>> >> 1009.4GB No No
>>> >> /data4/climate/downscale/CMIP5 65.0TB 80% 56.4TB
>>> >> 8.6TB Yes No
>>> >>
>>> >> I was so focused on the /projects-CanSISE quota not being accurate
>>> >> that I missed that the 'Used' space on /data4/climate is listed higher
>>> >> then the total gluster volume capacity.
>>> >>
>>> >> On Mon, Jan 25, 2016 at 10:52 AM, Steve Dainard <sdainard at spd1.com>
>>> wrote:
>>> >>> Hi Manikandan
>>> >>>
>>> >>> I'm using 'du' not df in this case.
>>> >>>
>>> >>> On Thu, Jan 21, 2016 at 9:20 PM, Manikandan Selvaganesh
>>> >>> <mselvaga at redhat.com> wrote:
>>> >>>> Hi Steve,
>>> >>>>
>>> >>>> If you would like disk usage using df utility by taking quota limits
>>> into
>>> >>>> consideration, then you are expected to run the following command.
>>> >>>>
>>> >>>> 'gluster volume set VOLNAME quota-deem-statfs on'
>>> >>>>
>>> >>>> with older versions where quota-deem-statfs is OFF by default.
>>> However with
>>> >>>> the latest versions, quota-deem-statfs is by default ON. In this
>>> case, the total
>>> >>>> disk space of the directory is taken as the quota hard limit set on
>>> the directory
>>> >>>> of the volume and disk utility would display accordingly. This
>>> answers why there is
>>> >>>> a mismatch in disk utility.
>>> >>>>
>>> >>>> Next, answering to quota mechanism and accuracy: There is something
>>> called timeouts
>>> >>>> in quota. For performance reasons, quota caches the directory size on
>>> client. You can
>>> >>>> set timeout indicating the maximum valid duration of directory sizes
>>> in cache,
>>> >>>> from the time they are populated. By default the hard-timeout is 5s
>>> and soft timeout
>>> >>>> is 60s. Setting a timeout of zero will do a force fetching of
>>> directory sizes from server
>>> >>>> for every operation that modifies file data and will effectively
>>> disables directory size
>>> >>>> caching on client side. If you do not have a timeout of 0(which we do
>>> not encourage due to
>>> >>>> performance reasons), then till you reach soft-limit, soft timeout
>>> will be taken into
>>> >>>> consideration, and only for every 60s operations will be synced and
>>> that could cause the
>>> >>>> usage to exceed more than the hard-limit specified. If you would like
>>> quota to
>>> >>>> strictly enforce then please run the following commands,
>>> >>>>
>>> >>>> 'gluster v quota VOLNAME hard-timeout 0s'
>>> >>>> 'gluster v quota VOLNAME soft-timeout 0s'
>>> >>>>
>>> >>>> Appreciate your curiosity in exploring and if you would like to know
>>> more about quota
>>> >>>> please refer[1]
>>> >>>>
>>> >>>> [1]
>>> http://gluster.readthedocs.org/en/release-3.7.0-1/Administrator%20Guide/Directory%20Quota/
>>> >>>>
>>> >>>> --
>>> >>>> Thanks & Regards,
>>> >>>> Manikandan Selvaganesh.
>>> >>>>
>>> >>>> ----- Original Message -----
>>> >>>> From: "Steve Dainard" <sdainard at spd1.com>
>>> >>>> To: "gluster-users at gluster.org List" <gluster-users at gluster.org>
>>> >>>> Sent: Friday, January 22, 2016 1:40:07 AM
>>> >>>> Subject: Re: [Gluster-users] Quota list not reflecting disk usage
>>> >>>>
>>> >>>> This is gluster 3.6.6.
>>> >>>>
>>> >>>> I've attempted to disable and re-enable quota's on the volume, but
>>> >>>> when I re-apply the quotas on each directory the same 'Used' value is
>>> >>>> present as before.
>>> >>>>
>>> >>>> Where is quotad getting its information from, and how can I clean
>>> >>>> up/regenerate that info?
>>> >>>>
>>> >>>> On Thu, Jan 21, 2016 at 10:07 AM, Steve Dainard <sdainard at spd1.com>
>>> wrote:
>>> >>>>> I have a distributed volume with quota's enabled:
>>> >>>>>
>>> >>>>> Volume Name: storage
>>> >>>>> Type: Distribute
>>> >>>>> Volume ID: 26d355cb-c486-481f-ac16-e25390e73775
>>> >>>>> Status: Started
>>> >>>>> Number of Bricks: 4
>>> >>>>> Transport-type: tcp
>>> >>>>> Bricks:
>>> >>>>> Brick1: 10.0.231.50:/mnt/raid6-storage/storage
>>> >>>>> Brick2: 10.0.231.51:/mnt/raid6-storage/storage
>>> >>>>> Brick3: 10.0.231.52:/mnt/raid6-storage/storage
>>> >>>>> Brick4: 10.0.231.53:/mnt/raid6-storage/storage
>>> >>>>> Options Reconfigured:
>>> >>>>> performance.cache-size: 1GB
>>> >>>>> performance.readdir-ahead: on
>>> >>>>> features.quota: on
>>> >>>>> diagnostics.brick-log-level: WARNING
>>> >>>>>
>>> >>>>> Here is a partial list of quotas:
>>> >>>>> # /usr/sbin/gluster volume quota storage list
>>> >>>>> Path Hard-limit Soft-limit Used
>>> >>>>> Available Soft-limit exceeded? Hard-limit exceeded?
>>> >>>>>
>>> ---------------------------------------------------------------------------------------------------------------------------
>>> >>>>> ...
>>> >>>>> /projects-CanSISE 10.0TB 80%
>>> 11.9TB
>>> >>>>> 0Bytes Yes Yes
>>> >>>>> ...
>>> >>>>>
>>> >>>>> If I du on that location I do not get 11.9TB of space used (fuse
>>> mount point):
>>> >>>>> [root at storage projects-CanSISE]# du -hs
>>> >>>>> 9.5T .
>>> >>>>>
>>> >>>>> Can someone provide an explanation for how the quota mechanism tracks
>>> >>>>> disk usage? How often does the quota mechanism check its accuracy?
>>> And
>>> >>>>> how could it get so far off?
>>> >>>>>
>>> >>>>> Can I get gluster to rescan that location and update the quota usage?
>>> >>>>>
>>> >>>>> Thanks,
>>> >>>>> Steve
>>> >>>> _______________________________________________
>>> >>>> Gluster-users mailing list
>>> >>>> Gluster-users at gluster.org
>>> >>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>> >> _______________________________________________
>>> >> Gluster-users mailing list
>>> >> Gluster-users at gluster.org
>>> >> http://www.gluster.org/mailman/listinfo/gluster-users
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>
More information about the Gluster-users
mailing list