[Gluster-devel] [Gluster-users] Quota list not reflecting disk usage
Vijaikumar Mallikarjuna
vmallika at redhat.com
Fri Feb 12 06:08:48 UTC 2016
Hi Steve,
Here is how quota usage accounting works
For each file, below extended attributes are set:
trusted.glusterfs.quota......contri -> This value tells how much size this
file/dir has contributed to its parent (key will have a gfid of parent)
For each directory, below extended attributes are set:
trusted.glusterfs.quota......contri (not on root, as root doesn't have
parent)
trusted.glusterfs.quota.dirty -> this attribute is used for recovery when
brick crashes during metadata update
trusted.glusterfs.quota.size -> it is the total size of all the files,
directories and sub-directories till the leaf node.
When a file "/a/b/f1" is changed, then the change in the size needs to be
updated in the extended attributes of file 'f1' and crawl upwards till the
root
and update the extended attributes of the ancestors of file 'f1'. This is
done independently on each brick.
Here is the pseudo code
1) Begin metadata update file update '/a/b/f1'
2) inode = f1
3) parent_inode = b
4) if parent_inode is root goto end
5) take lock on parent_inode
6) get new size of inode
if inode is a file get size from statbuf
if inode is a dir get size from extended attribute
7) get contribution value from inode
8) find delta value
delta = size - contri
9) if delta is zero, no update goto end
10) set dirty flag on parent_inode
11) add delta to the inode contri
12) add delta to size attribute of parent_inode
13) clear dirty flag on parent_inode
14) release lock on parent_inode
15) inode = parent_inode
16) parent_inode = parent (inode)
17) goto step 4
18) End
As mentioned above, if there is a change in any file we get the old
metadata value and add the delta to this
value. So when quota is disable and some stale xattrs are leftover, this
value will be used when adding the delta.
Quota marker cannot identify if the xattr leftover is a newly created
attribute or a stale attribute.
This problem is now solved in 3.7 by using a version number as part of the
xattr key. This version number is
incremented every-time quota is disabled and enabled and even if old
entries are not cleaned, looking at the
version number quota marker identifies that as stale entry and creates a
new xattrs with the current version number
Step 11 & 12 should be atomic, hence we use dirty flag. In case if there is
a crash during this step.
When the brick is back online and during a lookup on a directory if dound
that dirty flag is set
below operation is performed
1) If dirty flag set on a 'inode'
a) readdir on inode
b) get sum of contri attribute of all file/dir entries of inode
c) update size attribute of 'inode'
d) clear dirty flag
In our previous email, we have provided a workaround to just fix a selected
directory from the back-end.
If volume if not stopped, make sure that no IO happens. Because when you
are manually updating the
xattrs in the back-end, and if IO is happening brick process can also
update the xattrs and again you end-up with
inconsistent accounting.
Thanks,
Vijay
On Fri, Feb 12, 2016 at 1:28 AM, Steve Dainard <sdainard at spd1.com> wrote:
> What would happen if I:
> - Did not disable quotas
> - Did not stop the volume (140T volume takes at least 3-4 days to do
> any find operations, which is too much downtime)
> - Find and remove all xattrs:
> trusted.glusterfs.quota.242dcfd9-6aea-4cb8-beb2-c0ed91ad70d3.contri on
> the /brick/volumename/modules
> - set the dirty bit on /brick/volumename/modules
>
> As far as an upgrade to 3.7, I'm not comfortable with running the
> newest release - which version is RHGS based on? I typically like to
> follow supported product version if I can, so I know most of the kinks
> are worked out :)
>
> On Wed, Feb 10, 2016 at 11:02 PM, Manikandan Selvaganesh
> <mselvaga at redhat.com> wrote:
> > Hi Steve,
> >
> > We suspect the mismatching in accounting is probably because of the
> > xattr's being not cleaned up properly. Please ensure you do the following
> > steps and make sure the xattr's are cleaned up properly before quota
> > is enabled for the next time.
> >
> > 1) stop the volume
> > 2) on each brick in the backend do
> > Find and remove all the xattrs and make sure they are not present
> > # find <brickpath>/module | xargs getfattr -d -m . -e hex | grep
> quota | grep -E 'contri|size'
> > # setxattr -x xattrname <path>
> >
> > 3) set dirty on <brickpath>/
> > # setxattr -n trusted.glusterfs.quota.dirty -v 0x3100
> > By setting dirty value on root as 1(0x3100), the contri will be
> calculated again
> > and the proper contri will be crawled and updated again.
> >
> > 4) Start volume and from a fuse mount
> > # stat /mountpath
> >
> > If you have ever performed a rename, then there is a possibility of two
> contributions
> > getting created for a single entry.
> >
> > We have fixed quite some rename issues and have refactored the marker
> approach. Also
> > as I have mentioned already we have also done Versioning of xattr's
> which solves the
> > issue you are facing in 3.7. It would be really helpful in a production
> environment if
> > you could upgrade to 3.7
> >
> > --
> > Thanks & Regards,
> > Manikandan Selvaganesh.
> >
> > ----- Original Message -----
> > From: "Steve Dainard" <sdainard at spd1.com>
> > To: "Manikandan Selvaganesh" <mselvaga at redhat.com>
> > Cc: "Vijaikumar Mallikarjuna" <vmallika at redhat.com>, "Gluster Devel" <
> gluster-devel at gluster.org>, "gluster-users at gluster.org List" <
> gluster-users at gluster.org>
> > Sent: Thursday, February 11, 2016 1:48:19 AM
> > Subject: Re: [Gluster-users] Quota list not reflecting disk usage
> >
> > So after waiting out the process of disabling quotas, waiting for the
> > xattrs to be cleaned up, re-enabling quotas and waiting for the
> > xattr's to be created, then applying quotas I'm running into the same
> > issue.
> >
> > Yesterday at ~2pm one of the quotas was listed as:
> > /modules|100.0GB|18.3GB|81.7GB
> >
> > I initiated a copy from that glusterfs fuse mount to another fuse
> > mount for a different volume, and now I'm seeing:
> > /modules|100.0GB|27.4GB|72.6GB
> >
> > So an increase of 9GB usage.
> >
> > There were no writes at all to this directory during or after the cp.
> >
> > I did a bit of digging through the /modules directory on one of the
> > gluster nodes and created this spreadsheet:
> >
> https://docs.google.com/spreadsheets/d/1l_6ze68TCOcx6LEh9MFwmqPZ9bM-70CUlSM_8tpQ654/edit?usp=sharing
> >
> > The /modules/R/3.2.2 directory quota value doesn't come close to
> > matching the du value.
> >
> > Funny bit, there are TWO quota contribution attributes:
> > # getfattr -d -m quota -e hex 3.2.2
> > # file: 3.2.2
> >
> trusted.glusterfs.quota.242dcfd9-6aea-4cb8-beb2-c0ed91ad70d3.contri=0x0000000009af6000
> >
> trusted.glusterfs.quota.c890be20-1bb9-4aec-a8d0-eacab0446f16.contri=0x0000000013fda800
> > trusted.glusterfs.quota.dirty=0x3000
> > trusted.glusterfs.quota.size=0x0000000013fda800
> >
> > For reference, another directory /modules/R/2.14.2 has only one
> > contribution attribute:
> > # getfattr -d -m quota -e hex 2.14.2
> > # file: 2.14.2
> >
> trusted.glusterfs.quota.c890be20-1bb9-4aec-a8d0-eacab0446f16.contri=0x0000000000692800
> > trusted.glusterfs.quota.dirty=0x3000
> > trusted.glusterfs.quota.size=0x0000000000692800
> >
> > Questions:
> > 1. Why wasn't the
> >
> trusted.glusterfs.quota.242dcfd9-6aea-4cb8-beb2-c0ed91ad70d3.contri=0x0000000009af6000
> > cleaned up?
> > 2A. How can I remove old attributes from the fs, and then force a
> > re-calculation of contributions for the quota path /modules once I've
> > done this on all gluster nodes?
> > 2B. Or am I stuck yet again removing quotas completely, waiting for
> > the automated setfattr to remove the quotas for
> > c890be20-1bb9-4aec-a8d0-eacab0446f16 ID, manually removing attrs for
> > 242dcfd9-6aea-4cb8-beb2-c0ed91ad70d3, re-enabling quotas, waiting for
> > xattrs to be generated, then enabling limits?
> > 3. Shouldn't there be a command to re-trigger quota accounting on a
> > directory that confirms the attrs are set correctly and checks that
> > the contribution attr actually match disk usage?
> >
> > On Tue, Feb 2, 2016 at 3:00 AM, Manikandan Selvaganesh
> > <mselvaga at redhat.com> wrote:
> >> Hi Steve,
> >>
> >> As you have mentioned, if you are using a glusterfs version lesser than
> 3.7,
> >> then you are doing it right. We are sorry to say but unfortunately
> that's the only
> >> way(manually going and cleaning up the xattr's before enabling quota or
> wait for
> >> the process to complete itself, which would take quite some time
> depending upon the
> >> files) that can be done so as not to mess up quota
> enforcing/accounting. Also, we could
> >> not find anything that could help us with the logs too. Thanks for the
> >> point. We are in the process of writing blogs and documenting clearly
> about quota and
> >> it's internal working. There is an initial blog[1] which we have
> written. More blogs will
> >> follow.
> >>
> >> With glusterfs-3.7, we have introduced something called "Quota
> versioning".
> >> So whenever you enable quota, we are suffixing a number(1..N) with the
> quota xattr's,
> >> say you enable quota for the first time and the xattr will be like,
> >> "trusted.glusterfs.quota.size.<suffix number from 1..N>". So all the
> quota related xattr's
> >> will have the number suffixed to the xattr. With the versioning
> patch[2], when you disable and
> >> enable quota again for the next time, it will be
> "trusted.glusterfs.quota.size.2"(Similarly
> >> for other quota related xattr's). So quota accounting can happen
> independently depending on
> >> the suffix and the cleanup process can go on independently which solves
> the issue that you
> >> have.
> >>
> >> [1] https://manikandanselvaganesh.wordpress.com/
> >>
> >> [2] http://review.gluster.org/12386
> >>
> >> --
> >> Thanks & Regards,
> >> Manikandan Selvaganesh.
> >>
> >> ----- Original Message -----
> >> From: "Vijaikumar Mallikarjuna" <vmallika at redhat.com>
> >> To: "Steve Dainard" <sdainard at spd1.com>
> >> Cc: "Manikandan Selvaganesh" <mselvaga at redhat.com>
> >> Sent: Tuesday, February 2, 2016 10:12:51 AM
> >> Subject: Re: [Gluster-users] Quota list not reflecting disk usage
> >>
> >> Hi Steve,
> >>
> >> Sorry for the delay. Mani and myself was busy with something else at
> work,
> >> we will update you on this by eod.
> >>
> >> Many quota issues has been fixed in 3.7, also version numbers are added
> to
> >> quota xattrs, so when quota is disabled we don't need to cleanup the
> xattrs.
> >>
> >> Thanks,
> >> Vijay
> >>
> >>
> >>
> >>
> >>
> >> On Tue, Feb 2, 2016 at 12:26 AM, Steve Dainard <sdainard at spd1.com>
> wrote:
> >>
> >>> I haven't heard anything back on this thread so here's where I've
> landed:
> >>>
> >>> It appears that the quota xattr's are not being cleared when quota's
> >>> are disabled, so when they are disabled and re-enabled the value for
> >>> size is added to the previous size, making it appear that the 'Used'
> >>> space is significantly greater than it should be. This seems like a
> >>> bug, but I don't know what to file it against, or if the logs I
> >>> attached prove this.
> >>>
> >>> Also; the documentation doesn't make mention of how the quota system
> >>> works, and what happens when quotas are enabled/disabled. There seems
> >>> to be a background task for both settings:
> >>> On enable: "/usr/bin/find . -exec /usr/bin/stat {} \ ;"
> >>> On disable: setfattr is removing quota xattrs
> >>>
> >>> The thing is neither of these tasks are listed in 'gluster volume
> >>> status <volume>' ie:
> >>>
> >>> Status of volume: storage
> >>> Gluster process Port Online Pid
> >>>
> >>>
> ------------------------------------------------------------------------------
> >>> Brick 10.0.231.50:/mnt/raid6-storage/storage 49156 Y 24899
> >>> Brick 10.0.231.51:/mnt/raid6-storage/storage 49156 Y 2991
> >>> Brick 10.0.231.52:/mnt/raid6-storage/storage 49156 Y 28853
> >>> Brick 10.0.231.53:/mnt/raid6-storage/storage 49153 Y 2705
> >>> NFS Server on localhost N/A N N/A
> >>> Quota Daemon on localhost N/A Y 30066
> >>> NFS Server on 10.0.231.52 N/A N N/A
> >>> Quota Daemon on 10.0.231.52 N/A Y 24976
> >>> NFS Server on 10.0.231.53 N/A N N/A
> >>> Quota Daemon on 10.0.231.53 N/A Y 30334
> >>> NFS Server on 10.0.231.51 N/A N N/A
> >>> Quota Daemon on 10.0.231.51 N/A Y 15781
> >>>
> >>> Task Status of Volume storage
> >>>
> >>>
> ------------------------------------------------------------------------------
> >>> ******There are no active volume tasks*******
> >>>
> >>> (I added the asterisks above)
> >>> So without any visibility into these running tasks, or knowing of
> >>> their existence (not documented) it becomes very difficult to know
> >>> what's going on. On any reasonably large storage system these tasks
> >>> take days to complete and there should be some indication of this.
> >>>
> >>> Where I'm at right now:
> >>> - I disabled the quota's on volume 'storage'
> >>> - I started to manually remove xattrs until I realized there is an
> >>> automated task to do this.
> >>> - After waiting for 'ps aux | grep setfattr' to return nothing, I
> >>> re-enabled quotas
> >>> - I'm currently waiting for the stat tasks to complete
> >>> - Once the entire filesystem has been stat'ed, I'm going to set limits
> >>> again.
> >>>
> >>> As a note, this is a pretty brutal process on a system with 140T of
> >>> storage, and I can't imagine how much worse this would be if my nodes
> >>> had more than 12 disks per, or if I was at PB scale.
> >>>
> >>> On Mon, Jan 25, 2016 at 12:31 PM, Steve Dainard <sdainard at spd1.com>
> wrote:
> >>> > Here's a l link to a tarball of one of the gluster hosts logs:
> >>> > https://dl.dropboxusercontent.com/u/21916057/gluster01.tar.gz
> >>> >
> >>> > I wanted to include past logs in case they were useful.
> >>> >
> >>> > Also, the volume I'm trying to get quota's working on is 'storage'
> >>> > you'll notice I have a brick issue on a different volume
> 'vm-storage'.
> >>> >
> >>> > In regards to the 3.7 upgrade. I'm a bit hesitant to move to the
> >>> > current release, I prefer to stay on a stable release with
> maintenance
> >>> > updates if possible.
> >>> >
> >>> > On Mon, Jan 25, 2016 at 12:09 PM, Manikandan Selvaganesh
> >>> > <mselvaga at redhat.com> wrote:
> >>> >> Hi Steve,
> >>> >>
> >>> >> Also, do you have any plans to upgrade to the latest version. With
> 3.7,
> >>> >> we have re factored some approaches used in quota and marker and
> that
> >>> have
> >>> >> fixed quite some issues.
> >>> >>
> >>> >> --
> >>> >> Thanks & Regards,
> >>> >> Manikandan Selvaganesh.
> >>> >>
> >>> >> ----- Original Message -----
> >>> >> From: "Manikandan Selvaganesh" <mselvaga at redhat.com>
> >>> >> To: "Steve Dainard" <sdainard at spd1.com>
> >>> >> Cc: "gluster-users at gluster.org List" <gluster-users at gluster.org>
> >>> >> Sent: Tuesday, January 26, 2016 1:31:10 AM
> >>> >> Subject: Re: [Gluster-users] Quota list not reflecting disk usage
> >>> >>
> >>> >> Hi Steve,
> >>> >>
> >>> >> Could you send us the glusterfs logs, it could help us debug the
> issue!!
> >>> >>
> >>> >> --
> >>> >> Thanks & Regards,
> >>> >> Manikandan Selvaganesh.
> >>> >>
> >>> >> ----- Original Message -----
> >>> >> From: "Steve Dainard" <sdainard at spd1.com>
> >>> >> To: "Manikandan Selvaganesh" <mselvaga at redhat.com>
> >>> >> Cc: "gluster-users at gluster.org List" <gluster-users at gluster.org>
> >>> >> Sent: Tuesday, January 26, 2016 12:56:22 AM
> >>> >> Subject: Re: [Gluster-users] Quota list not reflecting disk usage
> >>> >>
> >>> >> Something is seriously wrong with the quota output:
> >>> >>
> >>> >> # gluster volume quota storage list
> >>> >> Path Hard-limit Soft-limit
> Used
> >>> >> Available Soft-limit exceeded? Hard-limit exceeded?
> >>> >>
> >>>
> ---------------------------------------------------------------------------------------------------------------------------
> >>> >> /projects-CanSISE 10.0TB 80%
> 27.8TB
> >>> >> 0Bytes Yes Yes
> >>> >> /data4/climate 105.0TB 80%
> 307.1TB
> >>> >> 0Bytes Yes Yes
> >>> >> /data4/forestry 50.0GB 80%
> 61.9GB
> >>> >> 0Bytes Yes Yes
> >>> >> /data4/projects 800.0GB 80%
> 2.0TB
> >>> >> 0Bytes Yes Yes
> >>> >> /data4/strays 85.0GB 80%
> 230.5GB
> >>> >> 0Bytes Yes Yes
> >>> >> /data4/gis 2.2TB 80%
> 6.3TB
> >>> >> 0Bytes Yes Yes
> >>> >> /data4/modperl 1.0TB 80%
> 953.2GB
> >>> >> 70.8GB Yes No
> >>> >> /data4/dem 1.0GB 80%
> 0Bytes
> >>> >> 1.0GB No No
> >>> >> /projects-hydrology-archive0 5.0TB 80%
> 14.4TB
> >>> >> 0Bytes Yes Yes
> >>> >> /climate-downscale-idf-ec 7.5TB 80%
> 5.1TB
> >>> >> 2.4TB No No
> >>> >> /climate-downscale-idf 5.0TB 80%
> 6.1TB
> >>> >> 0Bytes Yes Yes
> >>> >> /home 5.0TB 80%
> 11.8TB
> >>> >> 0Bytes Yes Yes
> >>> >> /projects-hydrology-scratch0 7.0TB 80%
> 169.1GB
> >>> >> 6.8TB No No
> >>> >> /projects-rci-scratch 10.0TB 80%
> 1.9TB
> >>> >> 8.1TB No No
> >>> >> /projects-dataportal 1.0TB 80%
> 775.4GB
> >>> >> 248.6GB No No
> >>> >> /modules 1.0TB 80%
> 36.1GB
> >>> >> 987.9GB No No
> >>> >> /data4/climate/downscale/CMIP5 65.0TB 80%
> 56.4TB
> >>> >> 8.6TB Yes No
> >>> >>
> >>> >> Gluster is listing 'Used' space of over 307TB on /data4/climate, but
> >>> >> the volume capacity is only 146T.
> >>> >>
> >>> >> This has happened after disabling quotas on the volume, re-enabling
> >>> >> quotas, and then setting quotas again. There was a lot of glusterfsd
> >>> >> CPU usage afterwards, and now 3 days later the quota's I set were
> all
> >>> >> missing except
> >>> >>
> >>> >> /data4/projects|800.0GB|2.0TB|0Bytes
> >>> >>
> >>> >> So I re-set the quotas and the output above is what I have.
> >>> >>
> >>> >> Previous to disabling quota's this was the output:
> >>> >> # gluster volume quota storage list
> >>> >> Path Hard-limit Soft-limit
> Used
> >>> >> Available Soft-limit exceeded? Hard-limit exceeded?
> >>> >>
> >>>
> ---------------------------------------------------------------------------------------------------------------------------
> >>> >> /data4/climate 105.0TB 80%
> 151.6TB
> >>> >> 0Bytes Yes Yes
> >>> >> /data4/forestry 50.0GB 80%
> 45.4GB
> >>> >> 4.6GB Yes No
> >>> >> /data4/projects 800.0GB 80%
> 753.1GB
> >>> >> 46.9GB Yes No
> >>> >> /data4/strays 85.0GB 80%
> 80.8GB
> >>> >> 4.2GB Yes No
> >>> >> /data4/gis 2.2TB 80%
> 2.1TB
> >>> >> 91.8GB Yes No
> >>> >> /data4/modperl 1.0TB 80%
> 948.1GB
> >>> >> 75.9GB Yes No
> >>> >> /data4/dem 1.0GB 80%
> 0Bytes
> >>> >> 1.0GB No No
> >>> >> /projects-CanSISE 10.0TB 80%
> 11.9TB
> >>> >> 0Bytes Yes Yes
> >>> >> /projects-hydrology-archive0 5.0TB 80%
> 4.8TB
> >>> >> 174.0GB Yes No
> >>> >> /climate-downscale-idf-ec 7.5TB 80%
> 5.0TB
> >>> >> 2.5TB No No
> >>> >> /climate-downscale-idf 5.0TB 80%
> 3.8TB
> >>> >> 1.2TB No No
> >>> >> /home 5.0TB 80%
> 4.7TB
> >>> >> 283.8GB Yes No
> >>> >> /projects-hydrology-scratch0 7.0TB 80%
> 95.9GB
> >>> >> 6.9TB No No
> >>> >> /projects-rci-scratch 10.0TB 80%
> 1.7TB
> >>> >> 8.3TB No No
> >>> >> /projects-dataportal 1.0TB 80%
> 775.4GB
> >>> >> 248.6GB No No
> >>> >> /modules 1.0TB 80%
> 14.6GB
> >>> >> 1009.4GB No No
> >>> >> /data4/climate/downscale/CMIP5 65.0TB 80%
> 56.4TB
> >>> >> 8.6TB Yes No
> >>> >>
> >>> >> I was so focused on the /projects-CanSISE quota not being accurate
> >>> >> that I missed that the 'Used' space on /data4/climate is listed
> higher
> >>> >> then the total gluster volume capacity.
> >>> >>
> >>> >> On Mon, Jan 25, 2016 at 10:52 AM, Steve Dainard <sdainard at spd1.com>
> >>> wrote:
> >>> >>> Hi Manikandan
> >>> >>>
> >>> >>> I'm using 'du' not df in this case.
> >>> >>>
> >>> >>> On Thu, Jan 21, 2016 at 9:20 PM, Manikandan Selvaganesh
> >>> >>> <mselvaga at redhat.com> wrote:
> >>> >>>> Hi Steve,
> >>> >>>>
> >>> >>>> If you would like disk usage using df utility by taking quota
> limits
> >>> into
> >>> >>>> consideration, then you are expected to run the following command.
> >>> >>>>
> >>> >>>> 'gluster volume set VOLNAME quota-deem-statfs on'
> >>> >>>>
> >>> >>>> with older versions where quota-deem-statfs is OFF by default.
> >>> However with
> >>> >>>> the latest versions, quota-deem-statfs is by default ON. In this
> >>> case, the total
> >>> >>>> disk space of the directory is taken as the quota hard limit set
> on
> >>> the directory
> >>> >>>> of the volume and disk utility would display accordingly. This
> >>> answers why there is
> >>> >>>> a mismatch in disk utility.
> >>> >>>>
> >>> >>>> Next, answering to quota mechanism and accuracy: There is
> something
> >>> called timeouts
> >>> >>>> in quota. For performance reasons, quota caches the directory
> size on
> >>> client. You can
> >>> >>>> set timeout indicating the maximum valid duration of directory
> sizes
> >>> in cache,
> >>> >>>> from the time they are populated. By default the hard-timeout is
> 5s
> >>> and soft timeout
> >>> >>>> is 60s. Setting a timeout of zero will do a force fetching of
> >>> directory sizes from server
> >>> >>>> for every operation that modifies file data and will effectively
> >>> disables directory size
> >>> >>>> caching on client side. If you do not have a timeout of 0(which
> we do
> >>> not encourage due to
> >>> >>>> performance reasons), then till you reach soft-limit, soft timeout
> >>> will be taken into
> >>> >>>> consideration, and only for every 60s operations will be synced
> and
> >>> that could cause the
> >>> >>>> usage to exceed more than the hard-limit specified. If you would
> like
> >>> quota to
> >>> >>>> strictly enforce then please run the following commands,
> >>> >>>>
> >>> >>>> 'gluster v quota VOLNAME hard-timeout 0s'
> >>> >>>> 'gluster v quota VOLNAME soft-timeout 0s'
> >>> >>>>
> >>> >>>> Appreciate your curiosity in exploring and if you would like to
> know
> >>> more about quota
> >>> >>>> please refer[1]
> >>> >>>>
> >>> >>>> [1]
> >>>
> http://gluster.readthedocs.org/en/release-3.7.0-1/Administrator%20Guide/Directory%20Quota/
> >>> >>>>
> >>> >>>> --
> >>> >>>> Thanks & Regards,
> >>> >>>> Manikandan Selvaganesh.
> >>> >>>>
> >>> >>>> ----- Original Message -----
> >>> >>>> From: "Steve Dainard" <sdainard at spd1.com>
> >>> >>>> To: "gluster-users at gluster.org List" <gluster-users at gluster.org>
> >>> >>>> Sent: Friday, January 22, 2016 1:40:07 AM
> >>> >>>> Subject: Re: [Gluster-users] Quota list not reflecting disk usage
> >>> >>>>
> >>> >>>> This is gluster 3.6.6.
> >>> >>>>
> >>> >>>> I've attempted to disable and re-enable quota's on the volume, but
> >>> >>>> when I re-apply the quotas on each directory the same 'Used'
> value is
> >>> >>>> present as before.
> >>> >>>>
> >>> >>>> Where is quotad getting its information from, and how can I clean
> >>> >>>> up/regenerate that info?
> >>> >>>>
> >>> >>>> On Thu, Jan 21, 2016 at 10:07 AM, Steve Dainard <
> sdainard at spd1.com>
> >>> wrote:
> >>> >>>>> I have a distributed volume with quota's enabled:
> >>> >>>>>
> >>> >>>>> Volume Name: storage
> >>> >>>>> Type: Distribute
> >>> >>>>> Volume ID: 26d355cb-c486-481f-ac16-e25390e73775
> >>> >>>>> Status: Started
> >>> >>>>> Number of Bricks: 4
> >>> >>>>> Transport-type: tcp
> >>> >>>>> Bricks:
> >>> >>>>> Brick1: 10.0.231.50:/mnt/raid6-storage/storage
> >>> >>>>> Brick2: 10.0.231.51:/mnt/raid6-storage/storage
> >>> >>>>> Brick3: 10.0.231.52:/mnt/raid6-storage/storage
> >>> >>>>> Brick4: 10.0.231.53:/mnt/raid6-storage/storage
> >>> >>>>> Options Reconfigured:
> >>> >>>>> performance.cache-size: 1GB
> >>> >>>>> performance.readdir-ahead: on
> >>> >>>>> features.quota: on
> >>> >>>>> diagnostics.brick-log-level: WARNING
> >>> >>>>>
> >>> >>>>> Here is a partial list of quotas:
> >>> >>>>> # /usr/sbin/gluster volume quota storage list
> >>> >>>>> Path Hard-limit Soft-limit
> Used
> >>> >>>>> Available Soft-limit exceeded? Hard-limit exceeded?
> >>> >>>>>
> >>>
> ---------------------------------------------------------------------------------------------------------------------------
> >>> >>>>> ...
> >>> >>>>> /projects-CanSISE 10.0TB 80%
> >>> 11.9TB
> >>> >>>>> 0Bytes Yes Yes
> >>> >>>>> ...
> >>> >>>>>
> >>> >>>>> If I du on that location I do not get 11.9TB of space used (fuse
> >>> mount point):
> >>> >>>>> [root at storage projects-CanSISE]# du -hs
> >>> >>>>> 9.5T .
> >>> >>>>>
> >>> >>>>> Can someone provide an explanation for how the quota mechanism
> tracks
> >>> >>>>> disk usage? How often does the quota mechanism check its
> accuracy?
> >>> And
> >>> >>>>> how could it get so far off?
> >>> >>>>>
> >>> >>>>> Can I get gluster to rescan that location and update the quota
> usage?
> >>> >>>>>
> >>> >>>>> Thanks,
> >>> >>>>> Steve
> >>> >>>> _______________________________________________
> >>> >>>> Gluster-users mailing list
> >>> >>>> Gluster-users at gluster.org
> >>> >>>> http://www.gluster.org/mailman/listinfo/gluster-users
> >>> >> _______________________________________________
> >>> >> Gluster-users mailing list
> >>> >> Gluster-users at gluster.org
> >>> >> http://www.gluster.org/mailman/listinfo/gluster-users
> >>> _______________________________________________
> >>> Gluster-users mailing list
> >>> Gluster-users at gluster.org
> >>> http://www.gluster.org/mailman/listinfo/gluster-users
> >>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20160212/a515d076/attachment-0001.html>
More information about the Gluster-devel
mailing list