[Gluster-users] quota calculation gluster 3.2.3
Mohammed Junaid
junaid at gluster.com
Mon Sep 5 11:51:45 UTC 2011
Coming back to the problem of quota sizes not converging, the converging may
not complete if glusterd is restarted before converging process is complete
because a thread is spawned by glusterd to do the converging. Hence, if
glusterd is restarted converging will fail or to put it precisely it will
not be completed. So running ls -lR on the mount point will solve the
problem.
On Mon, Sep 5, 2011 at 4:40 PM, Mohammed Junaid <junaid at gluster.com> wrote:
> Hi Anthony,
>
> This is quiet unusual, to debug this further can you get the output of "du
> -h" of mount-point and of directory soft (which is under mount point). This
> is will help a lot in debugging this issue. Also, I noticed that the volume
> info output provided is of a distributed-replicate setup which is different
> from the one provided previously (it was plain replicate). Also please
> provide the output of getfattr for the bricks and soft directory.
>
> Junaid
>
> On Mon, Sep 5, 2011 at 2:58 PM, anthony garnier <sokar6012 at hotmail.com>wrote:
>
>>
>> Volume configuration:
>>
>> yval1000:/ # gluster volume info
>>
>> Volume Name: venus
>> Type: Distributed-Replicate
>> Status: Started
>> Number of Bricks: 2 x 2 = 4
>>
>> Transport-type: tcp
>> Bricks:
>> Brick1: ylal3020:/soft/venus
>> Brick2: ylal3030:/soft/venus
>> Brick3: yval1000:/soft/venus
>> Brick4: yval1010:/soft/venus
>> Options Reconfigured:
>> features.quota-timeout: 120
>> features.limit-usage: /test:100MB,/psa:200MB,/:7GB,/soft:5GB
>> features.quota: on
>> geo-replication.indexing: on
>> network.ping-timeout: 10
>> performance.cache-size: 1GB
>> performance.cache-refresh-timeout: 60
>> nfs.port: 2049
>>
>>
>> Indeed values didn't converged automaticaly for big folder, for small
>> folder like /test and /psa it was fine (I waited 1 day before checking the
>> value). I was forced to run ls -lRa several times, but even with this
>> workaround, values for big folder like /soft are not rightfully calculated :
>>
>>
>> gluster volume quota venus list
>> path limit_set size
>>
>> ----------------------------------------------------------------------------------
>> /test 100MB 4.5MB <= OK
>> /psa 200MB 167.0MB <=OK
>> / 7GB 5.2GB <=Strange
>> /soft 5GB 4.7GB <= Strange
>>
>> => the size of the entire filesystem is 1.4GB
>>
>>
>>
>> Anthony
>>
>>
>> ------------------------------
>> From: junaid at gluster.com
>> Date: Mon, 5 Sep 2011 14:38:06 +0530
>>
>> Subject: Re: [Gluster-users] quota calculation gluster 3.2.3
>> To: sokar6012 at hotmail.com
>> CC: gluster-users at gluster.org
>>
>> Hi Anthony,
>>
>> By volume configuration, I meant 'is the volume setup dht, afr, or
>> dht-afr', I would like to know the output of
>>
>> volume info volume-name
>>
>> When data is already present before the activation of quota, it takes some
>> time for quota values to converge because it has to crawl through the entire
>> file-system to calculate the sizes. This is automatically done when you run
>>
>> volume quota vol-name enable
>>
>> The output of
>> volume quota vol-name list
>>
>> will reflect the aggregated value at that point in time. If the values are
>> not converging then its a issue. A simple work-around is to run ls -lR on
>> the mount point. Please confirm if the values were not converging
>> automatically, we will have a look into it.
>>
>> Junaid
>>
>>
>> On Mon, Sep 5, 2011 at 12:51 PM, anthony garnier <sokar6012 at hotmail.com>wrote:
>>
>> Junaid,
>>
>> What do you mean by " the volume configuration on other two servers" ?
>> Configuration is the same on all the server..
>> I'm creating data with "dd" cmd, and yes, data was already present before
>> activation of quota.
>>
>>
>>
>> ------------------------------
>> From: junaid at gluster.com
>> Date: Fri, 2 Sep 2011 22:55:02 +0530
>>
>> Subject: Re: [Gluster-users] quota calculation gluster 3.2.3
>> To: sokar6012 at hotmail.com
>> CC: gluster-users at gluster.org
>>
>> Hi Anthony,
>>
>> Thanks for providing the input. What is the volume configuration on other
>> two servers (yval1000 and yval1010)? My question is same, what data where
>> you creating on these two machines? Was the data already present before
>> quota was enabled?
>>
>> On Fri, Sep 2, 2011 at 1:49 PM, anthony garnier <sokar6012 at hotmail.com>wrote:
>>
>>
>> I've done a ls -lRa throught the mount point and now i get :
>> /soft/venus # gluster volume quota venus list
>> path limit_set size
>>
>> ----------------------------------------------------------------------------------
>> / 2GB 1.3GB
>> /test 100MB 8.0KB
>> /psa 200MB 151.0MB
>> /soft 1500MB 826.3MB
>>
>> =>Seems to be consistent
>>
>> and now command getfattr on yval1000 and yval1010 show :
>>
>> yval1000 :
>> getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/
>> trusted.glusterfs.quota.size=0x0000000028f9ca00
>>
>>
>> getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/soft
>> trusted.glusterfs.quota.size=0x000000001421d400
>>
>>
>> getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/psa
>> trusted.glusterfs.quota.size=0x00000000053d6000
>>
>>
>> yval1010 :
>> getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/
>> trusted.glusterfs.quota.size=0x000000002904ea00
>>
>>
>> getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/soft
>> trusted.glusterfs.quota.size=0x000000002292ba00
>>
>>
>> getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/psa
>> trusted.glusterfs.quota.size=0x00000000053d6000
>>
>> Regards,
>>
>> Anthony
>>
>>
>> ------------------------------
>> From: sokar6012 at hotmail.com
>> To: junaid at gluster.com
>> CC: gluster-users at gluster.org
>> Subject: RE: [Gluster-users] quota calculation gluster 3.2.3
>> Date: Fri, 2 Sep 2011 07:57:19 +0000
>>
>>
>> I forgot to check on the 2 other server.
>> So ylal3020 and ylal3030 show similar output send previously.
>>
>> But yval1000 and yval1010 show with all command :
>> trusted.glusterfs.quota.size=0x0000000000000000
>>
>> Is it normal ?
>>
>> Regards,
>>
>> Anthony
>> ------------------------------
>> From: sokar6012 at hotmail.com
>> To: junaid at gluster.com
>> CC: gluster-users at gluster.org
>> Subject: RE: [Gluster-users] quota calculation gluster 3.2.3
>> Date: Fri, 2 Sep 2011 07:51:52 +0000
>>
>> Hi Junaid,
>>
>> Here is the output of the diffrent command executed on both server.
>> Command show identical output on both server
>>
>> # getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus
>> getfattr: Removing leading '/' from absolute path names
>> # file: soft/venus
>> trusted.glusterfs.quota.size=0x0000000022ccac00
>>
>> # getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/test
>> getfattr: Removing leading '/' from absolute path names
>> # file: soft/venus/test
>> trusted.glusterfs.quota.size=0x0000000000000000
>>
>> # getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/psa
>> getfattr: Removing leading '/' from absolute path names
>> # file: soft/venus/psa
>> trusted.glusterfs.quota.size=0x000000000013d000
>>
>> # getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/soft
>> getfattr: Removing leading '/' from absolute path names
>> # file: soft/venus/soft
>> trusted.glusterfs.quota.size=0x000000000d15ce00
>>
>>
>> # gluster volume quota venus list
>> path limit_set size
>>
>> ----------------------------------------------------------------------------------
>> / 2GB 556.8MB
>> /test 100MB 0Bytes
>> /psa 200MB 1.2MB
>> /soft 1500MB 209.4MB
>>
>>
>> Indeed there is 90% of small file (~60000 files).
>>
>> Thx
>>
>> Anthony
>>
>>
>> ------------------------------
>> From: junaid at gluster.com
>> Date: Fri, 2 Sep 2011 06:18:30 +0530
>> Subject: Re: [Gluster-users] quota calculation gluster 3.2.3
>> To: sokar6012 at hotmail.com
>> CC: gluster-users at gluster.org
>>
>> Hi Anthony,
>>
>> To debug this further, can you send the output of
>>
>> getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus
>> getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/test
>> getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/psa
>> getfattr -n trusted.glusterfs.quota.size -e hex /soft/venus/soft
>>
>> from both the machine. Also check the output of
>>
>> gluster volume quota Replicate list
>>
>> Sometimes it takes a small amount of time to bring up the sizes. Also,
>> what kind of data where you creating (I mean large amount of small files or
>> large files because in case of small files, the directory sizes are not
>> accounted by quota to calculate the size unlike the du -h command which uses
>> the directory size as well).
>>
>> Junaid
>> On Thu, Sep 1, 2011 at 1:37 PM, anthony garnier <sokar6012 at hotmail.com>wrote:
>>
>> Hi all,
>>
>> I've enable quota but I'm a bit confused by values displayed by GlusterFS
>>
>> Here is my volume :
>>
>> Volume Name: venus
>> Type: Replicate
>> Status: Started
>> Number of Bricks: 2
>> Transport-type: tcp
>> Bricks:
>> Brick1: ylal3020:/soft/venus
>> Brick2: ylal3030:/soft/venus
>> Options Reconfigured:
>> features.limit-usage: /test:100MB,/psa:200MB,/soft:1500MB,/:2GB
>> features.quota: on
>> diagnostics.client-log-level: ERROR
>> diagnostics.brick-log-level: ERROR
>> network.ping-timeout: 10
>> performance.cache-size: 2GB
>> nfs.port: 2049
>>
>> I've got 3 folders in the backend (/soft/venus) :
>> psa 160MB (with du -sh)
>> soft 1.2GB
>> test 12KB
>> Total 1.4GB
>>
>> But when I list the quota with gluster I got :
>> # gluster volume quota venus list
>> path limit_set size
>>
>> ----------------------------------------------------------------------------------
>> /test 100MB 12.0KB <=
>> This one is OK
>> /psa 200MB 64.4MB <= not OK
>> /soft 1500MB 281.8MB <= not OK
>> / 2GB 346.2MB <= not
>> OK
>>
>> Any idea ?
>>
>> Regards,
>>
>> Anthony Garnier
>>
>>
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>
>>
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20110905/3513075c/attachment.html>
More information about the Gluster-users
mailing list