[Gluster-users] Issue with Gluster Quota
Brian Smith
brs at usf.edu
Mon Jul 18 17:02:26 UTC 2011
After further tests, it appears both now work after the update. Seems
the attributes set while using the git source build on the 'brs'
directory were gummed up. When I recreated the directory and remounted
with -o acl, ACLs worked and so did quota enforcement. I'll keep
testing and post if anything else comes up. So far, so good.
-Brian
Brian Smith
Senior Systems Administrator
IT Research Computing, University of South Florida
4202 E. Fowler Ave. ENB308
Office Phone: +1 813 974-1467
Organization URL: http://rc.usf.edu
On 07/18/2011 11:57 AM, Brian Smith wrote:
> Updated to the 3.2.2 release and found that Quotas do not work when
> using the POSIX-ACL translator. In fact, after I disabled the ACLs, I
> had to remove the quota'd directory (presumably, to remove some
> attributes) and start over in order to get them to work. Once I
> disabled ACLs and re-created my directory, quotas worked as expected.
> Is this a known limitation of using POSIX ACLs? I happen to need both
> features, so that could pose an issue :)
>
> -Brian
>
> Brian Smith
> Senior Systems Administrator
> IT Research Computing, University of South Florida
> 4202 E. Fowler Ave. ENB308
> Office Phone: +1 813 974-1467
> Organization URL: http://rc.usf.edu
>
> On 07/11/2011 12:48 PM, Brian Smith wrote:
>> According to the logs, the last commit was:
>>
>> commit 5c20eb3bbf870edadd22d06babb5d38dad222533
>> Author: shishir gowda <shishirng at gluster.com>
>> Date: Tue Jul 5 03:41:51 2011 +0000
>>
>> [root at gluster1 glusterfs-3.2git]# gluster volume quota home list
>> path limit_set size
>> ----------------------------------------------------------------------------------
>> /brs 10485760 81965056
>>
>> [root at gluster1 glusterfs-3.2git]# gluster volume info
>>
>> Volume Name: home
>> Type: Distribute
>> Status: Started
>> Number of Bricks: 2
>> Transport-type: tcp,rdma
>> Bricks:
>> Brick1: gluster1:/glusterfs/home
>> Brick2: gluster2:/glusterfs/home
>> Options Reconfigured:
>> features.limit-usage: /brs:10MB
>> features.quota: on
>>
>> -Brian
>>
>> Brian Smith
>> Senior Systems Administrator
>> IT Research Computing, University of South Florida
>> 4202 E. Fowler Ave. ENB308
>> Office Phone: +1 813 974-1467
>> Organization URL: http://rc.usf.edu
>>
>> On 07/11/2011 08:29 AM, Saurabh Jain wrote:
>>> Hello Brian,
>>>
>>>
>>> I synced my gluster repository back to July 5th and tried quota on a certain dir of a distribute and the quota was implemeted properly on that, here are the logs,
>>>
>>> [root at centos-qa-client-3 glusterfs]# /root/july6git/inst/sbin/gluster volume quota dist list
>>> path limit_set size
>>> ----------------------------------------------------------------------------------
>>> /dir 10485760 10485760
>>>
>>>
>>> [root at centos-qa-client-3 glusterfs]# /root/july6git/inst/sbin/gluster volume info
>>>
>>> Volume Name: dist
>>> Type: Distribute
>>> Status: Started
>>> Number of Bricks: 2
>>> Transport-type: tcp,rdma
>>> Bricks:
>>> Brick1: 10.1.12.134:/mnt/dist
>>> Brick2: 10.1.12.135:/mnt/dist
>>> Options Reconfigured:
>>> features.limit-usage: /dir:10MB
>>> features.quota: on
>>> [root at centos-qa-client-3 glusterfs]#
>>>
>>> requesting you to please inform us about the <commit id> to which your workspace is synced.
>>>
>>> Thanks,
>>> Saurabh
>>> ________________________________________
>>> From: gluster-users-bounces at gluster.org [gluster-users-bounces at gluster.org] on behalf of gluster-users-request at gluster.org [gluster-users-request at gluster.org]
>>> Sent: Friday, July 08, 2011 12:30 AM
>>> To: gluster-users at gluster.org
>>> Subject: Gluster-users Digest, Vol 39, Issue 13
>>>
>>> Send Gluster-users mailing list submissions to
>>> gluster-users at gluster.org
>>>
>>> To subscribe or unsubscribe via the World Wide Web, visit
>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>> or, via email, send a message with subject or body 'help' to
>>> gluster-users-request at gluster.org
>>>
>>> You can reach the person managing the list at
>>> gluster-users-owner at gluster.org
>>>
>>> When replying, please edit your Subject line so it is more specific
>>> than "Re: Contents of Gluster-users digest..."
>>>
>>>
>>> Today's Topics:
>>>
>>> 1. Re: Issue with Gluster Quota (Brian Smith)
>>> 2. Re: Issues with geo-rep (Carl Chenet)
>>>
>>>
>>> ----------------------------------------------------------------------
>>>
>>> Message: 1
>>> Date: Thu, 07 Jul 2011 13:10:06 -0400
>>> From: Brian Smith <brs at usf.edu>
>>> Subject: Re: [Gluster-users] Issue with Gluster Quota
>>> To: gluster-users at gluster.org
>>> Message-ID: <4E15E86E.6030407 at usf.edu>
>>> Content-Type: text/plain; charset=ISO-8859-1
>>>
>>> Sorry about that. I re-populated with an 82MB dump from dd:
>>>
>>> [root at gluster1 ~]# gluster volume quota home list
>>> path limit_set size
>>> ----------------------------------------------------------------------------------
>>> /brs 10485760 81965056
>>>
>>> [root at gluster1 ~]# getfattr -m . -d -e hex /glusterfs/home/brs
>>> getfattr: Removing leading '/' from absolute path names
>>> # file: glusterfs/home/brs
>>> security.selinux=0x726f6f743a6f626a6563745f723a66696c655f743a733000
>>> trusted.gfid=0x1bbcb9a08bf64406b440f3bb3ad334ed
>>> trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
>>> trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000000000006000
>>> trusted.glusterfs.quota.dirty=0x3000
>>> trusted.glusterfs.quota.size=0x0000000000006000
>>>
>>> [root at gluster2 ~]# getfattr -m . -d -e hex /glusterfs/home/brs
>>> getfattr: Removing leading '/' from absolute path names
>>> # file: glusterfs/home/brs
>>> security.selinux=0x726f6f743a6f626a6563745f723a66696c655f743a733000
>>> trusted.gfid=0x1bbcb9a08bf64406b440f3bb3ad334ed
>>> trusted.glusterfs.dht=0x0000000100000000000000007ffffffe
>>> trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000000004e25000
>>> trusted.glusterfs.quota.dirty=0x3000
>>> trusted.glusterfs.quota.size=0x0000000004e25000
>>>
>>> Brian Smith
>>> Senior Systems Administrator
>>> IT Research Computing, University of South Florida
>>> 4202 E. Fowler Ave. ENB308
>>> Office Phone: +1 813 974-1467
>>> Organization URL: http://rc.usf.edu
>>>
>>> On 07/07/2011 04:50 AM, Mohammed Junaid wrote:
>>>>>
>>>>> [root at gluster1 ~]# getfattr -m . -d -e hex /glusterfs/home/brs
>>>>> getfattr: Removing leading '/' from absolute path names
>>>>> # file: glusterfs/home/brs
>>>>> security.selinux=0x726f6f743a6f626a6563745f723a66696c655f743a733000
>>>>> trusted.gfid=0x1bbcb9a08bf64406b440f3bb3ad334ed
>>>>> trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
>>>>>
>>>>> trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000000000006000
>>>>> trusted.glusterfs.quota.dirty=0x3000
>>>>> trusted.glusterfs.quota.size=0x0000000000006000
>>>>>
>>>>> and
>>>>>
>>>>> [root at gluster2 ~]# getfattr -m . -d -e hex /glusterfs/home/brs
>>>>> getfattr: Removing leading '/' from absolute path names
>>>>> # file: glusterfs/home/brs
>>>>> security.selinux=0x726f6f743a6f626a6563745f723a66696c655f743a733000
>>>>> trusted.gfid=0x1bbcb9a08bf64406b440f3bb3ad334ed
>>>>> trusted.glusterfs.dht=0x0000000100000000000000007ffffffe
>>>>>
>>>>> trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000000000002000
>>>>> trusted.glusterfs.quota.dirty=0x3000
>>>>> trusted.glusterfs.quota.size=0x0000000000002000
>>>>
>>>>
>>>> trusted.glusterfs.quota.size=0x0000000000006000
>>>> trusted.glusterfs.quota.size=0x0000000000002000
>>>>
>>>> So, quota adds these values to calculate the size of the directory. They are
>>>> in hex so when I add them the value comes upto 32kb. So I suspect that you
>>>> have deleted some data from your volume. These values wont be helpful now.
>>>> Can you please re-run the same test and report the values when such a
>>>> problem occurs again.
>>>>
>>>
>>>
>>> ------------------------------
>>>
>>> Message: 2
>>> Date: Thu, 07 Jul 2011 19:56:46 +0200
>>> From: Carl Chenet <chaica at ohmytux.com>
>>> Subject: Re: [Gluster-users] Issues with geo-rep
>>> To: gluster-users at gluster.org
>>> Message-ID: <4E15F35E.1060404 at ohmytux.com>
>>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>>
>>> On 07/07/2011 15:25, Kaushik BV wrote:
>>>> Hi Chaica,
>>>>
>>>> This primarily means that the RPC communtication between the master
>>>> gsyncd module and slave gsyncd module is broken, this could happen to
>>>> various reasons. Check if it satisies all the pre-requisites:
>>>>
>>>> - If FUSE is installed in the machine, since Geo-replication module
>>>> mounts the GlusterFS volume using FUSE to sync data.
>>>> - If the Slave is a volume, check if the volume is started.
>>>> - If the Slave is a plain directory, check if the directory has been
>>>> created already with the desired permissions (Not applicable in your case)
>>>> - If Glusterfs 3.2 is not installed in the default location (in Master)
>>>> and has been prefixed to be installed in a custom location, configure
>>>> the *gluster-command* for it to point to exact location.
>>>> - If Glusterfs 3.2 is not installed in the default location (in slave)
>>>> and has been prefixed to be installed in a custom location, configure
>>>> the *remote-gsyncd-command* for it to point to exact place where gsyncd
>>>> is located.
>>>> - locate the slave log and see if it has any anomalies.
>>>> - Passwordless SSH is set up properly between the host and the remote
>>>> machine ( Not applicable in your case)
>>>
>>> Ok the situation has slightly evolved. Now I do have a slave log and
>>> clearer error message on the master :
>>>
>>>
>>> [2011-07-07 19:53:16.258866] I [monitor(monitor):42:monitor] Monitor:
>>> ------------------------------------------------------------
>>> [2011-07-07 19:53:16.259073] I [monitor(monitor):43:monitor] Monitor:
>>> starting gsyncd worker
>>> [2011-07-07 19:53:16.332720] I [gsyncd:286:main_i] <top>: syncing:
>>> gluster://localhost:test-volume -> ssh://192.168.1.32::test-volume
>>> [2011-07-07 19:53:16.343554] D [repce:131:push] RepceClient: call
>>> 6302:140305661662976:1310061196.34 __repce_version__() ...
>>> [2011-07-07 19:53:20.931523] D [repce:141:__call__] RepceClient: call
>>> 6302:140305661662976:1310061196.34 __repce_version__ -> 1.0
>>> [2011-07-07 19:53:20.932172] D [repce:131:push] RepceClient: call
>>> 6302:140305661662976:1310061200.93 version() ...
>>> [2011-07-07 19:53:20.933662] D [repce:141:__call__] RepceClient: call
>>> 6302:140305661662976:1310061200.93 version -> 1.0
>>> [2011-07-07 19:53:20.933861] D [repce:131:push] RepceClient: call
>>> 6302:140305661662976:1310061200.93 pid() ...
>>> [2011-07-07 19:53:20.934525] D [repce:141:__call__] RepceClient: call
>>> 6302:140305661662976:1310061200.93 pid -> 10075
>>> [2011-07-07 19:53:20.957355] E [syncdutils:131:log_raise_exception]
>>> <top>: FAIL:
>>> Traceback (most recent call last):
>>> File "/usr/lib/glusterfs/glusterfs/python/syncdaemon/gsyncd.py", line
>>> 102, in main
>>> main_i()
>>> File "/usr/lib/glusterfs/glusterfs/python/syncdaemon/gsyncd.py", line
>>> 293, in main_i
>>> local.connect()
>>> File "/usr/lib/glusterfs/glusterfs/python/syncdaemon/resource.py",
>>> line 379, in connect
>>> raise RuntimeError("command failed: " + " ".join(argv))
>>> RuntimeError: command failed: /usr/sbin/glusterfs --xlator-option
>>> *-dht.assert-no-child-down=true -L DEBUG -l
>>> /var/log/glusterfs/geo-replication/test-volume/ssh%3A%2F%2Froot%40192.168.1.32%3Agluster%3A%2F%2F127.0.0.1%3Atest-volume.gluster.log
>>> -s localhost --volfile-id test-volume --client-pid=-1
>>> /tmp/gsyncd-aux-mount-hy6T_w
>>> [2011-07-07 19:53:20.960621] D [monitor(monitor):58:monitor] Monitor:
>>> worker seems to be connected (?? racy check)
>>> [2011-07-07 19:53:21.962501] D [monitor(monitor):62:monitor] Monitor:
>>> worker died in startup phase
>>>
>>> The command launched by glusterfs returns a 255 error shell code, which
>>> I belive means the command is terminated by a signal. On the slave log I
>>> have :
>>>
>>> [2011-07-07 19:54:49.571549] I [fuse-bridge.c:3218:fuse_thread_proc]
>>> 0-fuse: unmounting /tmp/gsyncd-aux-mount-z2Q2Hg
>>> [2011-07-07 19:54:49.572459] W [glusterfsd.c:712:cleanup_and_exit]
>>> (-->/lib/libc.so.6(clone+0x6d) [0x7f2c8998b02d]
>>> (-->/lib/libpthread.so.0(+0x68ba) [0x7f2c89c238ba]
>>> (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xc5) [0x7f2c8a8f51b5])))
>>> 0-: received signum (15), shutting down
>>> [2011-07-07 19:54:51.280207] W [write-behind.c:3029:init]
>>> 0-test-volume-write-behind: disabling write-behind for first 0 bytes
>>> [2011-07-07 19:54:51.291669] I [client.c:1935:notify]
>>> 0-test-volume-client-0: parent translators are ready, attempting connect
>>> on transport
>>> [2011-07-07 19:54:51.292329] I [client.c:1935:notify]
>>> 0-test-volume-client-1: parent translators are ready, attempting connect
>>> on transport
>>> [2011-07-07 19:55:38.582926] I [rpc-clnt.c:1531:rpc_clnt_reconfig]
>>> 0-test-volume-client-0: changing port to 24009 (from 0)
>>> [2011-07-07 19:55:38.583456] I [rpc-clnt.c:1531:rpc_clnt_reconfig]
>>> 0-test-volume-client-1: changing port to 24009 (from 0)
>>>
>>> Bye,
>>> Carl Chenet
>>>
>>>
>>> ------------------------------
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>
>>>
>>> End of Gluster-users Digest, Vol 39, Issue 13
>>> *********************************************
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
More information about the Gluster-users
mailing list