[Gluster-devel] Problem with GlusterFS quotas (Debian Wheezy/x86_64)

Fri Apr 20 05:17:49 UTC 2012

Dear list,

I'm testing a little GlusterFS setup on debian wheezy:

=== snip ===
muzzy:~# cat /etc/debian_version
wheezy/sid

muzzy:~# dpkg -l | grep gluster
ii glusterfs-client 3.2.6-1 clustered file-system (client package)
ii glusterfs-common 3.2.6-1 GlusterFS common libraries and translator 
modules
ii glusterfs-server 3.2.6-1 clustered file-system (server package)
=== snip ===

My volume has 3 bricks and is a distributed one:

=== snip ===
muzzy:~# gluster volume info

Volume Name: m3d
Type: Distribute
Status: Started
Number of Bricks: 3
Transport-type: tcp
Bricks:
Brick1: g0:/g0
Brick2: g1:/g1
Brick3: g2:/g2
Options Reconfigured:
features.quota-timeout: 0
features.limit-usage: 
/1Gg:1GB,/10Gg:10GB,/5Gg:5GB,/512Mb:512MB,/dav/256MB:256MB,/dav/1GB:1GB,/dav/5GB:5GB,/dav/10GB:10GB,/cifs/256Mb:256MB,/dav/128mb:128MB,/dav/192mb:192MB,/cifs/192mb:192MB,/cifs/64mb:64MB
features.quota: on
auth.allow: 194.12.*,130.204.198.50
=== snip ===

One of nodes, (g0), also mounts m3d volume (mount -t glusterfs) and 
"exports" it via samba and webdav. I'm using two different directories 
under volume root for each of them, i.e. dav and cifs. /This is not the 
best way to do export the volume, but I do not have more nodes at the 
moment :(/

I'm mounting the volume with the following parameters:
=== snip ===
mount -t glusterfs g0:/m3d /mnt/m3d -o 
acl,log-level=WARNING,log-file=/var/log/gluster.log
=== snip ===

According the glusterfs documentation 
(Gluster_File_System-3.2.5-Administration_Guide-en-US.pdf, p. 67) we can 
set directory limit even on non-existing directory:
=== quote ===
Note
You can set the disk limit on the directory even if it is not created. 
The disk limit is enforced
immediately after creating that directory. For more information on 
setting disk limit, see
Section 10.3, “Setting or Replacing Disk Limit ”.
=== quote ===

Here comes my problem. When I set directory quota on non-existing (or 
even existing directory) the quota limit is not enforced. I have to 
unmount the volume (i.e. stop apache and samba daemons before that) and 
mount it again (respectively starting back apache and samba daemons). 
Then quota limit is enforced correctly on existing directories. If I 
create a new directory and set quota limit, i have to umount and mount 
again in order to have working quota for that directory.

Do you have any ideas, what may cause this problem? Will it be 
appropriate to blame Debian Wheezy /because I'm using Its testing branch/?

I hade spet some time, searching the mail list archives of glusterfs and 
I found these two descriptions of problems quite close to mine:

https://bugzilla.redhat.com/show_bug.cgi?id=765267
https://bugzilla.redhat.com/show_bug.cgi?id=790389

So, will it be appropriate to relate my quota problem to these two?

Regards
Hristo Hristov

p.s. More verbose info about my glusterfs installation follows:

=== snip ===
muzzy:~# cat /etc/debian_version
wheezy/sid

muzzy:~# uname -a
Linux muzzy 3.2.0-1-amd64 #1 SMP Fri Feb 17 05:17:36 UTC 2012 x86_64 
GNU/Linux

muzzy:~# cat /var/log/gluster.log
2012-04-19 16:36:17.663509] W [write-behind.c:3023:init] 
0-m3d-write-behind: disabling write-behind for first 0 bytes
Given volfile:
+------------------------------------------------------------------------------+
1: volume m3d-client-0
2: type protocol/client
3: option remote-host g0
4: option remote-subvolume /g0
5: option transport-type tcp
6: end-volume
7:
8: volume m3d-client-1
9: type protocol/client
10: option remote-host g1
11: option remote-subvolume /g1
12: option transport-type tcp
13: end-volume
14:
15: volume m3d-client-2
16: type protocol/client
17: option remote-host g2
18: option remote-subvolume /g2
19: option transport-type tcp
20: end-volume
21:
22: volume m3d-dht
23: type cluster/distribute
24: subvolumes m3d-client-0 m3d-client-1 m3d-client-2
25: end-volume
26:
27: volume m3d-quota
28: type features/quota
29: option limit-set 
/1Gg:1GB,/10Gg:10GB,/5Gg:5GB,/512Mb:512MB,/dav/256MB:256MB,/dav/1GB:1GB,/dav/5GB:5GB,/dav/10GB:10GB,/cifs/256Mb:256MB,/dav/128mb:128MB,/dav/192mb:192MB,/cifs/192mb:192MB
30: option timeout 0
31: subvolumes m3d-dht
32: end-volume
33:
34: volume m3d-write-behind
35: type performance/write-behind
36: subvolumes m3d-quota
37: end-volume
38:
39: volume m3d-read-ahead
40: type performance/read-ahead
41: subvolumes m3d-write-behind
42: end-volume
43:
44: volume m3d-io-cache
45: type performance/io-cache
46: subvolumes m3d-read-ahead
47: end-volume
48:
49: volume m3d-quick-read
50: type performance/quick-read
51: subvolumes m3d-io-cache
52: end-volume
53:
54: volume m3d-stat-prefetch
55: type performance/stat-prefetch
56: subvolumes m3d-quick-read
57: end-volume
58:
59: volume m3d
60: type debug/io-stats
61: option latency-measurement off
62: option count-fop-hits off
63: subvolumes m3d-stat-prefetch
64: end-volume

+------------------------------------------------------------------------------+
[2012-04-19 16:37:12.729422] W [dict.c:1153:data_to_str] 
(-->/usr/lib/libglusterfs.so.0(+0x18ddc) [0x7ff9c3025ddc] 
(-->/usr/lib/libglusterfs.so.0(+0x18e6a) [0x7ff9c3025e6a] 
(-->/usr/lib/glusterfs/3.2.6/xlator/performance/io-cache.so(reconfigure+0x443) 
[0x7ff9bf0f7913]))) 0-dict: data is NULL
[2012-04-19 16:37:12.729481] W [dict.c:1153:data_to_str] 
(-->/usr/lib/libglusterfs.so.0(+0x18ddc) [0x7ff9c3025ddc] 
(-->/usr/lib/libglusterfs.so.0(+0x18e6a) [0x7ff9c3025e6a] 
(-->/usr/lib/glusterfs/3.2.6/xlator/performance/io-cache.so(reconfigure+0x59d) 
[0x7ff9bf0f7a6d]))) 0-dict: data is NULL
[2012-04-19 16:37:12.729531] W [dict.c:1153:data_to_str] 
(-->/usr/lib/libglusterfs.so.0(+0x18ddc) [0x7ff9c3025ddc] 
(-->/usr/lib/libglusterfs.so.0(+0x18e6a) [0x7ff9c3025e6a] 
(-->/usr/lib/glusterfs/3.2.6/xlator/performance/io-cache.so(reconfigure+0x72f) 
[0x7ff9bf0f7bff]))) 0-dict: data is NULL
[2012-04-19 16:37:12.729580] W [dict.c:1153:data_to_str] 
(-->/usr/lib/libglusterfs.so.0(+0x18ddc) [0x7ff9c3025ddc] 
(-->/usr/lib/libglusterfs.so.0(+0x18e6a) [0x7ff9c3025e6a] 
(-->/usr/lib/glusterfs/3.2.6/xlator/performance/io-cache.so(reconfigure+0x74b) 
[0x7ff9bf0f7c1b]))) 0-dict: data is NULL
=== snip ===

Info about extended fs attributes for one directory with quota in my 
glusterfs setup follows:

=== snip ===
(o) muzzy, aka g0
muzzy:~# getfattr -m . -d -e hex /g0/dav/128mb
getfattr: Removing leading '/' from absolute path names
# file: g0/dav/128mb
trusted.gfid=0xd226869ff0ce4e519403be39a5842ebf
trusted.glusterfs.dht=0x0000000100000000aaaaaaaaffffffff
trusted.glusterfs.quota.c4218eca-9bbf-468d-8dba-3f2f135d8f38.contri=0x0000000000003000
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.size=0x0000000000003000

root at g1:~# getfattr -m . -d -e hex /g1/dav/128mb
getfattr: Removing leading '/' from absolute path names
# file: g1/dav/128mb
trusted.gfid=0xd226869ff0ce4e519403be39a5842ebf
trusted.glusterfs.dht=0x00000001000000000000000055555554
trusted.glusterfs.quota.c4218eca-9bbf-468d-8dba-3f2f135d8f38.contri=0x000000000539f000
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.size=0x000000000539f000

root at mdx:~# getfattr -m . -d -e hex /g2/dav/128mb
getfattr: Removing leading '/' from absolute path names
# file: g2/dav/128mb
trusted.gfid=0xd226869ff0ce4e519403be39a5842ebf
trusted.glusterfs.dht=0x000000010000000055555555aaaaaaa9
trusted.glusterfs.quota.c4218eca-9bbf-468d-8dba-3f2f135d8f38.contri=0x0000000000004000
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.size=0x0000000000004000
=== snip ===

Also, there is a list of all quotas I've set, during my initial testing.

=== snip ===
muzzy:~# gluster volume quota m3d list
path limit_set size
----------------------------------------------------------------------------------
/1Gg 1GB
/10Gg 10GB
/5Gg 5GB
/512Mb 512MB 0Bytes
/dav/256MB 256MB 208.8MB
/dav/1GB 1GB 0Bytes
/dav/5GB 5GB 0Bytes
/dav/10GB 10GB 0Bytes
/cifs/256Mb 256MB 167.5MB
/dav/128mb 128MB 83.6MB
/dav/192mb 192MB 365.4MB
/cifs/192mb 192MB 210.9MB
/cifs/64mb 64MB 365.4MB
=== snip ===