[Gluster-users] Upgraded from 3.4.1 to 3.5.2, quota no longer working

Tue Dec 2 07:28:20 UTC 2014

Hi, 

Are you sure the post-upgrade script ran to completion? 
Here is one way to confirm whether that is the case: check if the quota configured directories have an xattr called "trusted.glusterfs.quota.limit-set" set on them in the respective bricks. 

For example, here's what mine looks like: 

[root at haddock 1]# pwd 
/brick/1 
[root at haddock 1]# getfattr -d -m . -e hex 1 
# file: 1 
security.selinux=0x73797374656d5f753a6f626a6563745f723a64656661756c745f743a733000 
trusted.gfid=0x57d0a561ca574d1cb0428f38d1c06e85 
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff 
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri=0x0000000000000a00 
trusted.glusterfs.quota.dirty=0x3000 
trusted.glusterfs.quota.limit-set=0x0000000006400000ffffffffffffffff 
trusted.glusterfs.quota.size=0x0000000000000a00 

where /brick/1 is the brick directory and under it "1" is the name of one of the quota-configured directories. 

I believe your quota configurations are backed up at /var/tmp/glusterfs/quota-config-backup/vol_<volname> which you can use to get the quota-configured directory names. 
As for operating version, I think it is sufficient for it to be at 3 for the 3.5.x quota to work. 

-Krutika 

----- Original Message -----

> From: "David Gibbons" <david.c.gibbons at gmail.com>
> To: "Krutika Dhananjay" <kdhananj at redhat.com>
> Cc: "gluster-users" <gluster-users at gluster.org>
> Sent: Monday, December 1, 2014 6:35:55 PM
> Subject: Re: [Gluster-users] Upgraded from 3.4.1 to 3.5.2, quota no longer
> working

> Certainly, thank you for your response:

> Quotad is running on all nodes:

> > > [root at gfs-a-1 ~]# ps aux | grep quotad
> > 
> 
> > > root 3004 0.0 0.4 241368 68552 ? Ssl Nov30 0:07 /usr/local/sbin/glusterfs
> > > -s
> > > localhost --volfile-id gluster/quotad -p
> > > /var/lib/glusterd/quotad/run/quotad.pid -l
> > > /usr/local/var/log/glusterfs/quotad.log -S
> > > /var/run/9d02605105ef0e74d913a4671c1143a1.socket --xlator-option
> > > *replicate*.data-self-heal=off --xlator-option
> > > *replicate*.metadata-self-heal=off --xlator-option
> > > *replicate*.entry-self-heal=off
> > 
> 
> And the relevant output from gluster volume status shares per your request:

> > [root at gfs-a-1 ~]# gluster volume status shares | grep Quota
> 
> > Quota Daemon on localhost N/A Y 3004
> 
> > Quota Daemon on gfs-a-3 N/A Y 32307
> 
> > Quota Daemon on gfs-a-4 N/A Y 10818
> 
> > Quota Daemon on gfs-a-2 N/A Y 12292
> 
> No log entries are created in /var/log/glusterfs/quotad.log when I run a
> quota list; all of the log entries are from yesterday. They do indicate a
> version mis-match, although I can't seem to locate where that version is
> specified:

> > > [2014-11-30 13:21:55.173081] I
> > > [client-handshake.c:1474:client_setvolume_cbk]
> > > 0-shares-client-14: Server and Client lk-version numbers are not same,
> > > reopening the fds
> > 
> 
> > > [2014-11-30 13:21:55.173170] I
> > > [client-handshake.c:450:client_set_lk_version_cbk] 0-shares-client-14:
> > > Server lk version = 1
> > 
> 
> > > [2014-11-30 13:21:55.178739] I [rpc-clnt.c:1729:rpc_clnt_reconfig]
> > > 0-shares-client-9: changing port to 49154 (from 0)
> > 
> 
> > > [2014-11-30 13:21:55.181170] I
> > > [client-handshake.c:1677:select_server_supported_programs]
> > > 0-shares-client-9: Using Program GlusterFS 3.3, Num (1298437), Version
> > > (330)
> > 
> 
> > > [2014-11-30 13:21:55.181386] I
> > > [client-handshake.c:1462:client_setvolume_cbk]
> > > 0-shares-client-9: Connected to 172.16.10.13:49154 , attached to remote
> > > volume '/mnt/a-3-shares-brick-3/brick'.
> > 
> 
> > > [2014-11-30 13:21:55.181401] I
> > > [client-handshake.c:1474:client_setvolume_cbk]
> > > 0-shares-client-9: Server and Client lk-version numbers are not same,
> > > reopening the fds
> > 
> 
> > > [2014-11-30 13:21:55.181535] I
> > > [client-handshake.c:450:client_set_lk_version_cbk] 0-shares-client-9:
> > > Server
> > > lk version = 1
> > 
> 
> I see the operational mode for the volume as "3". I saw a non-related thread
> that indicated this number should be more digits on a cluster running 3.5.2.
> The other thread also indicated that quota may not work if the volume
> version number was not compatible with the quota version running on the
> cluster. I can't seem to find the link right now.

> It's almost as if the volume version did not get upgraded when the server
> version was upgraded. Is that possible?

> Cheers,
> Dave

> On Sun, Nov 30, 2014 at 11:46 PM, Krutika Dhananjay < kdhananj at redhat.com >
> wrote:

> > Hi,
> 

> > Could you confirm whether quotad (Quota Daemon) is online from the output
> > of
> > `gluster volume status shares`?
> 

> > Also, could you share quota daemon's log file from the node where you
> > executed `quota list` command, which you will find at @
> > /var/log/glusterfs/quotad.log?
> 

> > -Krutika
> 

> > > From: "David Gibbons" < david.c.gibbons at gmail.com >
> > 
> 
> > > To: "gluster-users" < gluster-users at gluster.org >
> > 
> 
> > > Sent: Sunday, November 30, 2014 9:57:02 PM
> > 
> 
> > > Subject: [Gluster-users] Upgraded from 3.4.1 to 3.5.2, quota no longer
> > > working
> > 
> 

> > > Hi All,
> > 
> 

> > > I performed a long-awaited upgrade from 3.4.1 to 3.5.2 today following
> > > the
> > > instructions for an offline upgrade outlined here:
> > 
> 
> > > http://www.gluster.org/community/documentation/index.php/Upgrade_to_3.5
> > 
> 

> > > I ran the pre- and post- upgrade scripts as instructed, intending to move
> > > the
> > > quotas over to the new version. The upgrade seemed to go well, the volume
> > > is
> > > online and it appears to be functioning properly.
> > 
> 

> > > When I attempt to check quotas, the list is empty:
> > 
> 

> > > > > [root at gfs-a-1 glusterfs]# gluster volume quota shares list
> > > > 
> > > 
> > 
> 
> > > > > Path Hard-limit Soft-limit Used Available
> > > > 
> > > 
> > 
> 
> > > > > --------------------------------------------------------------------------------
> > > > 
> > > 
> > 
> 
> > > > > [root at gfs-a-1 glusterfs]#
> > > > 
> > > 
> > 
> 
> > > And upon execution of that command, the cli.log file fills up with
> > > entries
> > > like this. I am assuming it's one cli log entry per quota entry:
> > 
> 

> > > > > [2014-11-30 14:00:02.154143] W
> > > > > [cli-rpc-ops.c:2469:print_quota_list_from_quotad] 0-cli: path key is
> > > > > not
> > > > > present in dict
> > > > 
> > > 
> > 
> 
> > > > > [2014-11-30 14:00:02.160507] W
> > > > > [cli-rpc-ops.c:2469:print_quota_list_from_quotad] 0-cli: path key is
> > > > > not
> > > > > present in dict
> > > > 
> > > 
> > 
> 
> > > > > [2014-11-30 14:00:02.167947] W
> > > > > [cli-rpc-ops.c:2469:print_quota_list_from_quotad] 0-cli: path key is
> > > > > not
> > > > > present in dic
> > > > 
> > > 
> > 
> 
> > > So, it appears that somehow the quota database has become offline or
> > > corrupt.
> > > Any thoughts on what I can do to resolve this?
> > 
> 

> > > I have checked all of the binaries on all 4 machines in the cluster, and
> > > they
> > > all appear to be running the correct version:
> > 
> 

> > > > > [root at gfs-a-1 glusterfs]# glusterfsd --version
> > > > 
> > > 
> > 
> 
> > > > > glusterfs 3.5.2 built on Nov 30 2014 08:16:37
> > > > 
> > > 
> > 
> 
> > > Cheers,
> > 
> 
> > > Dave
> > 
> 

> > > _______________________________________________
> > 
> 
> > > Gluster-users mailing list
> > 
> 
> > > Gluster-users at gluster.org
> > 
> 
> > > http://supercolony.gluster.org/mailman/listinfo/gluster-users
> > 
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20141202/6266c396/attachment.html>