[Gluster-users] Why is Gluster 3.4+ limited to 32 groups when Gluster 3.0 was not?

Tue Feb 3 04:14:46 UTC 2015

A discussion thread on this list from last May 2014,
http://www.gluster.org/pipermail/gluster-users.old/2014-May/017283.html,
discussed how Gluster is limited to 32 groups, due to FUSE, or maybe to 96
groups, due to the AUTH header size in the RPC library being used, unless
the administrator enables the server.manage-gids option, which causes
user/group membership to be resolved on the server instead of on the
client, thus requiring the same user/group membership information to be
maintained across all clients and servers.

As far as I can tell, Gluster 3.0 did not have this limitation. What
changed?

Here is my evidence: I have created a Gluster 3.0 and 3.4 installation on
separate clients and servers, mounted at /mnt/gfs in each case. I create
>32 groups and add one user (bjaspan) to all of them. I create a directory
named for the group and owned by root:<group> mode 0750. Then, as the user,
I try to ls all of them. On Gluster 3.0, it works fine:

bjaspan at web-47:~$ glusterfs --version | head -n 1
glusterfs 3.0.0git built on Oct 28 2013 16:38:44
bjaspan at web-47:~$ groups
bjaspan www-data ahops g30000 g30001 g30002 g30003 g30004 g30005 g30006
g30007 g30008 g30009 g30010 g30011 g30012 g30013 g30014 g30015 g30016
g30017 g30018 g30019 g30020 g30021 g30022 g30023 g30024 g30025 g30026
g30027 g30028 g30029 g30030 g30031 g30032 g30033 g30034 g30035 g30036
g30037 g30038 g30039 g30040 g30041 g30042 g30043 g30044 g30045 g30046
g30047 g30048 g30049 g30050 g30051 g30052 g30053 g30054 g30055 g30056
g30057 g30058 g30059 g30060 g30061 g30062 g30063 g30064 g30065 g30066
g30067 g30068 g30069 g30070 g30071 g30072 g30073 g30074 g30075 g30076
g30077 g30078 g30079 g30080 g30081 g30082 g30083 g30084 g30085 g30086
g30087 g30088 g30089 g30090 g30091 g30092 g30093 g30094 g30095 g30096
g30097 g30098 g30099 g30100
bjaspan at web-47:~$ groups | wc
      1     104     730
bjaspan at web-47:~$ ls -ld /mnt/gfs/g* | head
drwxr-x--- 2 root g30000 6 Feb  3 02:41 /mnt/gfs/g30000
drwxr-x--- 2 root g30001 6 Feb  3 02:41 /mnt/gfs/g30001
drwxr-x--- 2 root g30002 6 Feb  3 02:41 /mnt/gfs/g30002
drwxr-x--- 2 root g30003 6 Feb  3 02:41 /mnt/gfs/g30003
drwxr-x--- 2 root g30004 6 Feb  3 02:41 /mnt/gfs/g30004
drwxr-x--- 2 root g30005 6 Feb  3 02:41 /mnt/gfs/g30005
drwxr-x--- 2 root g30006 6 Feb  3 02:41 /mnt/gfs/g30006
drwxr-x--- 2 root g30007 6 Feb  3 02:41 /mnt/gfs/g30007
drwxr-x--- 2 root g30008 6 Feb  3 02:41 /mnt/gfs/g30008
drwxr-x--- 2 root g30009 6 Feb  3 02:41 /mnt/gfs/g30009
bjaspan at web-47:~$ ls -l /mnt/gfs/g*/. > /dev/null
bjaspan at web-47:~$

However, the same test on Gluster 3.4 gives Permission Denied errors
(notice that it gives exactly 18 such errors, and my user is in 50 groups,
50-18=32):

bjaspan at web-28:~$ glusterfs --version | head -n 1
glusterfs 3.4git built on Nov 21 2013 14:10:04
bjaspan at web-28:~$ groups
bjaspan www-data ahops test0003shared693 test0040shared000 remotealias795
test1010shared000 test1011shared576 test1011shared981 g30000 g30001 g30002
g30003 g30004 g30005 g30006 g30007 g30008 g30009 g30010 g30011 g30012
g30013 g30014 g30015 g30016 g30017 g30018 g30019 g30020 g30021 g30022
g30023 g30024 g30025 g30026 g30027 g30028 g30029 g30030 g30031 g30032
g30033 g30034 g30035 g30036 g30037 g30038 g30039 g30040
bjaspan at web-28:~$ groups | wc
      1      50     415
bjaspan at web-28:~$ ls -ld /mnt/gfs/g* | head
drwxr-x--- 2 root g30000 6 Feb  2 21:52 /mnt/gfs/g30000
drwxr-x--- 2 root g30001 6 Feb  2 21:52 /mnt/gfs/g30001
drwxr-x--- 2 root g30002 6 Feb  2 21:52 /mnt/gfs/g30002
drwxr-x--- 2 root g30003 6 Feb  2 21:52 /mnt/gfs/g30003
drwxr-x--- 2 root g30004 6 Feb  2 21:52 /mnt/gfs/g30004
drwxr-x--- 2 root g30005 6 Feb  2 21:52 /mnt/gfs/g30005
drwxr-x--- 2 root g30006 6 Feb  2 21:52 /mnt/gfs/g30006
drwxr-x--- 2 root g30007 6 Feb  2 21:52 /mnt/gfs/g30007
drwxr-x--- 2 root g30008 6 Feb  2 21:52 /mnt/gfs/g30008
drwxr-x--- 2 root g30009 6 Feb  2 21:52 /mnt/gfs/g30009
bjaspan at web-28:~$ ls -l /mnt/gfs/g*/. > /dev/null
ls: cannot open directory /mnt/gfs/g30023/.: Permission denied
ls: cannot open directory /mnt/gfs/g30024/.: Permission denied
ls: cannot open directory /mnt/gfs/g30025/.: Permission denied
ls: cannot open directory /mnt/gfs/g30026/.: Permission denied
ls: cannot open directory /mnt/gfs/g30027/.: Permission denied
ls: cannot open directory /mnt/gfs/g30028/.: Permission denied
ls: cannot open directory /mnt/gfs/g30029/.: Permission denied
ls: cannot open directory /mnt/gfs/g30030/.: Permission denied
ls: cannot open directory /mnt/gfs/g30031/.: Permission denied
ls: cannot open directory /mnt/gfs/g30032/.: Permission denied
ls: cannot open directory /mnt/gfs/g30033/.: Permission denied
ls: cannot open directory /mnt/gfs/g30034/.: Permission denied
ls: cannot open directory /mnt/gfs/g30035/.: Permission denied
ls: cannot open directory /mnt/gfs/g30036/.: Permission denied
ls: cannot open directory /mnt/gfs/g30037/.: Permission denied
ls: cannot open directory /mnt/gfs/g30038/.: Permission denied
ls: cannot open directory /mnt/gfs/g30039/.: Permission denied
ls: cannot open directory /mnt/gfs/g30040/.: Permission denied
bjaspan at web-28:~$

Both of these systems are Ubuntu 12.04. I note that in the previous thread,
someone asserted that /proc/$$/status only contains the first 32 groups for
a user, and FUSE is using that. On Ubuntu 10.04, /proc/$$/status only
contained 32 groups (and Gluster 3.0 worked fine with >32 anyway), but on
Ubuntu 12.04, it contains all of the user's groups:

bjaspan at web-28:~$ grep Groups: /proc/$$/status
Groups: 33 2000 3004 10847 10859 10905 11017 11029 11031 30000 30001 30002
30003 30004 30005 30006 30007 30008 30009 30010 30011 30012 30013 30014
30015 30016 30017 30018 30019 30020 30021 30022 30023 30024 30025 30026
30027 30028 30029 30030 30031 30032 30033 30034 30035 30036 30037 30038
30039 30040

So this rules out /proc/$$/status as a factor. My understanding is that
Gluster 3.0 uses FUSE, which rules out FUSE as a factor.

The one factor mentioned in the thread linked above is the RPC AUTH header
structure being limited to 400 bytes. So one guess is that the wire
protocol changed between Gluster 3.0 and 3.4 changed; perhaps 3.0 did not
use Sun/ONC RPC and so is not limited to the 400-byte AUTH header. I
haven't checked the code, though, so that is just a guess.

Does anyone know *for sure* why Gluster 3.4+ is limited to 32 groups and
Gluster 3.0 was not?

Thanks,

Barry

-- 
Barry Jaspan
Chief Software Architect | Acquia <http://acquia.com>
barry.jaspan at acquia.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150202/926165d7/attachment.html>