[Gluster-users] Permission denied after upgrade and rebalancing

[Jochen Lillich]

Hi,

when our two-node distributed backup storage cluster, running on
GlusterFS 3.0.2 (Ubuntu Lucid packages), started running out of space,
I added a third node and tried rebalancing them. The scale-n-defrag
script didn't work, though. I always got a "stale NFS handle".

So I decided to upgrade and installed the 3.2.4 packages from
semiosis' Launchpad PPA, followed the upgrade guide for 3.0 -> 3.2 and
then started "gluster volume rebalance". After two days, the three
nodes are now about equally full, so the rebalancing worked.

Unfortunately, it seems that it also broke the cluster filesystem. We
get a lot of "permission denied" errors although file system
permissions are correct. The log contains a lot of these entries:

[2011-10-17 10:11:48.979801] I
[client3_1-fops.c:2228:client3_1_lookup_cbk] 0-glfs-client-2: remote
operation failed: Permission denied
[2011-10-17 10:11:48.979892] I [dht-common.c:478:dht_revalidate_cbk]
0-glfs-dht: subvolume glfs-client-2 for /bacula/mail02 returned -1
(Permission denied)
[2011-10-17 10:11:48.980216] I [dht-layout.c:192:dht_layout_search]
0-glfs-dht: no subvolume for hash (value) = 1486987642
[2011-10-17 10:11:48.981101] I
[client3_1-fops.c:2228:client3_1_lookup_cbk] 0-glfs-client-2: remote
operation failed: Permission denied
[2011-10-17 10:11:48.981581] I [dht-layout.c:192:dht_layout_search]
0-glfs-dht: no subvolume for hash (value) = 1486987642
[2011-10-17 10:11:48.982468] I
[client3_1-fops.c:2228:client3_1_lookup_cbk] 0-glfs-client-2: remote
operation failed: Permission denied

How can I repair this?

Best regards,
 Jochen