[Gluster-users] some nodes mount unwritable /subvolumes/ of glusterfs (3.3b1)

Harry Mangalam harry.mangalam at uci.edu
Fri Oct 28 21:00:54 UTC 2011

Just set up and mounted a glusterfs consisting of 6 nodes:

$ gluster volume info 

Volume Name: g6
Type: Distribute
Status: Started
Number of Bricks: 6
Transport-type: tcp
Brick1: pbs1:/data2
Brick2: pbs2:/data2
Brick3: pbs3:/data2
Brick4: pbs3:/data
Brick5: dabrick:/data2
Brick6: hef:/data2
Options Reconfigured:
auth.allow: 128.*

It's mounted on 9 similar (tho not identical) ubuntu 10.04.3 nodes 
(64b Kubuntu Dektop) in a cluster (claw[1-9]), which are running a 
self-compiled and installed version of 3.3b1. The gluster server 
bricks are also running 10.04.3 (64b server).

However, the glusterfs mounts differentially on the nodes:

root at bduc-login:~
576 $ for ITER in  $(seq 1 9); do    echo -n "claw${ITER}:";   ssh 
claw${ITER} 'df -h |egrep  "gluz|pbs"' ; done; 

claw1:pbs3:/g6               21T  114G   21T   1% /mnt/gluz
                       21T  114G   21T   1% /mnt/gluz
                       21T  114G   21T   1% /mnt/gluz
                       21T  114G   21T   1% /mnt/gluz
                      5.5T  2.7G  5.5T   1% /mnt/gluz <---
                       21T  112G   21T   1% /mnt/gluz
                       21T  114G   21T   1% /mnt/gluz
                      5.5T  2.7G  5.5T   1% /mnt/gluz <---
                       21T  112G   21T   1% /mnt/gluz

Obviously claw5 and claw8 are ... different.  It looks like they're 
mounting a component of the glusterfs (from the numbers, it's the 
contribution of "Brick5: dabrick:/data2"), but not the whole thing. 
The glusterfs was mounted on all of them with a script:

mount -t glusterfs pbs3.calit2.uci.edu:/g6 /mnt/gluz

Ive unmounted and remounted the glusterfs on claw8 with no difference.
I've also unmounted all instances of the glusterfs, then gluster-
stopped and restarted the volume on the gluster server node and then 
remounted the fs as well.  No diff.

When mounted, claw[58] can see some of the files not all of them (the 
same ones that are on the 'dabrick' volume, as you might guess) but 
cannot write to the mounted filesystem, even tho the user has 
permissions to write.


The nodes that are having the problem have never had a gluster fs 
mounted on them before.
Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
[ZOT 2225] / 92697  Google Voice Multiplexer: (949) 478-4487 
MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
This signature has been OCCUPIED!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20111028/d589fa81/attachment.html>

More information about the Gluster-users mailing list