[Gluster-users] SOLVED: some nodes mount unwritable /subvolumes/ of glusterfs (3.3b1)

[Harry Mangalam]

Answering my own question but it reveals an oddity:

The problem was that the 2 nodes (claw[58]) that showed the mount of 
the subvolume did not have the /etc/hosts entries that the rest of the 
nodes did (I know, a hanging offense to use /etc/hosts to do DNS 
bookeeping and this is one reason why).

But it seems like an bug if mounts appear to succeed instead of fail 
with the appropriate error (which did show up in the log when I went 
looking for it).

Partial successes should be prefaced with some kind of error or 
explanation; just sayin'.  Instead, the erroneous mount responded the 
same way as a successful mount and showed up as a successful mount 
with a 'df'.  A user would only know that it was a failure when he 
tried to write or compared the df value to a successful mount.

So, onwards!

Harry

On Friday 28 October 2011 14:00:54 Harry Mangalam wrote:
> Just set up and mounted a glusterfs consisting of 6 nodes:
> 
> $ gluster volume info
> 
> Volume Name: g6
> Type: Distribute
> Status: Started
> Number of Bricks: 6
> Transport-type: tcp
> Bricks:
> Brick1: pbs1:/data2
> Brick2: pbs2:/data2
> Brick3: pbs3:/data2
> Brick4: pbs3:/data
> Brick5: dabrick:/data2
> Brick6: hef:/data2
> Options Reconfigured:
> auth.allow: 128.*
> 
> 
> It's mounted on 9 similar (tho not identical) ubuntu 10.04.3 nodes
> (64b Kubuntu Dektop) in a cluster (claw[1-9]), which are running a
> self-compiled and installed version of 3.3b1. The gluster server
> bricks are also running 10.04.3 (64b server).
> 
> However, the glusterfs mounts differentially on the nodes:
> 
> root at bduc-login:~
> 576 $ for ITER in  $(seq 1 9); do    echo -n "claw${ITER}:";   ssh
> claw${ITER} 'df -h |egrep  "gluz|pbs"' ; done;
> 
> claw1:pbs3:/g6               21T  114G   21T   1% /mnt/gluz
> claw2:pbs3.calit2.uci.edu:/g6
>                        21T  114G   21T   1% /mnt/gluz
> claw3:pbs3.calit2.uci.edu:/g6
>                        21T  114G   21T   1% /mnt/gluz
> claw4:pbs3.calit2.uci.edu:/g6
>                        21T  114G   21T   1% /mnt/gluz
> claw5:pbs3.calit2.uci.edu:/g6
>                       5.5T  2.7G  5.5T   1% /mnt/gluz <---
> claw6:pbs3.calit2.uci.edu:/g6
>                        21T  112G   21T   1% /mnt/gluz
> claw7:pbs3.calit2.uci.edu:/g6
>                        21T  114G   21T   1% /mnt/gluz
> claw8:pbs3.calit2.uci.edu:/g6
>                       5.5T  2.7G  5.5T   1% /mnt/gluz <---
> claw9:pbs3.calit2.uci.edu:/g6
>                        21T  112G   21T   1% /mnt/gluz
> 
> Obviously claw5 and claw8 are ... different.  It looks like they're
> mounting a component of the glusterfs (from the numbers, it's the
> contribution of "Brick5: dabrick:/data2"), but not the whole thing.
> The glusterfs was mounted on all of them with a script:
> 
> mount -t glusterfs pbs3.calit2.uci.edu:/g6 /mnt/gluz
> 
> Ive unmounted and remounted the glusterfs on claw8 with no
> difference. I've also unmounted all instances of the glusterfs,
> then gluster- stopped and restarted the volume on the gluster
> server node and then remounted the fs as well.  No diff.
> 
> When mounted, claw[58] can see some of the files not all of them
> (the same ones that are on the 'dabrick' volume, as you might
> guess) but cannot write to the mounted filesystem, even tho the
> user has permissions to write.
> 
> ...?!
> 
> The nodes that are having the problem have never had a gluster fs
> mounted on them before.

-- 
Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
[ZOT 2225] / 92697  Google Voice Multiplexer: (949) 478-4487 
MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
--
This signature has been OCCUPIED!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20111028/bd3081c0/attachment.html>