[Gluster-users] strange error hangs hangs any access to gluster mount

[Jeff Darcy]

On 03/28/2011 12:43 PM, Burnash, James wrote:
> I am receiving an error on a client trying to access a gluster mount
> (/pfs2, in this case).
> 
> [2011-03-28 12:26:17.897887] I
> [dht-layout.c:588:dht_layout_normalize] pfs-ro1-dht: found anomalies
> in /. holes=1 overlaps=2
> 
> This is seen on the client in the /var/log/glusterfs/pfs2.log, which
> is the mount point associated with that storage.
> 
> All other clients accessing the same storage do not have the hanging
> symptom, and have no such entry in their logs.
> 
> One possibly helpful note - this node worked fine until I upgraded
> the client from 3.1.1-1 to 3.1.3-1 on the x86_64 architecture,
> running CentOS 5.2. Even after I completely uninstalled GlusterFS
> from this node and reinstalled 3.1.1-1, the problem persisted.
> 
> Here is the RPM info:
> 
> root at jc1lnxsamm33:~# rpm -qa fuse fuse-2.7.4-8.el5.x86_64 
> root at jc1lnxsamm33:~# rpm -qa "glusterfs*" 
> glusterfs-fuse-3.1.1-1.x86_64 glusterfs-core-3.1.1-1.x86_64 
> glusterfs-debuginfo-3.1.1-1.x86_64
> 
> Servers are 4 Replicated-Distribute machines running CentOS 5.5 and
> GlusterFs 3.1.3-1.
> 
> Volume Name: pfs-ro1 Type: Distributed-Replicate Status: Started 
> Number of Bricks: 20 x 2 = 40 Transport-type: tcp Bricks: Brick1:
> jc1letgfs17-pfs1:/export/read-only/g01 Brick2:
> jc1letgfs18-pfs1:/export/read-only/g01 Brick3:
> jc1letgfs17-pfs1:/export/read-only/g02 Brick4:
> jc1letgfs18-pfs1:/export/read-only/g02 ... Brick35:
> jc1letgfs14-pfs1:/export/read-only/g08 Brick36:
> jc1letgfs15-pfs1:/export/read-only/g08 Brick37:
> jc1letgfs14-pfs1:/export/read-only/g09 Brick38:
> jc1letgfs15-pfs1:/export/read-only/g09 Brick39:
> jc1letgfs14-pfs1:/export/read-only/g10 Brick40:
> jc1letgfs15-pfs1:/export/read-only/g10 Options Reconfigured: 
> performance.stat-prefetch: on performance.cache-size: 2GB 
> network.ping-timeout: 10
> 
> Any help greatly appreciated.

Can you execute the following command on each of the brick roots?

	 getfattr -d -e hex -n trusted.glusterfs.dht $brick_root

That should give a clearer picture of what the layouts look like, and
what those gaps/overlaps are.  How they happened is a bit of another
story.  I see this kind of thing pretty often, but I know it's because
of some Weird Stuff (tm) I do in CloudFS.  I'm not aware of any bugs
etc. that would cause this in other contexts.