[Gluster-users] strange error hangs hangs any access to gluster mount

[Burnash, James]

Hi Jeff.

Thanks for the quick response. You can find the output here:

http://pastebin.com/PVzBTSxq

I should note that the mirror pair jc1letgfs17 and 18 were created first, and then the pair jc1letgfs14 and 15 were added using the gluster volume add-brick command below:

gluster volume add-brick pfs-ro1 jc1letgfs14-pfs1:/export/read-only/g01 jc1letgfs15-pfs1:/export/read-only/g01 jc1letgfs14-pfs1:/export/read-only/g02 jc1letgfs15-pfs1:/export/read-only/g02 jc1letgfs14-pfs1:/export/read-only/g03 jc1letgfs15-pfs1:/export/read-only/g03 jc1letgfs14-pfs1:/export/read-only/g04 jc1letgfs15-pfs1:/export/read-only/g04 jc1letgfs14-pfs1:/export/read-only/g05 jc1letgfs15-pfs1:/export/read-only/g05 jc1letgfs14-pfs1:/export/read-only/g06 jc1letgfs15-pfs1:/export/read-only/g06 jc1letgfs14-pfs1:/export/read-only/g07 jc1letgfs15-pfs1:/export/read-only/g07 jc1letgfs14-pfs1:/export/read-only/g08 jc1letgfs15-pfs1:/export/read-only/g08 jc1letgfs14-pfs1:/export/read-only/g09 jc1letgfs15-pfs1:/export/read-only/g09 jc1letgfs14-pfs1:/export/read-only/g10 jc1letgfs15-pfs1:/export/read-only/g10

Thanks,

James Burnash, Unix Engineering

-----Original Message-----
From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Jeff Darcy
Sent: Monday, March 28, 2011 12:53 PM
To: gluster-users at gluster.org
Subject: Re: [Gluster-users] strange error hangs hangs any access to gluster mount

On 03/28/2011 12:43 PM, Burnash, James wrote:
> I am receiving an error on a client trying to access a gluster mount
> (/pfs2, in this case).
> 
> [2011-03-28 12:26:17.897887] I
> [dht-layout.c:588:dht_layout_normalize] pfs-ro1-dht: found anomalies
> in /. holes=1 overlaps=2
> 
> This is seen on the client in the /var/log/glusterfs/pfs2.log, which
> is the mount point associated with that storage.
> 
> All other clients accessing the same storage do not have the hanging
> symptom, and have no such entry in their logs.
> 
> One possibly helpful note - this node worked fine until I upgraded
> the client from 3.1.1-1 to 3.1.3-1 on the x86_64 architecture,
> running CentOS 5.2. Even after I completely uninstalled GlusterFS
> from this node and reinstalled 3.1.1-1, the problem persisted.
> 
> Here is the RPM info:
> 
> root at jc1lnxsamm33:~# rpm -qa fuse fuse-2.7.4-8.el5.x86_64 
> root at jc1lnxsamm33:~# rpm -qa "glusterfs*" 
> glusterfs-fuse-3.1.1-1.x86_64 glusterfs-core-3.1.1-1.x86_64 
> glusterfs-debuginfo-3.1.1-1.x86_64
> 
> Servers are 4 Replicated-Distribute machines running CentOS 5.5 and
> GlusterFs 3.1.3-1.
> 
> Volume Name: pfs-ro1 Type: Distributed-Replicate Status: Started 
> Number of Bricks: 20 x 2 = 40 Transport-type: tcp Bricks: Brick1:
> jc1letgfs17-pfs1:/export/read-only/g01 Brick2:
> jc1letgfs18-pfs1:/export/read-only/g01 Brick3:
> jc1letgfs17-pfs1:/export/read-only/g02 Brick4:
> jc1letgfs18-pfs1:/export/read-only/g02 ... Brick35:
> jc1letgfs14-pfs1:/export/read-only/g08 Brick36:
> jc1letgfs15-pfs1:/export/read-only/g08 Brick37:
> jc1letgfs14-pfs1:/export/read-only/g09 Brick38:
> jc1letgfs15-pfs1:/export/read-only/g09 Brick39:
> jc1letgfs14-pfs1:/export/read-only/g10 Brick40:
> jc1letgfs15-pfs1:/export/read-only/g10 Options Reconfigured: 
> performance.stat-prefetch: on performance.cache-size: 2GB 
> network.ping-timeout: 10
> 
> Any help greatly appreciated.

Can you execute the following command on each of the brick roots?

	 getfattr -d -e hex -n trusted.glusterfs.dht $brick_root

That should give a clearer picture of what the layouts look like, and
what those gaps/overlaps are.  How they happened is a bit of another
story.  I see this kind of thing pretty often, but I know it's because
of some Weird Stuff (tm) I do in CloudFS.  I'm not aware of any bugs
etc. that would cause this in other contexts.
_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users


DISCLAIMER: 
This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this in error, please immediately notify me and permanently delete the original and any copy of any e-mail and any printout thereof. E-mail transmission cannot be guaranteed to be secure or error-free. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. 
NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may, at its discretion, monitor and review the content of all e-mail communications. http://www.knight.com