[Gluster-users] strange error hangs hangs any access to gluster mount

[Burnash, James]

I am receiving an error on a client trying to access a gluster mount (/pfs2, in this case).

[2011-03-28 12:26:17.897887] I [dht-layout.c:588:dht_layout_normalize] pfs-ro1-dht: found anomalies in /. holes=1 overlaps=2

This is seen on the client in the /var/log/glusterfs/pfs2.log, which is the mount point associated with that storage.

All other clients accessing the same storage do not have the hanging symptom, and have no such entry in their logs.

One possibly helpful note - this node worked fine until I upgraded the client from 3.1.1-1 to 3.1.3-1 on the x86_64 architecture, running CentOS 5.2. Even after I completely uninstalled GlusterFS from this node and reinstalled 3.1.1-1, the problem persisted.

Here is the RPM info:

root at jc1lnxsamm33:~# rpm -qa fuse
fuse-2.7.4-8.el5.x86_64
root at jc1lnxsamm33:~# rpm -qa "glusterfs*"
glusterfs-fuse-3.1.1-1.x86_64
glusterfs-core-3.1.1-1.x86_64
glusterfs-debuginfo-3.1.1-1.x86_64

Servers are 4 Replicated-Distribute machines running CentOS 5.5 and GlusterFs 3.1.3-1.

Volume Name: pfs-ro1
Type: Distributed-Replicate
Status: Started
Number of Bricks: 20 x 2 = 40
Transport-type: tcp
Bricks:
Brick1: jc1letgfs17-pfs1:/export/read-only/g01
Brick2: jc1letgfs18-pfs1:/export/read-only/g01
Brick3: jc1letgfs17-pfs1:/export/read-only/g02
Brick4: jc1letgfs18-pfs1:/export/read-only/g02
...
Brick35: jc1letgfs14-pfs1:/export/read-only/g08
Brick36: jc1letgfs15-pfs1:/export/read-only/g08
Brick37: jc1letgfs14-pfs1:/export/read-only/g09
Brick38: jc1letgfs15-pfs1:/export/read-only/g09
Brick39: jc1letgfs14-pfs1:/export/read-only/g10
Brick40: jc1letgfs15-pfs1:/export/read-only/g10
Options Reconfigured:
performance.stat-prefetch: on
performance.cache-size: 2GB
network.ping-timeout: 10

Any help greatly appreciated.

Thanks,

James Burnash, Unix Engineering



DISCLAIMER: 
This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this in error, please immediately notify me and permanently delete the original and any copy of any e-mail and any printout thereof. E-mail transmission cannot be guaranteed to be secure or error-free. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. 
NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may, at its discretion, monitor and review the content of all e-mail communications. http://www.knight.com