[Gluster-users] Load goes up every 3-5 days

[Tobias Wilken]

Hey all,

I have a curious problem with a simple glusterfs Installation. After about
3-5 days the load of the glusterfs process goes up to 100% CPU usage and
takes about 1 hour until it comes back to normal operation. Actions on the
glusterfs mount point are mainly creating, reading and writing into bzr
repositories, but I can't assoziate any action to the problem. It occurs on
operations which I done multiple times before without problems.

The configuration files are created by:
glusterfs-volgen --name repstore1 --raid 1 hostname1:/data/share
hostname2:/data/share
Only the iocache cache-size is reduced to 256MB and a user/password
authentication is added and of course the hostnames are adapted. I'm using
two glusterfs daemons/server as replication and the glusterfs is mounted on
these nodes, too. Additional two nodes have mounted the glusterfs. As
version glusterfs 3.0.4 is used, compiled without any special flags.

The only specialty the nodes are amazon ec2 instances, but I don't think it
should make a difference.

The last week I tried very hard to reproduce the problem by settings the
nodes under cpu load, memory usage, stressing the filesystem with writing
and reading and destroying the network connection and many permutations of
that :-), but I can't reproduce it. Today a simple bzr export operation
"crashes" it again.

Any idea how I can reproduce such a problem for further debugging?? Any
other ideas? Maybe some pity? :-)

Best regards
Tobias Wilken

P.S. The logs from the today "crash", on Host1 the load of the glusterfs
process goes up.
Host1:
/var/log/glusterfs/data-share.log
[2010-06-14 08:52:07] W [fuse-bridge.c:793:fuse_getattr] glusterfs-fuse:
14750285: GETATTR 140270343725136 (fuse_loc_fill() failed)
[2010-06-14 08:52:07] W [fuse-bridge.c:1529:fuse_rename_cbk] glusterfs-fuse:
14750288:
/applications/moodletest/repository/.bzr/branch/lock/jodfk6p0iu.tmp ->
/applications/moodletest/repository/.bzr/branch/lock/held => -1 (Directory
not empty)
[2010-06-14 08:52:07] W [fuse-bridge.c:793:fuse_getattr] glusterfs-fuse:
14750295: GETATTR 140270343725136 (fuse_loc_fill() failed)
[2010-06-14 08:52:07] W [fuse-bridge.c:793:fuse_getattr] glusterfs-fuse:
14750298: GETATTR 140270343725136 (fuse_loc_fill() failed)
[2010-06-14 08:52:07] W [fuse-bridge.c:1529:fuse_rename_cbk] glusterfs-fuse:
14750300:
/applications/moodletest/repository/.bzr/branch/lock/jhogp2wpi2.tmp ->
/applications/moodletest/repository/.bzr/branch/lock/held => -1 (Directory
not empty)
[2010-06-14 08:52:07] W [fuse-bridge.c:1529:fuse_rename_cbk] glusterfs-fuse:
14750303:
/applications/moodletest/repository/.bzr/branch/lock/18ytmffmsv.tmp ->
/applications/moodletest/repository/.bzr/branch/lock/held => -1 (Directory
not empty)
[2010-06-14 08:52:07] W [fuse-bridge.c:793:fuse_getattr] glusterfs-fuse:
14750308: GETATTR 140270343725136 (fuse_loc_fill() failed)
[2010-06-14 08:52:07] W [fuse-bridge.c:1529:fuse_rename_cbk] glusterfs-fuse:
14750311:
/applications/moodletest/repository/.bzr/branch/lock/lzrpezkqno.tmp ->
/applications/moodletest/repository/.bzr/branch/lock/held => -1 (Directory
not empty)
[2010-06-14 08:52:47] W [fuse-bridge.c:722:fuse_attr_cbk] glusterfs-fuse:
14756735: FSTAT() ERR => -1 (File descriptor in bad state)


/var/log/glusterfs/glusterfsd.log
[2010-06-14 08:52:15] N [server-protocol.c:6788:notify] server-tcp:
10.227.26.95:1017 disconnected
[2010-06-14 08:52:15] N [server-protocol.c:6788:notify] server-tcp:
10.227.26.95:1016 disconnected
[2010-06-14 08:52:15] N [server-helpers.c:842:server_connection_destroy]
server-tcp: destroyed connection of
ip-10-227-26-95-6105-2010/06/10-01:28:54:815100-hostname2-1

Host2:
/var/log/glusterfs/data-share.log
[2010-06-14 08:52:15] W [fuse-bridge.c:1848:fuse_readv_cbk] glusterfs-fuse:
16468676: READ => -1 (File descriptor in bad state)
pending frames:

patchset: v3.0.4
signal received: 6
time of crash: 2010-06-14 08:52:15
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.0.4
/lib/libc.so.6(+0x33af0)[0x7ff48e9a3af0]
/lib/libc.so.6(gsignal+0x35)[0x7ff48e9a3a75]
/lib/libc.so.6(abort+0x180)[0x7ff48e9a75c0]
/lib/libc.so.6(+0x6d4fb)[0x7ff48e9dd4fb]
/lib/libc.so.6(+0x775b6)[0x7ff48e9e75b6]
/lib/libc.so.6(cfree+0x73)[0x7ff48e9ede53]
/usr/local/lib/glusterfs/3.0.4/xlator/performance/quick-read.so(qr_readv+0x252)[0x7ff48d4e3842]
/usr/local/lib/glusterfs/3.0.4/xlator/performance/stat-prefetch.so(sp_readv+0x142)[0x7ff48d2d2082]
/usr/local/lib/glusterfs/3.0.4/xlator/mount/fuse.so(+0x5fa7)[0x7ff48d0b7fa7]
/usr/local/lib/glusterfs/3.0.4/xlator/mount/fuse.so(+0x4b54)[0x7ff48d0b6b54]
/lib/libpthread.so.0(+0x69ca)[0x7ff48ecf89ca]
/lib/libc.so.6(clone+0x6d)[0x7ff48ea566cd]
---------

/var/log/glusterfs/glusterfsd.log
[2010-06-14 08:52:15] N [server-protocol.c:6788:notify] server-tcp:
10.227.26.95:1019 disconnected
[2010-06-14 08:52:15] N [server-protocol.c:6788:notify] server-tcp:
10.227.26.95:1018 disconnected
[2010-06-14 08:52:15] N [server-helpers.c:842:server_connection_destroy]
server-tcp: destroyed connection of
ip-10-227-26-95-6105-2010/06/10-01:28:54:815100-hostname1-1