[Gluster-devel] very strange issue with 2.0.1

[Steve]

I have a very strange issue with 2.0.1. I have 2 systems. On each system there is server AND client running. The 2 servers are using serverside afr/replicate and the client on each is connected to a single brick/volume exported on his local server. The client does not know anything about the other server.

Now when I benchmark and look what get's over the wire, then I can not get more then +/- 14MB/s. No matter what performance translators I enable/disable. The speed is always around 14MB/s.

Now if I put NUFA on top of replicate, things change. I get faster transfer but the file written does not get transferred to the other server (I see NO network traffic in that regard). But if I go on the other server and do a simple "ls", then network traffic goes up to +/- 50MB/s and the file shows up on the other server/client.

That sound sounds to me normal (well... that's probably NUFA responsible in favoring the local disc).

However... after the "ls" command on the server/client where the file was transferred to, the other server/client crashes with the following log:
+------------------------------------------------------------------------------+
[2009-06-05 22:31:48] N [afr.c:2190:notify] gfs-srv-ds-replicate: Subvolume 'gfs-srv-ds-locks' came back up; going online.
[2009-06-05 22:31:48] N [afr.c:2190:notify] gfs-srv-ds-replicate: Subvolume 'gfs-srv-ds-locks' came back up; going online.
[2009-06-05 22:31:48] N [afr.c:2190:notify] gfs-srv-ds-replicate: Subvolume 'gfs-srv-ds-locks' came back up; going online.
[2009-06-05 22:31:48] N [afr.c:2190:notify] gfs-srv-ds-replicate: Subvolume 'gfs-srv-ds-locks' came back up; going online.
[2009-06-05 22:31:48] N [glusterfsd.c:1152:main] glusterfs: Successfully started
[2009-06-05 22:31:48] N [client-protocol.c:5557:client_setvolume_cbk] gfs-srv-ds-remote: Connected to 192.168.0.77:6997, attached to remote volume 'gfs-srv-ds-locks'.
[2009-06-05 22:31:48] N [client-protocol.c:5557:client_setvolume_cbk] gfs-srv-ds-remote: Connected to 192.168.0.77:6997, attached to remote volume 'gfs-srv-ds-locks'.
[2009-06-05 22:31:51] N [server-protocol.c:7035:mop_setvolume] gfs-srv-ds-server: accepted client from 192.168.0.77:1021
[2009-06-05 22:31:51] N [server-protocol.c:7035:mop_setvolume] gfs-srv-ds-server: accepted client from 127.0.0.1:1023
[2009-06-05 22:31:51] N [server-protocol.c:7035:mop_setvolume] gfs-srv-ds-server: accepted client from 127.0.0.1:1022
[2009-06-05 22:31:51] N [server-protocol.c:7035:mop_setvolume] gfs-srv-ds-server: accepted client from 192.168.0.77:1020
pending frames:
frame : type(1) op(LOOKUP)

patchset: 5c1d9108c1529a1155963cb1911f8870a674ab5b
signal received: 11
configuration details:argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 2.0.1
[0xffffe400]
/usr/lib/glusterfs/2.0.1/xlator/protocol/client.so(client_lookup+0x96)[0xb75573fb]
/usr/lib/glusterfs/2.0.1/xlator/cluster/replicate.so(afr_lookup+0x22f)[0xb7517a05]
/usr/lib/glusterfs/2.0.1/xlator/cluster/nufa.so(nufa_lookup+0x3ea)[0xb7503fa9]
/usr/lib/glusterfs/2.0.1/xlator/performance/io-threads.so(iot_lookup_wrapper+0xa5)[0xb74e8415]
/usr/lib/libglusterfs.so.0(call_resume+0x344)[0xb7f47e01]
/usr/lib/glusterfs/2.0.1/xlator/performance/io-threads.so(iot_worker_unordered+0x20)[0xb74e5895]
/lib/libpthread.so.0[0xb7f154cf]
/lib/libc.so.6(clone+0x5e)[0xb7e9b27e]
---------


Now my questions:

1) Is this issue known? I can reproduce that error and therefore I could send more info if needed.

2) Why is Server 1 AFR/Replicate <-> Server 2 AFR/Replicate so slow? Just 14MB/s on GigE seems slow to me. Writing directly to the local disk (without GlusterFS) delivers +/- 57MB/s. Going over NFSv4 delivers +/- 45MB/s. Going over SSH delivers +/- 33MB/s. Transfer from Server 1 to Server 2 with the GlusterFS log delivers +/- 45MB/s. Just pure Server to Server with AFR/Replicate only delivers 14MB/s. Why?


// Steve


-- 
GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT!
Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01