[Gluster-devel] very strange issue with 2.0.1

[Steve]

-------- Original-Nachricht --------
> Datum: Fri, 05 Jun 2009 22:54:49 +0200
> Von: "Steve" <steeeeeveee at gmx.net>
> An: gluster-devel at nongnu.org
> Betreff: [Gluster-devel] very strange issue with 2.0.1

> I have a very strange issue with 2.0.1. I have 2 systems. On each system
> there is server AND client running. The 2 servers are using serverside
> afr/replicate and the client on each is connected to a single brick/volume
> exported on his local server. The client does not know anything about the other
> server.
> 
> Now when I benchmark and look what get's over the wire, then I can not get
> more then +/- 14MB/s. No matter what performance translators I
> enable/disable. The speed is always around 14MB/s.
> 
> Now if I put NUFA on top of replicate, things change. I get faster
> transfer but the file written does not get transferred to the other server (I see
> NO network traffic in that regard). But if I go on the other server and do
> a simple "ls", then network traffic goes up to +/- 50MB/s and the file
> shows up on the other server/client.
> 
> That sound sounds to me normal (well... that's probably NUFA responsible
> in favoring the local disc).
> 
> However... after the "ls" command on the server/client where the file was
> transferred to, the other server/client crashes with the following log:
> +------------------------------------------------------------------------------+
> [2009-06-05 22:31:48] N [afr.c:2190:notify] gfs-srv-ds-replicate:
> Subvolume 'gfs-srv-ds-locks' came back up; going online.
> [2009-06-05 22:31:48] N [afr.c:2190:notify] gfs-srv-ds-replicate:
> Subvolume 'gfs-srv-ds-locks' came back up; going online.
> [2009-06-05 22:31:48] N [afr.c:2190:notify] gfs-srv-ds-replicate:
> Subvolume 'gfs-srv-ds-locks' came back up; going online.
> [2009-06-05 22:31:48] N [afr.c:2190:notify] gfs-srv-ds-replicate:
> Subvolume 'gfs-srv-ds-locks' came back up; going online.
> [2009-06-05 22:31:48] N [glusterfsd.c:1152:main] glusterfs: Successfully
> started
> [2009-06-05 22:31:48] N [client-protocol.c:5557:client_setvolume_cbk]
> gfs-srv-ds-remote: Connected to 192.168.0.77:6997, attached to remote volume
> 'gfs-srv-ds-locks'.
> [2009-06-05 22:31:48] N [client-protocol.c:5557:client_setvolume_cbk]
> gfs-srv-ds-remote: Connected to 192.168.0.77:6997, attached to remote volume
> 'gfs-srv-ds-locks'.
> [2009-06-05 22:31:51] N [server-protocol.c:7035:mop_setvolume]
> gfs-srv-ds-server: accepted client from 192.168.0.77:1021
> [2009-06-05 22:31:51] N [server-protocol.c:7035:mop_setvolume]
> gfs-srv-ds-server: accepted client from 127.0.0.1:1023
> [2009-06-05 22:31:51] N [server-protocol.c:7035:mop_setvolume]
> gfs-srv-ds-server: accepted client from 127.0.0.1:1022
> [2009-06-05 22:31:51] N [server-protocol.c:7035:mop_setvolume]
> gfs-srv-ds-server: accepted client from 192.168.0.77:1020
> pending frames:
> frame : type(1) op(LOOKUP)
> 
> patchset: 5c1d9108c1529a1155963cb1911f8870a674ab5b
> signal received: 11
> configuration details:argp 1
> backtrace 1
> dlfcn 1
> fdatasync 1
> libpthread 1
> llistxattr 1
> setfsid 1
> spinlock 1
> xattr.h 1
> st_atim.tv_nsec 1
> package-string: glusterfs 2.0.1
> [0xffffe400]
> /usr/lib/glusterfs/2.0.1/xlator/protocol/client.so(client_lookup+0x96)[0xb75573fb]
> /usr/lib/glusterfs/2.0.1/xlator/cluster/replicate.so(afr_lookup+0x22f)[0xb7517a05]
> /usr/lib/glusterfs/2.0.1/xlator/cluster/nufa.so(nufa_lookup+0x3ea)[0xb7503fa9]
> /usr/lib/glusterfs/2.0.1/xlator/performance/io-threads.so(iot_lookup_wrapper+0xa5)[0xb74e8415]
> /usr/lib/libglusterfs.so.0(call_resume+0x344)[0xb7f47e01]
> /usr/lib/glusterfs/2.0.1/xlator/performance/io-threads.so(iot_worker_unordered+0x20)[0xb74e5895]
> /lib/libpthread.so.0[0xb7f154cf]
> /lib/libc.so.6(clone+0x5e)[0xb7e9b27e]
> ---------
> 
> 
> Now my questions:
> 
> 1) Is this issue known? I can reproduce that error and therefore I could
> send more info if needed.
> 
> 2) Why is Server 1 AFR/Replicate <-> Server 2 AFR/Replicate so slow? Just
> 14MB/s on GigE seems slow to me. Writing directly to the local disk
> (without GlusterFS) delivers +/- 57MB/s. Going over NFSv4 delivers +/- 45MB/s.
> Going over SSH delivers +/- 33MB/s. Transfer from Server 1 to Server 2 with
> the GlusterFS log delivers +/- 45MB/s. Just pure Server to Server with
> AFR/Replicate only delivers 14MB/s. Why?
> 
Ach! I did more testing and the conclusion is:
Using 128KB chunks for writing changes the speed. Local disk write is then almost 90MB/s and GlusterFS is around 45MB/s. I guess I have no real speed issue. I would love to come close to 90MB/s but 45MB/s is fine.

However... the problem with the crash is still there if using NUFA. Maybe I just messed up and tried to many different (and obscure) combinations because NUFA never failed on me in the past?


> // Steve
> 
// Steve
-- 
GRATIS für alle GMX-Mitglieder: Die maxdome Movie-FLAT!
Jetzt freischalten unter http://portal.gmx.net/de/go/maxdome01