[Gluster-users] Random disconnect

Fri Apr 13 09:59:01 UTC 2012

Amar,

I'm facing an other issue, don't know if there is a link with the quota issue.

I got random disconnections during worload : 

I use the same scripts as previoulsy. (https://bugzilla.redhat.com/show_bug.cgi?id=812230 )

Config info :
I'm accessing the mount point with nfs. I'm connected to ylal3510

10.80.49.10 ylal3510
10.80.49.11 ylal3520
10.80.49.12 ylal3530
10.80.49.13 ylal3540

Volume Name: poolsave
Type: Distributed-Replicate
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: ylal3510:/users3/poolsave
Brick2: ylal3530:/users3/poolsave
Brick3: ylal3520:/users3/poolsave
Brick4: ylal3540:/users3/poolsave
Options Reconfigured:
features.limit-usage: /yval9000:1TB
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
performance.io-thread-count: 64
nfs.port: 2049
performance.cache-refresh-timeout: 2
performance.cache-max-file-size: 4GB
performance.cache-min-file-size: 1KB
network.ping-timeout: 60
performance.cache-size: 6GB
features.quota: on
features.quota-timeout: 30
nfs.enable-ino32: off

Here is the log on the first storage node (ylal3510): nfs.log

First disconnection : 

[2012-04-12 03:01:40.700893] W [quota.c:2165:quota_fstat_cbk] 0-poolsave-quota: quota context not set in inode (ino:-2126980194, gfid:ec0c049e-78b9-4972-b14c-2fa60f503e29)
[2012-04-12 03:01:40.708029] W [quota.c:2165:quota_fstat_cbk] 0-poolsave-quota: quota context not set in inode (ino:-2126980194, gfid:ec0c049e-78b9-4972-b14c-2fa60f503e29)

[2012-04-12 03:09:17.454283] C [client-handshake.c:121:rpc_client_ping_timer_expired] 0-poolsave-client-1: server 10.80.49.12:24010 has not responded in the last 30 seconds, disconnecting.
[2012-04-12 03:09:17.455307] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame type(GlusterFS 3.1) op(FSYNC(16)) called at 2012-04-12 03:08:17.264052
[2012-04-12 03:09:17.455413] I [client3_1-fops.c:779:client3_1_fsync_cbk] 0-poolsave-client-1: remote operation failed: Transport endpoint is not connected
[2012-04-12 03:09:17.480558] I [socket.c:2275:socket_submit_request] 0-poolsave-client-1: not connected (priv->connected = 0)
[2012-04-12 03:09:17.480659] W [rpc-clnt.c:1417:rpc_clnt_submit] 0-poolsave-client-1: failed to submit rpc-request (XID: 0x10709557x Program: GlusterFS 3.1, ProgVers: 310, Proc: 16) to rpc-transport (poolsave-client-1)
[2012-04-12 03:09:17.480710] I [client3_1-fops.c:779:client3_1_fsync_cbk] 0-poolsave-client-1: remote operation failed: Transport endpoint is not connected
[2012-04-12 03:09:17.480875] W [rpc-clnt.c:1417:rpc_clnt_submit] 0-poolsave-client-1: failed to submit rpc-request (XID: 0x10709558x Program: GlusterFS 3.1, ProgVers: 310, Proc: 16) to rpc-transport (poolsave-client-1)
[2012-04-12 03:09:17.480918] I [client3_1-fops.c:779:client3_1_fsync_cbk] 0-poolsave-client-1: remote operation failed: Transport endpoint is not connected
[2012-04-12 03:09:17.481006] W [rpc-clnt.c:1417:rpc_clnt_submit] 0-poolsave-client-1: failed to submit rpc-request (XID: 0x10709559x Program: GlusterFS 3.1, ProgVers: 310, Proc: 16) to rpc-transport (poolsave-client-1)
[2012-04-12 03:09:17.481117] I [client3_1-fops.c:779:client3_1_fsync_cbk] 0-poolsave-client-1: remote operation failed: Transport endpoint is not connected
[2012-04-12 03:09:17.481198] W [rpc-clnt.c:1417:rpc_clnt_submit] 0-poolsave-client-1: failed to submit rpc-request (XID: 0x10709560x Program: GlusterFS 3.1, ProgVers: 310, Proc: 16) to rpc-transport (poolsave-client-1)
[2012-04-12 03:09:17.481237] I [client3_1-fops.c:779:client3_1_fsync_cbk] 0-poolsave-client-1: remote operation failed: Transport endpoint is not connected
[2012-04-12 03:09:17.481326] W [rpc-clnt.c:1417:rpc_clnt_submit] 0-poolsave-client-1: failed to submit rpc-request (XID: 0x10709561x Program: GlusterFS 3.1, ProgVers: 310, Proc: 16) to rpc-transport (poolsave-client-1)
[2012-04-12 03:09:17.481364] I [client3_1-fops.c:779:client3_1_fsync_cbk] 0-poolsave-client-1: remote operation failed: Transport endpoint is not connected
[2012-04-12 03:09:17.481448] W [rpc-clnt.c:1417:rpc_clnt_submit] 0-poolsave-client-1: failed to submit rpc-request (XID: 0x10709562x Program: GlusterFS 3.1, ProgVers: 310, Proc: 16) to rpc-transport (poolsave-client-1)
[2012-04-12 03:09:17.481487] I [client3_1-fops.c:779:client3_1_fsync_cbk] 0-poolsave-client-1: remote operation failed: Transport endpoint is not connected
[2012-04-12 03:09:17.481565] W [rpc-clnt.c:1417:rpc_clnt_submit] 0-poolsave-client-1: failed to submit rpc-request (XID: 0x10709563x Program: GlusterFS 3.1, ProgVers: 310, Proc: 16) to rpc-transport (poolsave-client-1)
[2012-04-12 03:09:17.481604] I [client3_1-fops.c:779:client3_1_fsync_cbk] 0-poolsave-client-1: remote operation failed: Transport endpoint is not connected
[2012-04-12 03:09:17.481820] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-04-12 03:08:37.235705
[2012-04-12 03:09:17.481872] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-poolsave-client-1: remote operation failed: Transport endpoint is not connected
[2012-04-12 03:09:17.482093] W [rpc-clnt.c:1417:rpc_clnt_submit] 0-poolsave-client-1: failed to submit rpc-request (XID: 0x10709564x Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport (poolsave-client-1)
[2012-04-12 03:09:17.482198] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-04-12 03:08:37.235895
[2012-04-12 03:09:17.482243] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-poolsave-client-1: remote operation failed: Transport endpoint is not connected
[2012-04-12 03:09:17.482417] W [rpc-clnt.c:1417:rpc_clnt_submit] 0-poolsave-client-1: failed to submit rpc-request (XID: 0x10709565x Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport (poolsave-client-1)
[2012-04-12 03:09:17.482513] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.236009
[2012-04-12 03:09:17.482562] E [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote operation failed: Transport endpoint is not connected
[2012-04-12 03:09:17.482650] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.236370
[2012-04-12 03:09:17.482722] E [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote operation failed: Transport endpoint is not connected
[2012-04-12 03:09:17.482856] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.236455
[2012-04-12 03:09:17.482901] E [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote operation failed: Transport endpoint is not connected
[2012-04-12 03:09:17.482986] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.236631
[2012-04-12 03:09:17.483028] E [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote operation failed: Transport endpoint is not connected
[2012-04-12 03:09:17.483147] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.237248
[2012-04-12 03:09:17.483191] E [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote operation failed: Transport endpoint is not connected
[2012-04-12 03:09:17.483357] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.237537
[2012-04-12 03:09:17.483402] E [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote operation failed: Transport endpoint is not connected
[2012-04-12 03:09:17.483523] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.237664
[2012-04-12 03:09:17.483568] E [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote operation failed: Transport endpoint is not connected
[2012-04-12 03:09:17.483689] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.237960
[2012-04-12 03:09:17.483734] E [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote operation failed: Transport endpoint is not connected
[2012-04-12 03:09:17.483855] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.238258
[2012-04-12 03:09:17.483924] E [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote operation failed: Transport endpoint is not connected
[2012-04-12 03:09:17.484045] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.238554
[2012-04-12 03:09:17.484090] E [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote operation failed: Transport endpoint is not connected
[2012-04-12 03:09:17.484207] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.238848
[2012-04-12 03:09:17.484252] E [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote operation failed: Transport endpoint is not connected
[2012-04-12 03:09:17.484373] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.239070
[2012-04-12 03:09:17.484417] E [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote operation failed: Transport endpoint is not connected
[2012-04-12 03:09:17.484536] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame type(GlusterFS Handshake) op(PING(3)) called at 2012-04-12 03:08:47.450735
[2012-04-12 03:09:17.484587] W [client-handshake.c:264:client_ping_cbk] 0-poolsave-client-1: timer must have expired
[2012-04-12 03:09:17.484622] I [client.c:1883:client_rpc_notify] 0-poolsave-client-1: disconnected
[2012-04-12 03:09:17.485639] W [client3_1-fops.c:4868:client3_1_finodelk] 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD
[2012-04-12 03:09:17.485717] W [client3_1-fops.c:4915:client3_1_finodelk] 0-poolsave-client-1: failed to send the fop: File descriptor in bad state
[2012-04-12 03:09:17.485834] W [client3_1-fops.c:4868:client3_1_finodelk] 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD
[2012-04-12 03:09:17.485873] W [client3_1-fops.c:4915:client3_1_finodelk] 0-poolsave-client-1: failed to send the fop: File descriptor in bad state
[2012-04-12 03:09:17.486432] W [client3_1-fops.c:4868:client3_1_finodelk] 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD
[2012-04-12 03:09:17.486471] W [client3_1-fops.c:4915:client3_1_finodelk] 0-poolsave-client-1: failed to send the fop: File descriptor in bad state
[2012-04-12 03:09:17.486620] W [client3_1-fops.c:4868:client3_1_finodelk] 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD
[2012-04-12 03:09:17.486659] W [client3_1-fops.c:4915:client3_1_finodelk] 0-poolsave-client-1: failed to send the fop: File descriptor in bad state
[2012-04-12 03:09:17.486820] W [client3_1-fops.c:4868:client3_1_finodelk] 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD
[2012-04-12 03:09:17.486933] W [client3_1-fops.c:4915:client3_1_finodelk] 0-poolsave-client-1: failed to send the fop: File descriptor in bad state
[2012-04-12 03:09:17.487075] W [client3_1-fops.c:4868:client3_1_finodelk] 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD
[2012-04-12 03:09:17.487120] W [client3_1-fops.c:4915:client3_1_finodelk] 0-poolsave-client-1: failed to send the fop: File descriptor in bad state
[2012-04-12 03:09:17.487245] W [client3_1-fops.c:4868:client3_1_finodelk] 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD
[2012-04-12 03:09:17.487283] W [client3_1-fops.c:4915:client3_1_finodelk] 0-poolsave-client-1: failed to send the fop: File descriptor in bad state
[2012-04-12 03:09:17.487429] W [client3_1-fops.c:4868:client3_1_finodelk] 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD
[2012-04-12 03:09:17.487489] W [client3_1-fops.c:4915:client3_1_finodelk] 0-poolsave-client-1: failed to send the fop: File descriptor in bad state
[2012-04-12 03:09:17.487628] W [client3_1-fops.c:4868:client3_1_finodelk] 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD
[2012-04-12 03:09:17.487671] W [client3_1-fops.c:4915:client3_1_finodelk] 0-poolsave-client-1: failed to send the fop: File descriptor in bad state
[2012-04-12 03:09:17.487827] W [client3_1-fops.c:4868:client3_1_finodelk] 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD
[2012-04-12 03:09:17.487868] W [client3_1-fops.c:4915:client3_1_finodelk] 0-poolsave-client-1: failed to send the fop: File descriptor in bad state
[2012-04-12 03:09:17.488009] W [client3_1-fops.c:4868:client3_1_finodelk] 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD
[2012-04-12 03:09:17.488046] W [client3_1-fops.c:4915:client3_1_finodelk] 0-poolsave-client-1: failed to send the fop: File descriptor in bad state
[2012-04-12 03:09:17.488174] W [client3_1-fops.c:4868:client3_1_finodelk] 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD
[2012-04-12 03:09:17.488211] W [client3_1-fops.c:4915:client3_1_finodelk] 0-poolsave-client-1: failed to send the fop: File descriptor in bad state
[2012-04-12 03:09:23.347623] I [client-handshake.c:1090:select_server_supported_programs] 0-poolsave-client-1: Using Program GlusterFS 3.2.5, Num (1298437), Version (310)
[2012-04-12 03:09:23.348507] I [client-handshake.c:913:client_setvolume_cbk] 0-poolsave-client-1: Connected to 10.80.49.12:24010, attached to remote volume '/users3/poolsave'.
[2012-04-12 03:09:23.348540] I [client-handshake.c:779:client_post_handshake] 0-poolsave-client-1: 3 fds open - Delaying child_up until they are re-opened
[2012-04-12 03:09:23.348632] W [inode.c:1044:inode_path] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/client.so(client_setvolume_cbk+0x1b2) [0x7fc2a205aee2] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/client.so(client_post_handshake+0x193) [0x7fc2a205acf3] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/client.so(protocol_client_reopen+0x75) [0x7fc2a205a1b5]))) 0-poolsave/inode: no dentry for non-root inode -6906554876: d0344eb6-6f41-4d14-8855-e94fa9d541b3
[2012-04-12 03:09:23.348665] W [client-handshake.c:683:protocol_client_reopen] 0-poolsave-client-1: couldn't build path from inode d0344eb6-6f41-4d14-8855-e94fa9d541b3
[2012-04-12 03:09:23.348680] E [client-handshake.c:722:protocol_client_reopen] 0-: failed to send the re-open request
[2012-04-12 03:09:23.348724] W [inode.c:1044:inode_path] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/client.so(client_setvolume_cbk+0x1b2) [0x7fc2a205aee2] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/client.so(client_post_handshake+0x193) [0x7fc2a205acf3] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/client.so(protocol_client_reopen+0x75) [0x7fc2a205a1b5]))) 0-poolsave/inode: no dentry for non-root inode 800038244: 70c18c81-cc2d-4698-b28d-2fba654a9ef8
[2012-04-12 03:09:23.348749] W [client-handshake.c:683:protocol_client_reopen] 0-poolsave-client-1: couldn't build path from inode 70c18c81-cc2d-4698-b28d-2fba654a9ef8
[2012-04-12 03:09:23.348802] E [client-handshake.c:722:protocol_client_reopen] 0-: failed to send the re-open request
[2012-04-12 03:09:23.349450] I [client-lk.c:617:decrement_reopen_fd_count] 0-poolsave-client-1: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2012-04-12 03:09:23.352548] I [afr-open.c:432:afr_openfd_sh] 0-poolsave-replicate-0:  data missing-entry gfid self-heal triggered. path: /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up down flush, data lock is held
[2012-04-12 03:09:23.354603] I [afr-self-heal-common.c:1203:sh_missing_entries_create] 0-poolsave-replicate-0: no missing files - /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to metadata check
[2012-04-12 03:09:23.356460] W [dict.c:418:dict_unref] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0xa2) [0x7fc2a3fa6712] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/client.so(client3_1_fstat_cbk+0x2c9) [0x7fc2a20551a9] (-->/usr/local/lib//glusterfs/3.2.5/xlator/cluster/replicate.so(afr_sh_data_fstat_cbk+0x1d0) [0x7fc2a1efa590]))) 0-dict: dict is NULL
[2012-04-12 03:09:41.442063] I [afr-self-heal-algorithm.c:520:sh_diff_loop_driver_done] 0-poolsave-replicate-0: diff self-heal on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar: completed. (5249 blocks of 73670 were different (7.13%))
[2012-04-12 03:09:41.443242] I [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] 0-poolsave-replicate-0: background  data missing-entry gfid self-heal completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
[2012-04-12 03:09:41.443660] I [afr-open.c:432:afr_openfd_sh] 0-poolsave-replicate-0:  data missing-entry gfid self-heal triggered. path: /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up down flush, data lock is held
[2012-04-12 03:09:41.444944] I [afr-self-heal-common.c:1203:sh_missing_entries_create] 0-poolsave-replicate-0: no missing files - /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to metadata check
[2012-04-12 03:09:41.446530] I [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] 0-poolsave-replicate-0: background  data missing-entry gfid self-heal completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
[2012-04-12 03:09:41.446960] I [afr-open.c:432:afr_openfd_sh] 0-poolsave-replicate-0:  data missing-entry gfid self-heal triggered. path: /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up down flush, data lock is held
[2012-04-12 03:09:41.448516] I [afr-self-heal-common.c:1203:sh_missing_entries_create] 0-poolsave-replicate-0: no missing files - /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to metadata check
[2012-04-12 03:09:41.450345] I [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] 0-poolsave-replicate-0: background  data missing-entry gfid self-heal completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
[2012-04-12 03:09:41.450774] I [afr-open.c:432:afr_openfd_sh] 0-poolsave-replicate-0:  data missing-entry gfid self-heal triggered. path: /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up down flush, data lock is held
[2012-04-12 03:09:41.452061] I [afr-self-heal-common.c:1203:sh_missing_entries_create] 0-poolsave-replicate-0: no missing files - /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to metadata check
[2012-04-12 03:09:41.453921] I [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] 0-poolsave-replicate-0: background  data missing-entry gfid self-heal completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
[2012-04-12 03:09:41.454448] I [afr-open.c:432:afr_openfd_sh] 0-poolsave-replicate-0:  data missing-entry gfid self-heal triggered. path: /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up down flush, data lock is held
[2012-04-12 03:09:41.455766] I [afr-self-heal-common.c:1203:sh_missing_entries_create] 0-poolsave-replicate-0: no missing files - /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to metadata check
[2012-04-12 03:09:41.457581] I [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] 0-poolsave-replicate-0: background  data missing-entry gfid self-heal completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
[2012-04-12 03:09:41.457959] I [afr-open.c:432:afr_openfd_sh] 0-poolsave-replicate-0:  data missing-entry gfid self-heal triggered. path: /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up down flush, data lock is held
[2012-04-12 03:09:41.459287] I [afr-self-heal-common.c:1203:sh_missing_entries_create] 0-poolsave-replicate-0: no missing files - /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to metadata check
[2012-04-12 03:09:41.460707] I [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] 0-poolsave-replicate-0: background  data missing-entry gfid self-heal completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
[2012-04-12 03:09:41.460955] I [afr-open.c:432:afr_openfd_sh] 0-poolsave-replicate-0:  data missing-entry gfid self-heal triggered. path: /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up down flush, data lock is held
[2012-04-12 03:09:41.462187] I [afr-self-heal-common.c:1203:sh_missing_entries_create] 0-poolsave-replicate-0: no missing files - /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to metadata check
[2012-04-12 03:09:41.463562] I [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] 0-poolsave-replicate-0: background  data missing-entry gfid self-heal completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
[2012-04-12 03:09:41.463885] I [afr-open.c:432:afr_openfd_sh] 0-poolsave-replicate-0:  data missing-entry gfid self-heal triggered. path: /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up down flush, data lock is held
[2012-04-12 03:09:41.465196] I [afr-self-heal-common.c:1203:sh_missing_entries_create] 0-poolsave-replicate-0: no missing files - /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to metadata check
[2012-04-12 03:09:41.466648] I [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] 0-poolsave-replicate-0: background  data missing-entry gfid self-heal completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
[2012-04-12 03:09:41.466945] I [afr-open.c:432:afr_openfd_sh] 0-poolsave-replicate-0:  data missing-entry gfid self-heal triggered. path: /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up down flush, data lock is held
[2012-04-12 03:09:41.468313] I [afr-self-heal-common.c:1203:sh_missing_entries_create] 0-poolsave-replicate-0: no missing files - /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to metadata check
[2012-04-12 03:09:41.469879] I [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] 0-poolsave-replicate-0: background  data missing-entry gfid self-heal completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
[2012-04-12 03:09:41.470142] I [afr-open.c:432:afr_openfd_sh] 0-poolsave-replicate-0:  data missing-entry gfid self-heal triggered. path: /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up down flush, data lock is held
[2012-04-12 03:09:41.471485] I [afr-self-heal-common.c:1203:sh_missing_entries_create] 0-poolsave-replicate-0: no missing files - /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to metadata check
[2012-04-12 03:09:41.472874] I [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] 0-poolsave-replicate-0: background  data missing-entry gfid self-heal completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
[2012-04-12 03:09:41.473142] I [afr-open.c:432:afr_openfd_sh] 0-poolsave-replicate-0:  data missing-entry gfid self-heal triggered. path: /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up down flush, data lock is held
[2012-04-12 03:09:41.474379] I [afr-self-heal-common.c:1203:sh_missing_entries_create] 0-poolsave-replicate-0: no missing files - /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to metadata check
[2012-04-12 03:09:41.475981] I [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] 0-poolsave-replicate-0: background  data missing-entry gfid self-heal completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
[2012-04-12 03:09:41.476246] I [afr-open.c:432:afr_openfd_sh] 0-poolsave-replicate-0:  data missing-entry gfid self-heal triggered. path: /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up down flush, data lock is held
[2012-04-12 03:09:41.477659] I [afr-self-heal-common.c:1203:sh_missing_entries_create] 0-poolsave-replicate-0: no missing files - /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to metadata check
[2012-04-12 03:09:41.479086] I [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] 0-poolsave-replicate-0: background  data missing-entry gfid self-heal completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
[2012-04-12 03:09:41.479400] I [afr-open.c:432:afr_openfd_sh] 0-poolsave-replicate-0:  data missing-entry gfid self-heal triggered. path: /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up down flush, data lock is held
[2012-04-12 03:09:41.480587] I [afr-self-heal-common.c:1203:sh_missing_entries_create] 0-poolsave-replicate-0: no missing files - /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to metadata check
[2012-04-12 03:09:41.482176] I [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] 0-poolsave-replicate-0: background  data missing-entry gfid self-heal completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
[2012-04-12 03:09:41.482444] I [afr-open.c:432:afr_openfd_sh] 0-poolsave-replicate-0:  data missing-entry gfid self-heal triggered. path: /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up down flush, data lock is held
[2012-04-12 03:09:41.483829] I [afr-self-heal-common.c:1203:sh_missing_entries_create] 0-poolsave-replicate-0: no missing files - /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to metadata check
[2012-04-12 03:09:41.485441] I [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] 0-poolsave-replicate-0: background  data missing-entry gfid self-heal completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
[2012-04-12 03:09:41.485728] I [afr-open.c:432:afr_openfd_sh] 0-poolsave-replicate-0:  data missing-entry gfid self-heal triggered. path: /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up down flush, data lock is held
[2012-04-12 03:09:41.487107] I [afr-self-heal-common.c:1203:sh_missing_entries_create] 0-poolsave-replicate-0: no missing files - /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to metadata check
[2012-04-12 03:09:41.488585] I [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] 0-poolsave-replicate-0: background  data missing-entry gfid self-heal completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
[2012-04-12 03:09:41.488858] I [afr-open.c:432:afr_openfd_sh] 0-poolsave-replicate-0:  data missing-entry gfid self-heal triggered. path: /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up down flush, data lock is held
[2012-04-12 03:09:41.490300] I [afr-self-heal-common.c:1203:sh_missing_entries_create] 0-poolsave-replicate-0: no missing files - /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to metadata check
[2012-04-12 03:09:41.491699] I [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk] 0-poolsave-replicate-0: background  data missing-entry gfid self-heal completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
[2012-04-12 03:13:57.87121] W [quota.c:2165:quota_fstat_cbk] 0-poolsave-quota: quota context not set in inode (ino:-2969972712, gfid:9f59f832-ee26-455f-8230-a42e254c43de)
[2012-04-12 03:13:57.90601] W [quota.c:2165:quota_fstat_cbk] 0-poolsave-quota: quota context not set in inode (ino:-2969972712, gfid:9f59f832-ee26-455f-8230-a42e254c43de)
[2012-04-12 03:30:01.295850] W [quota.c:2165:quota_fstat_cbk] 0-poolsave-quota: quota context not set in inode (ino:-1737201406, gfid:a85246c1-e3b2-4dd4-afff-0e4d1c3b2634)

A lot of those warning "quota context not set in inode" and then an another disconnection : 

[2012-04-12 05:09:43.310624] C [client-handshake.c:121:rpc_client_ping_timer_expired] 0-poolsave-client-3: server 10.80.49.13:24010 has not responded in the last 30 seconds, disconnecting.
[2012-04-12 05:09:43.311087] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fc2a3fa605e]))) 0-poolsave-client-3: forced unwinding frame type(GlusterFS 3.1) op(FSYNC(16)) called at 2012-04-12 05:08:50.912592
[2012-04-12 05:09:43.311243] I [client3_1-fops.c:779:client3_1_fsync_cbk] 0-poolsave-client-3: remote operation failed: Transport endpoint is not connected
[2012-04-12 05:09:43.311341] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fc2a3fa605e]))) 0-poolsave-client-3: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-04-12 05:09:05.303333
[2012-04-12 05:09:43.311387] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-poolsave-client-3: remote operation failed: Transport endpoint is not connected
[2012-04-12 05:09:43.332543] I [socket.c:2275:socket_submit_request] 0-poolsave-client-3: not connected (priv->connected = 0)
[2012-04-12 05:09:43.332650] W [rpc-clnt.c:1417:rpc_clnt_submit] 0-poolsave-client-3: failed to submit rpc-request (XID: 0x13367995x Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport (poolsave-client-3)
[2012-04-12 05:09:43.332763] W [rpc-clnt.c:1417:rpc_clnt_submit] 0-poolsave-client-3: failed to submit rpc-request (XID: 0x13367996x Program: GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport (poolsave-client-3)
[2012-04-12 05:09:43.332821] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fc2a3fa605e]))) 0-poolsave-client-3: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-04-12 05:09:05.303419
[2012-04-12 05:09:43.332891] E [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-3: remote operation failed: Transport endpoint is not connected
[2012-04-12 05:09:43.333319] I [client3_1-fops.c:683:client3_1_writev_cbk] 0-poolsave-client-3: remote operation failed: Transport endpoint is not connected
[2012-04-12 05:09:43.333730] W [rpc-clnt.c:1417:rpc_clnt_submit] 0-poolsave-client-3: failed to submit rpc-request (XID: 0x13367997x Program: GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport (poolsave-client-3)
[2012-04-12 05:09:43.333852] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fc2a3fa605e]))) 0-poolsave-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 05:09:05.303477
[2012-04-12 05:09:43.333900] E [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-3: remote operation failed: Transport endpoint is not connected
[2012-04-12 05:09:43.334017] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fc2a3fa605e]))) 0-poolsave-client-3: forced unwinding frame type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 05:09:05.303508
[2012-04-12 05:09:43.334062] E [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-3: remote operation failed: Transport endpoint is not connected
[2012-04-12 05:09:43.334154] E [rpc-clnt.c:341:saved_frames_unwind] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939] (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe) [0x7fc2a3fa605e]))) 0-poolsave-client-3: forced unwinding frame type(GlusterFS 3.1) op(WRITE(13)) called at 2012-04-12 05:09:05.303758

Log of ylal3530 for fisrt disconnection of 3:09am : (log/glusterfs/bricks/users3-poolsave.log)

[2012-04-12 03:09:23.346130] W [socket.c:204:__socket_rwv] 0-tcp.poolsave-server: writev failed (Broken pipe)
[2012-04-12 03:09:23.346328] I [socket.c:1768:socket_event_handler] 0-transport: disconnecting now
[2012-04-12 03:09:23.346452] I [server-helpers.c:343:do_lock_table_cleanup] 0-poolsave-server: finodelk released on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
[2012-04-12 03:09:23.346530] I [server-helpers.c:343:do_lock_table_cleanup] 0-poolsave-server: finodelk released on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
[2012-04-12 03:09:23.346551] I [server-helpers.c:343:do_lock_table_cleanup] 0-poolsave-server: finodelk released on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
[2012-04-12 03:09:23.346604] I [socket.c:2338:socket_submit_reply] 0-tcp.poolsave-server: not connected (priv->connected = 255)
[2012-04-12 03:09:23.346625] W [inode.c:1044:inode_path] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_connection_cleanup+0xa6) [0x7f5ec426d106] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_connection_cleanup+0x139) [0x7f5ec426ce09] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_fd_cleanup+0x32c) [0x7f5ec426bb6c]))) 0-/users3/poolsave/inode: no dentry for non-root inode 400019122: 70c18c81-cc2d-4698-b28d-2fba654a9ef8
[2012-04-12 03:09:23.346714] E [rpcsvc.c:1033:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x10709543x, Program: GlusterFS 3.2.5, ProgVers: 310, Proc: 13) to rpc-transport (tcp.poolsave-server)
[2012-04-12 03:09:23.346780] I [server-helpers.c:492:do_fd_cleanup] 0-poolsave-server: fd cleanup on ino 400019122 with gfid 70c18c81-cc2d-4698-b28d-2fba654a9ef8
[2012-04-12 03:09:23.347047] E [server.c:136:server_submit_reply] (-->/usr/local/lib//glusterfs/3.2.5/xlator/features/marker.so(marker_writev_cbk+0xa9) [0x7f5ec44ae109] (-->/usr/local/lib//glusterfs/3.2.5/xlator/debug/io-stats.so(io_stats_writev_cbk+0x1b3) [0x7f5ec439be73] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_writev_cbk+0xa8) [0x7f5ec42760e8]))) 0-: Reply submission failed
[2012-04-12 03:09:23.347074] W [inode.c:1044:inode_path] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_connection_cleanup+0xa6) [0x7f5ec426d106] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_connection_cleanup+0x139) [0x7f5ec426ce09] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_fd_cleanup+0x32c) [0x7f5ec426bb6c]))) 0-/users3/poolsave/inode: no dentry for non-root inode -3453277438: d0344eb6-6f41-4d14-8855-e94fa9d541b3
[2012-04-12 03:09:23.347162] I [server-helpers.c:492:do_fd_cleanup] 0-poolsave-server: fd cleanup on ino -3453277438 with gfid d0344eb6-6f41-4d14-8855-e94fa9d541b3
[2012-04-12 03:09:23.347187] I [server-helpers.c:485:do_fd_cleanup] 0-poolsave-server: fd cleanup on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
[2012-04-12 03:09:23.347204] I [server.c:438:server_rpc_notify] 0-poolsave-server: disconnected connection from 10.80.49.10:1015
[2012-04-12 03:09:23.347226] I [server-helpers.c:783:server_connection_destroy] 0-poolsave-server: destroyed connection of ylal3510-25300-2012/04/11-09:11:19:688358-poolsave-client-1
[2012-04-12 03:09:23.348129] I [server-handshake.c:542:server_setvolume] 0-poolsave-server: accepted client from 10.80.49.10:1008 (version: 3.2.5)

log of ylal3540 for disconnection of 5:09 am : 

[2012-04-12 05:09:47.540357] W [socket.c:1494:__socket_proto_state_machine] 0-tcp.poolsave-server: reading from socket failed. Error (Transport endpoint is not connected), peer (10.80.49.10:1012)
[2012-04-12 05:09:47.540473] I [server-helpers.c:343:do_lock_table_cleanup] 0-poolsave-server: finodelk released on /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar
[2012-04-12 05:09:47.540507] I [server-helpers.c:343:do_lock_table_cleanup] 0-poolsave-server: finodelk released on /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar
[2012-04-12 05:09:47.540525] I [server-helpers.c:343:do_lock_table_cleanup] 0-poolsave-server: finodelk released on /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar
[2012-04-12 05:09:47.540543] I [server-helpers.c:343:do_lock_table_cleanup] 0-poolsave-server: finodelk released on /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar
[2012-04-12 05:09:47.540564] I [server-helpers.c:343:do_lock_table_cleanup] 0-poolsave-server: finodelk released on /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar
[2012-04-12 05:09:47.540582] I [server-helpers.c:343:do_lock_table_cleanup] 0-poolsave-server: finodelk released on /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar
[2012-04-12 05:09:47.540583] I [socket.c:2338:socket_submit_reply] 0-tcp.poolsave-server: not connected (priv->connected = 255)
[2012-04-12 05:09:47.540602] I [server-helpers.c:343:do_lock_table_cleanup] 0-poolsave-server: finodelk released on /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar
[2012-04-12 05:09:47.540807] I [server-helpers.c:343:do_lock_table_cleanup] 0-poolsave-server: finodelk released on /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar
[2012-04-12 05:09:47.540705] E [rpcsvc.c:1033:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x13367977x, Program: GlusterFS 3.2.5, ProgVers: 310, Proc: 13) to rpc-transport (tcp.poolsave-server)
[2012-04-12 05:09:47.540831] I [server-helpers.c:343:do_lock_table_cleanup] 0-poolsave-server: finodelk released on /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar
[2012-04-12 05:09:47.540739] E [rpcsvc.c:1033:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x13367984x, Program: GlusterFS 3.2.5, ProgVers: 310, Proc: 13) to rpc-transport (tcp.poolsave-server)
[2012-04-12 05:09:47.540935] I [server-helpers.c:343:do_lock_table_cleanup] 0-poolsave-server: finodelk released on /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar
[2012-04-12 05:09:47.541141] E [server.c:136:server_submit_reply] (-->/usr/local/lib//glusterfs/3.2.5/xlator/features/marker.so(marker_writev_cbk+0xa9) [0x7f9952011109] (-->/usr/local/lib//glusterfs/3.2.5/xlator/debug/io-stats.so(io_stats_writev_cbk+0x1b3) [0x7f9951efee73] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_writev_cbk+0xa8) [0x7f9951dd90e8]))) 0-: Reply submission failed
[2012-04-12 05:09:47.541059] E [server.c:136:server_submit_reply] (-->/usr/local/lib//glusterfs/3.2.5/xlator/features/marker.so(marker_writev_cbk+0xa9) [0x7f9952011109] (-->/usr/local/lib//glusterfs/3.2.5/xlator/debug/io-stats.so(io_stats_writev_cbk+0x1b3) [0x7f9951efee73] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_writev_cbk+0xa8) [0x7f9951dd90e8]))) 0-: Reply submission failed
[2012-04-12 05:09:47.541252] E [server.c:136:server_submit_reply] (-->/usr/local/lib//glusterfs/3.2.5/xlator/features/marker.so(marker_writev_cbk+0xa9) [0x7f9952011109] (-->/usr/local/lib//glusterfs/3.2.5/xlator/debug/io-stats.so(io_stats_writev_cbk+0x1b3) [0x7f9951efee73] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_writev_cbk+0xa8) [0x7f9951dd90e8]))) 0-: Reply submission failed
[2012-04-12 05:09:47.541210] E [rpcsvc.c:1033:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x13367987x, Program: GlusterFS 3.2.5, ProgVers: 310, Proc: 13) to rpc-transport (tcp.poolsave-server)
[2012-04-12 05:09:47.541164] I [server-helpers.c:343:do_lock_table_cleanup] 0-poolsave-server: finodelk released on /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar
[2012-04-12 05:09:47.541552] E [server.c:136:server_submit_reply] (-->/usr/local/lib//glusterfs/3.2.5/xlator/features/marker.so(marker_writev_cbk+0xa9) [0x7f9952011109] (-->/usr/local/lib//glusterfs/3.2.5/xlator/debug/io-stats.so(io_stats_writev_cbk+0x1b3) [0x7f9951efee73] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_writev_cbk+0xa8) [0x7f9951dd90e8]))) 0-: Reply submission failed
[2012-04-12 05:09:47.541593] I [server-helpers.c:343:do_lock_table_cleanup] 0-poolsave-server: finodelk released on /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar
[2012-04-12 05:09:47.541618] E [rpcsvc.c:1033:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x13367988x, Program: GlusterFS 3.2.5, ProgVers: 310, Proc: 13) to rpc-transport (tcp.poolsave-server)
[2012-04-12 05:09:47.541665] I [server-helpers.c:343:do_lock_table_cleanup] 0-poolsave-server: finodelk released on /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar
[2012-04-12 05:09:47.541788] E [server.c:136:server_submit_reply] (-->/usr/local/lib//glusterfs/3.2.5/xlator/features/marker.so(marker_writev_cbk+0xa9) [0x7f9952011109] (-->/usr/local/lib//glusterfs/3.2.5/xlator/debug/io-stats.so(io_stats_writev_cbk+0x1b3) [0x7f9951efee73] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_writev_cbk+0xa8) [0x7f9951dd90e8]))) 0-: Reply submission failed
[2012-04-12 05:09:47.541837] W [inode.c:1044:inode_path] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_connection_cleanup+0xa6) [0x7f9951dd0106] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_connection_cleanup+0x139) [0x7f9951dcfe09] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_fd_cleanup+0x32c) [0x7f9951dceb6c]))) 0-/users3/poolsave/inode: no dentry for non-root inode -450063628: bfdedf4b-671e-4d2f-ba7a-27362ab26bbe
[2012-04-12 05:09:47.541929] I [server-helpers.c:492:do_fd_cleanup] 0-poolsave-server: fd cleanup on ino -450063628 with gfid bfdedf4b-671e-4d2f-ba7a-27362ab26bbe
[2012-04-12 05:09:47.541983] W [inode.c:1044:inode_path] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_connection_cleanup+0xa6) [0x7f9951dd0106] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_connection_cleanup+0x139) [0x7f9951dcfe09] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_fd_cleanup+0x32c) [0x7f9951dceb6c]))) 0-/users3/poolsave/inode: no dentry for non-root inode 393879387: 157da663-ff30-4ba9-bd24-0dc45a561197
[2012-04-12 05:09:47.542007] I [server-helpers.c:492:do_fd_cleanup] 0-poolsave-server: fd cleanup on ino 393879387 with gfid 157da663-ff30-4ba9-bd24-0dc45a561197
[2012-04-12 05:09:47.542106] W [inode.c:1044:inode_path] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_connection_cleanup+0xa6) [0x7f9951dd0106] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_connection_cleanup+0x139) [0x7f9951dcfe09] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_fd_cleanup+0x32c) [0x7f9951dceb6c]))) 0-/users3/poolsave/inode: no dentry for non-root inode -1147302459: e88b34c2-9453-4e6b-b4e0-066706bd875e
[2012-04-12 05:09:47.542129] I [server-helpers.c:492:do_fd_cleanup] 0-poolsave-server: fd cleanup on ino -1147302459 with gfid e88b34c2-9453-4e6b-b4e0-066706bd875e
[2012-04-12 05:09:47.542173] W [inode.c:1044:inode_path] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_connection_cleanup+0xa6) [0x7f9951dd0106] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_connection_cleanup+0x139) [0x7f9951dcfe09] (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_fd_cleanup+0x32c) [0x7f9951dceb6c]))) 0-/users3/poolsave/inode: no dentry for non-root inode -462949261: dbec18c6-4f55-442d-ba76-267929f1cdfa
[2012-04-12 05:09:47.542194] I [server-helpers.c:492:do_fd_cleanup] 0-poolsave-server: fd cleanup on ino -462949261 with gfid dbec18c6-4f55-442d-ba76-267929f1cdfa
[2012-04-12 05:09:47.542231] I [server-helpers.c:485:do_fd_cleanup] 0-poolsave-server: fd cleanup on /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar
[2012-04-12 05:09:47.542247] I [server.c:438:server_rpc_notify] 0-poolsave-server: disconnected connection from 10.80.49.10:1012
[2012-04-12 05:09:47.542282] I [server-helpers.c:783:server_connection_destroy] 0-poolsave-server: destroyed connection of ylal3510-25300-2012/04/11-09:11:19:688358-poolsave-client-3
[2012-04-12 05:09:47.542349] I [server-handshake.c:542:server_setvolume] 0-poolsave-server: accepted client from 10.80.49.10:1015 (version: 3.2.5)

Regards,

Anthony

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120413/dff822b0/attachment.html>