[Gluster-users] Random disconnect

Fri Apr 13 12:44:29 UTC 2012

I have lots of clients hitting my bricks and I noticed that the bricks
thought that they were being syn flooded so they were sending out syn
cookies. I was also having these sorts of disconnects so I turned off
syn cookies, not sure yet if it helped but I haven't had a disconnect
since then.

On Fri, Apr 13, 2012 at 4:59 AM, anthony garnier <sokar6012 at hotmail.com> wrote:
> Amar,
>
>
> I'm facing an other issue, don't know if there is a link with the quota
> issue.
>
> I got random disconnections during worload :
>
> I use the same scripts as previoulsy.
> (https://bugzilla.redhat.com/show_bug.cgi?id=812230 )
>
> Config info :
> I'm accessing the mount point with nfs. I'm connected to ylal3510
>
>
> 10.80.49.10 ylal3510
> 10.80.49.11 ylal3520
> 10.80.49.12 ylal3530
> 10.80.49.13 ylal3540
>
> Volume Name: poolsave
> Type: Distributed-Replicate
> Status: Started
> Number of Bricks: 2 x 2 = 4
> Transport-type: tcp
> Bricks:
> Brick1: ylal3510:/users3/poolsave
> Brick2: ylal3530:/users3/poolsave
> Brick3: ylal3520:/users3/poolsave
> Brick4: ylal3540:/users3/poolsave
> Options Reconfigured:
> features.limit-usage: /yval9000:1TB
> diagnostics.count-fop-hits: on
> diagnostics.latency-measurement: on
> performance.io-thread-count: 64
> nfs.port: 2049
> performance.cache-refresh-timeout: 2
> performance.cache-max-file-size: 4GB
> performance.cache-min-file-size: 1KB
> network.ping-timeout: 60
> performance.cache-size: 6GB
> features.quota: on
> features.quota-timeout: 30
> nfs.enable-ino32: off
>
>
>
>
> Here is the log on the first storage node (ylal3510): nfs.log
>
> First disconnection :
>
> [2012-04-12 03:01:40.700893] W [quota.c:2165:quota_fstat_cbk]
> 0-poolsave-quota: quota context not set in inode (ino:-2126980194,
> gfid:ec0c049e-78b9-4972-b14c-2fa60f503e29)
> [2012-04-12 03:01:40.708029] W [quota.c:2165:quota_fstat_cbk]
> 0-poolsave-quota: quota context not set in inode (ino:-2126980194,
> gfid:ec0c049e-78b9-4972-b14c-2fa60f503e29)
>
> [2012-04-12 03:09:17.454283] C
> [client-handshake.c:121:rpc_client_ping_timer_expired] 0-poolsave-client-1:
> server 10.80.49.12:24010 has not responded in the last 30 seconds,
> disconnecting.
> [2012-04-12 03:09:17.455307] E [rpc-clnt.c:341:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)
> [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)
> [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame
> type(GlusterFS 3.1) op(FSYNC(16)) called at 2012-04-12 03:08:17.264052
> [2012-04-12 03:09:17.455413] I [client3_1-fops.c:779:client3_1_fsync_cbk]
> 0-poolsave-client-1: remote operation failed: Transport endpoint is not
> connected
> [2012-04-12 03:09:17.480558] I [socket.c:2275:socket_submit_request]
> 0-poolsave-client-1: not connected (priv->connected = 0)
> [2012-04-12 03:09:17.480659] W [rpc-clnt.c:1417:rpc_clnt_submit]
> 0-poolsave-client-1: failed to submit rpc-request (XID: 0x10709557x Program:
> GlusterFS 3.1, ProgVers: 310, Proc: 16) to rpc-transport (poolsave-client-1)
> [2012-04-12 03:09:17.480710] I [client3_1-fops.c:779:client3_1_fsync_cbk]
> 0-poolsave-client-1: remote operation failed: Transport endpoint is not
> connected
> [2012-04-12 03:09:17.480875] W [rpc-clnt.c:1417:rpc_clnt_submit]
> 0-poolsave-client-1: failed to submit rpc-request (XID: 0x10709558x Program:
> GlusterFS 3.1, ProgVers: 310, Proc: 16) to rpc-transport (poolsave-client-1)
> [2012-04-12 03:09:17.480918] I [client3_1-fops.c:779:client3_1_fsync_cbk]
> 0-poolsave-client-1: remote operation failed: Transport endpoint is not
> connected
> [2012-04-12 03:09:17.481006] W [rpc-clnt.c:1417:rpc_clnt_submit]
> 0-poolsave-client-1: failed to submit rpc-request (XID: 0x10709559x Program:
> GlusterFS 3.1, ProgVers: 310, Proc: 16) to rpc-transport (poolsave-client-1)
> [2012-04-12 03:09:17.481117] I [client3_1-fops.c:779:client3_1_fsync_cbk]
> 0-poolsave-client-1: remote operation failed: Transport endpoint is not
> connected
> [2012-04-12 03:09:17.481198] W [rpc-clnt.c:1417:rpc_clnt_submit]
> 0-poolsave-client-1: failed to submit rpc-request (XID: 0x10709560x Program:
> GlusterFS 3.1, ProgVers: 310, Proc: 16) to rpc-transport (poolsave-client-1)
> [2012-04-12 03:09:17.481237] I [client3_1-fops.c:779:client3_1_fsync_cbk]
> 0-poolsave-client-1: remote operation failed: Transport endpoint is not
> connected
> [2012-04-12 03:09:17.481326] W [rpc-clnt.c:1417:rpc_clnt_submit]
> 0-poolsave-client-1: failed to submit rpc-request (XID: 0x10709561x Program:
> GlusterFS 3.1, ProgVers: 310, Proc: 16) to rpc-transport (poolsave-client-1)
> [2012-04-12 03:09:17.481364] I [client3_1-fops.c:779:client3_1_fsync_cbk]
> 0-poolsave-client-1: remote operation failed: Transport endpoint is not
> connected
> [2012-04-12 03:09:17.481448] W [rpc-clnt.c:1417:rpc_clnt_submit]
> 0-poolsave-client-1: failed to submit rpc-request (XID: 0x10709562x Program:
> GlusterFS 3.1, ProgVers: 310, Proc: 16) to rpc-transport (poolsave-client-1)
> [2012-04-12 03:09:17.481487] I [client3_1-fops.c:779:client3_1_fsync_cbk]
> 0-poolsave-client-1: remote operation failed: Transport endpoint is not
> connected
> [2012-04-12 03:09:17.481565] W [rpc-clnt.c:1417:rpc_clnt_submit]
> 0-poolsave-client-1: failed to submit rpc-request (XID: 0x10709563x Program:
> GlusterFS 3.1, ProgVers: 310, Proc: 16) to rpc-transport (poolsave-client-1)
> [2012-04-12 03:09:17.481604] I [client3_1-fops.c:779:client3_1_fsync_cbk]
> 0-poolsave-client-1: remote operation failed: Transport endpoint is not
> connected
> [2012-04-12 03:09:17.481820] E [rpc-clnt.c:341:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)
> [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)
> [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame
> type(GlusterFS 3.1) op(WRITE(13)) called at 2012-04-12 03:08:37.235705
> [2012-04-12 03:09:17.481872] I [client3_1-fops.c:683:client3_1_writev_cbk]
> 0-poolsave-client-1: remote operation failed: Transport endpoint is not
> connected
> [2012-04-12 03:09:17.482093] W [rpc-clnt.c:1417:rpc_clnt_submit]
> 0-poolsave-client-1: failed to submit rpc-request (XID: 0x10709564x Program:
> GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport (poolsave-client-1)
> [2012-04-12 03:09:17.482198] E [rpc-clnt.c:341:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)
> [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)
> [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame
> type(GlusterFS 3.1) op(WRITE(13)) called at 2012-04-12 03:08:37.235895
> [2012-04-12 03:09:17.482243] I [client3_1-fops.c:683:client3_1_writev_cbk]
> 0-poolsave-client-1: remote operation failed: Transport endpoint is not
> connected
> [2012-04-12 03:09:17.482417] W [rpc-clnt.c:1417:rpc_clnt_submit]
> 0-poolsave-client-1: failed to submit rpc-request (XID: 0x10709565x Program:
> GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport (poolsave-client-1)
> [2012-04-12 03:09:17.482513] E [rpc-clnt.c:341:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)
> [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)
> [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame
> type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.236009
> [2012-04-12 03:09:17.482562] E
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote
> operation failed: Transport endpoint is not connected
> [2012-04-12 03:09:17.482650] E [rpc-clnt.c:341:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)
> [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)
> [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame
> type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.236370
> [2012-04-12 03:09:17.482722] E
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote
> operation failed: Transport endpoint is not connected
> [2012-04-12 03:09:17.482856] E [rpc-clnt.c:341:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)
> [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)
> [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame
> type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.236455
> [2012-04-12 03:09:17.482901] E
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote
> operation failed: Transport endpoint is not connected
> [2012-04-12 03:09:17.482986] E [rpc-clnt.c:341:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)
> [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)
> [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame
> type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.236631
> [2012-04-12 03:09:17.483028] E
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote
> operation failed: Transport endpoint is not connected
> [2012-04-12 03:09:17.483147] E [rpc-clnt.c:341:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)
> [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)
> [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame
> type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.237248
> [2012-04-12 03:09:17.483191] E
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote
> operation failed: Transport endpoint is not connected
> [2012-04-12 03:09:17.483357] E [rpc-clnt.c:341:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)
> [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)
> [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame
> type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.237537
> [2012-04-12 03:09:17.483402] E
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote
> operation failed: Transport endpoint is not connected
> [2012-04-12 03:09:17.483523] E [rpc-clnt.c:341:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)
> [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)
> [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame
> type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.237664
> [2012-04-12 03:09:17.483568] E
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote
> operation failed: Transport endpoint is not connected
> [2012-04-12 03:09:17.483689] E [rpc-clnt.c:341:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)
> [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)
> [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame
> type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.237960
> [2012-04-12 03:09:17.483734] E
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote
> operation failed: Transport endpoint is not connected
> [2012-04-12 03:09:17.483855] E [rpc-clnt.c:341:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)
> [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)
> [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame
> type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.238258
> [2012-04-12 03:09:17.483924] E
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote
> operation failed: Transport endpoint is not connected
> [2012-04-12 03:09:17.484045] E [rpc-clnt.c:341:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)
> [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)
> [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame
> type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.238554
> [2012-04-12 03:09:17.484090] E
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote
> operation failed: Transport endpoint is not connected
> [2012-04-12 03:09:17.484207] E [rpc-clnt.c:341:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)
> [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)
> [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame
> type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.238848
> [2012-04-12 03:09:17.484252] E
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote
> operation failed: Transport endpoint is not connected
> [2012-04-12 03:09:17.484373] E [rpc-clnt.c:341:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)
> [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)
> [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame
> type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 03:08:37.239070
> [2012-04-12 03:09:17.484417] E
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-1: remote
> operation failed: Transport endpoint is not connected
> [2012-04-12 03:09:17.484536] E [rpc-clnt.c:341:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)
> [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)
> [0x7fc2a3fa605e]))) 0-poolsave-client-1: forced unwinding frame
> type(GlusterFS Handshake) op(PING(3)) called at 2012-04-12 03:08:47.450735
> [2012-04-12 03:09:17.484587] W [client-handshake.c:264:client_ping_cbk]
> 0-poolsave-client-1: timer must have expired
> [2012-04-12 03:09:17.484622] I [client.c:1883:client_rpc_notify]
> 0-poolsave-client-1: disconnected
> [2012-04-12 03:09:17.485639] W [client3_1-fops.c:4868:client3_1_finodelk]
> 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD
> [2012-04-12 03:09:17.485717] W [client3_1-fops.c:4915:client3_1_finodelk]
> 0-poolsave-client-1: failed to send the fop: File descriptor in bad state
> [2012-04-12 03:09:17.485834] W [client3_1-fops.c:4868:client3_1_finodelk]
> 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD
> [2012-04-12 03:09:17.485873] W [client3_1-fops.c:4915:client3_1_finodelk]
> 0-poolsave-client-1: failed to send the fop: File descriptor in bad state
> [2012-04-12 03:09:17.486432] W [client3_1-fops.c:4868:client3_1_finodelk]
> 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD
> [2012-04-12 03:09:17.486471] W [client3_1-fops.c:4915:client3_1_finodelk]
> 0-poolsave-client-1: failed to send the fop: File descriptor in bad state
> [2012-04-12 03:09:17.486620] W [client3_1-fops.c:4868:client3_1_finodelk]
> 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD
> [2012-04-12 03:09:17.486659] W [client3_1-fops.c:4915:client3_1_finodelk]
> 0-poolsave-client-1: failed to send the fop: File descriptor in bad state
> [2012-04-12 03:09:17.486820] W [client3_1-fops.c:4868:client3_1_finodelk]
> 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD
> [2012-04-12 03:09:17.486933] W [client3_1-fops.c:4915:client3_1_finodelk]
> 0-poolsave-client-1: failed to send the fop: File descriptor in bad state
> [2012-04-12 03:09:17.487075] W [client3_1-fops.c:4868:client3_1_finodelk]
> 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD
> [2012-04-12 03:09:17.487120] W [client3_1-fops.c:4915:client3_1_finodelk]
> 0-poolsave-client-1: failed to send the fop: File descriptor in bad state
> [2012-04-12 03:09:17.487245] W [client3_1-fops.c:4868:client3_1_finodelk]
> 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD
> [2012-04-12 03:09:17.487283] W [client3_1-fops.c:4915:client3_1_finodelk]
> 0-poolsave-client-1: failed to send the fop: File descriptor in bad state
> [2012-04-12 03:09:17.487429] W [client3_1-fops.c:4868:client3_1_finodelk]
> 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD
> [2012-04-12 03:09:17.487489] W [client3_1-fops.c:4915:client3_1_finodelk]
> 0-poolsave-client-1: failed to send the fop: File descriptor in bad state
> [2012-04-12 03:09:17.487628] W [client3_1-fops.c:4868:client3_1_finodelk]
> 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD
> [2012-04-12 03:09:17.487671] W [client3_1-fops.c:4915:client3_1_finodelk]
> 0-poolsave-client-1: failed to send the fop: File descriptor in bad state
> [2012-04-12 03:09:17.487827] W [client3_1-fops.c:4868:client3_1_finodelk]
> 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD
> [2012-04-12 03:09:17.487868] W [client3_1-fops.c:4915:client3_1_finodelk]
> 0-poolsave-client-1: failed to send the fop: File descriptor in bad state
> [2012-04-12 03:09:17.488009] W [client3_1-fops.c:4868:client3_1_finodelk]
> 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD
> [2012-04-12 03:09:17.488046] W [client3_1-fops.c:4915:client3_1_finodelk]
> 0-poolsave-client-1: failed to send the fop: File descriptor in bad state
> [2012-04-12 03:09:17.488174] W [client3_1-fops.c:4868:client3_1_finodelk]
> 0-poolsave-client-1: (-1117219266): failed to get fd ctx. EBADFD
> [2012-04-12 03:09:17.488211] W [client3_1-fops.c:4915:client3_1_finodelk]
> 0-poolsave-client-1: failed to send the fop: File descriptor in bad state
> [2012-04-12 03:09:23.347623] I
> [client-handshake.c:1090:select_server_supported_programs]
> 0-poolsave-client-1: Using Program GlusterFS 3.2.5, Num (1298437), Version
> (310)
> [2012-04-12 03:09:23.348507] I [client-handshake.c:913:client_setvolume_cbk]
> 0-poolsave-client-1: Connected to 10.80.49.12:24010, attached to remote
> volume '/users3/poolsave'.
> [2012-04-12 03:09:23.348540] I
> [client-handshake.c:779:client_post_handshake] 0-poolsave-client-1: 3 fds
> open - Delaying child_up until they are re-opened
> [2012-04-12 03:09:23.348632] W [inode.c:1044:inode_path]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/client.so(client_setvolume_cbk+0x1b2)
> [0x7fc2a205aee2]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/client.so(client_post_handshake+0x193)
> [0x7fc2a205acf3]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/client.so(protocol_client_reopen+0x75)
> [0x7fc2a205a1b5]))) 0-poolsave/inode: no dentry for non-root inode
> -6906554876: d0344eb6-6f41-4d14-8855-e94fa9d541b3
> [2012-04-12 03:09:23.348665] W
> [client-handshake.c:683:protocol_client_reopen] 0-poolsave-client-1:
> couldn't build path from inode d0344eb6-6f41-4d14-8855-e94fa9d541b3
> [2012-04-12 03:09:23.348680] E
> [client-handshake.c:722:protocol_client_reopen] 0-: failed to send the
> re-open request
> [2012-04-12 03:09:23.348724] W [inode.c:1044:inode_path]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/client.so(client_setvolume_cbk+0x1b2)
> [0x7fc2a205aee2]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/client.so(client_post_handshake+0x193)
> [0x7fc2a205acf3]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/client.so(protocol_client_reopen+0x75)
> [0x7fc2a205a1b5]))) 0-poolsave/inode: no dentry for non-root inode
> 800038244: 70c18c81-cc2d-4698-b28d-2fba654a9ef8
> [2012-04-12 03:09:23.348749] W
> [client-handshake.c:683:protocol_client_reopen] 0-poolsave-client-1:
> couldn't build path from inode 70c18c81-cc2d-4698-b28d-2fba654a9ef8
> [2012-04-12 03:09:23.348802] E
> [client-handshake.c:722:protocol_client_reopen] 0-: failed to send the
> re-open request
> [2012-04-12 03:09:23.349450] I [client-lk.c:617:decrement_reopen_fd_count]
> 0-poolsave-client-1: last fd open'd/lock-self-heal'd - notifying CHILD-UP
> [2012-04-12 03:09:23.352548] I [afr-open.c:432:afr_openfd_sh]
> 0-poolsave-replicate-0:  data missing-entry gfid self-heal triggered. path:
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up
> down flush, data lock is held
> [2012-04-12 03:09:23.354603] I
> [afr-self-heal-common.c:1203:sh_missing_entries_create]
> 0-poolsave-replicate-0: no missing files -
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to
> metadata check
> [2012-04-12 03:09:23.356460] W [dict.c:418:dict_unref]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0xa2)
> [0x7fc2a3fa6712]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/client.so(client3_1_fstat_cbk+0x2c9)
> [0x7fc2a20551a9]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/cluster/replicate.so(afr_sh_data_fstat_cbk+0x1d0)
> [0x7fc2a1efa590]))) 0-dict: dict is NULL
> [2012-04-12 03:09:41.442063] I
> [afr-self-heal-algorithm.c:520:sh_diff_loop_driver_done]
> 0-poolsave-replicate-0: diff self-heal on
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar: completed. (5249
> blocks of 73670 were different (7.13%))
> [2012-04-12 03:09:41.443242] I
> [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk]
> 0-poolsave-replicate-0: background  data missing-entry gfid self-heal
> completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
> [2012-04-12 03:09:41.443660] I [afr-open.c:432:afr_openfd_sh]
> 0-poolsave-replicate-0:  data missing-entry gfid self-heal triggered. path:
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up
> down flush, data lock is held
> [2012-04-12 03:09:41.444944] I
> [afr-self-heal-common.c:1203:sh_missing_entries_create]
> 0-poolsave-replicate-0: no missing files -
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to
> metadata check
> [2012-04-12 03:09:41.446530] I
> [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk]
> 0-poolsave-replicate-0: background  data missing-entry gfid self-heal
> completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
> [2012-04-12 03:09:41.446960] I [afr-open.c:432:afr_openfd_sh]
> 0-poolsave-replicate-0:  data missing-entry gfid self-heal triggered. path:
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up
> down flush, data lock is held
> [2012-04-12 03:09:41.448516] I
> [afr-self-heal-common.c:1203:sh_missing_entries_create]
> 0-poolsave-replicate-0: no missing files -
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to
> metadata check
> [2012-04-12 03:09:41.450345] I
> [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk]
> 0-poolsave-replicate-0: background  data missing-entry gfid self-heal
> completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
> [2012-04-12 03:09:41.450774] I [afr-open.c:432:afr_openfd_sh]
> 0-poolsave-replicate-0:  data missing-entry gfid self-heal triggered. path:
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up
> down flush, data lock is held
> [2012-04-12 03:09:41.452061] I
> [afr-self-heal-common.c:1203:sh_missing_entries_create]
> 0-poolsave-replicate-0: no missing files -
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to
> metadata check
> [2012-04-12 03:09:41.453921] I
> [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk]
> 0-poolsave-replicate-0: background  data missing-entry gfid self-heal
> completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
> [2012-04-12 03:09:41.454448] I [afr-open.c:432:afr_openfd_sh]
> 0-poolsave-replicate-0:  data missing-entry gfid self-heal triggered. path:
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up
> down flush, data lock is held
> [2012-04-12 03:09:41.455766] I
> [afr-self-heal-common.c:1203:sh_missing_entries_create]
> 0-poolsave-replicate-0: no missing files -
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to
> metadata check
> [2012-04-12 03:09:41.457581] I
> [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk]
> 0-poolsave-replicate-0: background  data missing-entry gfid self-heal
> completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
> [2012-04-12 03:09:41.457959] I [afr-open.c:432:afr_openfd_sh]
> 0-poolsave-replicate-0:  data missing-entry gfid self-heal triggered. path:
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up
> down flush, data lock is held
> [2012-04-12 03:09:41.459287] I
> [afr-self-heal-common.c:1203:sh_missing_entries_create]
> 0-poolsave-replicate-0: no missing files -
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to
> metadata check
> [2012-04-12 03:09:41.460707] I
> [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk]
> 0-poolsave-replicate-0: background  data missing-entry gfid self-heal
> completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
> [2012-04-12 03:09:41.460955] I [afr-open.c:432:afr_openfd_sh]
> 0-poolsave-replicate-0:  data missing-entry gfid self-heal triggered. path:
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up
> down flush, data lock is held
> [2012-04-12 03:09:41.462187] I
> [afr-self-heal-common.c:1203:sh_missing_entries_create]
> 0-poolsave-replicate-0: no missing files -
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to
> metadata check
> [2012-04-12 03:09:41.463562] I
> [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk]
> 0-poolsave-replicate-0: background  data missing-entry gfid self-heal
> completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
> [2012-04-12 03:09:41.463885] I [afr-open.c:432:afr_openfd_sh]
> 0-poolsave-replicate-0:  data missing-entry gfid self-heal triggered. path:
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up
> down flush, data lock is held
> [2012-04-12 03:09:41.465196] I
> [afr-self-heal-common.c:1203:sh_missing_entries_create]
> 0-poolsave-replicate-0: no missing files -
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to
> metadata check
> [2012-04-12 03:09:41.466648] I
> [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk]
> 0-poolsave-replicate-0: background  data missing-entry gfid self-heal
> completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
> [2012-04-12 03:09:41.466945] I [afr-open.c:432:afr_openfd_sh]
> 0-poolsave-replicate-0:  data missing-entry gfid self-heal triggered. path:
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up
> down flush, data lock is held
> [2012-04-12 03:09:41.468313] I
> [afr-self-heal-common.c:1203:sh_missing_entries_create]
> 0-poolsave-replicate-0: no missing files -
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to
> metadata check
> [2012-04-12 03:09:41.469879] I
> [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk]
> 0-poolsave-replicate-0: background  data missing-entry gfid self-heal
> completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
> [2012-04-12 03:09:41.470142] I [afr-open.c:432:afr_openfd_sh]
> 0-poolsave-replicate-0:  data missing-entry gfid self-heal triggered. path:
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up
> down flush, data lock is held
> [2012-04-12 03:09:41.471485] I
> [afr-self-heal-common.c:1203:sh_missing_entries_create]
> 0-poolsave-replicate-0: no missing files -
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to
> metadata check
> [2012-04-12 03:09:41.472874] I
> [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk]
> 0-poolsave-replicate-0: background  data missing-entry gfid self-heal
> completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
> [2012-04-12 03:09:41.473142] I [afr-open.c:432:afr_openfd_sh]
> 0-poolsave-replicate-0:  data missing-entry gfid self-heal triggered. path:
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up
> down flush, data lock is held
> [2012-04-12 03:09:41.474379] I
> [afr-self-heal-common.c:1203:sh_missing_entries_create]
> 0-poolsave-replicate-0: no missing files -
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to
> metadata check
> [2012-04-12 03:09:41.475981] I
> [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk]
> 0-poolsave-replicate-0: background  data missing-entry gfid self-heal
> completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
> [2012-04-12 03:09:41.476246] I [afr-open.c:432:afr_openfd_sh]
> 0-poolsave-replicate-0:  data missing-entry gfid self-heal triggered. path:
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up
> down flush, data lock is held
> [2012-04-12 03:09:41.477659] I
> [afr-self-heal-common.c:1203:sh_missing_entries_create]
> 0-poolsave-replicate-0: no missing files -
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to
> metadata check
> [2012-04-12 03:09:41.479086] I
> [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk]
> 0-poolsave-replicate-0: background  data missing-entry gfid self-heal
> completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
> [2012-04-12 03:09:41.479400] I [afr-open.c:432:afr_openfd_sh]
> 0-poolsave-replicate-0:  data missing-entry gfid self-heal triggered. path:
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up
> down flush, data lock is held
> [2012-04-12 03:09:41.480587] I
> [afr-self-heal-common.c:1203:sh_missing_entries_create]
> 0-poolsave-replicate-0: no missing files -
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to
> metadata check
> [2012-04-12 03:09:41.482176] I
> [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk]
> 0-poolsave-replicate-0: background  data missing-entry gfid self-heal
> completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
> [2012-04-12 03:09:41.482444] I [afr-open.c:432:afr_openfd_sh]
> 0-poolsave-replicate-0:  data missing-entry gfid self-heal triggered. path:
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up
> down flush, data lock is held
> [2012-04-12 03:09:41.483829] I
> [afr-self-heal-common.c:1203:sh_missing_entries_create]
> 0-poolsave-replicate-0: no missing files -
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to
> metadata check
> [2012-04-12 03:09:41.485441] I
> [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk]
> 0-poolsave-replicate-0: background  data missing-entry gfid self-heal
> completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
> [2012-04-12 03:09:41.485728] I [afr-open.c:432:afr_openfd_sh]
> 0-poolsave-replicate-0:  data missing-entry gfid self-heal triggered. path:
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up
> down flush, data lock is held
> [2012-04-12 03:09:41.487107] I
> [afr-self-heal-common.c:1203:sh_missing_entries_create]
> 0-poolsave-replicate-0: no missing files -
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to
> metadata check
> [2012-04-12 03:09:41.488585] I
> [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk]
> 0-poolsave-replicate-0: background  data missing-entry gfid self-heal
> completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
> [2012-04-12 03:09:41.488858] I [afr-open.c:432:afr_openfd_sh]
> 0-poolsave-replicate-0:  data missing-entry gfid self-heal triggered. path:
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar, reason: Replicate up
> down flush, data lock is held
> [2012-04-12 03:09:41.490300] I
> [afr-self-heal-common.c:1203:sh_missing_entries_create]
> 0-poolsave-replicate-0: no missing files -
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar. proceeding to
> metadata check
> [2012-04-12 03:09:41.491699] I
> [afr-self-heal-common.c:2077:afr_self_heal_completion_cbk]
> 0-poolsave-replicate-0: background  data missing-entry gfid self-heal
> completed on /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
> [2012-04-12 03:13:57.87121] W [quota.c:2165:quota_fstat_cbk]
> 0-poolsave-quota: quota context not set in inode (ino:-2969972712,
> gfid:9f59f832-ee26-455f-8230-a42e254c43de)
> [2012-04-12 03:13:57.90601] W [quota.c:2165:quota_fstat_cbk]
> 0-poolsave-quota: quota context not set in inode (ino:-2969972712,
> gfid:9f59f832-ee26-455f-8230-a42e254c43de)
> [2012-04-12 03:30:01.295850] W [quota.c:2165:quota_fstat_cbk]
> 0-poolsave-quota: quota context not set in inode (ino:-1737201406,
> gfid:a85246c1-e3b2-4dd4-afff-0e4d1c3b2634)
>
> A lot of those warning "quota context not set in inode" and then an another
> disconnection :
>
> [2012-04-12 05:09:43.310624] C
> [client-handshake.c:121:rpc_client_ping_timer_expired] 0-poolsave-client-3:
> server 10.80.49.13:24010 has not responded in the last 30 seconds,
> disconnecting.
> [2012-04-12 05:09:43.311087] E [rpc-clnt.c:341:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)
> [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)
> [0x7fc2a3fa605e]))) 0-poolsave-client-3: forced unwinding frame
> type(GlusterFS 3.1) op(FSYNC(16)) called at 2012-04-12 05:08:50.912592
> [2012-04-12 05:09:43.311243] I [client3_1-fops.c:779:client3_1_fsync_cbk]
> 0-poolsave-client-3: remote operation failed: Transport endpoint is not
> connected
> [2012-04-12 05:09:43.311341] E [rpc-clnt.c:341:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)
> [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)
> [0x7fc2a3fa605e]))) 0-poolsave-client-3: forced unwinding frame
> type(GlusterFS 3.1) op(WRITE(13)) called at 2012-04-12 05:09:05.303333
> [2012-04-12 05:09:43.311387] I [client3_1-fops.c:683:client3_1_writev_cbk]
> 0-poolsave-client-3: remote operation failed: Transport endpoint is not
> connected
> [2012-04-12 05:09:43.332543] I [socket.c:2275:socket_submit_request]
> 0-poolsave-client-3: not connected (priv->connected = 0)
> [2012-04-12 05:09:43.332650] W [rpc-clnt.c:1417:rpc_clnt_submit]
> 0-poolsave-client-3: failed to submit rpc-request (XID: 0x13367995x Program:
> GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport (poolsave-client-3)
> [2012-04-12 05:09:43.332763] W [rpc-clnt.c:1417:rpc_clnt_submit]
> 0-poolsave-client-3: failed to submit rpc-request (XID: 0x13367996x Program:
> GlusterFS 3.1, ProgVers: 310, Proc: 30) to rpc-transport (poolsave-client-3)
> [2012-04-12 05:09:43.332821] E [rpc-clnt.c:341:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)
> [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)
> [0x7fc2a3fa605e]))) 0-poolsave-client-3: forced unwinding frame
> type(GlusterFS 3.1) op(WRITE(13)) called at 2012-04-12 05:09:05.303419
> [2012-04-12 05:09:43.332891] E
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-3: remote
> operation failed: Transport endpoint is not connected
> [2012-04-12 05:09:43.333319] I [client3_1-fops.c:683:client3_1_writev_cbk]
> 0-poolsave-client-3: remote operation failed: Transport endpoint is not
> connected
> [2012-04-12 05:09:43.333730] W [rpc-clnt.c:1417:rpc_clnt_submit]
> 0-poolsave-client-3: failed to submit rpc-request (XID: 0x13367997x Program:
> GlusterFS 3.1, ProgVers: 310, Proc: 34) to rpc-transport (poolsave-client-3)
> [2012-04-12 05:09:43.333852] E [rpc-clnt.c:341:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)
> [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)
> [0x7fc2a3fa605e]))) 0-poolsave-client-3: forced unwinding frame
> type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 05:09:05.303477
> [2012-04-12 05:09:43.333900] E
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-3: remote
> operation failed: Transport endpoint is not connected
> [2012-04-12 05:09:43.334017] E [rpc-clnt.c:341:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)
> [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)
> [0x7fc2a3fa605e]))) 0-poolsave-client-3: forced unwinding frame
> type(GlusterFS 3.1) op(FINODELK(30)) called at 2012-04-12 05:09:05.303508
> [2012-04-12 05:09:43.334062] E
> [client3_1-fops.c:1264:client3_1_finodelk_cbk] 0-poolsave-client-3: remote
> operation failed: Transport endpoint is not connected
> [2012-04-12 05:09:43.334154] E [rpc-clnt.c:341:saved_frames_unwind]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x7fc2a3fa6939]
> (-->/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e)
> [0x7fc2a3fa60ee] (-->/usr/local/lib/libgfrpc.so.0(saved_frames_destroy+0xe)
> [0x7fc2a3fa605e]))) 0-poolsave-client-3: forced unwinding frame
> type(GlusterFS 3.1) op(WRITE(13)) called at 2012-04-12 05:09:05.303758
>
>
> Log of ylal3530 for fisrt disconnection of 3:09am :
> (log/glusterfs/bricks/users3-poolsave.log)
>
> [2012-04-12 03:09:23.346130] W [socket.c:204:__socket_rwv]
> 0-tcp.poolsave-server: writev failed (Broken pipe)
> [2012-04-12 03:09:23.346328] I [socket.c:1768:socket_event_handler]
> 0-transport: disconnecting now
> [2012-04-12 03:09:23.346452] I [server-helpers.c:343:do_lock_table_cleanup]
> 0-poolsave-server: finodelk released on
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
> [2012-04-12 03:09:23.346530] I [server-helpers.c:343:do_lock_table_cleanup]
> 0-poolsave-server: finodelk released on
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
> [2012-04-12 03:09:23.346551] I [server-helpers.c:343:do_lock_table_cleanup]
> 0-poolsave-server: finodelk released on
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
> [2012-04-12 03:09:23.346604] I [socket.c:2338:socket_submit_reply]
> 0-tcp.poolsave-server: not connected (priv->connected = 255)
> [2012-04-12 03:09:23.346625] W [inode.c:1044:inode_path]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_connection_cleanup+0xa6)
> [0x7f5ec426d106]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_connection_cleanup+0x139)
> [0x7f5ec426ce09]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_fd_cleanup+0x32c)
> [0x7f5ec426bb6c]))) 0-/users3/poolsave/inode: no dentry for non-root inode
> 400019122: 70c18c81-cc2d-4698-b28d-2fba654a9ef8
> [2012-04-12 03:09:23.346714] E [rpcsvc.c:1033:rpcsvc_submit_generic]
> 0-rpc-service: failed to submit message (XID: 0x10709543x, Program:
> GlusterFS 3.2.5, ProgVers: 310, Proc: 13) to rpc-transport
> (tcp.poolsave-server)
> [2012-04-12 03:09:23.346780] I [server-helpers.c:492:do_fd_cleanup]
> 0-poolsave-server: fd cleanup on ino 400019122 with gfid
> 70c18c81-cc2d-4698-b28d-2fba654a9ef8
> [2012-04-12 03:09:23.347047] E [server.c:136:server_submit_reply]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/features/marker.so(marker_writev_cbk+0xa9)
> [0x7f5ec44ae109]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/debug/io-stats.so(io_stats_writev_cbk+0x1b3)
> [0x7f5ec439be73]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_writev_cbk+0xa8)
> [0x7f5ec42760e8]))) 0-: Reply submission failed
> [2012-04-12 03:09:23.347074] W [inode.c:1044:inode_path]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_connection_cleanup+0xa6)
> [0x7f5ec426d106]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_connection_cleanup+0x139)
> [0x7f5ec426ce09]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_fd_cleanup+0x32c)
> [0x7f5ec426bb6c]))) 0-/users3/poolsave/inode: no dentry for non-root inode
> -3453277438: d0344eb6-6f41-4d14-8855-e94fa9d541b3
> [2012-04-12 03:09:23.347162] I [server-helpers.c:492:do_fd_cleanup]
> 0-poolsave-server: fd cleanup on ino -3453277438 with gfid
> d0344eb6-6f41-4d14-8855-e94fa9d541b3
> [2012-04-12 03:09:23.347187] I [server-helpers.c:485:do_fd_cleanup]
> 0-poolsave-server: fd cleanup on
> /yval9000/test/tmp/12-04-03-00_test_glusterfs_save.tar
> [2012-04-12 03:09:23.347204] I [server.c:438:server_rpc_notify]
> 0-poolsave-server: disconnected connection from 10.80.49.10:1015
> [2012-04-12 03:09:23.347226] I
> [server-helpers.c:783:server_connection_destroy] 0-poolsave-server:
> destroyed connection of
> ylal3510-25300-2012/04/11-09:11:19:688358-poolsave-client-1
> [2012-04-12 03:09:23.348129] I [server-handshake.c:542:server_setvolume]
> 0-poolsave-server: accepted client from 10.80.49.10:1008 (version: 3.2.5)
>
>
> log of ylal3540 for disconnection of 5:09 am :
>
> [2012-04-12 05:09:47.540357] W [socket.c:1494:__socket_proto_state_machine]
> 0-tcp.poolsave-server: reading from socket failed. Error (Transport endpoint
> is not connected), peer (10.80.49.10:1012)
> [2012-04-12 05:09:47.540473] I [server-helpers.c:343:do_lock_table_cleanup]
> 0-poolsave-server: finodelk released on
> /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar
> [2012-04-12 05:09:47.540507] I [server-helpers.c:343:do_lock_table_cleanup]
> 0-poolsave-server: finodelk released on
> /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar
> [2012-04-12 05:09:47.540525] I [server-helpers.c:343:do_lock_table_cleanup]
> 0-poolsave-server: finodelk released on
> /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar
> [2012-04-12 05:09:47.540543] I [server-helpers.c:343:do_lock_table_cleanup]
> 0-poolsave-server: finodelk released on
> /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar
> [2012-04-12 05:09:47.540564] I [server-helpers.c:343:do_lock_table_cleanup]
> 0-poolsave-server: finodelk released on
> /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar
> [2012-04-12 05:09:47.540582] I [server-helpers.c:343:do_lock_table_cleanup]
> 0-poolsave-server: finodelk released on
> /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar
> [2012-04-12 05:09:47.540583] I [socket.c:2338:socket_submit_reply]
> 0-tcp.poolsave-server: not connected (priv->connected = 255)
> [2012-04-12 05:09:47.540602] I [server-helpers.c:343:do_lock_table_cleanup]
> 0-poolsave-server: finodelk released on
> /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar
> [2012-04-12 05:09:47.540807] I [server-helpers.c:343:do_lock_table_cleanup]
> 0-poolsave-server: finodelk released on
> /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar
> [2012-04-12 05:09:47.540705] E [rpcsvc.c:1033:rpcsvc_submit_generic]
> 0-rpc-service: failed to submit message (XID: 0x13367977x, Program:
> GlusterFS 3.2.5, ProgVers: 310, Proc: 13) to rpc-transport
> (tcp.poolsave-server)
> [2012-04-12 05:09:47.540831] I [server-helpers.c:343:do_lock_table_cleanup]
> 0-poolsave-server: finodelk released on
> /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar
> [2012-04-12 05:09:47.540739] E [rpcsvc.c:1033:rpcsvc_submit_generic]
> 0-rpc-service: failed to submit message (XID: 0x13367984x, Program:
> GlusterFS 3.2.5, ProgVers: 310, Proc: 13) to rpc-transport
> (tcp.poolsave-server)
> [2012-04-12 05:09:47.540935] I [server-helpers.c:343:do_lock_table_cleanup]
> 0-poolsave-server: finodelk released on
> /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar
> [2012-04-12 05:09:47.541141] E [server.c:136:server_submit_reply]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/features/marker.so(marker_writev_cbk+0xa9)
> [0x7f9952011109]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/debug/io-stats.so(io_stats_writev_cbk+0x1b3)
> [0x7f9951efee73]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_writev_cbk+0xa8)
> [0x7f9951dd90e8]))) 0-: Reply submission failed
> [2012-04-12 05:09:47.541059] E [server.c:136:server_submit_reply]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/features/marker.so(marker_writev_cbk+0xa9)
> [0x7f9952011109]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/debug/io-stats.so(io_stats_writev_cbk+0x1b3)
> [0x7f9951efee73]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_writev_cbk+0xa8)
> [0x7f9951dd90e8]))) 0-: Reply submission failed
> [2012-04-12 05:09:47.541252] E [server.c:136:server_submit_reply]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/features/marker.so(marker_writev_cbk+0xa9)
> [0x7f9952011109]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/debug/io-stats.so(io_stats_writev_cbk+0x1b3)
> [0x7f9951efee73]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_writev_cbk+0xa8)
> [0x7f9951dd90e8]))) 0-: Reply submission failed
> [2012-04-12 05:09:47.541210] E [rpcsvc.c:1033:rpcsvc_submit_generic]
> 0-rpc-service: failed to submit message (XID: 0x13367987x, Program:
> GlusterFS 3.2.5, ProgVers: 310, Proc: 13) to rpc-transport
> (tcp.poolsave-server)
> [2012-04-12 05:09:47.541164] I [server-helpers.c:343:do_lock_table_cleanup]
> 0-poolsave-server: finodelk released on
> /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar
> [2012-04-12 05:09:47.541552] E [server.c:136:server_submit_reply]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/features/marker.so(marker_writev_cbk+0xa9)
> [0x7f9952011109]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/debug/io-stats.so(io_stats_writev_cbk+0x1b3)
> [0x7f9951efee73]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_writev_cbk+0xa8)
> [0x7f9951dd90e8]))) 0-: Reply submission failed
> [2012-04-12 05:09:47.541593] I [server-helpers.c:343:do_lock_table_cleanup]
> 0-poolsave-server: finodelk released on
> /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar
> [2012-04-12 05:09:47.541618] E [rpcsvc.c:1033:rpcsvc_submit_generic]
> 0-rpc-service: failed to submit message (XID: 0x13367988x, Program:
> GlusterFS 3.2.5, ProgVers: 310, Proc: 13) to rpc-transport
> (tcp.poolsave-server)
> [2012-04-12 05:09:47.541665] I [server-helpers.c:343:do_lock_table_cleanup]
> 0-poolsave-server: finodelk released on
> /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar
> [2012-04-12 05:09:47.541788] E [server.c:136:server_submit_reply]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/features/marker.so(marker_writev_cbk+0xa9)
> [0x7f9952011109]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/debug/io-stats.so(io_stats_writev_cbk+0x1b3)
> [0x7f9951efee73]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_writev_cbk+0xa8)
> [0x7f9951dd90e8]))) 0-: Reply submission failed
> [2012-04-12 05:09:47.541837] W [inode.c:1044:inode_path]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_connection_cleanup+0xa6)
> [0x7f9951dd0106]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_connection_cleanup+0x139)
> [0x7f9951dcfe09]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_fd_cleanup+0x32c)
> [0x7f9951dceb6c]))) 0-/users3/poolsave/inode: no dentry for non-root inode
> -450063628: bfdedf4b-671e-4d2f-ba7a-27362ab26bbe
> [2012-04-12 05:09:47.541929] I [server-helpers.c:492:do_fd_cleanup]
> 0-poolsave-server: fd cleanup on ino -450063628 with gfid
> bfdedf4b-671e-4d2f-ba7a-27362ab26bbe
> [2012-04-12 05:09:47.541983] W [inode.c:1044:inode_path]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_connection_cleanup+0xa6)
> [0x7f9951dd0106]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_connection_cleanup+0x139)
> [0x7f9951dcfe09]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_fd_cleanup+0x32c)
> [0x7f9951dceb6c]))) 0-/users3/poolsave/inode: no dentry for non-root inode
> 393879387: 157da663-ff30-4ba9-bd24-0dc45a561197
> [2012-04-12 05:09:47.542007] I [server-helpers.c:492:do_fd_cleanup]
> 0-poolsave-server: fd cleanup on ino 393879387 with gfid
> 157da663-ff30-4ba9-bd24-0dc45a561197
> [2012-04-12 05:09:47.542106] W [inode.c:1044:inode_path]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_connection_cleanup+0xa6)
> [0x7f9951dd0106]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_connection_cleanup+0x139)
> [0x7f9951dcfe09]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_fd_cleanup+0x32c)
> [0x7f9951dceb6c]))) 0-/users3/poolsave/inode: no dentry for non-root inode
> -1147302459: e88b34c2-9453-4e6b-b4e0-066706bd875e
> [2012-04-12 05:09:47.542129] I [server-helpers.c:492:do_fd_cleanup]
> 0-poolsave-server: fd cleanup on ino -1147302459 with gfid
> e88b34c2-9453-4e6b-b4e0-066706bd875e
> [2012-04-12 05:09:47.542173] W [inode.c:1044:inode_path]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(server_connection_cleanup+0xa6)
> [0x7f9951dd0106]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_connection_cleanup+0x139)
> [0x7f9951dcfe09]
> (-->/usr/local/lib//glusterfs/3.2.5/xlator/protocol/server.so(do_fd_cleanup+0x32c)
> [0x7f9951dceb6c]))) 0-/users3/poolsave/inode: no dentry for non-root inode
> -462949261: dbec18c6-4f55-442d-ba76-267929f1cdfa
> [2012-04-12 05:09:47.542194] I [server-helpers.c:492:do_fd_cleanup]
> 0-poolsave-server: fd cleanup on ino -462949261 with gfid
> dbec18c6-4f55-442d-ba76-267929f1cdfa
> [2012-04-12 05:09:47.542231] I [server-helpers.c:485:do_fd_cleanup]
> 0-poolsave-server: fd cleanup on
> /yval9000/test/tmp/12-04-05-00_test_glusterfs_save.tar
> [2012-04-12 05:09:47.542247] I [server.c:438:server_rpc_notify]
> 0-poolsave-server: disconnected connection from 10.80.49.10:1012
> [2012-04-12 05:09:47.542282] I
> [server-helpers.c:783:server_connection_destroy] 0-poolsave-server:
> destroyed connection of
> ylal3510-25300-2012/04/11-09:11:19:688358-poolsave-client-3
> [2012-04-12 05:09:47.542349] I [server-handshake.c:542:server_setvolume]
> 0-poolsave-server: accepted client from 10.80.49.10:1015 (version: 3.2.5)
>
>
>
> Regards,
>
> Anthony
>
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>