[Gluster-users] Glusterfs distributed volume stops responding with bailing out frame type(GlusterFS 3.1) op(LOOKUP(27))

[Stanislav Alekseev]

Glusterfs 3.1.1 volume stops responding after some time until I perform 
"volume stop; volume start". The following messages are displayed in the 
client's log:

04:29:41.286651. timeout = 1800
[2011-05-02 05:29:44.952153] E [rpc-clnt.c:199:call_bail] 
cccbstore1-client-0: bailing out frame type(GlusterFS 3.1) 
op(LOOKUP(27)) xid = 0x311c sent = 2011-05-02 04:59:43.110723. timeout = 
1800
[2011-05-02 05:59:46.773153] E [rpc-clnt.c:199:call_bail] 
cccbstore1-client-0: bailing out frame type(GlusterFS 3.1) 
op(LOOKUP(27)) xid = 0x311d sent = 2011-05-02 05:29:44.952650. timeout = 
1800
[2011-05-02 06:29:48.597153] E [rpc-clnt.c:199:call_bail] 
cccbstore1-client-0: bailing out frame type(GlusterFS 3.1) 
op(LOOKUP(27)) xid = 0x311e sent = 2011-05-02 05:59:46.773627. timeout = 
1800
[2011-05-02 06:59:50.435152] E [rpc-clnt.c:199:call_bail] 
cccbstore1-client-0: bailing out frame type(GlusterFS 3.1) 
op(LOOKUP(27)) xid = 0x311f sent = 2011-05-02 06:29:48.597779. timeout = 
1800
[2011-05-02 07:19:46.460771] C 
[rpc-clnt.c:430:rpc_clnt_fill_request_info] rpc-clnt: cannot lookup the 
saved frame corresponding to xid (12556) for msg arrived on transport 
cccbstore1-client-0
[2011-05-02 07:19:46.461094] E [rpc-clnt.c:338:saved_frames_unwind] 
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0xb9) [0x2b512972f769] 
(-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x7e) 
[0x2b512972ef1e] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) 
[0x2b512972ee8e]))) rpc-clnt: forced unwinding frame type(GlusterFS 3.1) 
op(LOOKUP(27)) called at 2011-05-02 06:59:50.435645
[2011-05-02 07:19:46.658640] I 
[client-handshake.c:993:select_server_supported_programs] 
cccbstore1-client-0: Using Program GlusterFS-3.1.0, Num (1298437), 
Version (310)
[2011-05-02 07:19:46.659111] I 
[client-handshake.c:829:client_setvolume_cbk] cccbstore1-client-0: 
Connected to 155.52.49.32:24010, attached to remote volume 
'/u01/cccbstore1'.
[2011-05-02 07:20:41.726069] I 
[client-handshake.c:993:select_server_supported_programs] 
cccbstore1-client-0: Using Program GlusterFS-3.1.0, Num (1298437), 
Version (310)
[2011-05-02 07:20:41.726583] I 
[client-handshake.c:829:client_setvolume_cbk] cccbstore1-client-0: 
Connected to 155.52.49.32:24010, attached to remote volume 
'/u01/cccbstore1'.
[2011-05-02 07:20:42.737992] I 
[client-handshake.c:993:select_server_supported_programs] 
cccbstore1-client-3: Using Program GlusterFS-3.1.0, Num (1298437), 
Version (310)
[2011-05-02 07:20:42.738430] I 
[client-handshake.c:829:client_setvolume_cbk] cccbstore1-client-3: 
Connected to 155.52.49.35:24010, attached to remote volume 
'/u01/cccbstore1'.
[2011-05-02 07:20:42.748585] I 
[client-handshake.c:993:select_server_supported_programs] 
cccbstore1-client-2: Using Program GlusterFS-3.1.0, Num (1298437), 
Version (310)
[2011-05-02 07:20:42.749036] I 
[client-handshake.c:829:client_setvolume_cbk] cccbstore1-client-2: 
Connected to 155.52.49.34:24010, attached to remote volume 
'/u01/cccbstore1'.
[2011-05-02 07:20:42.762387] I 
[client-handshake.c:993:select_server_supported_programs] 
cccbstore1-client-1: Using Program GlusterFS-3.1.0, Num (1298437), 
Version (310)
[2011-05-02 07:20:42.763134] I 
[client-handshake.c:829:client_setvolume_cbk] cccbstore1-client-1: 
Connected to 155.52.49.33:24010, attached to remote volume 
'/u01/cccbstore1'.

Please advice.

Thanks.