[Gluster-devel] Unify/AFR crashes

Mon Aug 11 10:58:07 UTC 2008

Hi!

2008/8/11 Anand Avati <avati at zresearch.com>:
> can you please repost the logs "with" the timestamps?
>

Here there are:
-----------
2008-07-28 16:04:47 E [client-protocol.c:4430:client_lookup_cbk] c54:
no proper reply from server, returning ENOTCONN
2008-07-28 16:05:38 W [client-protocol.c:205:call_bail] c-ns:
activating bail-out. pending frames = 11. last sent = 2008-07-28
16:04:47. last received = 2008-07-28 16:03:17 transport-timeout = 42
2008-07-28 16:05:38 C [client-protocol.c:212:call_bail] c-ns: bailing transport
2008-07-28 16:05:38 W [client-protocol.c:4784:client_protocol_cleanup]
c-ns: cleaning up state in transport object 0x64d310
2008-07-28 16:05:38 E [client-protocol.c:4834:client_protocol_cleanup]
c-ns: forced unwinding frame type(1) op(36) reply=@0xc165b0
2008-07-28 16:05:38 E [client-protocol.c:4215:client_setdents_cbk]
c-ns: no proper reply from server, returning ENOTCONN
2008-07-28 16:05:38 E [client-protocol.c:4834:client_protocol_cleanup]
c-ns: forced unwinding frame type(1) op(36) reply=@0xc165b0
2008-07-28 16:05:38 E [client-protocol.c:4215:client_setdents_cbk]
c-ns: no proper reply from server, returning ENOTCONN
2008-07-28 16:05:38 E [client-protocol.c:4834:client_protocol_cleanup]
c-ns: forced unwinding frame type(1) op(36) reply=@0xc165b0
2008-07-28 16:05:38 E [client-protocol.c:4215:client_setdents_cbk]
c-ns: no proper reply from server, returning ENOTCONN
2008-07-28 16:05:38 E [client-protocol.c:4834:client_protocol_cleanup]
c-ns: forced unwinding frame type(1) op(36) reply=@0xc165b0
2008-07-28 16:05:38 E [client-protocol.c:4215:client_setdents_cbk]
c-ns: no proper reply from server, returning ENOTCONN
2008-07-28 16:05:38 E [client-protocol.c:4834:client_protocol_cleanup]
c-ns: forced unwinding frame type(1) op(36) reply=@0xc165b0
2008-07-28 16:05:38 E [client-protocol.c:4215:client_setdents_cbk]
c-ns: no proper reply from server, returning ENOTCONN
2008-07-28 16:05:38 E [client-protocol.c:4834:client_protocol_cleanup]
c-ns: forced unwinding frame type(1) op(23) reply=@0xc165b0
2008-07-28 16:05:38 E [client-protocol.c:3310:client_getdents_cbk]
c-ns: no proper reply from server, returning ENOTCONN
2008-07-28 16:05:38 W [client-protocol.c:1711:client_closedir] c51: no
proper fd found, returning
2008-07-28 16:05:38 W [client-protocol.c:1711:client_closedir] c-ns:
no proper fd found, returning
2008-07-28 16:05:38 E [client-protocol.c:4834:client_protocol_cleanup]
c-ns: forced unwinding frame type(1) op(34) reply=@0xc165b0
2008-07-28 16:05:38 E [client-protocol.c:4430:client_lookup_cbk] c-ns:
no proper reply from server, returning ENOTCONN
2008-07-28 16:05:38 E [unify.c:182:unify_lookup_cbk] bricks: c-ns returned 107
2008-07-28 16:05:38 E [fuse-bridge.c:468:fuse_entry_cbk]
glusterfs-fuse: 1976200: (34) /nova => -1 (107)
2008-07-28 16:05:38 W [client-protocol.c:280:client_protocol_xfer]
c51: attempting to pipeline request type(1) op(34) with handshake
2008-07-28 16:05:38 W [client-protocol.c:280:client_protocol_xfer]
c37: attempting to pipeline request type(1) op(34) with handshake
2008-07-28 16:05:38 W [client-protocol.c:280:client_protocol_xfer]
c54: attempting to pipeline request type(1) op(34) with handshake
2008-07-28 16:05:38 E [client-protocol.c:325:client_protocol_xfer]
c-ns: transport_submit failed
2008-07-28 16:05:38 E [client-protocol.c:4834:client_protocol_cleanup]
c-ns: forced unwinding frame type(1) op(34) reply=@0xc165b0
2008-07-28 16:05:38 E [client-protocol.c:4430:client_lookup_cbk] c-ns:
no proper reply from server, returning ENOTCONN
2008-07-28 16:05:38 E [fuse-bridge.c:468:fuse_entry_cbk]
glusterfs-fuse: 1976201: (34) / => -1 (2)
2008-07-28 16:05:38 E [client-protocol.c:4834:client_protocol_cleanup]
c-ns: forced unwinding frame type(1) op(0) reply=@0xc165b0
2008-07-28 16:05:38 E [client-protocol.c:2688:client_stat_cbk] c-ns:
no proper reply from server, returning ENOTCONN
2008-07-28 16:05:38 E [unify.c:118:unify_buf_cbk] bricks: c-ns returned 107
2008-07-28 16:05:38 E [fuse-bridge.c:589:fuse_attr_cbk]
glusterfs-fuse: 1976202: (0) /nova/nio1/new_free/m085am200_low_turb =>
-1 (107)
2008-07-28 16:05:38 E [client-protocol.c:4834:client_protocol_cleanup]
c-ns: forced unwinding frame type(2) op(9) reply=@0xc165b0
2008-07-28 16:05:38 E [client-protocol.c:4613:client_checksum_cbk]
c-ns: no proper reply from server, returning ENOTCONN
2008-07-28 16:05:38 E [client-protocol.c:4834:client_protocol_cleanup]
c-ns: forced unwinding frame type(2) op(9) reply=@0xc165b0
2008-07-28 16:05:38 E [client-protocol.c:4613:client_checksum_cbk]
c-ns: no proper reply from server, returning ENOTCONN
2008-07-28 16:05:38 W [client-protocol.c:280:client_protocol_xfer]
c54: attempting to pipeline request type(1) op(35) with handshake
2008-07-28 16:05:38 W [client-protocol.c:280:client_protocol_xfer]
c-ns: attempting to pipeline request type(1) op(35) with handshake
2008-07-28 16:05:38 W [client-protocol.c:280:client_protocol_xfer]
c37: attempting to pipeline request type(1) op(35) with handshake
2008-07-28 16:05:38 W [client-protocol.c:280:client_protocol_xfer]
c51: attempting to pipeline request type(1) op(35) with handshake
2008-07-28 16:05:38 E [unify.c:325:unify_lookup] bricks: returning
ESTALE for /nova/nio1 [translator generation (27) inode generation
(25)]
2008-07-28 16:05:38 E [fuse-bridge.c:468:fuse_entry_cbk]
glusterfs-fuse: 1976204: (34) /nova/nio1 => -1 (116)
2008-07-28 16:05:38 W [client-protocol.c:280:client_protocol_xfer]
c51: attempting to pipeline request type(1) op(34) with handshake
2008-07-28 16:05:38 W [client-protocol.c:280:client_protocol_xfer]
c37: attempting to pipeline request type(1) op(34) with handshake
2008-07-28 16:05:38 W [client-protocol.c:280:client_protocol_xfer]
c54: attempting to pipeline request type(1) op(34) with handshake
2008-07-28 16:05:38 W [client-protocol.c:280:client_protocol_xfer]
c-ns: attempting to pipeline request type(1) op(34) with handshake
2008-07-28 16:06:24 W [client-protocol.c:205:call_bail] c37:
activating bail-out. pending frames = 3. last sent = 2008-07-28
16:05:38. last received = 1970-01-01 03:00:00 transport-timeout = 42
-----------
2008-07-28 16:06:24 E [client-protocol.c:4430:client_lookup_cbk] c37:
no proper reply from server, returning ENOTCONN
2008-07-28 16:06:28 W [client-protocol.c:205:call_bail] c54:
activating bail-out. pending frames = 3. last sent = 2008-07-28
16:05:38. last received = 1970-01-01 03:00:00 transport-timeout = 42
2008-07-28 16:06:28 C [client-protocol.c:212:call_bail] c54: bailing transport
2008-07-28 16:06:28 W [client-protocol.c:205:call_bail] c-ns:
activating bail-out. pending frames = 3. last sent = 2008-07-28
16:05:38. last received = 1970-01-01 03:00:00 transport-timeout = 42
2008-07-28 16:06:28 C [client-protocol.c:212:call_bail] c-ns: bailing transport
-----------
2008-07-28 16:06:28 W [client-protocol.c:332:client_protocol_xfer]
c-ns: not connected at the moment to submit frame type(1) op(35)
2008-07-28 16:06:28 W [client-protocol.c:332:client_protocol_xfer]
c-ns: not connected at the moment to submit frame type(1) op(35)
2008-07-28 16:06:28 W [client-protocol.c:332:client_protocol_xfer]
c-ns: not connected at the moment to submit frame type(1) op(35)
And hundreds of the same lines during this and following second.
-----------
2008-07-28 16:10:36 E [client-protocol.c:4430:client_lookup_cbk] c54:
no proper reply from server, returning ENOTCONN
2008-07-28 16:11:21 W [client-protocol.c:205:call_bail] c-ns:
activating bail-out. pending frames = 5. last sent = 2008-07-28
16:10:36. last received = 2008-07-28 16:10:36 transport-timeout = 42
2008-07-28 16:11:21 C [client-protocol.c:212:call_bail] c-ns: bailing transport
2008-07-28 16:11:21 W [client-protocol.c:4784:client_protocol_cleanup]
c-ns: cleaning up state in transport object 0x64d310
2008-07-28 16:11:21 E [client-protocol.c:4834:client_protocol_cleanup]
c-ns: forced unwinding frame type(1) op(36) reply=@0x2aaab0048720
2008-07-28 16:11:21 E [client-protocol.c:4215:client_setdents_cbk]
c-ns: no proper reply from server, returning ENOTCONN
2008-07-28 16:11:21 E [client-protocol.c:4834:client_protocol_cleanup]
c-ns: forced unwinding frame type(1) op(36) reply=@0x2aaab0048720
2008-07-28 16:11:21 E [client-protocol.c:4215:client_setdents_cbk]
c-ns: no proper reply from server, returning ENOTCONN
2008-07-28 16:11:21 E [client-protocol.c:4834:client_protocol_cleanup]
c-ns: forced unwinding frame type(1) op(23) reply=@0x2aaab0048720
2008-07-28 16:11:21 E [client-protocol.c:3310:client_getdents_cbk]
c-ns: no proper reply from server, returning ENOTCONN
2008-07-28 16:11:21 W [client-protocol.c:1711:client_closedir] c-ns:
no proper fd found, returning
2008-07-28 16:11:21 E [client-protocol.c:4834:client_protocol_cleanup]
c-ns: forced unwinding frame type(1) op(34) reply=@0x2aaab0048720
2008-07-28 16:11:21 E [client-protocol.c:4430:client_lookup_cbk] c-ns:
no proper reply from server, returning ENOTCONN
2008-07-28 16:11:21 E [client-protocol.c:4834:client_protocol_cleanup]
c-ns: forced unwinding frame type(1) op(22) reply=@0x2aaab0048720
2008-07-28 16:11:21 E [client-protocol.c:3767:client_opendir_cbk]
c-ns: no proper reply from server, returning ENOTCONN
2008-07-28 16:12:08 W [client-protocol.c:205:call_bail] c51:
activating bail-out. pending frames = 3. last sent = 2008-07-28
16:11:21. last received = 2008-07-28 16:10:36 transport-timeout = 42
2008-07-28 16:12:08 C [client-protocol.c:212:call_bail] c51: bailing transport
2008-07-28 16:12:08 W [client-protocol.c:4784:client_protocol_cleanup]
c51: cleaning up state in transport object 0x614600
2008-07-28 16:12:08 E [client-protocol.c:4834:client_protocol_cleanup]
c51: forced unwinding frame type(1) op(34) reply=@0x2aaab00180c0
2008-07-28 16:12:08 E [client-protocol.c:4430:client_lookup_cbk] c51:
no proper reply from server, returning ENOTCONN
2008-07-28 16:12:08 E [fuse-bridge.c:468:fuse_entry_cbk]
glusterfs-fuse: 1978748: (34) /nova => -1 (2)
2008-07-28 16:12:08 E [client-protocol.c:4834:client_protocol_cleanup]
c51: forced unwinding frame type(1) op(22) reply=@0x2aaab00180c0
----------
2008-08-04 13:10:09 E [unify.c:118:unify_buf_cbk] bricks: c41 returned 77
2008-08-04 13:10:09 E [client-protocol.c:1884:client_fstat] c-ns: :
returning EBADFD
2008-08-04 13:10:09 E [unify.c:118:unify_buf_cbk] bricks: c-ns returned 77
2008-08-04 13:10:09 E [fuse-bridge.c:589:fuse_attr_cbk]
glusterfs-fuse: 9731: (29)
/ansys11/ansys_inc/shared_files/licensing/linem64t/ansyslmd => -1 (77)
----------

Hope this helps. I can upload full logs somewhere.

Regards!
  Andrey

>
> 2008/8/11 NovA <av.nova at gmail.com>
>>
>> Hi!
>>
>> I'm using glusterfs-1.3.9tla790 unify (without AFR) and also
>> periodically has troubles with NS. Sometimes, when many files are
>> copied to unify volume, the operation is stalled. And the client log
>> is flooded by messages like:
>> W [client-protocol.c:332:client_protocol_xfer] c-ns: not connected at
>> the moment to submit frame type(1) op(35)
>> The glusterfs server not really crash, but eat 100% CPU. Restarting it
>> restores the normal work.
>>
>> There are also many other messages concerning namespace, like
>> ------
>> W [client-protocol.c:1711:client_closedir] c-ns: no proper fd found,
>> returning
>> E [client-protocol.c:4834:client_protocol_cleanup] c-ns: forced
>> unwinding frame type(1) op(34) reply=@0xc165b0
>> E [client-protocol.c:4430:client_lookup_cbk] c-ns: no proper reply
>> from server, returning ENOTCONN
>> E [unify.c:182:unify_lookup_cbk] bricks: c-ns returned 107
>> ------
>> W [client-protocol.c:205:call_bail] c-ns: activating bail-out. pending
>> frames = 3. last sent = 2008-07-28 16:05:38. last received =
>> 1970-01-01 03:00:00 transport-timeout = 42
>> C [client-protocol.c:212:call_bail] c-ns: bailing transport
>> ------
>>      Note the strange last received date...
>> -----
>> W [client-protocol.c:280:client_protocol_xfer] c-ns: attempting to
>> pipeline request type(1) op(34) with handshake
>> -----
>> W [client-protocol.c:4784:client_protocol_cleanup] c-ns: cleaning up
>> state in transport object 0x64
>> E [client-protocol.c:4834:client_protocol_cleanup] c-ns: forced
>> unwinding frame type(1) op(34)  reply=@0x66c140
>> E [client-protocol.c:4430:client_lookup_cbk] c-ns: no proper reply
>> from server, returning ENOTCONN
>> ----
>> [client-protocol.c:4784:client_protocol_cleanup] c-ns: cleaning up
>> state in transport object 0x64d310
>> [client-protocol.c:4834:client_protocol_cleanup] c-ns: forced
>> unwinding frame type(1) op(36) reply=@0x2aaab0048720
>> [client-protocol.c:4215:client_setdents_cbk] c-ns: no proper reply
>> from server, returning ENOTCONN
>> [client-protocol.c:4834:client_protocol_cleanup] c-ns: forced
>> unwinding frame type(1) op(36) reply=@0x2aaab0048720
>> [client-protocol.c:4215:client_setdents_cbk] c-ns: no proper reply
>> from server, returning ENOTCONN
>> [client-protocol.c:4834:client_protocol_cleanup] c-ns: forced
>> unwinding frame type(1) op(23) reply=@0x2aaab0048720
>> [client-protocol.c:3310:client_getdents_cbk] c-ns: no proper reply
>> from server, returning ENOTCONN
>> [client-protocol.c:1711:client_closedir] c-ns: no proper fd found,
>> returning
>> [client-protocol.c:4834:client_protocol_cleanup] c-ns: forced
>> unwinding frame type(1) op(34) reply=@0x2aaab0048720
>> [client-protocol.c:4430:client_lookup_cbk] c-ns: no proper reply from
>> server, returning ENOTCONN
>> [client-protocol.c:4834:client_protocol_cleanup] c-ns: forced
>> unwinding frame type(1) op(22) reply=@0x2aaab0048720
>> [client-protocol.c:3767:client_opendir_cbk] c-ns: no proper reply from
>> server, returning ENOTCONN
>> ----
>> E [client-protocol.c:1884:client_fstat] c-ns: : returning EBADFD
>> E [unify.c:118:unify_buf_cbk] bricks: c-ns returned 77
>> ----
>>
>> All these are from different places of the log (rather huge now), and
>> probably not connected to each other. It's just to show different
>> types of warnings and errors I've really saw.
>>
>> The name-space in my case is just a folder on common ext3 volume. It's
>> for testing purposes, I'm going to use separate reiserfs volume for it
>> later. But probably it's somehow connected with NS problems which I'm
>> seeing now.
>>