[Gluster-users] 答复: glusterd crashed
Atin Mukherjee
amukherj at redhat.com
Fri May 8 16:41:22 UTC 2015
The gluster version is pretty old i.e. 3.4 and we have already moved to
3.6 and 3.7 is round the corner. I would recommend you to upgrade your
cluster to 3.6 and see if you hit the same issue again ?
~Atin
On 05/08/2015 03:15 PM, vyyy杨雨阳 wrote:
> Following is backTrace of the etc-glusterfs-glusterd.vol.log file, thanks
>
>
> [2015-05-07 11:56:32.228232] E [glusterd-utils.c:332:glusterd_lock] 0-management: Unable to get lock for uuid: ea17d7f9-d737-4472-ab9a-feed3cfac57c, lock held by: ea17d7f9-d737-4472-ab9a-feed3cfac57c
>
> [2015-05-07 11:56:32.228267] E [glusterd-syncop.c:1023:gd_sync_task_begin] 0-management: Unable to acquire lock
>
> [2015-05-07 11:56:41.802737] E [rpc-clnt.c:207:call_bail] 0-management: bailing out frame type(GLUSTERD-DUMP) op(DUMP(1)) xid = 0x280776x sent = 2015-05-07 11:46:31.743195. timeout = 600
>
> [2015-05-07 11:56:41.802772] E [glusterd-handshake.c:1074:__glusterd_peer_dump_version_cbk] 0-: Error through RPC layer, retry again later
>
> [2015-05-07 11:56:41.802836] W [socket.c:514:__socket_rwv] 0-management: readv failed (No data available)
>
> [2015-05-07 11:56:41.844795] E [rpc-clnt.c:368:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13d) [0x39d9a0ec8d] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x39d9a0e7f3] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x39d9a0e70e]))) 0-management: forced unwinding frame type(glusterd mgmt) op(--(1)) called at 2015-05-07 11:47:47.530276 (xid=0x280777x)
>
> [2015-05-07 11:56:41.844833] E [rpc-clnt.c:368:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13d) [0x39d9a0ec8d] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x39d9a0e7f3] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x39d9a0e70e]))) 0-management: forced unwinding frame type(glusterd mgmt) op(--(1)) called at 2015-05-07 11:48:48.544182 (xid=0x280778x)
>
> [2015-05-07 11:56:41.844854] E [rpc-clnt.c:368:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13d) [0x39d9a0ec8d] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x39d9a0e7f3] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x39d9a0e70e]))) 0-management: forced unwinding frame type(glusterd mgmt) op(--(1)) called at 2015-05-07 11:53:48.340948 (xid=0x280779x)
>
> [2015-05-07 11:56:41.844861] E [glusterd-syncop.c:715:gd_lock_op_phase] 0-management: Failed to acquire lock
>
> [2015-05-07 11:56:41.844898] E [glusterd-syncop.c:715:gd_lock_op_phase] 0-management: Failed to acquire lock
>
> [2015-05-07 11:56:41.844960] E [rpc-clnt.c:368:saved_frames_unwind] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x13d) [0x39d9a0ec8d] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xc3) [0x39d9a0e7f3] (-->/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe) [0x39d9a0e70e]))) 0-management: forced unwinding frame type(glusterd mgmt) op(--(1)) called at 2015-05-07 11:55:47.461721 (xid=0x280780x)
>
> [2015-05-07 11:56:41.848677] I [socket.c:3027:socket_submit_request] 0-management: not connected (priv->connected = 0)
>
> [2015-05-07 11:56:41.848697] W [rpc-clnt.c:1488:rpc_clnt_submit] 0-management: failed to submit rpc-request (XID: 0x280781x Program: glusterd mgmt, ProgVers: 2, Proc: 2) to rpc-transport (management)
>
> [2015-05-07 11:56:41.848744] W [rpc-clnt.c:1488:rpc_clnt_submit] 0-management: failed to submit rpc-request (XID: 0x280782x Program: glusterd mgmt, ProgVers: 2, Proc: 2) to rpc-transport (management)
>
> [2015-05-07 11:56:41.848770] E [glusterd-syncop.c:715:gd_lock_op_phase] 0-management: Failed to acquire lock
>
> [2015-05-07 11:56:41.848812] I [socket.c:3101:socket_submit_reply] 0-socket.management: not connected (priv->connected = -1)
>
> [2015-05-07 11:56:41.848822] E [rpcsvc.c:1113:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x1x, Program: GlusterD svc cli, ProgVers: 2, Proc: 27) to rpc-transport (socket.management)
>
> [2015-05-07 11:56:41.848852] E [glusterd-utils.c:586:glusterd_submit_reply] 0-: Reply submission failed
>
> [2015-05-07 11:56:41.848868] E [glusterd-utils.c:365:glusterd_unlock] 0-management: Cluster lock not held!
>
> [2015-05-07 11:56:41.848885] E [glusterd-syncop.c:715:gd_lock_op_phase] 0-management: Failed to acquire lock
>
> [2015-05-07 11:56:41.848909] I [socket.c:3101:socket_submit_reply] 0-socket.management: not connected (priv->connected = -1)
>
> [2015-05-07 11:56:41.848918] E [rpcsvc.c:1113:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x1x, Program: GlusterD svc cli, ProgVers: 2, Proc: 27) to rpc-transport (socket.management)
>
> [2015-05-07 11:56:41.848933] E [glusterd-utils.c:586:glusterd_submit_reply] 0-: Reply submission failed
>
> [2015-05-07 11:56:41.848942] E [glusterd-utils.c:365:glusterd_unlock] 0-management: Cluster lock not held!
>
> [2015-05-07 11:56:41.849903] I [socket.c:3101:socket_submit_reply] 0-socket.management: not connected (priv->connected = -1)
>
> [2015-05-07 11:56:41.849913] E [rpcsvc.c:1113:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x1x, Program: GlusterD svc cli, ProgVers: 2, Proc: 27) to rpc-transport (socket.management)
>
> [2015-05-07 11:56:41.849924] E [glusterd-utils.c:586:glusterd_submit_reply] 0-: Reply submission failed
>
> [2015-05-07 11:56:41.849935] E [glusterd-utils.c:365:glusterd_unlock] 0-management: Cluster lock not held!
>
> [2015-05-07 11:56:49.706897] I [glusterd-handler.c:952:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
> [2015-05-07 11:56:50.640610] I [glusterd-handler.c:952:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
> [2015-05-07 11:56:51.689354] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 11:56:51.689903] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 11:56:51.690286] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 11:56:51.690660] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 11:56:51.691090] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 11:56:51.691519] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 11:56:51.691868] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 11:57:40.505580] E [glusterd-utils.c:332:glusterd_lock] 0-management: Unable to get lock for uuid: 674a78b5-0590-48d4-8752-d4608832ed1d, lock held by: ea17d7f9-d737-4472-ab9a-feed3cfac57c
>
> [2015-05-07 11:57:40.505712] E [glusterd-op-sm.c:5445:glusterd_op_sm] 0-management: handler returned: -1
>
> [2015-05-07 11:57:49.848485] I [glusterd-handler.c:952:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
> [2015-05-07 11:57:50.614735] I [glusterd-handler.c:952:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
> [2015-05-07 11:57:51.737376] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 11:57:51.755171] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 11:57:51.755740] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 11:57:51.756165] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 11:57:51.756637] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 11:57:51.757055] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 11:57:51.757453] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 11:58:25.351136] E [glusterd-utils.c:332:glusterd_lock] 0-management: Unable to get lock for uuid: 674a78b5-0590-48d4-8752-d4608832ed1d, lock held by: ea17d7f9-d737-4472-ab9a-feed3cfac57c
>
> [2015-05-07 11:58:25.367614] E [glusterd-op-sm.c:5445:glusterd_op_sm] 0-management: handler returned: -1
>
> [2015-05-07 11:58:53.655999] I [glusterd-handler.c:952:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
> [2015-05-07 11:58:53.675171] I [glusterd-handler.c:952:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
> [2015-05-07 11:58:54.068535] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 11:58:56.079742] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 11:58:56.092418] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 11:58:56.101670] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 11:58:56.922748] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 11:58:56.929828] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 11:58:56.939039] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 11:59:54.303643] I [glusterd-handler.c:952:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
> [2015-05-07 11:59:54.316523] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 11:59:54.324661] I [glusterd-handler.c:952:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
> [2015-05-07 11:59:54.334902] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 11:59:54.346829] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 11:59:54.358660] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 11:59:54.359401] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 11:59:54.365991] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 11:59:54.371101] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 12:00:50.916410] E [glusterd-utils.c:332:glusterd_lock] 0-management: Unable to get lock for uuid: 674a78b5-0590-48d4-8752-d4608832ed1d, lock held by: ea17d7f9-d737-4472-ab9a-feed3cfac57c
>
> [2015-05-07 12:00:50.943695] E [glusterd-op-sm.c:5445:glusterd_op_sm] 0-management: handler returned: -1
>
> [2015-05-07 12:00:50.954392] I [glusterd-handler.c:952:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
> [2015-05-07 12:00:55.394885] I [glusterd-handler.c:952:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
> [2015-05-07 12:00:55.403735] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 12:00:55.431832] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 12:00:55.441505] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 12:00:55.453781] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 12:00:55.459636] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 12:00:55.467670] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
> [2015-05-07 12:00:55.474254] I [glusterd-handler.c:1007:__glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
>
>
>
> Best Regards
>
> Young Yang
>
>
>
>
>
> On 05/08/2015 10:37 AM, vyyy杨雨阳 wrote:
>
>> Hi,
>
>>
>
>>
>
>>
>
>> We've been using glusterfs for 1 years，It’s greadt except occasional
>
>> glusterd crashed, and just need restart
>
>>
>
>> But nowadays, Glusterd crashed more frequently, sometimes several
>
>> glusterds crashed and cause split-brain and gfid different
>
>>
>
>>
>
>>
>
>> Our environment: Centos 6.4 Gluster 3.4.2
>
>>
>
>>
>
>>
>
>>
>
>>
>
>> [root at SVR4666HW2285 ~]# service glusterd status
>
>>
>
>> glusterd dead but pid file exists
>
>>
>
>>
>
>>
>
>>
>
>>
>
>> When glusterd crashed, We can see linux swap exhausted and then
>
>> released
>
> We would require the backtrace and the glusterd log file to further analyze it.
>
>
>
> ~Atin
>
>>
>
>>
>
>>
>
>> Then glustershd.log filled with “readv failed”
>
>>
>
>>
>
>>
>
>> [2015-05-07 12:40:53.580605] W [socket.c:514:__socket_rwv]
>
>> 0-glusterfs: readv failed (No data available)
>
>>
>
>> [2015-05-07 12:40:53.609217] W
>
>> [socket.c:1962:__socket_proto_state_machine] 0-glusterfs: reading from
>
>> socket failed. Error (No data available), peer (127.0.0.1:24007)
>
>>
>
>> [2015-05-07 12:41:03.837798] E [socket.c:2157:socket_connect_finish]
>
>> 0-glusterfs: connection to 127.0.0.1:24007 failed (Connection refused)
>
>>
>
>> [2015-05-07 12:41:03.837853] W [socket.c:514:__socket_rwv]
>
>> 0-glusterfs: readv failed (No data available)
>
>>
>
>> [2015-05-07 12:41:06.844262] W [socket.c:514:__socket_rwv]
>
>> 0-glusterfs: readv failed (No data available)
>
>>
>
>> [2015-05-07 12:41:09.847699] W [socket.c:514:__socket_rwv]
>
>> 0-glusterfs: readv failed (No data available)
>
>>
>
>> [2015-05-07 12:41:12.854243] W [socket.c:514:__socket_rwv]
>
>> 0-glusterfs: readv failed (No data available)
>
>>
>
>>
>
>>
>
>>
>
>>
>
>> Any help appreciated.
>
>>
>
>> Best Regards
>
>> Young Yang
>
>>
>
>>
>
>>
>
>> _______________________________________________
>
>> Gluster-users mailing list
>
>> Gluster-users at gluster.org<mailto:Gluster-users at gluster.org>
>
>> http://www.gluster.org/mailman/listinfo/gluster-users
>
>>
>
>
>
> --
>
> ~Atin
>
--
~Atin
More information about the Gluster-users
mailing list