[Gluster-devel] deadlock? Unable to get lock for uuid which is held by itself
Yandong Yao
yydzero at gmail.com
Sat Jan 25 12:57:37 UTC 2014
Hi Guys,
I am using gluster 3.4.2 and doing some test with replica=4, and got
following 'unable to get lock error', while it seems the lock has been
acquired by itself already. is this a deadlock?
any other volume command will be fail with error message like 'another
transaction is in progress'.
$ sudo gluster volume status x9db
Status of volume: testvol
Gluster process Port Online Pid
------------------------------------------------------------------------------
Brick ip1:/home/cluster/db 49152 Y 2244
Brick ip2:/home/cluster/db 49152 Y 2225
Brick ip3:/home/cluster/db 49152 Y 2302
Brick ip4:/home/cluster/db 49152 Y 2302
NFS Server on localhost 2049 Y 5411
Self-heal Daemon on localhost N/A Y 5418
NFS Server on ip2 2049 Y 2279
Self-heal Daemon ip2 N/A Y 2284
NFS Server on ip3 2049 Y 2316
Self-heal Daemon on ip3 N/A Y 2321
NFS Server on ip4 2049 Y 2316
Self-heal Daemon on ip4 N/A Y 2321
There are no active volume tasks
s$ sudo gluster volume info x9db
Volume Name: testvol
Type: Replicate
Volume ID: 6a1159aa-3116-45c3-8611-16aaab76265a
Status: Started
Number of Bricks: 1 x 4 = 4
Transport-type: tcp
Bricks:
Brick1: ip1:/home/cluster/test
Brick2: ip2:/home/cluster/test
Brick3: ip3:/home/cluster/test
Brick4: ip4:/home/cluster/test
[2014-01-25 11:12:31.711699] I
[glusterd-handler.c:952:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
[2014-01-25 11:12:42.143165] I
[glusterd-handler.c:3260:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume x9db
[2014-01-25 11:12:46.061212] W [socket.c:514:__socket_rwv] 0-management:
readv failed (No data available)
[2014-01-25 11:12:47.547211] I
[glusterd-handshake.c:557:__glusterd_mgmt_hndsk_versions_ack] 0-management:
using the op-version 2
[2014-01-25 11:12:47.655510] I
[glusterd-volume-ops.c:478:__glusterd_handle_cli_heal_volume] 0-management:
Received heal vol req for volume x9db
[2014-01-25 11:12:47.766356] I
[glusterd-handler.c:2118:__glusterd_handle_friend_update] 0-glusterd:
Received friend update from uuid: 03bbe0ee-0c1b-44fc-8e41-2f15139ae614
[2014-01-25 11:12:47.766386] I
[glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received
uuid: b2c16491-ba2c-424c-bfce-19e3acabd140, hostname:172.16.1.105
[2014-01-25 11:12:47.766398] I
[glusterd-handler.c:2172:__glusterd_handle_friend_update] 0-: Received my
uuid as Friend
[2014-01-25 11:12:47.766410] I
[glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received
uuid: caf2f668-fc3a-4f12-a5f2-35ecbb72dfe6, hostname:172.16.1.104
[2014-01-25 11:12:47.766426] I
[glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received
uuid: 73de9fcc-70cb-4153-b369-6be56edc60e4, hostname:172.16.1.108
[2014-01-25 11:12:47.848456] I
[glusterd-handler.c:1956:__glusterd_handle_incoming_friend_req] 0-glusterd:
Received probe from uuid: 73de9fcc-70cb-4153-b369-6be56edc60e4
[2014-01-25 11:12:47.883926] I
[glusterd-handler.c:2987:glusterd_xfer_friend_add_resp] 0-glusterd:
Responded to 172.16.1.105 (0), ret: 0
[2014-01-25 11:12:47.997692] I
[glusterd-sm.c:494:glusterd_ac_send_friend_update] 0-: Added uuid:
caf2f668-fc3a-4f12-a5f2-35ecbb72dfe6, host: 172.16.1.104
[2014-01-25 11:12:47.997715] I
[glusterd-sm.c:494:glusterd_ac_send_friend_update] 0-: Added uuid:
03bbe0ee-0c1b-44fc-8e41-2f15139ae614, host: 172.16.1.107
[2014-01-25 11:12:47.997729] I
[glusterd-sm.c:494:glusterd_ac_send_friend_update] 0-: Added uuid:
73de9fcc-70cb-4153-b369-6be56edc60e4, host: 172.16.1.108
[2014-01-25 11:12:48.098660] I
[glusterd-rpc-ops.c:542:__glusterd_friend_update_cbk] 0-management:
Received ACC from uuid: 03bbe0ee-0c1b-44fc-8e41-2f15139ae614
[2014-01-25 11:12:48.098696] I
[glusterd-handler.c:2118:__glusterd_handle_friend_update] 0-glusterd:
Received friend update from uuid: caf2f668-fc3a-4f12-a5f2-35ecbb72dfe6
[2014-01-25 11:12:48.098711] I
[glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received
uuid: b2c16491-ba2c-424c-bfce-19e3acabd140, hostname:172.16.1.105
[2014-01-25 11:12:48.098718] I
[glusterd-handler.c:2172:__glusterd_handle_friend_update] 0-: Received my
uuid as Friend
[2014-01-25 11:12:48.098727] I
[glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received
uuid: 73de9fcc-70cb-4153-b369-6be56edc60e4, hostname:172.16.1.108
[2014-01-25 11:12:48.098738] I
[glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received
uuid: 03bbe0ee-0c1b-44fc-8e41-2f15139ae614, hostname:172.16.1.107
[2014-01-25 11:12:48.098764] I
[glusterd-rpc-ops.c:542:__glusterd_friend_update_cbk] 0-management:
Received ACC from uuid: caf2f668-fc3a-4f12-a5f2-35ecbb72dfe6
[2014-01-25 11:12:49.036676] I
[glusterd-handler.c:2118:__glusterd_handle_friend_update] 0-glusterd:
Received friend update from uuid: 73de9fcc-70cb-4153-b369-6be56edc60e4
[2014-01-25 11:12:49.036722] I
[glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received
uuid: b2c16491-ba2c-424c-bfce-19e3acabd140, hostname:172.16.1.105
[2014-01-25 11:12:49.036734] I
[glusterd-handler.c:2172:__glusterd_handle_friend_update] 0-: Received my
uuid as Friend
[2014-01-25 11:12:49.036746] I
[glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received
uuid: 03bbe0ee-0c1b-44fc-8e41-2f15139ae614, hostname:172.16.1.107
[2014-01-25 11:12:49.036759] I
[glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received
uuid: caf2f668-fc3a-4f12-a5f2-35ecbb72dfe6, hostname:172.16.1.104
[2014-01-25 11:12:51.052062] I
[glusterd-rpc-ops.c:345:__glusterd_friend_add_cbk] 0-glusterd: Received ACC
from uuid: 73de9fcc-70cb-4153-b369-6be56edc60e4, host: 172.16.1.108, port: 0
[2014-01-25 11:12:51.152936] E [glusterd-utils.c:332:glusterd_lock]
0-management: Unable to get lock for uuid:
b2c16491-ba2c-424c-bfce-19e3acabd140, lock held by:
b2c16491-ba2c-424c-bfce-19e3acabd140
[2014-01-25 11:12:51.152983] I
[glusterd-handler.c:2118:__glusterd_handle_friend_update] 0-glusterd:
Received friend update from uuid: 73de9fcc-70cb-4153-b369-6be56edc60e4
[2014-01-25 11:12:51.153006] I
[glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received
uuid: b2c16491-ba2c-424c-bfce-19e3acabd140, hostname:172.16.1.105
[2014-01-25 11:12:51.153017] I
[glusterd-handler.c:2172:__glusterd_handle_friend_update] 0-: Received my
uuid as Friend
[2014-01-25 11:12:51.153029] I
[glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received
uuid: 03bbe0ee-0c1b-44fc-8e41-2f15139ae614, hostname:172.16.1.107
[2014-01-25 11:12:51.153042] I
[glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received
uuid: caf2f668-fc3a-4f12-a5f2-35ecbb72dfe6, hostname:172.16.1.104
[2014-01-25 11:13:18.695488] I
[glusterd-handler.c:2118:__glusterd_handle_friend_update] 0-glusterd:
Received friend update from uuid: 73de9fcc-70cb-4153-b369-6be56edc60e4
[2014-01-25 11:13:18.695534] I
[glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received
uuid: b2c16491-ba2c-424c-bfce-19e3acabd140, hostname:172.16.1.105
[2014-01-25 11:13:18.695547] I
[glusterd-handler.c:2172:__glusterd_handle_friend_update] 0-: Received my
uuid as Friend
[2014-01-25 11:13:18.695561] I
[glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received
uuid: 03bbe0ee-0c1b-44fc-8e41-2f15139ae614, hostname:172.16.1.107
[2014-01-25 11:13:18.695575] I
[glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received
uuid: caf2f668-fc3a-4f12-a5f2-35ecbb72dfe6, hostname:172.16.1.104
[2014-01-25 11:13:46.049416] E [glusterd-utils.c:332:glusterd_lock]
0-management: Unable to get lock for uuid:
73de9fcc-70cb-4153-b369-6be56edc60e4, lock held by:
b2c16491-ba2c-424c-bfce-19e3acabd140
[2014-01-25 11:13:46.049481] E [glusterd-op-sm.c:5445:glusterd_op_sm]
0-management: handler returned: -1
[2014-01-25 11:13:46.049985] E [glusterd-utils.c:374:glusterd_unlock]
0-management: Cluster lock held by b2c16491-ba2c-424c-bfce-19e3acabd140
,unlock req from 73de9fcc-70cb-4153-b369-6be56edc60e4!
[2014-01-25 11:13:46.050019] E [glusterd-utils.c:332:glusterd_lock]
0-management: Unable to get lock for uuid:
b2c16491-ba2c-424c-bfce-19e3acabd140, lock held by:
b2c16491-ba2c-424c-bfce-19e3acabd140
[2014-01-25 11:13:46.050032] E [glusterd-op-sm.c:5445:glusterd_op_sm]
0-management: handler returned: -1
[2014-01-25 11:17:37.710091] E [glusterd-utils.c:332:glusterd_lock]
0-management: Unable to get lock for uuid:
73de9fcc-70cb-4153-b369-6be56edc60e4, lock held by:
b2c16491-ba2c-424c-bfce-19e3acabd140
[2014-01-25 11:17:37.710159] E [glusterd-op-sm.c:5445:glusterd_op_sm]
0-management: handler returned: -1
[2014-01-25 11:17:37.710605] E [glusterd-utils.c:374:glusterd_unlock]
0-management: Cluster lock held by b2c16491-ba2c-424c-bfce-19e3acabd140
,unlock req from 73de9fcc-70cb-4153-b369-6be56edc60e4!
[2014-01-25 11:17:37.710638] E [glusterd-utils.c:332:glusterd_lock]
0-management: Unable to get lock for uuid:
b2c16491-ba2c-424c-bfce-19e3acabd140, lock held by:
b2c16491-ba2c-424c-bfce-19e3acabd140
[2014-01-25 11:17:37.710651] E [glusterd-op-sm.c:5445:glusterd_op_sm]
0-management: handler returned: -1
[2014-01-25 11:18:25.706653] E [glusterd-utils.c:332:glusterd_lock]
0-management: Unable to get lock for uuid:
73de9fcc-70cb-4153-b369-6be56edc60e4, lock held by:
b2c16491-ba2c-424c-bfce-19e3acabd140
[2014-01-25 11:18:25.706716] E [glusterd-op-sm.c:5445:glusterd_op_sm]
0-management: handler returned: -1
[2014-01-25 11:18:25.707379] E [glusterd-utils.c:374:glusterd_unlock]
0-management: Cluster lock held by b2c16491-ba2c-424c-bfce-19e3acabd140
,unlock req from 73de9fcc-70cb-4153-b369-6be56edc60e4!
[2014-01-25 11:18:25.707413] E [glusterd-utils.c:332:glusterd_lock]
0-management: Unable to get lock for uuid:
b2c16491-ba2c-424c-bfce-19e3acabd140, lock held by:
b2c16491-ba2c-424c-bfce-19e3acabd140
[2014-01-25 11:18:25.707426] E [glusterd-op-sm.c:5445:glusterd_op_sm]
0-management: handler returned: -1
[2014-01-25 11:20:59.713622] I
[glusterd-handler.c:952:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req
[2014-01-25 11:21:03.963492] E [glusterd-utils.c:332:glusterd_lock]
0-management: Unable to get lock for uuid:
b2c16491-ba2c-424c-bfce-19e3acabd140, lock held by:
b2c16491-ba2c-424c-bfce-19e3acabd140
[2014-01-25 11:21:03.963524] E [glusterd-syncop.c:1023:gd_sync_task_begin]
0-management: Unable to acquire lock
[2014-01-25 11:22:49.047317] E [rpc-clnt.c:207:call_bail] 0-management:
bailing out frame type(glusterd mgmt) op(--(4)) xid = 0x29x sent =
2014-01-25 11:12:48.102044. timeout = 600
[2014-01-25 11:22:49.048264] E
[glusterd-rpc-ops.c:664:__glusterd_cluster_unlock_cbk] 0-management:
Received unlock RJT from uuid: 73de9fcc-70cb-4153-b369-6be56edc60e4
[2014-01-25 11:22:49.048287] E [glusterd-utils.c:365:glusterd_unlock]
0-management: Cluster lock not held!
[2014-01-25 11:22:49.048299] C
[glusterd-op-sm.c:3648:glusterd_op_txn_complete] 0-management: Unable to
clear local lock, ret: -1
[2014-01-25 11:22:49.048350] E
[glusterd-rpc-ops.c:126:glusterd_op_send_cli_response] 0-management:
invalid operation
[2014-01-25 11:22:49.048392] I [socket.c:3101:socket_submit_reply]
0-socket.management: not connected (priv->connected = -1)
[2014-01-25 11:22:49.048405] E [rpcsvc.c:1113:rpcsvc_submit_generic]
0-rpc-service: failed to submit message (XID: 0x1x, Program: GlusterD svc
cli, ProgVers: 2, Proc: 31) to rpc-transport (socket.management)
[2014-01-25 11:22:49.048421] E [glusterd-utils.c:586:glusterd_submit_reply]
0-: Reply submission failed
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20140125/f5263e71/attachment-0001.html>
More information about the Gluster-devel
mailing list