[Gluster-users] held cluster lock blocking volume operations

[Matthew Nicholson]

No, no duplicate UUIDs:

[root at ox60-gstore01 ~]# gluster peer status |grep -i uuid | uniq -c
      1 Uuid: 055a13fe-e40a-46ff-9011-6c81832e3ba1
      1 Uuid: e0c267e6-3dc2-4623-89f1-4516f1285c1a
      1 Uuid: e503bd2e-b2b2-49d4-ae05-45090e24acca
      1 Uuid: 974a503e-4f0f-44f2-81df-5383c28cdf20
      1 Uuid: 5517a055-c5f5-41b7-95d2-dedf6900be21
      1 Uuid: 13cfacc1-65a4-4151-91d5-bc7977e01654
      1 Uuid: a5de08c0-e761-45ee-a7ad-e8c556f2540b
      1 Uuid: 428e11bc-5a80-41cb-af1d-a9023e2bc11b
      1 Uuid: 113562a1-e521-4747-ae75-477614ea28cf
      1 Uuid: 04c6c37b-743d-4f87-9bdc-3dfe1b573709
      1 Uuid: 2225df4c-4510-457c-9958-0b6506ff25e4
      1 Uuid: 6456206b-fe19-4b65-b7ab-0c9e7ce6221e
      1 Uuid: 0edce15e-0de2-4496-a520-58c65dbbc7da
      1 Uuid: a327cd38-f98a-4554-ae62-97a21153f4d3
      1 Uuid: a7d3a064-1bb4-4da0-a680-180db8150e4c
      1 Uuid: 757297b4-5648-4e31-88f4-00fc167a43e4
      1 Uuid: 725a2567-b668-4a5f-b2c9-5c7dcc90c846
      1 Uuid: 303f4cc4-c8ae-42c7-b8cd-eafee8f95122
      1 Uuid: 439f3ffa-e468-4a8b-801e-e2f20062e6f0
      1 Uuid: cdba3b89-e804-4bf1-afb9-d7c231399955

glusterd (as well as glusterfs and the nfs server, which seemingly never
dies if glusterd is shutdown) have all been restarted. Actually, we just
went so fas as to bounce one replica then another (reboot).



--
Matthew Nicholson
Research Computing Specialist
Harvard FAS Research Computing
matthew_nicholson at harvard.edu



On Tue, Jun 4, 2013 at 10:30 AM, Vijay Bellur <vbellur at redhat.com> wrote:

> On 06/04/2013 07:57 PM, Matthew Nicholson wrote:
>
>> So, we've got a volume that is mostly functioning fine (its up
>> accessible, etc etc). However, volume operations fail/don't return on it.
>>
>>
>> what i mean is
>>
>> gluster peer status//probe/etc : works
>> gluster volume info : works
>> gluster volume status/remove-brick/etc : sit for a long time and return
>> nothing.
>>
>> The only things coming up in logs are:
>>
>> [2013-06-04 10:21:36.398072] I [glusterd-utils.c:285:**glusterd_lock]
>> 0-glusterd: Cluster lock held by 757297b4-5648-4e31-88f4-**00fc167a43e4
>> [2013-06-04 10:21:36.398123] I
>> [glusterd-handler.c:463:**glusterd_op_txn_begin] 0-management: Acquired
>> local lock
>> [2013-06-04 10:21:36.398424] I
>> [glusterd-handler.c:502:**glusterd_handle_cluster_lock] 0-glusterd:
>> Received LOCK from uuid: 757297b4-5648-4e31-88f4-**00fc167a43e4
>> [2013-06-04 10:21:36.398448] E [glusterd-utils.c:277:**glusterd_lock]
>> 0-glusterd: Unable to get lock for uuid:
>> 757297b4-5648-4e31-88f4-**00fc167a43e4, lock held by:
>> 757297b4-5648-4e31-88f4-**00fc167a43e4
>> [2013-06-04 10:21:36.398483] I
>> [glusterd-handler.c:1322:**glusterd_op_lock_send_resp] 0-glusterd:
>> Responded, ret: 0
>> [2013-06-04 10:21:36.398498] E [glusterd-op-sm.c:4624:**glusterd_op_sm]
>> 0-glusterd: handler returned: -1
>>
>> If you notice, the UUID holding the lock, and the uuid requesting the
>> lock, are the same. So it seems like a lock was "forgotten" about?
>>
>> any thoughts on clearing this?
>>
>
> Does gluster peer status list the same UUID more than once?
>
> If not, restarting the glusterd which is the lock owner should address it.
>
> -Vijay
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130604/aaa45a97/attachment.html>