[Gluster-devel] [Gluster-users] Unable to get lock for uuid peers

[Sergio Traldi]

On 06/10/2015 10:27 AM, Krishnan Parthasarathi wrote:
>> Hi all,
>> I two servers with 3.7.1 and have the same problem of this issue:
>> http://comments.gmane.org/gmane.comp.file-systems.gluster.user/20693
>>
>> My servers packages:
>> # rpm -qa | grep gluster | sort
>> glusterfs-3.7.1-1.el6.x86_64
>> glusterfs-api-3.7.1-1.el6.x86_64
>> glusterfs-cli-3.7.1-1.el6.x86_64
>> glusterfs-client-xlators-3.7.1-1.el6.x86_64
>> glusterfs-fuse-3.7.1-1.el6.x86_64
>> glusterfs-geo-replication-3.7.1-1.el6.x86_64
>> glusterfs-libs-3.7.1-1.el6.x86_64
>> glusterfs-server-3.7.1-1.el6.x86_64
>>
>> Command:
>> # gluster volume status
>> Another transaction is in progress. Please try again after sometime.
>>
>>
>> In /var/log/gluster/etc-glusterfs-glusterd.vol.log I found:
>>
>> [2015-06-09 16:12:38.949842] E [glusterd-utils.c:164:glusterd_lock]
>> 0-management: Unable to get lock for uuid:
>> 99a41a2a-2ce5-461c-aec0-510bd5b37bf2, lock held by:
>> 04a7d2bb-bdd9-4e0d-b460-87ad4adbe12c
>> [2015-06-09 16:12:38.949864] E [glusterd-syncop.c:1766:gd_sync_task_begin]
>> 0-management: Unable to acquire lock
>>
>> I check the files:
>>   From server 1:
>> # cat /var/lib/glusterd/peers/04a7d2bb-bdd9-4e0d-b460-87ad4adbe12c
>> uuid=04a7d2bb-bdd9-4e0d-b460-87ad4adbe12c
>> state=3
>> hostname1=192.168.61.101
>>
>>   From server 2:
>> # cat /var/lib/glusterd/peers/99a41a2a-2ce5-461c-aec0-510bd5b37bf2
>> uuid=99a41a2a-2ce5-461c-aec0-510bd5b37bf2
>> state=3
>> hostname1=192.168.61.100
> Could you attach the complete glusterd log file and cmd-history.log
> file under /var/log/glusterfs directory? Could you provide a more
> detailed listing of things you did before hitting this issue?
Hi Krishnan,
thanks to a quick answer.
In attach you can found the two log you request:
cmd_history.log
etc-glusterfs-glusterd.vol.log

We use the gluster volume as openstack nova, glance, cinder backend.

The volume is configured using 2 bricks mounted by an iscsi device:
[root at cld-stg-01 glusterfs]# gluster volume info volume-nova-prod
Volume Name: volume-nova-prod
Type: Distribute
Volume ID: 4bbef4c8-0441-4e81-a2c5-559401adadc0
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: 192.168.61.100:/brickOpenstack/nova-prod/mpathb
Brick2: 192.168.61.101:/brickOpenstack/nova-prod/mpathb
Options Reconfigured:
storage.owner-gid: 162
storage.owner-uid: 162

Last week we update openstack from havana to icehouse and we rename the 
storage hosts but we didn't change the IP.
All volume have been created using ip addresses.

So last week we stop all services (openstack gluster and also iscsi).
We change the name in DNS of private ip of the 2 nics.
We reboot the storage servers
We start agian iscsi, multipath, glusterd process.
We have to stop and start the volumes, but after that everything works fine.
Now we don't observe any other problems except this.

We have a nagios probe which check the volume status each 5 minutes to 
ensure all gluster process is working fine and so we find this problem I 
post.

Cheer
Sergio
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cmd_history.log
Type: text/x-log
Size: 161884 bytes
Desc: not available
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20150610/2af2a659/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: etc-glusterfs-glusterd.vol.log
Type: text/x-log
Size: 875307 bytes
Desc: not available
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20150610/2af2a659/attachment-0003.bin>