[Gluster-users] Stale locks on shards

Pranith Kumar Karampuri pkarampu at redhat.com
Mon Jan 29 08:20:11 UTC 2018 

On Mon, Jan 29, 2018 at 1:26 PM, Samuli Heinonen <samppah at neutraali.net>
wrote:

> Pranith Kumar Karampuri kirjoitti 29.01.2018 07:32:
>
>> On 29 Jan 2018 10:50 am, "Samuli Heinonen" <samppah at neutraali.net>
>> wrote:
>>
>> Hi!
>>>
>>> Yes, thank you for asking. I found out this line in the production
>>> environment:
>>>
>>> lgetxattr("/tmp/zone2-ssd1-vmstor1.s6jvPu//.shard/f349ffbd-
>> a423-4fb2-b83c-2d1d5e78e1fb.32",
>>
>>> "glusterfs.clrlk.tinode.kblocked", 0x7f2d7c4379f0, 4096) = -1 EPERM
>>> (Operation not permitted)
>>>
>>
>> I was expecting .kall instead of .blocked,
>> did you change the cli to kind blocked?
>>
>>
> Yes, I was testing this with different commands. Basicly it seems that
> name of the attribute is glusterfs.clrlk.t{posix,inode,entry}.k{all,blocked,granted},
> am I correct?


That is correct

Is it necessary to set any value or just reguest the attribute with
> getfattr?
>

Nope. No I/O is going on the file right?  Just request the attribute with
getfattr in that case.


>
>
>
> And this one in test environment (with posix locks):
>>> lgetxattr("/tmp/g1.gHj4Bw//file38",
>>> "glusterfs.clrlk.tposix.kblocked", "box1:/gluster/1/export/: posix
>>> blocked locks=1 granted locks=0", 4096) = 77
>>>
>>> In test environment I tried running following command which seemed
>>> to release gluster locks:
>>>
>>> getfattr -n glusterfs.clrlk.tposix.kblocked file38
>>>
>>> So I think it would go like this in production environment with
>>> locks on shards (using aux-gfid-mount mount option):
>>> getfattr -n glusterfs.clrlk.tinode.kall
>>> .shard/f349ffbd-a423-4fb2-b83c-2d1d5e78e1fb.32
>>>
>>> I haven't been able to try this out in production environment yet.
>>>
>>> Is there perhaps something else to notice?
>>>
>>> Would you be able to tell more about bricks crashing after releasing
>>> locks? Under what circumstances that does happen? Is it only process
>>> exporting the brick crashes or is there a possibility of data
>>> corruption?
>>>
>>
>> No data corruption. Brick process where you did clear-locks may crash.
>>
>> Best regards,
>>> Samuli Heinonen
>>>
>>> Pranith Kumar Karampuri wrote:
>>>
>>> Hi,
>>>> Did you find the command from strace?
>>>>
>>>> On 25 Jan 2018 1:52 pm, "Pranith Kumar Karampuri"
>>>> <pkarampu at redhat.com
>>>>
>>>> <mailto:pkarampu at redhat.com>> wrote:
>>>>
>>>> On Thu, Jan 25, 2018 at 1:49 PM, Samuli Heinonen
>>>>
>>>> <samppah at neutraali.net <mailto:samppah at neutraali.net>> wrote:
>>>>
>>>> Pranith Kumar Karampuri kirjoitti 25.01.2018 07:09:
>>>>
>>>> On Thu, Jan 25, 2018 at 2:27 AM, Samuli Heinonen
>>>>
>>>> <samppah at neutraali.net
>>>> <mailto:samppah at neutraali.net>> wrote:
>>>>
>>>> Hi!
>>>>
>>>> Thank you very much for your help so far. Could
>>>> you
>>>> please tell an
>>>> example command how to use aux-gid-mount to remove
>>>> locks? "gluster
>>>> vol clear-locks" seems to mount volume by itself.
>>>>
>>>> You are correct, sorry, this was implemented around 7
>>>> years
>>>> back and I
>>>> forgot that bit about it :-(. Essentially it becomes a
>>>> getxattr
>>>> syscall on the file.
>>>> Could you give me the clear-locks command you were
>>>> trying to
>>>> execute
>>>> and I can probably convert it to the getfattr command?
>>>>
>>>> I have been testing this in test environment and with
>>>> command:
>>>> gluster vol clear-locks g1
>>>> /.gfid/14341ccb-df7b-4f92-90d5-7814431c5a1c kind all inode
>>>>
>>>> Could you do strace of glusterd when this happens? It will
>>>> have a
>>>> getxattr with "glusterfs.clrlk" in the key. You need to
>>>> execute that
>>>> on the gfid-aux-mount
>>>>
>>>> Best regards,
>>>> Samuli Heinonen
>>>>
>>>> Pranith Kumar Karampuri
>>>> <mailto:pkarampu at redhat.com
>>>> <mailto:pkarampu at redhat.com>>
>>>> 23 January 2018 at 10.30
>>>>
>>>> On Tue, Jan 23, 2018 at 1:38 PM, Samuli
>>>> Heinonen
>>>> <samppah at neutraali.net
>>>> <mailto:samppah at neutraali.net>
>>>> <mailto:samppah at neutraali.net
>>>>
>>>> <mailto:samppah at neutraali.net>>> wrote:
>>>>
>>>> Pranith Kumar Karampuri kirjoitti 23.01.2018
>>>> 09:34:
>>>>
>>>> On Mon, Jan 22, 2018 at 12:33 AM, Samuli
>>>> Heinonen
>>>>
>>>> <samppah at neutraali.net
>>>> <mailto:samppah at neutraali.net>
>>>> <mailto:samppah at neutraali.net
>>>>
>>>> <mailto:samppah at neutraali.net>>>
>>>> wrote:
>>>>
>>>> Hi again,
>>>>
>>>> here is more information regarding issue
>>>> described
>>>> earlier
>>>>
>>>> It looks like self healing is stuck. According
>>>> to
>>>> "heal
>>>> statistics"
>>>> crawl began at Sat Jan 20 12:56:19 2018 and
>>>> it's still
>>>> going on
>>>> (It's around Sun Jan 21 20:30 when writing
>>>> this).
>>>> However
>>>> glustershd.log says that last heal was
>>>> completed at
>>>> "2018-01-20
>>>> 11:00:13.090697" (which is 13:00 UTC+2). Also
>>>> "heal
>>>> info"
>>>> has been
>>>> running now for over 16 hours without any
>>>> information.
>>>> In
>>>> statedump
>>>> I can see that storage nodes have locks on
>>>> files and
>>>> some
>>>> of those
>>>> are blocked. Ie. Here again it says that
>>>> ovirt8z2 is
>>>> having active
>>>> lock even ovirt8z2 crashed after the lock was
>>>> granted.:
>>>>
>>>>
>>>> [xlator.features.locks.zone2-ssd1-vmstor1-locks.inode]
>>>>
>>>> path=/.shard/3d55f8cc-cda9-489a-b0a3-fd0f43d67876.27
>>>> mandatory=0
>>>> inodelk-count=3
>>>>
>>>>
>>>> lock-dump.domain.domain=zone2-ssd1-vmstor1-replicate-0:self-heal
>>>> inodelk.inodelk[0](ACTIVE)=type=WRITE,
>>>> whence=0,
>>>> start=0,
>>>> len=0, pid
>>>> = 18446744073709551610,
>>>> owner=d0c6d857a87f0000,
>>>> client=0x7f885845efa0,
>>>>
>>>>
>>>>
>>>>
>>> connection-id=sto2z2.xxx-10975-2018/01/20-10:56:14:649541-
>> zone2-ssd1-vmstor1-client-0-0-0,
>>
>>>
>>>> granted at 2018-01-20 10:59:52
>>>>
>>>>
>>>> lock-dump.domain.domain=zone2-ssd1-vmstor1-replicate-0:metadata
>>>>
>>>> lock-dump.domain.domain=zone2-ssd1-vmstor1-replicate-0
>>>> inodelk.inodelk[0](ACTIVE)=type=WRITE,
>>>> whence=0,
>>>> start=0,
>>>> len=0, pid
>>>> = 3420, owner=d8b9372c397f0000,
>>>> client=0x7f8858410be0,
>>>>
>>>> connection-id=ovirt8z2.xxx.com [1]
>>>> <http://ovirt8z2.xxx.com> [1]
>>>>
>>>>
>>>>
>>>>
>>> <http://ovirt8z2.xxx.com>-5652-2017/12/27-09:49:02:946825-
>> zone2-ssd1-vmstor1-client-0-7-0,
>>
>>>
>>>> granted at 2018-01-20 08:57:23
>>>> inodelk.inodelk[1](BLOCKED)=type=WRITE,
>>>> whence=0,
>>>> start=0,
>>>> len=0,
>>>> pid = 18446744073709551610,
>>>> owner=d0c6d857a87f0000,
>>>> client=0x7f885845efa0,
>>>>
>>>>
>>>>
>>>>
>>> connection-id=sto2z2.xxx-10975-2018/01/20-10:56:14:649541-
>> zone2-ssd1-vmstor1-client-0-0-0,
>>
>>>
>>>> blocked at 2018-01-20 10:59:52
>>>>
>>>> I'd also like to add that volume had arbiter
>>>> brick
>>>> before
>>>> crash
>>>> happened. We decided to remove it because we
>>>> thought
>>>> that
>>>> it was
>>>> causing issues. However now I think that this
>>>> was
>>>> unnecessary. After
>>>> the crash arbiter logs had lots of messages
>>>> like this:
>>>> [2018-01-20 10:19:36.515717] I [MSGID: 115072]
>>>> [server-rpc-fops.c:1640:server_setattr_cbk]
>>>> 0-zone2-ssd1-vmstor1-server: 37374187: SETATTR
>>>> <gfid:a52055bd-e2e9-42dd-92a3-e96b693bcafe>
>>>> (a52055bd-e2e9-42dd-92a3-e96b693bcafe) ==>
>>>> (Operation
>>>> not
>>>> permitted)
>>>> [Operation not permitted]
>>>>
>>>> Is there anyways to force self heal to stop?
>>>> Any help
>>>> would be very
>>>> much appreciated :)
>>>>
>>>> Exposing .shard to a normal mount is opening a
>>>> can of
>>>> worms. You
>>>> should probably look at mounting the volume
>>>> with gfid
>>>> aux-mount where
>>>> you can access a file with
>>>> <path-to-mount>/.gfid/<gfid-string>to clear
>>>> locks on it.
>>>>
>>>> Mount command:  mount -t glusterfs -o
>>>> aux-gfid-mount
>>>> vm1:test
>>>> /mnt/testvol
>>>>
>>>> A gfid string will have some hyphens like:
>>>> 11118443-1894-4273-9340-4b212fa1c0e4
>>>>
>>>> That said. Next disconnect on the brick where
>>>> you
>>>> successfully
>>>> did the
>>>> clear-locks will crash the brick. There was a
>>>> bug in
>>>> 3.8.x
>>>> series with
>>>> clear-locks which was fixed in 3.9.0 with a
>>>> feature. The
>>>> self-heal
>>>> deadlocks that you witnessed also is fixed in
>>>> 3.10
>>>> version
>>>> of the
>>>> release.
>>>>
>>>> Thank you the answer. Could you please tell
>>>> more
>>>> about crash?
>>>> What
>>>> will actually happen or is there a bug report
>>>> about
>>>> it? Just
>>>> want
>>>> to make sure that we can do everything to
>>>> secure data on
>>>> bricks.
>>>> We will look into upgrade but we have to make
>>>> sure
>>>> that new
>>>> version works for us and of course get self
>>>> healing
>>>> working
>>>> before
>>>> doing anything :)
>>>>
>>>> Locks xlator/module maintains a list of locks
>>>> that
>>>> are granted to
>>>> a client. Clear locks had an issue where it
>>>> forgets
>>>> to remove the
>>>> lock from this list. So the connection list
>>>> ends up
>>>> pointing to
>>>> data that is freed in that list after a clear
>>>> lock.
>>>> When a
>>>> disconnect happens, all the locks that are
>>>> granted
>>>> to a client
>>>> need to be unlocked. So the process starts
>>>> traversing through this
>>>> list and when it starts trying to access this
>>>> freed
>>>> data it leads
>>>> to a crash. I found it while reviewing a
>>>> feature
>>>> patch sent by
>>>> facebook folks to locks xlator
>>>> (http://review.gluster.org/14816 [2]
>>>> <http://review.gluster.org/14816 [2]>
>>>>
>>>> [2]) for 3.9.0 and they also fixed this bug as
>>>> well
>>>> as part of
>>>>
>>>> that feature patch.
>>>>
>>>> Br,
>>>> Samuli
>>>>
>>>> 3.8.x is EOLed, so I recommend you to upgrade
>>>> to a
>>>> supported
>>>> version
>>>> soon.
>>>>
>>>> Best regards,
>>>> Samuli Heinonen
>>>>
>>>> Samuli Heinonen
>>>> 20 January 2018 at 21.57
>>>>
>>>> Hi all!
>>>>
>>>> One hypervisor on our virtualization
>>>> environment
>>>> crashed and now
>>>> some of the VM images cannot be accessed.
>>>> After
>>>> investigation we
>>>> found out that there was lots of images that
>>>> still
>>>> had
>>>> active lock
>>>> on crashed hypervisor. We were able to remove
>>>> locks
>>>> from "regular
>>>> files", but it doesn't seem possible to remove
>>>> locks
>>>> from shards.
>>>>
>>>> We are running GlusterFS 3.8.15 on all nodes.
>>>>
>>>> Here is part of statedump that shows shard
>>>> having
>>>> active lock on
>>>> crashed node:
>>>>
>>>>
>>>> [xlator.features.locks.zone2-ssd1-vmstor1-locks.inode]
>>>>
>>>>
>>>> path=/.shard/75353c17-d6b8-485d-9baf-fd6c700e39a1.21
>>>> mandatory=0
>>>> inodelk-count=1
>>>>
>>>>
>>>> lock-dump.domain.domain=zone2-ssd1-vmstor1-replicate-0:metadata
>>>>
>>>>
>>>> lock-dump.domain.domain=zone2-ssd1-vmstor1-replicate-0:self-heal
>>>>
>>>>
>>>> lock-dump.domain.domain=zone2-ssd1-vmstor1-replicate-0
>>>> inodelk.inodelk[0](ACTIVE)=type=WRITE,
>>>> whence=0,
>>>> start=0, len=0,
>>>> pid = 3568, owner=14ce372c397f0000,
>>>> client=0x7f3198388770,
>>>> connection-id
>>>>
>>>>
>>>>
>>>>
>>> ovirt8z2.xxx-5652-2017/12/27-09:49:02:946825-zone2-ssd1-vmst
>> or1-client-1-7-0,
>>
>>>
>>>> granted at 2018-01-20 08:57:24
>>>>
>>>> If we try to run clear-locks we get following
>>>> error
>>>> message:
>>>> # gluster volume clear-locks
>>>> zone2-ssd1-vmstor1
>>>>
>>>> /.shard/75353c17-d6b8-485d-9baf-fd6c700e39a1.21
>>>> kind
>>>> all inode
>>>> Volume clear-locks unsuccessful
>>>> clear-locks getxattr command failed. Reason:
>>>> Operation not
>>>> permitted
>>>>
>>>> Gluster vol info if needed:
>>>> Volume Name: zone2-ssd1-vmstor1
>>>> Type: Replicate
>>>> Volume ID:
>>>> b6319968-690b-4060-8fff-b212d2295208
>>>> Status: Started
>>>> Snapshot Count: 0
>>>> Number of Bricks: 1 x 2 = 2
>>>> Transport-type: rdma
>>>> Bricks:
>>>> Brick1: sto1z2.xxx:/ssd1/zone2-vmstor1/export
>>>> Brick2: sto2z2.xxx:/ssd1/zone2-vmstor1/export
>>>> Options Reconfigured:
>>>> cluster.shd-wait-qlength: 10000
>>>> cluster.shd-max-threads: 8
>>>> cluster.locking-scheme: granular
>>>> performance.low-prio-threads: 32
>>>> cluster.data-self-heal-algorithm: full
>>>> performance.client-io-threads: off
>>>> storage.linux-aio: off
>>>> performance.readdir-ahead: on
>>>> client.event-threads: 16
>>>> server.event-threads: 16
>>>> performance.strict-write-ordering: off
>>>> performance.quick-read: off
>>>> performance.read-ahead: on
>>>> performance.io-cache: off
>>>> performance.stat-prefetch: off
>>>> cluster.eager-lock: enable
>>>> network.remote-dio: on
>>>> cluster.quorum-type: none
>>>> network.ping-timeout: 22
>>>> performance.write-behind: off
>>>> nfs.disable: on
>>>> features.shard: on
>>>> features.shard-block-size: 512MB
>>>> storage.owner-uid: 36
>>>> storage.owner-gid: 36
>>>> performance.io-thread-count: 64
>>>> performance.cache-size: 2048MB
>>>> performance.write-behind-window-size: 256MB
>>>> server.allow-insecure: on
>>>> cluster.ensure-durability: off
>>>> config.transport: rdma
>>>> server.outstanding-rpc-limit: 512
>>>> diagnostics.brick-log-level: INFO
>>>>
>>>> Any recommendations how to advance from here?
>>>>
>>>> Best regards,
>>>> Samuli Heinonen
>>>>
>>>>
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> <mailto:Gluster-users at gluster.org>
>>>> <mailto:Gluster-users at gluster.org
>>>>
>>>> <mailto:Gluster-users at gluster.org>>
>>>>
>>>>
>>>> http://lists.gluster.org/mailman/listinfo/gluster-users [3]
>>>>
>>>> <http://lists.gluster.org/mailman/listinfo/gluster-users [3]>
>>>> [3]
>>>>
>>>> <http://lists.gluster.org/mailman/listinfo/gluster-users [3]
>>>>
>>>> <http://lists.gluster.org/mailman/listinfo/gluster-users [3]>
>>>> [3]>
>>>> [1]
>>>>
>>>>
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> <mailto:Gluster-users at gluster.org>
>>>> <mailto:Gluster-users at gluster.org
>>>>
>>>> <mailto:Gluster-users at gluster.org>>
>>>>
>>>>
>>>> http://lists.gluster.org/mailman/listinfo/gluster-users [3]
>>>>
>>>> <http://lists.gluster.org/mailman/listinfo/gluster-users [3]>
>>>> [3]
>>>>
>>>>
>>>> <http://lists.gluster.org/mailman/listinfo/gluster-users [3]
>>>>
>>>>
>>>> <http://lists.gluster.org/mailman/listinfo/gluster-users [3]>
>>>> [3]> [1]
>>>>
>>>> --
>>>>
>>>> Pranith
>>>>
>>>> Links:
>>>> ------
>>>> [1]
>>>>
>>>> http://lists.gluster.org/mailman/listinfo/gluster-users [3]
>>>>
>>>>
>>>> <http://lists.gluster.org/mailman/listinfo/gluster-users [3]>
>>>> [3]
>>>>
>>>> <http://lists.gluster.org/mailman/listinfo/gluster-users [3]
>>>>
>>>>
>>>> <http://lists.gluster.org/mailman/listinfo/gluster-users [3]>
>>>>
>>>> [3]>
>>>>
>>>> --
>>>> Pranith
>>>> Samuli Heinonen <mailto:samppah at neutraali.net
>>>> <mailto:samppah at neutraali.net>>
>>>> 21 January 2018 at 21.03
>>>> Hi again,
>>>>
>>>> here is more information regarding issue
>>>> described
>>>> earlier
>>>>
>>>> It looks like self healing is stuck. According
>>>> to "heal
>>>> statistics" crawl began at Sat Jan 20 12:56:19
>>>> 2018
>>>> and it's still
>>>> going on (It's around Sun Jan 21 20:30 when
>>>> writing
>>>> this). However
>>>> glustershd.log says that last heal was
>>>> completed at
>>>> "2018-01-20
>>>> 11:00:13.090697" (which is 13:00 UTC+2). Also
>>>> "heal
>>>> info" has been
>>>> running now for over 16 hours without any
>>>> information. In
>>>> statedump I can see that storage nodes have
>>>> locks on
>>>> files and
>>>> some of those are blocked. Ie. Here again it
>>>> says
>>>> that ovirt8z2 is
>>>> having active lock even ovirt8z2 crashed after
>>>> the
>>>> lock was
>>>> granted.:
>>>>
>>>>
>>>> [xlator.features.locks.zone2-ssd1-vmstor1-locks.inode]
>>>>
>>>> path=/.shard/3d55f8cc-cda9-489a-b0a3-fd0f43d67876.27
>>>> mandatory=0
>>>> inodelk-count=3
>>>>
>>>> lock-dump.domain.domain=zone2-ssd1-vmstor1-replicate-0:self-heal
>>>> inodelk.inodelk[0](ACTIVE)=type=WRITE,
>>>> whence=0,
>>>> start=0, len=0,
>>>> pid = 18446744073709551610,
>>>> owner=d0c6d857a87f0000,
>>>> client=0x7f885845efa0,
>>>>
>>>>
>>>>
>>>>
>>> connection-id=sto2z2.xxx-10975-2018/01/20-10:56:14:649541-
>> zone2-ssd1-vmstor1-client-0-0-0,
>>
>>>
>>>> granted at 2018-01-20 10:59:52
>>>>
>>>> lock-dump.domain.domain=zone2-ssd1-vmstor1-replicate-0:metadata
>>>>
>>>> lock-dump.domain.domain=zone2-ssd1-vmstor1-replicate-0
>>>> inodelk.inodelk[0](ACTIVE)=type=WRITE,
>>>> whence=0,
>>>> start=0, len=0,
>>>> pid = 3420, owner=d8b9372c397f0000,
>>>> client=0x7f8858410be0,
>>>> connection-id=ovirt8z2.xxx.com [1]
>>>> <http://ovirt8z2.xxx.com>
>>>>
>>>>
>>>>
>>>> [1]-5652-2017/12/27-09:49:02:946825-zone2-ssd1-vmstor1-client-0-7-0,
>>>
>>>>
>>>> granted at 2018-01-20 08:57:23
>>>> inodelk.inodelk[1](BLOCKED)=type=WRITE,
>>>> whence=0,
>>>> start=0, len=0,
>>>> pid = 18446744073709551610,
>>>> owner=d0c6d857a87f0000,
>>>> client=0x7f885845efa0,
>>>>
>>>>
>>>>
>>>>
>>> connection-id=sto2z2.xxx-10975-2018/01/20-10:56:14:649541-
>> zone2-ssd1-vmstor1-client-0-0-0,
>>
>>>
>>>> blocked at 2018-01-20 10:59:52
>>>>
>>>> I'd also like to add that volume had arbiter
>>>> brick
>>>> before crash
>>>> happened. We decided to remove it because we
>>>> thought
>>>> that it was
>>>> causing issues. However now I think that this
>>>> was
>>>> unnecessary.
>>>> After the crash arbiter logs had lots of
>>>> messages
>>>> like this:
>>>> [2018-01-20 10:19:36.515717] I [MSGID: 115072]
>>>> [server-rpc-fops.c:1640:server_setattr_cbk]
>>>> 0-zone2-ssd1-vmstor1-server: 37374187: SETATTR
>>>> <gfid:a52055bd-e2e9-42dd-92a3-e96b693bcafe>
>>>> (a52055bd-e2e9-42dd-92a3-e96b693bcafe) ==>
>>>> (Operation not
>>>> permitted) [Operation not permitted]
>>>>
>>>> Is there anyways to force self heal to stop?
>>>> Any
>>>> help would be
>>>> very much appreciated :)
>>>>
>>>> Best regards,
>>>> Samuli Heinonen
>>>>
>>>>
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> <mailto:Gluster-users at gluster.org>
>>>>
>>>> http://lists.gluster.org/mailman/listinfo/gluster-users [3]
>>>>
>>>>
>>>> <http://lists.gluster.org/mailman/listinfo/gluster-users [3]>
>>>>
>>>> [3]
>>>>
>>>> Samuli Heinonen <mailto:samppah at neutraali.net
>>>> <mailto:samppah at neutraali.net>>
>>>>
>>>> 20 January 2018 at 21.57
>>>> Hi all!
>>>>
>>>> One hypervisor on our virtualization
>>>> environment
>>>> crashed and now
>>>> some of the VM images cannot be accessed.
>>>> After
>>>> investigation we
>>>> found out that there was lots of images that
>>>> still
>>>> had active lock
>>>> on crashed hypervisor. We were able to remove
>>>> locks
>>>> from "regular
>>>> files", but it doesn't seem possible to remove
>>>> locks
>>>> from shards.
>>>>
>>>> We are running GlusterFS 3.8.15 on all nodes.
>>>>
>>>> Here is part of statedump that shows shard
>>>> having
>>>> active lock on
>>>> crashed node:
>>>>
>>>> [xlator.features.locks.zone2-ssd1-vmstor1-locks.inode]
>>>>
>>>> path=/.shard/75353c17-d6b8-485d-9baf-fd6c700e39a1.21
>>>> mandatory=0
>>>> inodelk-count=1
>>>>
>>>> lock-dump.domain.domain=zone2-ssd1-vmstor1-replicate-0:metadata
>>>>
>>>> lock-dump.domain.domain=zone2-ssd1-vmstor1-replicate-0:self-heal
>>>>
>>>> lock-dump.domain.domain=zone2-ssd1-vmstor1-replicate-0
>>>> inodelk.inodelk[0](ACTIVE)=type=WRITE,
>>>> whence=0,
>>>> start=0, len=0,
>>>> pid = 3568, owner=14ce372c397f0000,
>>>> client=0x7f3198388770,
>>>> connection-id
>>>>
>>>>
>>>>
>>>>
>>> ovirt8z2.xxx-5652-2017/12/27-09:49:02:946825-zone2-ssd1-vmst
>> or1-client-1-7-0,
>>
>>>
>>>> granted at 2018-01-20 08:57:24
>>>>
>>>> If we try to run clear-locks we get following
>>>> error
>>>> message:
>>>> # gluster volume clear-locks
>>>> zone2-ssd1-vmstor1
>>>>
>>>> /.shard/75353c17-d6b8-485d-9baf-fd6c700e39a1.21 kind
>>>> all inode
>>>> Volume clear-locks unsuccessful
>>>> clear-locks getxattr command failed. Reason:
>>>> Operation not
>>>> permitted
>>>>
>>>> Gluster vol info if needed:
>>>> Volume Name: zone2-ssd1-vmstor1
>>>> Type: Replicate
>>>> Volume ID:
>>>> b6319968-690b-4060-8fff-b212d2295208
>>>> Status: Started
>>>> Snapshot Count: 0
>>>> Number of Bricks: 1 x 2 = 2
>>>> Transport-type: rdma
>>>> Bricks:
>>>> Brick1: sto1z2.xxx:/ssd1/zone2-vmstor1/export
>>>> Brick2: sto2z2.xxx:/ssd1/zone2-vmstor1/export
>>>> Options Reconfigured:
>>>> cluster.shd-wait-qlength: 10000
>>>> cluster.shd-max-threads: 8
>>>> cluster.locking-scheme: granular
>>>> performance.low-prio-threads: 32
>>>> cluster.data-self-heal-algorithm: full
>>>> performance.client-io-threads: off
>>>> storage.linux-aio: off
>>>> performance.readdir-ahead: on
>>>> client.event-threads: 16
>>>> server.event-threads: 16
>>>> performance.strict-write-ordering: off
>>>> performance.quick-read: off
>>>> performance.read-ahead: on
>>>> performance.io-cache: off
>>>> performance.stat-prefetch: off
>>>> cluster.eager-lock: enable
>>>> network.remote-dio: on
>>>> cluster.quorum-type: none
>>>> network.ping-timeout: 22
>>>> performance.write-behind: off
>>>> nfs.disable: on
>>>> features.shard: on
>>>> features.shard-block-size: 512MB
>>>> storage.owner-uid: 36
>>>> storage.owner-gid: 36
>>>> performance.io-thread-count: 64
>>>> performance.cache-size: 2048MB
>>>> performance.write-behind-window-size: 256MB
>>>> server.allow-insecure: on
>>>> cluster.ensure-durability: off
>>>> config.transport: rdma
>>>> server.outstanding-rpc-limit: 512
>>>> diagnostics.brick-log-level: INFO
>>>>
>>>> Any recommendations how to advance from here?
>>>>
>>>> Best regards,
>>>> Samuli Heinonen
>>>>
>>>>
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> <mailto:Gluster-users at gluster.org>
>>>>
>>>> http://lists.gluster.org/mailman/listinfo/gluster-users [3]
>>>>
>>>>
>>>> <http://lists.gluster.org/mailman/listinfo/gluster-users [3]>
>>>> [3]
>>>>
>>>> --
>>>>
>>>> Pranith
>>>>
>>>> Links:
>>>> ------
>>>> [1] http://ovirt8z2.xxx.com
>>>> [2] http://review.gluster.org/14816 [2]
>>>> <http://review.gluster.org/14816 [2]>
>>>> [3]
>>>> http://lists.gluster.org/mailman/listinfo/gluster-users [3]
>>>>
>>>> <http://lists.gluster.org/mailman/listinfo/gluster-users [3]>
>>>>
>>>> --
>>>> Pranith
>>>>
>>>
>>
>>
>> Links:
>> ------
>> [1] http://ovirt8z2.xxx.com
>> [2] http://review.gluster.org/14816
>> [3] http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>


-- 
Pranith
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180129/f162aa3c/attachment.html>

More information about the Gluster-users mailing list