<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jan 23, 2018 at 1:04 PM, Pranith Kumar Karampuri <span dir="ltr"><<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="gmail-">On Mon, Jan 22, 2018 at 12:33 AM, Samuli Heinonen <span dir="ltr"><<a href="mailto:samppah@neutraali.net" target="_blank">samppah@neutraali.net</a>></span> wrote:<br></span><span class="gmail-"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">Hi again,<br>
<br>
here is more information regarding issue described earlier<br>
<br>
It looks like self healing is stuck. According to "heal statistics"
crawl began at Sat Jan 20 12:56:19 2018 and it's still going on (It's
around Sun Jan 21 20:30 when writing this). However glustershd.log says
that last heal was completed at "2018-01-20 11:00:13.090697" (which is
13:00 UTC+2). Also "heal info" has been running now for over 16 hours
without any information. In statedump I can see that storage nodes have
locks on files and some of those are blocked. Ie. Here again it says
that ovirt8z2 is having active lock even ovirt8z2 crashed after the lock
was granted.:<br>
<br>
[xlator.features.locks.zone2-s<wbr>sd1-vmstor1-locks.inode]<br>
path=/.shard/3d55f8cc-cda9-489<wbr>a-b0a3-fd0f43d67876.27<br>
mandatory=0<br>
inodelk-count=3<br>
lock-dump.domain.domain=zone2-<wbr>ssd1-vmstor1-replicate-0:self-<wbr>heal<br>
inodelk.inodelk[0](ACTIVE)=typ<wbr>e=WRITE, whence=0, start=0, len=0, pid =
18446744073709551610, owner=d0c6d857a87f0000, client=0x7f885845efa0,
connection-id=sto2z2.xxx-10975<wbr>-2018/01/20-10:56:14:649541-<wbr>zone2-ssd1-vmstor1-client-0-0-<wbr>0,
granted at 2018-01-20 10:59:52<br>
lock-dump.domain.domain=zone2-<wbr>ssd1-vmstor1-replicate-0:metad<wbr>ata<br>
lock-dump.domain.domain=zone2-<wbr>ssd1-vmstor1-replicate-0<br>
inodelk.inodelk[0](ACTIVE)=typ<wbr>e=WRITE, whence=0, start=0, len=0, pid =
3420, owner=d8b9372c397f0000, client=0x7f8858410be0,
connection-id=<a href="http://ovirt8z2.xxx.com">ovirt8z2.xxx.com</a><wbr>-5652-2017/12/27-09:49:02:9468<wbr>25-zone2-ssd1-vmstor1-client-<wbr>0-7-0,
granted at 2018-01-20 08:57:23<br>
inodelk.inodelk[1](BLOCKED)=ty<wbr>pe=WRITE, whence=0, start=0, len=0, pid =
18446744073709551610, owner=d0c6d857a87f0000, client=0x7f885845efa0,
connection-id=sto2z2.xxx-10975<wbr>-2018/01/20-10:56:14:649541-<wbr>zone2-ssd1-vmstor1-client-0-0-<wbr>0,
blocked at 2018-01-20 10:59:52<br>
<br>
I'd also like to add that volume had arbiter brick before crash
happened. We decided to remove it because we thought that it was causing
issues. However now I think that this was unnecessary. After the crash
arbiter logs had lots of messages like this:<br>
[2018-01-20 10:19:36.515717] I [MSGID: 115072]
[server-rpc-fops.c:1640:server<wbr>_setattr_cbk] 0-zone2-ssd1-vmstor1-server:
37374187: SETATTR <gfid:a52055bd-e2e9-42dd-92a3-<wbr>e96b693bcafe>
(a52055bd-e2e9-42dd-92a3-e96b6<wbr>93bcafe) ==> (Operation not permitted)
[Operation not permitted]<br>
<br>
Is there anyways to force self heal to stop? Any help would be very much
appreciated :)<br></div></blockquote><div><br></div></span><div>Exposing .shard to a normal mount is opening a can of worms. You should probably look at mounting the volume with gfid aux-mount where you can access a file with <path-to-mount>/.gfid/<gfid-<wbr>string>to clear locks on it.</div></div></div></div></blockquote><div><br></div><div>Please use this mount only for doing just this work and unmount it after that. But my recommendation would be to do an upgrade as soon as possible. Your bricks will crash on the next disconnect from '<span class="gmail-">sto2z2.xxx' if you are not lucky.<br></span></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div>Mount command: <code class="gmail-m_-4451317613014973479gmail-hljs gmail-m_-4451317613014973479gmail-puppet"><span class="gmail-m_-4451317613014973479gmail-hljs-keyword">mount</span> -t glusterfs -o aux-gfid-<span class="gmail-m_-4451317613014973479gmail-hljs-keyword">mount</span> vm1:test /mnt/testvol</code></div><div><pre><code class="gmail-m_-4451317613014973479gmail-hljs gmail-m_-4451317613014973479gmail-puppet"><span class="gmail-m_-4451317613014973479gmail-hljs-comment">A gfid string will have some hyphens like: 11118443-1894-4273-9340-<wbr>4b212fa1c0e4</span></code></pre></div><div>That said. Next disconnect on the brick where you successfully did the clear-locks will crash the brick. There was a bug in 3.8.x series with clear-locks which was fixed in 3.9.0 with a feature. The self-heal deadlocks that you witnessed also is fixed in 3.10 version of the release.</div><div><br></div><div>3.8.x is EOLed, so I recommend you to upgrade to a supported version soon.<br></div><div><div class="gmail-h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
<br>
Best regards,<br>
Samuli Heinonen<br>
<br>
<br>
<br>
<br>
<span>
</span><br>
<blockquote style="border-color:currentcolor;border-style:none;border-width:0px" type="cite">
<div style="margin:30px 25px 10px" class="gmail-m_-4451317613014973479gmail-m_-7566555307473636566__pbConvHr"><div style="width:100%;border-top:2px solid rgb(237,241,244);padding-top:10px"> <div style="display:inline-block;white-space:nowrap;vertical-align:middle;width:49%">
<a href="mailto:samppah@neutraali.net" style="color:rgb(72,86,100);padding-right:6px;font-weight:500;text-decoration:none" target="_blank">Samuli Heinonen</a></div> <div style="display:inline-block;white-space:nowrap;vertical-align:middle;width:48%;text-align:right"> <font color="#909AA4"><span style="padding-left:6px">20
January 2018 at 21.57</span></font></div> </div></div><div><div class="gmail-m_-4451317613014973479gmail-h5">
<div style="color:rgb(144,154,164);margin-left:24px;margin-right:24px" class="gmail-m_-4451317613014973479gmail-m_-7566555307473636566__pbConvBody">Hi all!
<br>
<br>One hypervisor on our virtualization environment crashed and now
some of
the VM images cannot be accessed. After investigation we found out that
there was lots of images that still had active lock on crashed
hypervisor. We were able to remove locks from "regular files", but it
doesn't seem possible to remove locks from shards.
<br>
<br>We are running GlusterFS 3.8.15 on all nodes.
<br>
<br>Here is part of statedump that shows shard having active lock on
crashed
node:
<br>[xlator.features.locks.zone2-s<wbr>sd1-vmstor1-locks.inode]
<br>path=/.shard/75353c17-d6b8-485<wbr>d-9baf-fd6c700e39a1.21
<br>mandatory=0
<br>inodelk-count=1
<br>lock-dump.domain.domain=zone2-<wbr>ssd1-vmstor1-replicate-0:metad<wbr>ata
<br>lock-dump.domain.domain=zone2-<wbr>ssd1-vmstor1-replicate-0:self-<wbr>heal
<br>lock-dump.domain.domain=zone2-<wbr>ssd1-vmstor1-replicate-0
<br>inodelk.inodelk[0](ACTIVE)=typ<wbr>e=WRITE, whence=0, start=0, len=0, pid
=
3568, owner=14ce372c397f0000, client=0x7f3198388770, connection-id
ovirt8z2.xxx-5652-2017/12/27-0<wbr>9:49:02:946825-zone2-ssd1-vmst<wbr>or1-client-1-7-0,
granted at 2018-01-20 08:57:24
<br>
<br>If we try to run clear-locks we get following error message:
<br># gluster volume clear-locks zone2-ssd1-vmstor1
/.shard/75353c17-d6b8-485d-9ba<wbr>f-fd6c700e39a1.21 kind all inode
<br>Volume clear-locks unsuccessful
<br>clear-locks getxattr command failed. Reason: Operation not permitted
<br>
<br>Gluster vol info if needed:
<br>Volume Name: zone2-ssd1-vmstor1
<br>Type: Replicate
<br>Volume ID: b6319968-690b-4060-8fff-b212d2<wbr>295208
<br>Status: Started
<br>Snapshot Count: 0
<br>Number of Bricks: 1 x 2 = 2
<br>Transport-type: rdma
<br>Bricks:
<br>Brick1: sto1z2.xxx:/ssd1/zone2-vmstor1<wbr>/export
<br>Brick2: sto2z2.xxx:/ssd1/zone2-vmstor1<wbr>/export
<br>Options Reconfigured:
<br>cluster.shd-wait-qlength: 10000
<br>cluster.shd-max-threads: 8
<br>cluster.locking-scheme: granular
<br>performance.low-prio-threads: 32
<br>cluster.data-self-heal-algorit<wbr>hm: full
<br>performance.client-io-threads: off
<br>storage.linux-aio: off
<br>performance.readdir-ahead: on
<br>client.event-threads: 16
<br>server.event-threads: 16
<br>performance.strict-write-order<wbr>ing: off
<br>performance.quick-read: off
<br>performance.read-ahead: on
<br>performance.io-cache: off
<br>performance.stat-prefetch: off
<br>cluster.eager-lock: enable
<br>network.remote-dio: on
<br>cluster.quorum-type: none
<br>network.ping-timeout: 22
<br>performance.write-behind: off
<br>nfs.disable: on
<br>features.shard: on
<br>features.shard-block-size: 512MB
<br>storage.owner-uid: 36
<br>storage.owner-gid: 36
<br>performance.io-thread-count: 64
<br>performance.cache-size: 2048MB
<br>performance.write-behind-windo<wbr>w-size: 256MB
<br>server.allow-insecure: on
<br>cluster.ensure-durability: off
<br>config.transport: rdma
<br>server.outstanding-rpc-limit: 512
<br>diagnostics.brick-log-level: INFO
<br>
<br>Any recommendations how to advance from here?
<br>
<br>Best regards,
<br>Samuli Heinonen
<br>
<br>______________________________<wbr>_________________
<br>Gluster-users mailing list
<br><a class="gmail-m_-4451317613014973479gmail-m_-7566555307473636566moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<br><a class="gmail-m_-4451317613014973479gmail-m_-7566555307473636566moz-txt-link-freetext" href="http://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a>
<br></div>
</div></div></blockquote>
<br>
</div>
<br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br></blockquote></div></div></div><span class="gmail-HOEnZb"><font color="#888888"><br><br clear="all"><br>-- <br><div class="gmail-m_-4451317613014973479gmail_signature"><div dir="ltr">Pranith<br></div></div>
</font></span></div></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature"><div dir="ltr">Pranith<br></div></div>
</div></div>