<div dir="auto"><div><br><div class="gmail_extra"><br><div class="gmail_quote">On 29 Jan 2018 10:50 am, "Samuli Heinonen" <<a href="mailto:samppah@neutraali.net">samppah@neutraali.net</a>> wrote:<br type="attribution"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi!<br>
<br>
Yes, thank you for asking. I found out this line in the production environment:<br>
lgetxattr("/tmp/zone2-ssd1-vms<wbr>tor1.s6jvPu//.shard/f349ffbd-<wbr>a423-4fb2-b83c-2d1d5e78e1fb.<wbr>32", "glusterfs.clrlk.tinode.kblock<wbr>ed", 0x7f2d7c4379f0, 4096) = -1 EPERM (Operation not permitted)<br></blockquote></div></div></div><div dir="auto"><br></div><div dir="auto">I was expecting .kall instead of .blocked,</div><div dir="auto">did you change the cli to kind blocked?</div><div dir="auto"><br></div><div dir="auto"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
And this one in test environment (with posix locks):<br>
lgetxattr("/tmp/g1.gHj4Bw//fil<wbr>e38", "glusterfs.clrlk.tposix.kblock<wbr>ed", "box1:/gluster/1/export/: posix blocked locks=1 granted locks=0", 4096) = 77<br>
<br>
In test environment I tried running following command which seemed to release gluster locks:<br>
<br>
getfattr -n glusterfs.clrlk.tposix.kblocke<wbr>d file38<br>
<br>
So I think it would go like this in production environment with locks on shards (using aux-gfid-mount mount option):<br>
getfattr -n glusterfs.clrlk.tinode.kall .shard/f349ffbd-a423-4fb2-b83c<wbr>-2d1d5e78e1fb.32<br>
<br>
I haven't been able to try this out in production environment yet.<br>
<br>
Is there perhaps something else to notice?<br>
<br>
Would you be able to tell more about bricks crashing after releasing locks? Under what circumstances that does happen? Is it only process exporting the brick crashes or is there a possibility of data corruption?<br></blockquote></div></div></div><div dir="auto"><br></div><div dir="auto">No data corruption. Brick process where you did clear-locks may crash.</div><div dir="auto"><br></div><div dir="auto"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Best regards,<br>
Samuli Heinonen<br>
<br>
<br>
Pranith Kumar Karampuri wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="quoted-text">
Hi,<br>
   Did you find the command from strace?<br>
<br>
On 25 Jan 2018 1:52 pm, "Pranith Kumar Karampuri" <<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a><br></div><div class="quoted-text">
<mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>>> wrote:<br>
<br>
<br>
<br>
  On Thu, Jan 25, 2018 at 1:49 PM, Samuli Heinonen<br></div><div class="quoted-text">
  <<a href="mailto:samppah@neutraali.net" target="_blank">samppah@neutraali.net</a> <mailto:<a href="mailto:samppah@neutraali.net" target="_blank">samppah@neutraali.net</a>><wbr>> wrote:<br>
<br>
    Pranith Kumar Karampuri kirjoitti 25.01.2018 07:09:<br>
<br>
      On Thu, Jan 25, 2018 at 2:27 AM, Samuli Heinonen<br></div><div class="elided-text">
      <<a href="mailto:samppah@neutraali.net" target="_blank">samppah@neutraali.net</a> <mailto:<a href="mailto:samppah@neutraali.net" target="_blank">samppah@neutraali.net</a>><wbr>> wrote:<br>
<br>
        Hi!<br>
<br>
        Thank you very much for your help so far. Could you<br>
        please tell an<br>
        example command how to use aux-gid-mount to remove<br>
        locks? "gluster<br>
        vol clear-locks" seems to mount volume by itself.<br>
<br>
<br>
      You are correct, sorry, this was implemented around 7 years<br>
      back and I<br>
      forgot that bit about it :-(. Essentially it becomes a getxattr<br>
      syscall on the file.<br>
      Could you give me the clear-locks command you were trying to<br>
      execute<br>
      and I can probably convert it to the getfattr command?<br>
<br>
<br>
    I have been testing this in test environment and with command:<br>
    gluster vol clear-locks g1<br>
    /.gfid/14341ccb-df7b-4f92-90d5<wbr>-7814431c5a1c kind all inode<br>
<br>
<br>
  Could you do strace of glusterd when this happens? It will have a<br>
  getxattr with "glusterfs.clrlk" in the key. You need to execute that<br>
  on the gfid-aux-mount<br>
<br>
<br>
<br>
<br>
        Best regards,<br>
        Samuli Heinonen<br>
<br>
          Pranith Kumar Karampuri <mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a><br>
          <mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>>><br>
          23 January 2018 at 10.30<br>
<br>
          On Tue, Jan 23, 2018 at 1:38 PM, Samuli Heinonen<br>
          <<a href="mailto:samppah@neutraali.net" target="_blank">samppah@neutraali.net</a><br>
          <mailto:<a href="mailto:samppah@neutraali.net" target="_blank">samppah@neutraali.net</a>><br></div>
          <mailto:<a href="mailto:samppah@neutraali.net" target="_blank">samppah@neutraali.net</a><div class="quoted-text"><br>
          <mailto:<a href="mailto:samppah@neutraali.net" target="_blank">samppah@neutraali.net</a>><wbr>>> wrote:<br>
<br>
          Pranith Kumar Karampuri kirjoitti 23.01.2018 09:34:<br>
<br>
          On Mon, Jan 22, 2018 at 12:33 AM, Samuli Heinonen<br>
<br>
          <<a href="mailto:samppah@neutraali.net" target="_blank">samppah@neutraali.net</a><br>
          <mailto:<a href="mailto:samppah@neutraali.net" target="_blank">samppah@neutraali.net</a>><br></div>
          <mailto:<a href="mailto:samppah@neutraali.net" target="_blank">samppah@neutraali.net</a><div class="elided-text"><br>
          <mailto:<a href="mailto:samppah@neutraali.net" target="_blank">samppah@neutraali.net</a>><wbr>>><br>
          wrote:<br>
<br>
          Hi again,<br>
<br>
          here is more information regarding issue described<br>
          earlier<br>
<br>
          It looks like self healing is stuck. According to<br>
          "heal<br>
          statistics"<br>
          crawl began at Sat Jan 20 12:56:19 2018 and it's still<br>
          going on<br>
          (It's around Sun Jan 21 20:30 when writing this).<br>
          However<br>
          glustershd.log says that last heal was completed at<br>
          "2018-01-20<br>
          11:00:13.090697" (which is 13:00 UTC+2). Also "heal<br>
          info"<br>
          has been<br>
          running now for over 16 hours without any information.<br>
          In<br>
          statedump<br>
          I can see that storage nodes have locks on files and<br>
          some<br>
          of those<br>
          are blocked. Ie. Here again it says that ovirt8z2 is<br>
          having active<br>
          lock even ovirt8z2 crashed after the lock was<br>
          granted.:<br>
<br>
          [xlator.features.locks.zone2-s<wbr>sd1-vmstor1-locks.inode]<br>
          path=/.shard/3d55f8cc-cda9-489<wbr>a-b0a3-fd0f43d67876.27<br>
          mandatory=0<br>
          inodelk-count=3<br>
<br>
          lock-dump.domain.domain=zone2-<wbr>ssd1-vmstor1-replicate-0:self-<wbr>heal<br>
          inodelk.inodelk[0](ACTIVE)=typ<wbr>e=WRITE, whence=0,<br>
          start=0,<br>
          len=0, pid<br>
          = 18446744073709551610, owner=d0c6d857a87f0000,<br>
          client=0x7f885845efa0,<br>
<br>
<br>
<br>
<br>
      connection-id=sto2z2.xxx-10975<wbr>-2018/01/20-10:56:14:649541-<wbr>zone2-ssd1-vmstor1-client-0-0-<wbr>0,<br>
<br>
<br>
          granted at 2018-01-20 10:59:52<br>
<br>
          lock-dump.domain.domain=zone2-<wbr>ssd1-vmstor1-replicate-0:metad<wbr>ata<br>
          lock-dump.domain.domain=zone2-<wbr>ssd1-vmstor1-replicate-0<br>
          inodelk.inodelk[0](ACTIVE)=typ<wbr>e=WRITE, whence=0,<br>
          start=0,<br>
          len=0, pid<br>
          = 3420, owner=d8b9372c397f0000, client=0x7f8858410be0,<br>
<br>
          connection-id=<a href="http://ovirt8z2.xxx.com" rel="noreferrer" target="_blank">ovirt8z2.xxx.com</a><br></div>
          <<a href="http://ovirt8z2.xxx.com" rel="noreferrer" target="_blank">http://ovirt8z2.xxx.com</a>> [1]<div class="elided-text"><br>
<br>
<br>
<br>
      <<a href="http://ovirt8z2.xxx.com" rel="noreferrer" target="_blank">http://ovirt8z2.xxx.com</a>>-5652<wbr>-2017/12/27-09:49:02:946825-<wbr>zone2-ssd1-vmstor1-client-0-7-<wbr>0,<br>
<br>
<br>
          granted at 2018-01-20 08:57:23<br>
          inodelk.inodelk[1](BLOCKED)=ty<wbr>pe=WRITE, whence=0,<br>
          start=0,<br>
          len=0,<br>
          pid = 18446744073709551610, owner=d0c6d857a87f0000,<br>
          client=0x7f885845efa0,<br>
<br>
<br>
<br>
<br>
      connection-id=sto2z2.xxx-10975<wbr>-2018/01/20-10:56:14:649541-<wbr>zone2-ssd1-vmstor1-client-0-0-<wbr>0,<br>
<br>
<br>
          blocked at 2018-01-20 10:59:52<br>
<br>
          I'd also like to add that volume had arbiter brick<br>
          before<br>
          crash<br>
          happened. We decided to remove it because we thought<br>
          that<br>
          it was<br>
          causing issues. However now I think that this was<br>
          unnecessary. After<br>
          the crash arbiter logs had lots of messages like this:<br>
          [2018-01-20 10:19:36.515717] I [MSGID: 115072]<br>
          [server-rpc-fops.c:1640:server<wbr>_setattr_cbk]<br>
          0-zone2-ssd1-vmstor1-server: 37374187: SETATTR<br>
          <gfid:a52055bd-e2e9-42dd-92a3-<wbr>e96b693bcafe><br>
          (a52055bd-e2e9-42dd-92a3-e96b6<wbr>93bcafe) ==> (Operation<br>
          not<br>
          permitted)<br>
          [Operation not permitted]<br>
<br>
          Is there anyways to force self heal to stop? Any help<br>
          would be very<br>
          much appreciated :)<br>
<br>
          Exposing .shard to a normal mount is opening a can of<br>
          worms. You<br>
          should probably look at mounting the volume with gfid<br>
          aux-mount where<br>
          you can access a file with<br>
          <path-to-mount>/.gfid/<gfid-st<wbr>ring>to clear<br>
          locks on it.<br>
<br>
          Mount command: mount -t glusterfs -o aux-gfid-mount<br>
          vm1:test<br>
          /mnt/testvol<br>
<br>
          A gfid string will have some hyphens like:<br>
          11118443-1894-4273-9340-4b212f<wbr>a1c0e4<br>
<br>
          That said. Next disconnect on the brick where you<br>
          successfully<br>
          did the<br>
          clear-locks will crash the brick. There was a bug in<br>
          3.8.x<br>
          series with<br>
          clear-locks which was fixed in 3.9.0 with a feature. The<br>
          self-heal<br>
          deadlocks that you witnessed also is fixed in 3.10<br>
          version<br>
          of the<br>
          release.<br>
<br>
          Thank you the answer. Could you please tell more<br>
          about crash?<br>
          What<br>
          will actually happen or is there a bug report about<br>
          it? Just<br>
          want<br>
          to make sure that we can do everything to secure data on<br>
          bricks.<br>
          We will look into upgrade but we have to make sure<br>
          that new<br>
          version works for us and of course get self healing<br>
          working<br>
          before<br>
          doing anything :)<br>
<br>
          Locks xlator/module maintains a list of locks that<br>
          are granted to<br>
          a client. Clear locks had an issue where it forgets<br>
          to remove the<br>
          lock from this list. So the connection list ends up<br>
          pointing to<br>
          data that is freed in that list after a clear lock.<br>
          When a<br>
          disconnect happens, all the locks that are granted<br>
          to a client<br>
          need to be unlocked. So the process starts<br>
          traversing through this<br>
          list and when it starts trying to access this freed<br>
          data it leads<br>
          to a crash. I found it while reviewing a feature<br>
          patch sent by<br>
          facebook folks to locks xlator<br>
          (<a href="http://review.gluster.org/14816" rel="noreferrer" target="_blank">http://review.gluster.org/148<wbr>16</a><br>
          <<a href="http://review.gluster.org/14816" rel="noreferrer" target="_blank">http://review.gluster.org/148<wbr>16</a>><br>
          [2]) for 3.9.0 and they also fixed this bug as well<br>
          as part of<br>
<br>
          that feature patch.<br>
<br>
          Br,<br>
          Samuli<br>
<br>
          3.8.x is EOLed, so I recommend you to upgrade to a<br>
          supported<br>
          version<br>
          soon.<br>
<br>
          Best regards,<br>
          Samuli Heinonen<br>
<br>
          Samuli Heinonen<br>
          20 January 2018 at 21.57<br>
<br>
          Hi all!<br>
<br>
          One hypervisor on our virtualization environment<br>
          crashed and now<br>
          some of the VM images cannot be accessed. After<br>
          investigation we<br>
          found out that there was lots of images that still<br>
          had<br>
          active lock<br>
          on crashed hypervisor. We were able to remove<br>
          locks<br>
          from "regular<br>
          files", but it doesn't seem possible to remove<br>
          locks<br>
          from shards.<br>
<br>
          We are running GlusterFS 3.8.15 on all nodes.<br>
<br>
          Here is part of statedump that shows shard having<br>
          active lock on<br>
          crashed node:<br>
<br>
          [xlator.features.locks.zone2-s<wbr>sd1-vmstor1-locks.inode]<br>
<br>
          path=/.shard/75353c17-d6b8-485<wbr>d-9baf-fd6c700e39a1.21<br>
          mandatory=0<br>
          inodelk-count=1<br>
<br>
          lock-dump.domain.domain=zone2-<wbr>ssd1-vmstor1-replicate-0:metad<wbr>ata<br>
<br>
          lock-dump.domain.domain=zone2-<wbr>ssd1-vmstor1-replicate-0:self-<wbr>heal<br>
<br>
          lock-dump.domain.domain=zone2-<wbr>ssd1-vmstor1-replicate-0<br>
          inodelk.inodelk[0](ACTIVE)=typ<wbr>e=WRITE, whence=0,<br>
          start=0, len=0,<br>
          pid = 3568, owner=14ce372c397f0000,<br>
          client=0x7f3198388770,<br>
          connection-id<br>
<br>
<br>
<br>
<br>
      ovirt8z2.xxx-5652-2017/12/27-0<wbr>9:49:02:946825-zone2-ssd1-vmst<wbr>or1-client-1-7-0,<br>
<br>
<br>
          granted at 2018-01-20 08:57:24<br>
<br>
          If we try to run clear-locks we get following<br>
          error<br>
          message:<br>
          # gluster volume clear-locks zone2-ssd1-vmstor1<br>
          /.shard/75353c17-d6b8-485d-9ba<wbr>f-fd6c700e39a1.21<br>
          kind<br>
          all inode<br>
          Volume clear-locks unsuccessful<br>
          clear-locks getxattr command failed. Reason:<br>
          Operation not<br>
          permitted<br>
<br>
          Gluster vol info if needed:<br>
          Volume Name: zone2-ssd1-vmstor1<br>
          Type: Replicate<br>
          Volume ID: b6319968-690b-4060-8fff-b212d2<wbr>295208<br>
          Status: Started<br>
          Snapshot Count: 0<br>
          Number of Bricks: 1 x 2 = 2<br>
          Transport-type: rdma<br>
          Bricks:<br>
          Brick1: sto1z2.xxx:/ssd1/zone2-vmstor1<wbr>/export<br>
          Brick2: sto2z2.xxx:/ssd1/zone2-vmstor1<wbr>/export<br>
          Options Reconfigured:<br>
          cluster.shd-wait-qlength: 10000<br>
          cluster.shd-max-threads: 8<br>
          cluster.locking-scheme: granular<br>
          performance.low-prio-threads: 32<br>
          cluster.data-self-heal-algorit<wbr>hm: full<br>
          performance.client-io-threads: off<br>
          storage.linux-aio: off<br>
          performance.readdir-ahead: on<br>
          client.event-threads: 16<br>
          server.event-threads: 16<br>
          performance.strict-write-order<wbr>ing: off<br>
          performance.quick-read: off<br>
          performance.read-ahead: on<br>
          performance.io-cache: off<br>
          performance.stat-prefetch: off<br>
          cluster.eager-lock: enable<br>
          network.remote-dio: on<br>
          cluster.quorum-type: none<br>
          network.ping-timeout: 22<br>
          performance.write-behind: off<br>
          nfs.disable: on<br>
          features.shard: on<br>
          features.shard-block-size: 512MB<br>
          storage.owner-uid: 36<br>
          storage.owner-gid: 36<br>
          performance.io-thread-count: 64<br>
          performance.cache-size: 2048MB<br>
          performance.write-behind-windo<wbr>w-size: 256MB<br>
          server.allow-insecure: on<br>
          cluster.ensure-durability: off<br>
          config.transport: rdma<br>
          server.outstanding-rpc-limit: 512<br>
          diagnostics.brick-log-level: INFO<br>
<br>
          Any recommendations how to advance from here?<br>
<br>
          Best regards,<br>
          Samuli Heinonen<br>
<br>
          ______________________________<wbr>_________________<br>
          Gluster-users mailing list<br>
          <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
          <mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.<wbr>org</a>><br></div>
          <mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.<wbr>org</a><div class="quoted-text"><br>
          <mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.<wbr>org</a>>><br>
<br>
          <a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
          <<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mail<wbr>man/listinfo/gluster-users</a>><br>
          [3]<br>
          <<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mail<wbr>man/listinfo/gluster-users</a><br>
          <<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mail<wbr>man/listinfo/gluster-users</a>><br>
          [3]><br>
          [1]<br>
<br>
          ______________________________<wbr>_________________<br>
          Gluster-users mailing list<br>
          <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
          <mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.<wbr>org</a>><br></div>
          <mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.<wbr>org</a><div class="quoted-text"><br>
          <mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.<wbr>org</a>>><br>
<br>
          <a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
          <<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mail<wbr>man/listinfo/gluster-users</a>><br>
          [3]<br>
<br>
          <<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mail<wbr>man/listinfo/gluster-users</a><br></div><div class="quoted-text">
          <<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mail<wbr>man/listinfo/gluster-users</a>><br>
          [3]> [1]<br>
<br>
          --<br>
<br>
          Pranith<br>
<br>
          Links:<br>
          ------<br>
          [1]<br>
          <a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br></div><div class="quoted-text">
          <<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mail<wbr>man/listinfo/gluster-users</a>><br>
          [3]<br>
          <<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mail<wbr>man/listinfo/gluster-users</a><br></div><div class="elided-text">
          <<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mail<wbr>man/listinfo/gluster-users</a>><br>
          [3]><br>
<br>
<br>
          --<br>
          Pranith<br>
          Samuli Heinonen <mailto:<a href="mailto:samppah@neutraali.net" target="_blank">samppah@neutraali.net</a><br>
          <mailto:<a href="mailto:samppah@neutraali.net" target="_blank">samppah@neutraali.net</a>><wbr>><br>
          21 January 2018 at 21.03<br>
          Hi again,<br>
<br>
          here is more information regarding issue described<br>
          earlier<br>
<br>
          It looks like self healing is stuck. According to "heal<br>
          statistics" crawl began at Sat Jan 20 12:56:19 2018<br>
          and it's still<br>
          going on (It's around Sun Jan 21 20:30 when writing<br>
          this). However<br>
          glustershd.log says that last heal was completed at<br>
          "2018-01-20<br>
          11:00:13.090697" (which is 13:00 UTC+2). Also "heal<br>
          info" has been<br>
          running now for over 16 hours without any<br>
          information. In<br>
          statedump I can see that storage nodes have locks on<br>
          files and<br>
          some of those are blocked. Ie. Here again it says<br>
          that ovirt8z2 is<br>
          having active lock even ovirt8z2 crashed after the<br>
          lock was<br>
          granted.:<br>
<br>
          [xlator.features.locks.zone2-s<wbr>sd1-vmstor1-locks.inode]<br>
          path=/.shard/3d55f8cc-cda9-489<wbr>a-b0a3-fd0f43d67876.27<br>
          mandatory=0<br>
          inodelk-count=3<br>
          lock-dump.domain.domain=zone2-<wbr>ssd1-vmstor1-replicate-0:self-<wbr>heal<br>
          inodelk.inodelk[0](ACTIVE)=typ<wbr>e=WRITE, whence=0,<br>
          start=0, len=0,<br>
          pid = 18446744073709551610, owner=d0c6d857a87f0000,<br>
          client=0x7f885845efa0,<br>
<br>
<br>
      connection-id=sto2z2.xxx-10975<wbr>-2018/01/20-10:56:14:649541-<wbr>zone2-ssd1-vmstor1-client-0-0-<wbr>0,<br>
<br>
          granted at 2018-01-20 10:59:52<br>
          lock-dump.domain.domain=zone2-<wbr>ssd1-vmstor1-replicate-0:metad<wbr>ata<br>
          lock-dump.domain.domain=zone2-<wbr>ssd1-vmstor1-replicate-0<br>
          inodelk.inodelk[0](ACTIVE)=typ<wbr>e=WRITE, whence=0,<br>
          start=0, len=0,<br>
          pid = 3420, owner=d8b9372c397f0000,<br>
          client=0x7f8858410be0,<br></div>
          connection-id=<a href="http://ovirt8z2.xxx.com" rel="noreferrer" target="_blank">ovirt8z2.xxx.com</a> <<a href="http://ovirt8z2.xxx.com" rel="noreferrer" target="_blank">http://ovirt8z2.xxx.com</a>><div class="elided-text"><br>
<br>
        [1]-5652-2017/12/27-09:49:02:9<wbr>46825-zone2-ssd1-vmstor1-clien<wbr>t-0-7-0,<br>
<br>
          granted at 2018-01-20 08:57:23<br>
          inodelk.inodelk[1](BLOCKED)=ty<wbr>pe=WRITE, whence=0,<br>
          start=0, len=0,<br>
          pid = 18446744073709551610, owner=d0c6d857a87f0000,<br>
          client=0x7f885845efa0,<br>
<br>
<br>
      connection-id=sto2z2.xxx-10975<wbr>-2018/01/20-10:56:14:649541-<wbr>zone2-ssd1-vmstor1-client-0-0-<wbr>0,<br>
<br>
          blocked at 2018-01-20 10:59:52<br>
<br>
          I'd also like to add that volume had arbiter brick<br>
          before crash<br>
          happened. We decided to remove it because we thought<br>
          that it was<br>
          causing issues. However now I think that this was<br>
          unnecessary.<br>
          After the crash arbiter logs had lots of messages<br>
          like this:<br>
          [2018-01-20 10:19:36.515717] I [MSGID: 115072]<br>
          [server-rpc-fops.c:1640:server<wbr>_setattr_cbk]<br>
          0-zone2-ssd1-vmstor1-server: 37374187: SETATTR<br>
          <gfid:a52055bd-e2e9-42dd-92a3-<wbr>e96b693bcafe><br>
          (a52055bd-e2e9-42dd-92a3-e96b6<wbr>93bcafe) ==><br>
          (Operation not<br>
          permitted) [Operation not permitted]<br>
<br>
          Is there anyways to force self heal to stop? Any<br>
          help would be<br>
          very much appreciated :)<br>
<br>
          Best regards,<br>
          Samuli Heinonen<br>
<br>
          ______________________________<wbr>_________________<br>
          Gluster-users mailing list<br>
          <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br></div>
          <mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.<wbr>org</a>><br>
          <a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><div class="elided-text"><br>
          <<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mail<wbr>man/listinfo/gluster-users</a>><br>
          [3]<br>
<br>
          Samuli Heinonen <mailto:<a href="mailto:samppah@neutraali.net" target="_blank">samppah@neutraali.net</a><br>
          <mailto:<a href="mailto:samppah@neutraali.net" target="_blank">samppah@neutraali.net</a>><wbr>><br>
<br>
          20 January 2018 at 21.57<br>
          Hi all!<br>
<br>
          One hypervisor on our virtualization environment<br>
          crashed and now<br>
          some of the VM images cannot be accessed. After<br>
          investigation we<br>
          found out that there was lots of images that still<br>
          had active lock<br>
          on crashed hypervisor. We were able to remove locks<br>
          from "regular<br>
          files", but it doesn't seem possible to remove locks<br>
          from shards.<br>
<br>
          We are running GlusterFS 3.8.15 on all nodes.<br>
<br>
          Here is part of statedump that shows shard having<br>
          active lock on<br>
          crashed node:<br>
          [xlator.features.locks.zone2-s<wbr>sd1-vmstor1-locks.inode]<br>
          path=/.shard/75353c17-d6b8-485<wbr>d-9baf-fd6c700e39a1.21<br>
          mandatory=0<br>
          inodelk-count=1<br>
          lock-dump.domain.domain=zone2-<wbr>ssd1-vmstor1-replicate-0:metad<wbr>ata<br>
          lock-dump.domain.domain=zone2-<wbr>ssd1-vmstor1-replicate-0:self-<wbr>heal<br>
          lock-dump.domain.domain=zone2-<wbr>ssd1-vmstor1-replicate-0<br>
          inodelk.inodelk[0](ACTIVE)=typ<wbr>e=WRITE, whence=0,<br>
          start=0, len=0,<br>
          pid = 3568, owner=14ce372c397f0000,<br>
          client=0x7f3198388770,<br>
          connection-id<br>
<br>
<br>
      ovirt8z2.xxx-5652-2017/12/27-0<wbr>9:49:02:946825-zone2-ssd1-vmst<wbr>or1-client-1-7-0,<br>
<br>
          granted at 2018-01-20 08:57:24<br>
<br>
          If we try to run clear-locks we get following error<br>
          message:<br>
          # gluster volume clear-locks zone2-ssd1-vmstor1<br>
          /.shard/75353c17-d6b8-485d-9ba<wbr>f-fd6c700e39a1.21 kind<br>
          all inode<br>
          Volume clear-locks unsuccessful<br>
          clear-locks getxattr command failed. Reason:<br>
          Operation not<br>
          permitted<br>
<br>
          Gluster vol info if needed:<br>
          Volume Name: zone2-ssd1-vmstor1<br>
          Type: Replicate<br>
          Volume ID: b6319968-690b-4060-8fff-b212d2<wbr>295208<br>
          Status: Started<br>
          Snapshot Count: 0<br>
          Number of Bricks: 1 x 2 = 2<br>
          Transport-type: rdma<br>
          Bricks:<br>
          Brick1: sto1z2.xxx:/ssd1/zone2-vmstor1<wbr>/export<br>
          Brick2: sto2z2.xxx:/ssd1/zone2-vmstor1<wbr>/export<br>
          Options Reconfigured:<br>
          cluster.shd-wait-qlength: 10000<br>
          cluster.shd-max-threads: 8<br>
          cluster.locking-scheme: granular<br>
          performance.low-prio-threads: 32<br>
          cluster.data-self-heal-algorit<wbr>hm: full<br>
          performance.client-io-threads: off<br>
          storage.linux-aio: off<br>
          performance.readdir-ahead: on<br>
          client.event-threads: 16<br>
          server.event-threads: 16<br>
          performance.strict-write-order<wbr>ing: off<br>
          performance.quick-read: off<br>
          performance.read-ahead: on<br>
          performance.io-cache: off<br>
          performance.stat-prefetch: off<br>
          cluster.eager-lock: enable<br>
          network.remote-dio: on<br>
          cluster.quorum-type: none<br>
          network.ping-timeout: 22<br>
          performance.write-behind: off<br>
          nfs.disable: on<br>
          features.shard: on<br>
          features.shard-block-size: 512MB<br>
          storage.owner-uid: 36<br>
          storage.owner-gid: 36<br>
          performance.io-thread-count: 64<br>
          performance.cache-size: 2048MB<br>
          performance.write-behind-windo<wbr>w-size: 256MB<br>
          server.allow-insecure: on<br>
          cluster.ensure-durability: off<br>
          config.transport: rdma<br>
          server.outstanding-rpc-limit: 512<br>
          diagnostics.brick-log-level: INFO<br>
<br>
          Any recommendations how to advance from here?<br>
<br>
          Best regards,<br>
          Samuli Heinonen<br>
<br>
          ______________________________<wbr>_________________<br>
          Gluster-users mailing list<br>
          <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br></div>
          <mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.<wbr>org</a>><br>
          <a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><div class="quoted-text"><br>
          <<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mail<wbr>man/listinfo/gluster-users</a>><br>
          [3]<br>
<br>
<br>
      --<br>
<br>
      Pranith<br>
<br>
<br>
      Links:<br>
      ------<br>
      [1] <a href="http://ovirt8z2.xxx.com" rel="noreferrer" target="_blank">http://ovirt8z2.xxx.com</a><br>
      [2] <a href="http://review.gluster.org/14816" rel="noreferrer" target="_blank">http://review.gluster.org/1481<wbr>6</a><br>
      <<a href="http://review.gluster.org/14816" rel="noreferrer" target="_blank">http://review.gluster.org/148<wbr>16</a>><br>
      [3] <a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br></div>
      <<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mail<wbr>man/listinfo/gluster-users</a>><br>
<br>
<br>
<br>
<br>
  --<br>
  Pranith<br>
<br>
</blockquote>
</blockquote></div><br></div></div></div>