<span style=" font-size:12pt;font-family:Arial">I upgraded late last week
to 4.1.2. Since then I've seen several posix health checks fail and bricks
drop offline but I'm not sure if that's related or a different root issue.</span>
<br>
<br><span style=" font-size:12pt;font-family:Arial">I haven't seen the
issue described below re-occur on 4.1.2 yet but it was intermittent to
begin with so I'll probably need to run for a week or more to be confident.</span>
<br>
<br><span style=" font-size:12pt;font-family:Arial">-Walter Deignan<br>
-Uline IT, Systems Architect</span>
<br>
<br>
<br>
<br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">From:
</span><span style=" font-size:9pt;font-family:sans-serif">"Claus
Jeppesen" <cjeppesen@datto.com></span>
<br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">To:
</span><span style=" font-size:9pt;font-family:sans-serif">WDeignan@uline.com</span>
<br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Cc:
</span><span style=" font-size:9pt;font-family:sans-serif">gluster-users@gluster.org</span>
<br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Date:
</span><span style=" font-size:9pt;font-family:sans-serif">08/20/2018
07:20 AM</span>
<br><span style=" font-size:9pt;color:#5f5f5f;font-family:sans-serif">Subject:
</span><span style=" font-size:9pt;font-family:sans-serif">Re:
[Gluster-users] KVM lockups on Gluster 4.1.1</span>
<br>
<hr noshade>
<br>
<br>
<br><span style=" font-size:12pt">I think I have seen this also on our
CentOS 7.5 systems using GlusterFS 4.1.1 (*) - has an upgrade to 4.1.2
helped out ? I'm trying this now.</span>
<br>
<br><span style=" font-size:12pt">Thanx,</span>
<br>
<br><span style=" font-size:12pt">Claus.</span>
<br>
<br><span style=" font-size:12pt">(*) libvirt/quemu log:</span>
<br><tt><span style=" font-size:12pt">[2018-08-19 16:45:54.275830] E [MSGID:
114031] [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk] 0-glu-vol01-lab-client-0:
remote operation failed [Invalid argument] <br>
[2018-08-19 16:45:54.276156] E [MSGID: 114031] [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk]
0-glu-vol01-lab-client-1: remote operation failed [Invalid argument] <br>
[2018-08-19 16:45:54.276159] E [MSGID: 108010] [afr-lk-common.c:284:afr_unlock_inodelk_cbk]
0-glu-vol01-lab-replicate-0: path=(null) gfid=00000000-0000-0000-0000-000000000000:
unlock failed on subvolume glu-vol<br>
01-lab-client-0 with lock owner 28ae497049560000 [Invalid argument] <br>
[2018-08-19 16:45:54.276183] E [MSGID: 108010] [afr-lk-common.c:284:afr_unlock_inodelk_cbk]
0-glu-vol01-lab-replicate-0: path=(null) gfid=00000000-0000-0000-0000-000000000000:
unlock failed on subvolume glu-vol<br>
01-lab-client-1 with lock owner 28ae497049560000 [Invalid argument] <br>
[2018-08-19 17:16:03.690808] E [rpc-clnt.c:184:call_bail] 0-glu-vol01-lab-client-0:
bailing out frame type(GlusterFS 4.x v1) op(FINODELK(30)) xid = 0x3071a5
sent = 2018-08-19 16:45:54.276560. timeout = 1800 for</span></tt><tt><span style=" font-size:12pt;color:blue"><u><br>
</u></span></tt><a href=http://192.168.13.131:49152/><tt><span style=" font-size:12pt;color:blue"><u>192.168.13.131:49152</u></span></tt></a><tt><span style=" font-size:12pt">
<br>
[2018-08-19 17:16:03.691113] E [MSGID: 114031] [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk]
0-glu-vol01-lab-client-0: remote operation failed [Transport endpoint is
not connected] <br>
[2018-08-19 17:46:03.855909] E [rpc-clnt.c:184:call_bail] 0-glu-vol01-lab-client-1:
bailing out frame type(GlusterFS 4.x v1) op(FINODELK(30)) xid = 0x301d0f
sent = 2018-08-19 17:16:03.691174. timeout = 1800 for</span></tt><tt><span style=" font-size:12pt;color:blue"><u><br>
</u></span></tt><a href=http://192.168.13.132:49152/><tt><span style=" font-size:12pt;color:blue"><u>192.168.13.132:49152</u></span></tt></a><tt><span style=" font-size:12pt">
<br>
[2018-08-19 17:46:03.856170] E [MSGID: 114031] [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk]
0-glu-vol01-lab-client-1: remote operation failed [Transport endpoint is
not connected] <br>
block I/O error in device 'drive-virtio-disk0': Operation not permitted
(1) <br>
... many repeats ... <br>
block I/O error in device 'drive-virtio-disk0': Operation not permitted
(1) <br>
[2018-08-19 18:16:04.022526] E [rpc-clnt.c:184:call_bail] 0-glu-vol01-lab-client-0:
bailing out frame type(GlusterFS 4.x v1) op(FINODELK(30)) xid = 0x307221
sent = 2018-08-19 17:46:03.861005. timeout = 1800 for</span></tt><tt><span style=" font-size:12pt;color:blue"><u><br>
</u></span></tt><a href=http://192.168.13.131:49152/><tt><span style=" font-size:12pt;color:blue"><u>192.168.13.131:49152</u></span></tt></a><tt><span style=" font-size:12pt">
<br>
[2018-08-19 18:16:04.022788] E [MSGID: 114031] [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk]
0-glu-vol01-lab-client-0: remote operation failed [Transport endpoint is
not connected] <br>
[2018-08-19 18:46:04.195590] E [rpc-clnt.c:184:call_bail] 0-glu-vol01-lab-client-1:
bailing out frame type(GlusterFS 4.x v1) op(FINODELK(30)) xid = 0x301d8a
sent = 2018-08-19 18:16:04.022838. timeout = 1800 for</span></tt><tt><span style=" font-size:12pt;color:blue"><u><br>
</u></span></tt><a href=http://192.168.13.132:49152/><tt><span style=" font-size:12pt;color:blue"><u>192.168.13.132:49152</u></span></tt></a><tt><span style=" font-size:12pt">
<br>
[2018-08-19 18:46:04.195881] E [MSGID: 114031] [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk]
0-glu-vol01-lab-client-1: remote operation failed [Transport endpoint is
not connected] <br>
block I/O error in device 'drive-virtio-disk0': Operation not permitted
(1) <br>
block I/O error in device 'drive-virtio-disk0': Operation not permitted
(1) <br>
block I/O error in device 'drive-virtio-disk0': Operation not permitted
(1) <br>
block I/O error in device 'drive-virtio-disk0': Operation not permitted
(1) <br>
block I/O error in device 'drive-virtio-disk0': Operation not permitted
(1) <br>
qemu: terminating on signal 15 from pid 507 <br>
2018-08-19 19:36:59.065+0000: shutting down, reason=destroyed <br>
2018-08-19 19:37:08.059+0000: starting up libvirt version: 3.9.0, package:
14.el7_5.6 (CentOS BuildSystem <</span></tt><a href=http://bugs.centos.org/><tt><span style=" font-size:12pt;color:blue"><u>http://bugs.centos.org</u></span></tt></a><tt><span style=" font-size:12pt">>,
2018-06-27-14:13:57, </span></tt><a href="http://x86-01.bsys.centos.org/"><tt><span style=" font-size:12pt;color:blue"><u>x86-01.bsys.centos.org</u></span></tt></a><tt><span style=" font-size:12pt">),
qemu version: 1.5.3 (qemu-kvm-1.<br>
5.3-156.el7_5.3)</span></tt>
<br>
<br><tt><span style=" font-size:12pt">At 19:37 the VM was restarted.</span></tt>
<br>
<br>
<br>
<br><span style=" font-size:12pt">On Wed, Aug 15, 2018 at 8:25 PM Walter
Deignan <</span><a href=mailto:WDeignan@uline.com><span style=" font-size:12pt;color:blue"><u>WDeignan@uline.com</u></span></a><span style=" font-size:12pt">>
wrote:</span>
<br><span style=" font-size:12pt;font-family:Arial">I am using gluster
to host KVM/QEMU images. I am seeing an intermittent issue where access
to an image will hang. I have to do a lazy dismount of the gluster volume
in order to break the lock and then reset the impacted virtual machine.</span><span style=" font-size:12pt">
<br>
</span><span style=" font-size:12pt;font-family:Arial"><br>
It happened again today and I caught the events below in the client side
logs. Any thoughts on what might cause this? It seemed to begin after I
upgraded from 3.12.10 to 4.1.1 a few weeks ago.</span><span style=" font-size:12pt">
<br>
</span><span style=" font-size:12pt;font-family:Arial"><br>
[2018-08-14 14:22:15.549501] E [MSGID: 114031] [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk]
2-gv1-client-4: remote operation failed [Invalid argument]</span><span style=" font-size:12pt">
</span><span style=" font-size:12pt;font-family:Arial"><br>
[2018-08-14 14:22:15.549576] E [MSGID: 114031] [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk]
2-gv1-client-5: remote operation failed [Invalid argument]</span><span style=" font-size:12pt">
</span><span style=" font-size:12pt;font-family:Arial"><br>
[2018-08-14 14:22:15.549583] E [MSGID: 108010] [afr-lk-common.c:284:afr_unlock_inodelk_cbk]
2-gv1-replicate-2: path=(null) gfid=00000000-0000-0000-0000-000000000000:
unlock failed on subvolume gv1-client-4 with lock owner d89caca92b7f0000
[Invalid argument]</span><span style=" font-size:12pt"> </span><span style=" font-size:12pt;font-family:Arial"><br>
[2018-08-14 14:22:15.549615] E [MSGID: 108010] [afr-lk-common.c:284:afr_unlock_inodelk_cbk]
2-gv1-replicate-2: path=(null) gfid=00000000-0000-0000-0000-000000000000:
unlock failed on subvolume gv1-client-5 with lock owner d89caca92b7f0000
[Invalid argument]</span><span style=" font-size:12pt"> </span><span style=" font-size:12pt;font-family:Arial"><br>
[2018-08-14 14:52:18.726219] E [rpc-clnt.c:184:call_bail] 2-gv1-client-4:
bailing out frame type(GlusterFS 4.x v1) op(FINODELK(30)) xid = 0xc5e00
sent = 2018-08-14 14:22:15.699082. timeout = 1800 for </span><a href=http://10.35.20.106:49159/ target=_blank><span style=" font-size:12pt;color:blue;font-family:Arial"><u>10.35.20.106:49159</u></span></a><span style=" font-size:12pt">
</span><span style=" font-size:12pt;font-family:Arial"><br>
[2018-08-14 14:52:18.726254] E [MSGID: 114031] [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk]
2-gv1-client-4: remote operation failed [Transport endpoint is not connected]</span><span style=" font-size:12pt">
</span><span style=" font-size:12pt;font-family:Arial"><br>
[2018-08-14 15:22:25.962546] E [rpc-clnt.c:184:call_bail] 2-gv1-client-5:
bailing out frame type(GlusterFS 4.x v1) op(FINODELK(30)) xid = 0xc4a6d
sent = 2018-08-14 14:52:18.726329. timeout = 1800 for </span><a href=http://10.35.20.107:49164/ target=_blank><span style=" font-size:12pt;color:blue;font-family:Arial"><u>10.35.20.107:49164</u></span></a><span style=" font-size:12pt">
</span><span style=" font-size:12pt;font-family:Arial"><br>
[2018-08-14 15:22:25.962587] E [MSGID: 114031] [client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk]
2-gv1-client-5: remote operation failed [Transport endpoint is not connected]</span><span style=" font-size:12pt">
</span><span style=" font-size:12pt;font-family:Arial"><br>
[2018-08-14 15:22:25.962618] W [MSGID: 108019] [afr-lk-common.c:601:is_blocking_locks_count_sufficient]
2-gv1-replicate-2: Unable to obtain blocking inode lock on even one child
for gfid:24a48cae-53fe-4634-8fb7-0254c85ad672.</span><span style=" font-size:12pt">
</span><span style=" font-size:12pt;font-family:Arial"><br>
[2018-08-14 15:22:25.962668] W [fuse-bridge.c:1441:fuse_err_cbk] 0-glusterfs-fuse:
3715808: FSYNC() ERR => -1 (Transport endpoint is not connected)</span><span style=" font-size:12pt">
<br>
</span><span style=" font-size:12pt;font-family:Arial"><br>
Volume configuration -</span><span style=" font-size:12pt"> <br>
</span><span style=" font-size:12pt;font-family:Arial"><br>
Volume Name: gv1</span><span style=" font-size:12pt"> </span><span style=" font-size:12pt;font-family:Arial"><br>
Type: Distributed-Replicate</span><span style=" font-size:12pt"> </span><span style=" font-size:12pt;font-family:Arial"><br>
Volume ID: 66ad703e-3bae-4e79-a0b7-29ea38e8fcfc</span><span style=" font-size:12pt">
</span><span style=" font-size:12pt;font-family:Arial"><br>
Status: Started</span><span style=" font-size:12pt"> </span><span style=" font-size:12pt;font-family:Arial"><br>
Snapshot Count: 0</span><span style=" font-size:12pt"> </span><span style=" font-size:12pt;font-family:Arial"><br>
Number of Bricks: 5 x 2 = 10</span><span style=" font-size:12pt"> </span><span style=" font-size:12pt;font-family:Arial"><br>
Transport-type: tcp</span><span style=" font-size:12pt"> </span><span style=" font-size:12pt;font-family:Arial"><br>
Bricks:</span><span style=" font-size:12pt"> </span><span style=" font-size:12pt;font-family:Arial"><br>
Brick1: dc-vihi44:/gluster/bricks/megabrick/data</span><span style=" font-size:12pt">
</span><span style=" font-size:12pt;font-family:Arial"><br>
Brick2: dc-vihi45:/gluster/bricks/megabrick/data</span><span style=" font-size:12pt">
</span><span style=" font-size:12pt;font-family:Arial"><br>
Brick3: dc-vihi44:/gluster/bricks/brick1/data</span><span style=" font-size:12pt">
</span><span style=" font-size:12pt;font-family:Arial"><br>
Brick4: dc-vihi45:/gluster/bricks/brick1/data</span><span style=" font-size:12pt">
</span><span style=" font-size:12pt;font-family:Arial"><br>
Brick5: dc-vihi44:/gluster/bricks/brick2_1/data</span><span style=" font-size:12pt">
</span><span style=" font-size:12pt;font-family:Arial"><br>
Brick6: dc-vihi45:/gluster/bricks/brick2/data</span><span style=" font-size:12pt">
</span><span style=" font-size:12pt;font-family:Arial"><br>
Brick7: dc-vihi44:/gluster/bricks/brick3/data</span><span style=" font-size:12pt">
</span><span style=" font-size:12pt;font-family:Arial"><br>
Brick8: dc-vihi45:/gluster/bricks/brick3/data</span><span style=" font-size:12pt">
</span><span style=" font-size:12pt;font-family:Arial"><br>
Brick9: dc-vihi44:/gluster/bricks/brick4/data</span><span style=" font-size:12pt">
</span><span style=" font-size:12pt;font-family:Arial"><br>
Brick10: dc-vihi45:/gluster/bricks/brick4/data</span><span style=" font-size:12pt">
</span><span style=" font-size:12pt;font-family:Arial"><br>
Options Reconfigured:</span><span style=" font-size:12pt"> </span><span style=" font-size:12pt;font-family:Arial"><br>
cluster.min-free-inodes: 6%</span><span style=" font-size:12pt"> </span><span style=" font-size:12pt;font-family:Arial"><br>
performance.client-io-threads: off</span><span style=" font-size:12pt">
</span><span style=" font-size:12pt;font-family:Arial"><br>
nfs.disable: on</span><span style=" font-size:12pt"> </span><span style=" font-size:12pt;font-family:Arial"><br>
transport.address-family: inet</span><span style=" font-size:12pt"> </span><span style=" font-size:12pt;font-family:Arial"><br>
performance.quick-read: off</span><span style=" font-size:12pt"> </span><span style=" font-size:12pt;font-family:Arial"><br>
performance.read-ahead: off</span><span style=" font-size:12pt"> </span><span style=" font-size:12pt;font-family:Arial"><br>
performance.io-cache: off</span><span style=" font-size:12pt"> </span><span style=" font-size:12pt;font-family:Arial"><br>
performance.low-prio-threads: 32</span><span style=" font-size:12pt"> </span><span style=" font-size:12pt;font-family:Arial"><br>
network.remote-dio: enable</span><span style=" font-size:12pt"> </span><span style=" font-size:12pt;font-family:Arial"><br>
cluster.eager-lock: enable</span><span style=" font-size:12pt"> </span><span style=" font-size:12pt;font-family:Arial"><br>
cluster.server-quorum-type: server</span><span style=" font-size:12pt">
</span><span style=" font-size:12pt;font-family:Arial"><br>
cluster.data-self-heal-algorithm: full</span><span style=" font-size:12pt">
</span><span style=" font-size:12pt;font-family:Arial"><br>
cluster.locking-scheme: granular</span><span style=" font-size:12pt"> </span><span style=" font-size:12pt;font-family:Arial"><br>
cluster.shd-max-threads: 8</span><span style=" font-size:12pt"> </span><span style=" font-size:12pt;font-family:Arial"><br>
cluster.shd-wait-qlength: 10000</span><span style=" font-size:12pt"> </span><span style=" font-size:12pt;font-family:Arial"><br>
user.cifs: off</span><span style=" font-size:12pt"> </span><span style=" font-size:12pt;font-family:Arial"><br>
cluster.choose-local: off</span><span style=" font-size:12pt"> </span><span style=" font-size:12pt;font-family:Arial"><br>
features.shard: on</span><span style=" font-size:12pt"> </span><span style=" font-size:12pt;font-family:Arial"><br>
cluster.server-quorum-ratio: 51%</span><span style=" font-size:12pt"> <br>
</span><span style=" font-size:12pt;font-family:Arial"><br>
-Walter Deignan<br>
-Uline IT, Systems Architect</span><span style=" font-size:12pt">_______________________________________________<br>
Gluster-users mailing list</span><span style=" font-size:12pt;color:blue"><u><br>
</u></span><a href="mailto:Gluster-users@gluster.org" target=_blank><span style=" font-size:12pt;color:blue"><u>Gluster-users@gluster.org</u></span></a><span style=" font-size:12pt;color:blue"><u><br>
</u></span><a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target=_blank><span style=" font-size:12pt;color:blue"><u>https://lists.gluster.org/mailman/listinfo/gluster-users</u></span></a>
<br>
<br>
<br><span style=" font-size:12pt">-- </span>
<table width=650 style="border-collapse:collapse;">
<tr height=8>
<td width=650 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:11pt;color:#2f2f2f;font-family:Arial"><b>Claus
Jeppesen</b></span>
<tr height=8>
<td width=650 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:11pt;color:#2f2f2f;font-family:Arial">Manager,
Network Services</span>
<tr height=8>
<td width=650 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:11pt;color:#2f2f2f;font-family:Arial">Datto,
Inc.</span>
<tr height=8>
<td width=650 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:11pt;color:#2f2f2f;font-family:Arial">p
+45 6170 5901 | Copenhagen Office</span>
<tr height=8>
<td width=650 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><a href=http://www.datto.com/ target=_blank><span style=" font-size:11pt;color:#0060a0;font-family:Arial"><u>www.datto.com</u></span></a>
<tr height=8>
<td width=650 style="border-style:none none none none;border-color:#000000;border-width:0px 0px 0px 0px;padding:0px 0px;"><span style=" font-size:11pt;color:#0082bf;font-family:Arial"><u><br>
</u></span><a href="http://www.datto.com/datto-signature/" target=_blank><img src="http://pages.datto.com/rs/572-ZRG-001/images/generic-sig.png" width=400 height=160 style="border:0px solid;"></a></table>
<p style="margin-top:0px;margin-Bottom:0px"></p>
<p style="margin-top:0px;margin-Bottom:0px"></p>
<p style="margin-top:0px;margin-Bottom:0px"></p>