<div dir="ltr">Thanks for this report! We will look into this. This is something new we are seeing, and not aware of a RCA yet!<div><br></div><div>-Amar</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Aug 20, 2018 at 1:08 PM, Claus Jeppesen <span dir="ltr"><<a href="mailto:cjeppesen@datto.com" target="_blank">cjeppesen@datto.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I think I have seen this also on our CentOS 7.5 systems using GlusterFS 4.1.1 (*) - has an upgrade to 4.1.2 helped out ? I'm trying this now.<div><br></div><div>Thanx,</div><div><br></div><div>Claus.<div><br><div>(*) libvirt/quemu log:</div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">[2018-08-19 16:45:54.275830] E [MSGID: 114031] [client-rpc-fops_v2.c:1352:<wbr>client4_0_finodelk_cbk] 0-glu-vol01-lab-client-0: remote operation failed [Invalid argument]
</span><br>[2018-08-19 16:45:54.276156] E [MSGID: 114031] [client-rpc-fops_v2.c:1352:<wbr>client4_0_finodelk_cbk] 0-glu-vol01-lab-client-1: remote operation failed [Invalid argument]
<br>[2018-08-19 16:45:54.276159] E [MSGID: 108010] [afr-lk-common.c:284:afr_<wbr>unlock_inodelk_cbk] 0-glu-vol01-lab-replicate-0: path=(null) gfid=00000000-0000-0000-0000-<wbr>000000000000: unlock failed on subvolume glu-vol<br>01-lab-client-0 with lock owner 28ae497049560000 [Invalid argument]
<br>[2018-08-19 16:45:54.276183] E [MSGID: 108010] [afr-lk-common.c:284:afr_<wbr>unlock_inodelk_cbk] 0-glu-vol01-lab-replicate-0: path=(null) gfid=00000000-0000-0000-0000-<wbr>000000000000: unlock failed on subvolume glu-vol<br>01-lab-client-1 with lock owner 28ae497049560000 [Invalid argument]
<br>[2018-08-19 17:16:03.690808] E [rpc-clnt.c:184:call_bail] 0-glu-vol01-lab-client-0: bailing out frame type(GlusterFS 4.x v1) op(FINODELK(30)) xid = 0x3071a5 sent = 2018-08-19 16:45:54.276560. timeout = 1800 for<br> <a href="http://192.168.13.131:49152" target="_blank">192.168.13.131:49152</a>
<br>[2018-08-19 17:16:03.691113] E [MSGID: 114031] [client-rpc-fops_v2.c:1352:<wbr>client4_0_finodelk_cbk] 0-glu-vol01-lab-client-0: remote operation failed [Transport endpoint is not connected]
<br>[2018-08-19 17:46:03.855909] E [rpc-clnt.c:184:call_bail] 0-glu-vol01-lab-client-1: bailing out frame type(GlusterFS 4.x v1) op(FINODELK(30)) xid = 0x301d0f sent = 2018-08-19 17:16:03.691174. timeout = 1800 for<br> <a href="http://192.168.13.132:49152" target="_blank">192.168.13.132:49152</a>
<br>[2018-08-19 17:46:03.856170] E [MSGID: 114031] [client-rpc-fops_v2.c:1352:<wbr>client4_0_finodelk_cbk] 0-glu-vol01-lab-client-1: remote operation failed [Transport endpoint is not connected]
<br>block I/O error in device 'drive-virtio-disk0': Operation not permitted (1)
<br>... many repeats ... <br>block I/O error in device 'drive-virtio-disk0': Operation not permitted (1)
<br>[2018-08-19 18:16:04.022526] E [rpc-clnt.c:184:call_bail] 0-glu-vol01-lab-client-0: bailing out frame type(GlusterFS 4.x v1) op(FINODELK(30)) xid = 0x307221 sent = 2018-08-19 17:46:03.861005. timeout = 1800 for<br> <a href="http://192.168.13.131:49152" target="_blank">192.168.13.131:49152</a>
<br>[2018-08-19 18:16:04.022788] E [MSGID: 114031] [client-rpc-fops_v2.c:1352:<wbr>client4_0_finodelk_cbk] 0-glu-vol01-lab-client-0: remote operation failed [Transport endpoint is not connected]
<br>[2018-08-19 18:46:04.195590] E [rpc-clnt.c:184:call_bail] 0-glu-vol01-lab-client-1: bailing out frame type(GlusterFS 4.x v1) op(FINODELK(30)) xid = 0x301d8a sent = 2018-08-19 18:16:04.022838. timeout = 1800 for<br> <a href="http://192.168.13.132:49152" target="_blank">192.168.13.132:49152</a>
<br>[2018-08-19 18:46:04.195881] E [MSGID: 114031] [client-rpc-fops_v2.c:1352:<wbr>client4_0_finodelk_cbk] 0-glu-vol01-lab-client-1: remote operation failed [Transport endpoint is not connected]
<br>block I/O error in device 'drive-virtio-disk0': Operation not permitted (1)
<br>block I/O error in device 'drive-virtio-disk0': Operation not permitted (1)
<br>block I/O error in device 'drive-virtio-disk0': Operation not permitted (1)
<br>block I/O error in device 'drive-virtio-disk0': Operation not permitted (1)
<br>block I/O error in device 'drive-virtio-disk0': Operation not permitted (1)
<br>qemu: terminating on signal 15 from pid 507
<br>2018-08-19 19:36:59.065+0000: shutting down, reason=destroyed
<br>2018-08-19 19:37:08.059+0000: starting up libvirt version: 3.9.0, package: 14.el7_5.6 (CentOS BuildSystem <<a href="http://bugs.centos.org" target="_blank">http://bugs.centos.org</a>>, 2018-06-27-14:13:57, <a href="http://x86-01.bsys.centos.org" target="_blank">x86-01.bsys.centos.org</a>), qemu version: 1.5.3 (qemu-kvm-1.<br>5.3-156.el7_5.3)<br></span></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace">At </span><font face="monospace">19:37 the VM was restarted.</font></div><div><font face="monospace"><br></font></div><div><font face="monospace"><br></font></div></div></div></div><br><div class="gmail_quote"><div><div class="h5"><div dir="ltr">On Wed, Aug 15, 2018 at 8:25 PM Walter Deignan <<a href="mailto:WDeignan@uline.com" target="_blank">WDeignan@uline.com</a>> wrote:<br></div></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5"><span style="font-size:12pt;font-family:Arial">I am using gluster to
host KVM/QEMU images. I am seeing an intermittent issue where access to
an image will hang. I have to do a lazy dismount of the gluster volume
in order to break the lock and then reset the impacted virtual machine.</span>
<br>
<br><span style="font-size:12pt;font-family:Arial">It happened again today
and I caught the events below in the client side logs. Any thoughts on
what might cause this? It seemed to begin after I upgraded from 3.12.10
to 4.1.1 a few weeks ago.</span>
<br>
<br><span style="font-size:12pt;font-family:Arial">[2018-08-14 14:22:15.549501]
E [MSGID: 114031] [client-rpc-fops_v2.c:1352:<wbr>client4_0_finodelk_cbk] 2-gv1-client-4:
remote operation failed [Invalid argument]</span>
<br><span style="font-size:12pt;font-family:Arial">[2018-08-14 14:22:15.549576]
E [MSGID: 114031] [client-rpc-fops_v2.c:1352:<wbr>client4_0_finodelk_cbk] 2-gv1-client-5:
remote operation failed [Invalid argument]</span>
<br><span style="font-size:12pt;font-family:Arial">[2018-08-14 14:22:15.549583]
E [MSGID: 108010] [afr-lk-common.c:284:afr_<wbr>unlock_inodelk_cbk] 2-gv1-replicate-2:
path=(null) gfid=00000000-0000-0000-0000-<wbr>000000000000: unlock failed on
subvolume gv1-client-4 with lock owner d89caca92b7f0000 [Invalid argument]</span>
<br><span style="font-size:12pt;font-family:Arial">[2018-08-14 14:22:15.549615]
E [MSGID: 108010] [afr-lk-common.c:284:afr_<wbr>unlock_inodelk_cbk] 2-gv1-replicate-2:
path=(null) gfid=00000000-0000-0000-0000-<wbr>000000000000: unlock failed on
subvolume gv1-client-5 with lock owner d89caca92b7f0000 [Invalid argument]</span>
<br><span style="font-size:12pt;font-family:Arial">[2018-08-14 14:52:18.726219]
E [rpc-clnt.c:184:call_bail] 2-gv1-client-4: bailing out frame type(GlusterFS
4.x v1) op(FINODELK(30)) xid = 0xc5e00 sent = 2018-08-14 14:22:15.699082.
timeout = 1800 for <a href="http://10.35.20.106:49159" target="_blank">10.35.20.106:49159</a></span>
<br><span style="font-size:12pt;font-family:Arial">[2018-08-14 14:52:18.726254]
E [MSGID: 114031] [client-rpc-fops_v2.c:1352:<wbr>client4_0_finodelk_cbk] 2-gv1-client-4:
remote operation failed [Transport endpoint is not connected]</span>
<br><span style="font-size:12pt;font-family:Arial">[2018-08-14 15:22:25.962546]
E [rpc-clnt.c:184:call_bail] 2-gv1-client-5: bailing out frame type(GlusterFS
4.x v1) op(FINODELK(30)) xid = 0xc4a6d sent = 2018-08-14 14:52:18.726329.
timeout = 1800 for <a href="http://10.35.20.107:49164" target="_blank">10.35.20.107:49164</a></span>
<br><span style="font-size:12pt;font-family:Arial">[2018-08-14 15:22:25.962587]
E [MSGID: 114031] [client-rpc-fops_v2.c:1352:<wbr>client4_0_finodelk_cbk] 2-gv1-client-5:
remote operation failed [Transport endpoint is not connected]</span>
<br><span style="font-size:12pt;font-family:Arial">[2018-08-14 15:22:25.962618]
W [MSGID: 108019] [afr-lk-common.c:601:is_<wbr>blocking_locks_count_<wbr>sufficient]
2-gv1-replicate-2: Unable to obtain blocking inode lock on even one child
for gfid:24a48cae-53fe-4634-8fb7-<wbr>0254c85ad672.</span>
<br><span style="font-size:12pt;font-family:Arial">[2018-08-14 15:22:25.962668]
W [fuse-bridge.c:1441:fuse_err_<wbr>cbk] 0-glusterfs-fuse: 3715808: FSYNC()
ERR => -1 (Transport endpoint is not connected)</span>
<br>
<br><span style="font-size:12pt;font-family:Arial">Volume configuration
-</span>
<br>
<br><span style="font-size:12pt;font-family:Arial">Volume Name: gv1</span>
<br><span style="font-size:12pt;font-family:Arial">Type: Distributed-Replicate</span>
<br><span style="font-size:12pt;font-family:Arial">Volume ID: 66ad703e-3bae-4e79-a0b7-<wbr>29ea38e8fcfc</span>
<br><span style="font-size:12pt;font-family:Arial">Status: Started</span>
<br><span style="font-size:12pt;font-family:Arial">Snapshot Count: 0</span>
<br><span style="font-size:12pt;font-family:Arial">Number of Bricks: 5
x 2 = 10</span>
<br><span style="font-size:12pt;font-family:Arial">Transport-type: tcp</span>
<br><span style="font-size:12pt;font-family:Arial">Bricks:</span>
<br><span style="font-size:12pt;font-family:Arial">Brick1: dc-vihi44:/gluster/bricks/<wbr>megabrick/data</span>
<br><span style="font-size:12pt;font-family:Arial">Brick2: dc-vihi45:/gluster/bricks/<wbr>megabrick/data</span>
<br><span style="font-size:12pt;font-family:Arial">Brick3: dc-vihi44:/gluster/bricks/<wbr>brick1/data</span>
<br><span style="font-size:12pt;font-family:Arial">Brick4: dc-vihi45:/gluster/bricks/<wbr>brick1/data</span>
<br><span style="font-size:12pt;font-family:Arial">Brick5: dc-vihi44:/gluster/bricks/<wbr>brick2_1/data</span>
<br><span style="font-size:12pt;font-family:Arial">Brick6: dc-vihi45:/gluster/bricks/<wbr>brick2/data</span>
<br><span style="font-size:12pt;font-family:Arial">Brick7: dc-vihi44:/gluster/bricks/<wbr>brick3/data</span>
<br><span style="font-size:12pt;font-family:Arial">Brick8: dc-vihi45:/gluster/bricks/<wbr>brick3/data</span>
<br><span style="font-size:12pt;font-family:Arial">Brick9: dc-vihi44:/gluster/bricks/<wbr>brick4/data</span>
<br><span style="font-size:12pt;font-family:Arial">Brick10: dc-vihi45:/gluster/bricks/<wbr>brick4/data</span>
<br><span style="font-size:12pt;font-family:Arial">Options Reconfigured:</span>
<br><span style="font-size:12pt;font-family:Arial">cluster.min-free-inodes:
6%</span>
<br><span style="font-size:12pt;font-family:Arial">performance.client-io-threads:
off</span>
<br><span style="font-size:12pt;font-family:Arial">nfs.disable: on</span>
<br><span style="font-size:12pt;font-family:Arial">transport.address-family:
inet</span>
<br><span style="font-size:12pt;font-family:Arial">performance.quick-read:
off</span>
<br><span style="font-size:12pt;font-family:Arial">performance.read-ahead:
off</span>
<br><span style="font-size:12pt;font-family:Arial">performance.io-cache:
off</span>
<br><span style="font-size:12pt;font-family:Arial">performance.low-prio-threads:
32</span>
<br><span style="font-size:12pt;font-family:Arial">network.remote-dio:
enable</span>
<br><span style="font-size:12pt;font-family:Arial">cluster.eager-lock:
enable</span>
<br><span style="font-size:12pt;font-family:Arial">cluster.server-quorum-type:
server</span>
<br><span style="font-size:12pt;font-family:Arial">cluster.data-self-heal-<wbr>algorithm:
full</span>
<br><span style="font-size:12pt;font-family:Arial">cluster.locking-scheme:
granular</span>
<br><span style="font-size:12pt;font-family:Arial">cluster.shd-max-threads:
8</span>
<br><span style="font-size:12pt;font-family:Arial">cluster.shd-wait-qlength:
10000</span>
<br><span style="font-size:12pt;font-family:Arial">user.cifs: off</span>
<br><span style="font-size:12pt;font-family:Arial">cluster.choose-local:
off</span>
<br><span style="font-size:12pt;font-family:Arial">features.shard: on</span>
<br><span style="font-size:12pt;font-family:Arial">cluster.server-quorum-ratio:
51%</span>
<br>
<br></div></div><span style="font-size:12pt;font-family:Arial">-Walter Deignan<br>
-Uline IT, Systems Architect</span>_____________________<wbr>__________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a></blockquote></div><span class="HOEnZb"><font color="#888888"><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_-6264156063395691700gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><table border="0" cellpadding="0" cellspacing="0" width="650" style="color:rgb(51,51,51);font-family:DattoDIN,Arial,sans-serif;font-size:14px"><tbody><tr><td height="17" width="270"><b>Claus Jeppesen</b></td></tr><tr><td height="16">Manager, Network Services<br></td></tr><tr><td height="16">Datto, Inc.</td></tr><tr><td height="16">p +45 6170 5901 | Copenhagen Office<br></td></tr><tr><td height="16"><a href="http://www.datto.com/" style="color:rgb(0,128,198)" target="_blank"><span style="color:rgb(11,83,148)">www.datto.com</span></a></td></tr><tr><td height="16"><br><a href="http://www.datto.com/datto-signature/" style="color:rgb(0,128,198)" target="_blank"><img alt="" src="http://pages.datto.com/rs/572-ZRG-001/images/generic-sig.png" style="margin:0px;padding:0px;border:0px"></a></td></tr></tbody></table></div></div>
</font></span><br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Amar Tumballi (amarts)<br></div></div></div></div></div>
</div>