<div dir="ltr"><div dir="ltr">Hi Strahil<div><br></div><div>first of all thanks a million for your help -- really appreciate it.</div><div>Thanks also for the pointers on the debug. I have tried it, and while I can't interpret the results I think I might have found something.</div><div><br></div><div>There is a lot of information so hopefully this is relevant. During the snapshot creation and deletion, I can see the following errors in the client log:</div><div><br></div><div><font face="monospace">[2020-07-07 21:23:06.837381] W [MSGID: 122019] [ec-helpers.c:401:ec_loc_gfid_check] 0-SSD_Storage-disperse-0: Mismatching GFID's in loc <br>[2020-07-07 21:23:06.837387] D [MSGID: 0] [defaults.c:1328:default_mknod_cbk] 0-stack-trace: stack-address: 0x7f0dc0001a78, SSD_Storage-disperse-0 returned -1 error: Input/output error [Input/output error]<br>[2020-07-07 21:23:06.837392] W [MSGID: 109002] [dht-rename.c:1019:dht_rename_links_create_cbk] 0-SSD_Storage-dht: link/file /8d49207e-f6b9-41d1-8d35-f6e0fb121980/images/4802e66e-a7e3-42df-a570-7155135566ad/b51133ee-54e0-4001-ab4b-9f0dc1e5c6fc.meta on SSD_Storage-disperse-0 failed [Input/output error]<br>[2020-07-07 21:23:06.837850] D [MSGID: 0] [stack.h:502:copy_frame] 0-stack: groups is null (ngrps: 0) [Invalid argument]<br>[2020-07-07 21:23:06.839252] D [dict.c:1168:data_to_uint32] (-->/lib64/libglusterfs.so.0(dict_foreach_match+0x77) [0x7f0ddb1855e7] -->/usr/lib64/glusterfs/7.5/xlator/cluster/disperse.so(+0x384cf) [0x7f0dd23c54cf] -->/lib64/libglusterfs.so.0(data_to_uint32+0x8e) [0x7f0ddb184f2e] ) 0-dict: key null, unsigned integer type asked, has integer type [Invalid argument]<br>[2020-07-07 21:23:06.839272] D [MSGID: 0] [dht-common.c:6674:dht_readdirp_cbk] 0-SSD_Storage-dht: Processing entries from SSD_Storage-disperse-0 <br>[2020-07-07 21:23:06.839281] D [MSGID: 0] [dht-common.c:6681:dht_readdirp_cbk] 0-SSD_Storage-dht: SSD_Storage-disperse-0: entry = ., type = 4 <br>[2020-07-07 21:23:06.839291] D [MSGID: 0] [dht-common.c:6813:dht_readdirp_cbk] 0-SSD_Storage-dht: SSD_Storage-disperse-0: Adding entry = . <br>[2020-07-07 21:23:06.839297] D [MSGID: 0] [dht-common.c:6681:dht_readdirp_cbk] 0-SSD_Storage-dht: SSD_Storage-disperse-0: entry = .., type = 4 <br>[2020-07-07 21:23:06.839324] D [MSGID: 0] [client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x7f0dc0034598, SSD_Storage-client-6 returned -1 error: Stale file handle [Stale file handle]<br>[2020-07-07 21:23:06.839327] D [dict.c:1800:dict_get_int32] (-->/usr/lib64/glusterfs/7.5/xlator/cluster/disperse.so(+0x227d6) [0x7f0dd23af7d6] -->/usr/lib64/glusterfs/7.5/xlator/cluster/disperse.so(+0x17661) [0x7f0dd23a4661] -->/lib64/libglusterfs.so.0(dict_get_int32+0x107) [0x7f0ddb186437] ) 0-dict: key glusterfs.inodelk-count, integer type asked, has unsigned integer type [Invalid argument]<br>[2020-07-07 21:23:06.839361] D [MSGID: 0] [client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x7f0dc0034598, SSD_Storage-client-11 returned -1 error: Stale file handle [Stale file handle]<br>[2020-07-07 21:23:06.839395] D [MSGID: 0] [client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x7f0dc00395a8, SSD_Storage-client-15 returned -1 error: Stale file handle [Stale file handle]<br>[2020-07-07 21:23:06.839419] D [MSGID: 0] [client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x7f0dc0034598, SSD_Storage-client-9 returned -1 error: Stale file handle [Stale file handle]<br>[2020-07-07 21:23:06.839473] D [MSGID: 0] [client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x7f0dc009c108, SSD_Storage-client-18 returned -1 error: Stale file handle [Stale file handle]<br>[2020-07-07 21:23:06.839471] D [MSGID: 0] [client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x7f0dc0034598, SSD_Storage-client-10 returned -1 error: Stale file handle [Stale file handle]<br>[2020-07-07 21:23:06.839491] D [dict.c:1800:dict_get_int32] (-->/usr/lib64/glusterfs/7.5/xlator/cluster/disperse.so(+0x256ad) [0x7f0dd23b26ad] -->/usr/lib64/glusterfs/7.5/xlator/cluster/disperse.so(+0x17661) [0x7f0dd23a4661] -->/lib64/libglusterfs.so.0(dict_get_int32+0x107) [0x7f0ddb186437] ) 0-dict: key glusterfs.inodelk-count, integer type asked, has unsigned integer type [Invalid argument]<br>[2020-07-07 21:23:06.839512] D [MSGID: 0] [client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x7f0dc0034598, SSD_Storage-client-7 returned -1 error: Stale file handle [Stale file handle]<br>[2020-07-07 21:23:06.839526] D [MSGID: 0] [client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x7f0dc009c108, SSD_Storage-client-23 returned -1 error: Stale file handle [Stale file handle]<br>[2020-07-07 21:23:06.839543] D [MSGID: 0] [client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x7f0dc009c108, SSD_Storage-client-22 returned -1 error: Stale file handle [Stale file handle]<br>[2020-07-07 21:23:06.839543] D [MSGID: 0] [client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x7f0dc00395a8, SSD_Storage-client-16 returned -1 error: Stale file handle [Stale file handle]<br>[2020-07-07 21:23:06.839556] D [MSGID: 0] [client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x7f0dc009c108, SSD_Storage-client-21 returned -1 error: Stale file handle [Stale file handle]<br>[2020-07-07 21:23:06.839596] D [MSGID: 0] [client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x7f0dc00395a8, SSD_Storage-client-12 returned -1 error: Stale file handle [Stale file handle]<br>[2020-07-07 21:23:06.839617] D [MSGID: 0] [client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x7f0dc00395a8, SSD_Storage-client-14 returned -1 error: Stale file handle [Stale file handle]<br>[2020-07-07 21:23:06.839631] D [MSGID: 0] [client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x7f0dc00395a8, SSD_Storage-client-13 returned -1 error: Stale file handle [Stale file handle]<br>[2020-07-07 21:23:06.839636] D [MSGID: 0] [client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x7f0dc00395a8, SSD_Storage-client-17 returned -1 error: Stale file handle [Stale file handle]<br>[2020-07-07 21:23:06.839643] D [MSGID: 0] [client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x7f0dc0034598, SSD_Storage-client-8 returned -1 error: Stale file handle [Stale file handle]<br>[2020-07-07 21:23:06.839656] D [MSGID: 0] [defaults.c:1548:default_lookup_cbk] 0-stack-trace: stack-address: 0x7f0dc007c428, SSD_Storage-disperse-2 returned -1 error: Stale file handle [Stale file handle]<br>[2020-07-07 21:23:06.839665] D [MSGID: 0] [dht-common.c:998:dht_discover_cbk] 0-SSD_Storage-dht: lookup of (null) on SSD_Storage-disperse-2 returned error [Stale file handle]<br>[2020-07-07 21:23:06.839666] D [MSGID: 0] [defaults.c:1548:default_lookup_cbk] 0-stack-trace: stack-address: 0x7f0dc007c428, SSD_Storage-disperse-1 returned -1 error: Stale file handle [Stale file handle]<br>[2020-07-07 21:23:06.839683] D [MSGID: 0] [dht-common.c:998:dht_discover_cbk] 0-SSD_Storage-dht: lookup of (null) on SSD_Storage-disperse-1 returned error [Stale file handle]<br>[2020-07-07 21:23:06.839686] D [dict.c:1168:data_to_uint32] (-->/lib64/libglusterfs.so.0(dict_foreach_match+0x77) [0x7f0ddb1855e7] -->/usr/lib64/glusterfs/7.5/xlator/cluster/disperse.so(+0x384cf) [0x7f0dd23c54cf] -->/lib64/libglusterfs.so.0(data_to_uint32+0x8e) [0x7f0ddb184f2e] ) 0-dict: key null, unsigned integer type asked, has integer type [Invalid argument]<br>[2020-07-07 21:23:06.839698] D [MSGID: 0] [client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x7f0dc009c108, SSD_Storage-client-19 returned -1 error: Stale file handle [Stale file handle]<br>[2020-07-07 21:23:06.839703] D [MSGID: 0] [dht-common.c:6674:dht_readdirp_cbk] 0-SSD_Storage-dht: Processing entries from SSD_Storage-disperse-0 <br>[2020-07-07 21:23:06.839714] D [MSGID: 0] [dht-common.c:6681:dht_readdirp_cbk] 0-SSD_Storage-dht: SSD_Storage-disperse-0: entry = .., type = 4 <br>[2020-07-07 21:23:06.839716] D [MSGID: 0] [client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x7f0dc0024b48, SSD_Storage-client-30 returned -1 error: Stale file handle [Stale file handle]<br>[2020-07-07 21:23:06.839724] D [MSGID: 0] [client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x7f0dc0024b48, SSD_Storage-client-34 returned -1 error: Stale file handle [Stale file handle]<br>[2020-07-07 21:23:06.839720] D [MSGID: 0] [client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x7f0dc0024b48, SSD_Storage-client-35 returned -1 error: Stale file handle [Stale file handle]<br>[2020-07-07 21:23:06.839755] D [MSGID: 0] [client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x7f0dc0024b48, SSD_Storage-client-31 returned -1 error: Stale file handle [Stale file handle]<br>[2020-07-07 21:23:06.839759] D [MSGID: 0] [client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x7f0dc009c108, SSD_Storage-client-20 returned -1 error: Stale file handle [Stale file handle]<br>[2020-07-07 21:23:06.839774] D [MSGID: 0] [defaults.c:1548:default_lookup_cbk] 0-stack-trace: stack-address: 0x7f0dc007c428, SSD_Storage-disperse-3 returned -1 error: Stale file handle [Stale file handle]<br>[2020-07-07 21:23:06.839775] D [MSGID: 0] [client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x7f0dc0024b48, SSD_Storage-client-32 returned -1 error: Stale file handle [Stale file handle]<br>[2020-07-07 21:23:06.839783] D [MSGID: 0] [dht-common.c:998:dht_discover_cbk] 0-SSD_Storage-dht: lookup of (null) on SSD_Storage-disperse-3 returned error [Stale file handle]<br>[2020-07-07 21:23:06.839798] D [MSGID: 0] [dht-common.c:601:dht_discover_complete] 0-SSD_Storage-dht: key = trusted.glusterfs.quota.read-only not present in dict <br>[2020-07-07 21:23:06.839807] D [MSGID: 0] [client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace: stack-address: 0x7f0dc0024b48, SSD_Storage-client-33 returned -1 error: Stale file handle [Stale file handle]<br>[2020-07-07 21:23:06.839807] D [MSGID: 0] [dht-layout.c:789:dht_layout_preset] 0-SSD_Storage-dht: file = 00000000-0000-0000-0000-000000000000, subvol = SSD_Storage-disperse-4 <br>[2020-07-07 21:23:06.839825] D [MSGID: 0] [defaults.c:1548:default_lookup_cbk] 0-stack-trace: stack-address: 0x7f0dc007c428, SSD_Storage-disperse-5 returned -1 error: Stale file handle [Stale file handle]<br>[2020-07-07 21:23:06.839835] D [MSGID: 0] [dht-common.c:998:dht_discover_cbk] 0-SSD_Storage-dht: lookup of (null) on SSD_Storage-disperse-5 returned error [Stale file handle]</font><br></div><div><br></div><div>The above is logged just shortly before the qemu-kvm process crashes with the usual error:</div><div><br></div><div><font face="monospace">Unexpected error in raw_check_lock_bytes() at block/file-posix.c:811:<br>2020-07-07T21:23:06.847336Z qemu-kvm: Failed to get shared "write" lock</font><br></div><div><br></div><div>I have looked also on the bricks logs, but there is too much information there and will need to know what to look for.</div><div><br></div><div>Not sure if there is any benefit in looking into this any further?</div><div><br>Thanks,<br>Marco</div><div><br></div></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, 2 Jul 2020 at 15:45, Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com" target="_blank">hunter86_bg@yahoo.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
<br>
На 2 юли 2020 г. 16:33:51 GMT+03:00, Marco Fais <<a href="mailto:evilmf@gmail.com" target="_blank">evilmf@gmail.com</a>> написа:<br>
>Hi Strahil,<br>
><br>
>WARNING: As you enabled sharding - NEVER DISABLE SHARDING, EVER !<br>
>><br>
><br>
>Thanks -- good to be reminded :)<br>
><br>
><br>
>> >When you say they will not be optimal are you referring mainly to<br>
>> >performance considerations? We did plenty of testing, and in terms<br>
>of<br>
>> >performance didn't have issues even with I/O intensive workloads<br>
>(using<br>
>> >SSDs, I had issues with spinning disks).<br>
>><br>
>> Yes, the client side has to connect to 6 bricks (4+2) at a time and<br>
>> calculate the data in order to obtain the necessary information.Same<br>
>is<br>
>> valid for writing.<br>
>> If you need to conserve space, you can test VDO without compression<br>
>(of<br>
>> even with it).<br>
>><br>
><br>
>Understood -- will explore VDO. Storage usage efficiency is less<br>
>important<br>
>than fault tolerance or performance for us -- disperse volumes seemed<br>
>to<br>
>tick all the boxes so we looked at them primarily.<br>
>But clearly I had missed that they are not used as mainstream VM<br>
>storage<br>
>for oVirt (I did know they weren't supported, but as explained thought<br>
>was<br>
>more on the management side).<br>
><br>
><br>
>><br>
>> Also with replica volumes, you can use 'choose-local' /in case<br>
>you<br>
>> have faster than the network storage (like NVMe)/ and increase the<br>
>read<br>
>> speed. Of course this feature is useful for Hyperconverged setup<br>
>(gluster<br>
>> + ovirt on the same node).<br>
>><br>
><br>
>Will explore this option as well, thanks for the suggestion.<br>
><br>
><br>
>> If you were using ovirt 4.3 , I would recommend you to focus on<br>
>> gluster. Yet, you use oVirt 4.4 which is quite newer and it needs<br>
> some<br>
>> polishing.<br>
>><br>
><br>
>Ovirt 4.3.9 (using the older Centos 7 qemu/libvirt) unfortunately had<br>
>similar issues with the disperse volumes. Not sure if exactly the same,<br>
>as<br>
>never looked deeper into it, but the results were similar.<br>
>Ovirt 4.4.0 has some issues with snapshot deletion that are independent<br>
>from Gluster (I have raised the issue here,<br>
><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1840414" rel="noreferrer" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1840414</a>, should be sorted<br>
>with<br>
>4.4.2 I guess), so at the moment it only works with the "testing" AV<br>
>repo.<br>
<br>
<br>
<br>
In such case I can recommend you to:<br>
1. Ensure you have enough space on all bricks for the logs (/var/log/gluster). Several gigs should be OK<br>
2. Enable all logs to 'TRACE' . Red Hat's documentation on the topic is quite good:<br>
<a href="https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3/html/administration_guide/configuring_the_log_level" rel="noreferrer" target="_blank">https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3/html/administration_guide/configuring_the_log_level</a><br>
3. Reproduce the issue on a fresh VM (never done snapshot deletion)<br>
4. Disable (switch to info) all logs as per the link in point 2<br>
<br>
The logs will be spread among all nodes. If you have remote logging available, you can also use it for analysis of the logs.<br>
<br>
Most probably the brick logs can provide useful information.<br>
<br>
<br>
><br>
>> Check ovirt engine logs (on the HostedEngine VM or your standalone<br>
>> engine) , vdsm logs on the host that was running the VM and next -<br>
>check<br>
>> the brick logs.<br>
>><br>
><br>
>Will do.<br>
><br>
>Thanks,<br>
>Marco<br>
<br>
<br>
About VDO - it might require some tuning and even afterwards it won't be very performant, so it depends on your needs.<br>
<br>
Best Regards,<br>
Strahil Nikolov<br>
</blockquote></div></div>