<html><head></head><body><div class="ydpcb96de78yahoo-style-wrap" style="font-family: courier new, courier, monaco, monospace, sans-serif; font-size: 16px;"><div></div>
<div dir="ltr" data-setdir="false">Hi Jiffin,</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">I'm observing the same behaviour like last time .</div><div dir="ltr" data-setdir="false">This time after upgrading from gluster v 7.0 to 7.2 as follows:</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><div><div>[2020-01-30 23:42:26.039822] T [MSGID: 0] [posix.c:2994:pl_lookup] 0-stack-trace: stack-address: 0x7f7b50003958, winding from data_fast-locks to data_f</div><div>ast-access-control </div><div>[2020-01-30 23:42:26.039837] I [MSGID: 139001] [posix-acl.c:262:posix_acl_log_permit_denied] 0-data_fast-access-control: client: CTX_ID:868c1fba-1f38-4</div><div>e91-a664-6e49fddb3f27-GRAPH_ID:0-PID:9525-HOST:ovirt2.localdomain-PC_NAME:data_fast-client-1-RECON_NO:-1, gfid: be318638-e8a0-4c6d-977d-7a937aa84806, r</div><div>eq(uid:107,gid:107,perm:1,ngrps:4), ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-) [Permission denied]</div><div>[2020-01-30 23:42:26.039857] D [MSGID: 0] [posix-acl.c:1055:posix_acl_lookup] 0-stack-trace: stack-address: 0x7f7b50003958, data_fast-access-control re</div><div>turned -1 error: Permission denied [Permission denied]</div><div>[2020-01-30 23:42:26.039868] D [MSGID: 0] [posix.c:2985:pl_lookup_cbk] 0-stack-trace: stack-address: 0x7f7b50003958, data_fast-locks returned -1 error:</div><div> Permission denied [Permission denied]</div><div>[2020-01-30 23:42:26.039879] D [MSGID: 0] [upcall.c:756:up_lookup_cbk] 0-stack-trace: stack-address: 0x7f7b50003958, data_fast-upcall returned -1 error</div><div>: Permission denied [Permission denied]</div><div>[2020-01-30 23:42:26.039889] D [MSGID: 0] [defaults.c:1349:default_lookup_cbk] 0-stack-trace: stack-address: 0x7f7b50003958, data_fast-io-threads retur</div><div>ned -1 error: Permission denied [Permission denied]</div><div>[2020-01-30 23:42:26.039901] T [marker.c:2907:marker_lookup_cbk] 0-data_fast-marker: lookup failed with Permission denied</div><div>[2020-01-30 23:42:26.039908] D [MSGID: 0] [marker.c:2944:marker_lookup_cbk] 0-stack-trace: stack-address: 0x7f7b50003958, data_fast-marker returned -1 </div><div>error: Permission denied [Permission denied]</div><div>[2020-01-30 23:42:26.039922] D [MSGID: 0] [index.c:2070:index_lookup_cbk] 0-stack-trace: stack-address: 0x7f7b50003958, data_fast-index returned -1 err</div><div>or: Permission denied [Permission denied]</div><div>[2020-01-30 23:42:26.039933] D [MSGID: 0] [io-stats.c:2228:io_stats_lookup_cbk] 0-stack-trace: stack-address: 0x7f7b50003958, /gluster_bricks/data_fast</div><div>/data_fast returned -1 error: Permission denied [Permission denied]</div><div>[2020-01-30 23:42:26.039949] E [MSGID: 115050] [server-rpc-fops_v2.c:157:server4_lookup_cbk] 0-data_fast-server: 16268: LOOKUP /.shard/f476698a-d8d2-4a</div><div>b2-b9c4-4c276c2eef43.79 (be318638-e8a0-4c6d-977d-7a937aa84806/f476698a-d8d2-4ab2-b9c4-4c276c2eef43.79), client: CTX_ID:868c1fba-1f38-4e91-a664-6e49fddb</div><div>3f27-GRAPH_ID:0-PID:9525-HOST:ovirt2.localdomain-PC_NAME:data_fast-client-1-RECON_NO:-1, error-xlator: data_fast-access-control [Permission denied]</div><div>[2020-01-30 23:42:26.039964] T [rpcsvc.c:1529:rpcsvc_submit_generic] 0-rpc-service: Tx message: 344</div><div>[2020-01-30 23:42:26.039971] T [rpcsvc.c:1065:rpcsvc_record_build_header] 0-rpc-service: Reply fraglen 368, payload: 344, rpc hdr: 24</div><div>[2020-01-30 23:42:26.040007] T [rpcsvc.c:1581:rpcsvc_submit_generic] 0-rpc-service: submitted reply for rpc-message (XID: 0x24c1, Program: GlusterFS 4.</div><div>x v1, ProgVers: 400, Proc: 27) to rpc-transport (tcp.data_fast-server)</div></div><br></div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">Are you sure it's fixed?</div><div dir="ltr" data-setdir="false">If yes, then this looks like a regresion.</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">By the way, I'm not using ACL - how can I permanently disable it ,so even if ACL gets corrupted -> my oVirt will survive ?</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">Thanks in advance for your answer.</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">Best Regards,</div><div dir="ltr" data-setdir="false">Strahil Nikolov</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><br></div><div><br></div>
</div><div id="ydpb224a27yahoo_quoted_0694770118" class="ydpb224a27yahoo_quoted">
<div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;">
<div>
В сряда, 4 декември 2019 г., 13:28:39 ч. Гринуич+2, Jiffin Thottan <jthottan@redhat.com> написа:
</div>
<div><br></div>
<div><br></div>
<div><div dir="ltr">Hi Krutika,<br></div><div dir="ltr"><br></div><div dir="ltr">Apparently, in context acl info got corrupted see brick logs<br></div><div dir="ltr"><br></div><div dir="ltr">[posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control:<br></div><div dir="ltr">client:<br></div><div dir="ltr">CTX_ID:dae9ffad-6acd-4a43-9372-229a3018fde9-GRAPH_ID:0-PID:11468-HOST:ovirt2.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0,<br></div><div dir="ltr">gfid: be318638-e8a0-4c6d-977d-7a937aa84806,<br></div><div dir="ltr">req(uid:107,gid:107,perm:1,ngrps:4),<br></div><div dir="ltr">ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-)<br></div><div dir="ltr">[Permission denied]<br></div><div dir="ltr"><br></div><div dir="ltr">which resulted in the situation. There was one bug similar was reported <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1668286 " rel="nofollow" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1668286 </a>and<br></div><div dir="ltr"><br></div><div dir="ltr">it got fixed in 6.6 release IMO <a href="https://review.gluster.org/#/c/glusterfs/+/23233/. " rel="nofollow" target="_blank">https://review.gluster.org/#/c/glusterfs/+/23233/. </a>But here he mentioned he saw the issue when<br></div><div dir="ltr"><br></div><div dir="ltr">he upgraded from 6.5 to 6.6<br></div><div dir="ltr"><br></div><div dir="ltr">One way to workaround is to perform a dummy setfacl(preferably using root) on the corrupted files which will forcefully fetch the acl<br></div><div dir="ltr"><br></div><div dir="ltr">info again from backend and update the context. Another approach to restart brick process(kill and vol start force)<br></div><div dir="ltr"><br></div><div dir="ltr">Regards,<br></div><div dir="ltr">Jiffin<br></div><div dir="ltr"><br></div><div dir="ltr">----- Original Message -----<br></div><div dir="ltr">From: "Krutika Dhananjay" <<a href="mailto:kdhananj@redhat.com" rel="nofollow" target="_blank">kdhananj@redhat.com</a>><br></div><div dir="ltr">To: "Strahil Nikolov" <<a href="mailto:hunter86_bg@yahoo.com" rel="nofollow" target="_blank">hunter86_bg@yahoo.com</a>>, "Jiffin Thottan" <<a href="mailto:jthottan@redhat.com" rel="nofollow" target="_blank">jthottan@redhat.com</a>>, "raghavendra talur" <<a href="mailto:rtalur@redhat.com" rel="nofollow" target="_blank">rtalur@redhat.com</a>><br></div><div dir="ltr">Cc: "Nir Soffer" <<a href="mailto:nsoffer@redhat.com" rel="nofollow" target="_blank">nsoffer@redhat.com</a>>, "Rafi Kavungal Chundattu Parambil" <<a href="mailto:rkavunga@redhat.com" rel="nofollow" target="_blank">rkavunga@redhat.com</a>>, "users" <<a href="mailto:users@ovirt.org" rel="nofollow" target="_blank">users@ovirt.org</a>>, "gluster-user" <<a href="mailto:gluster-users@gluster.org" rel="nofollow" target="_blank">gluster-users@gluster.org</a>><br></div><div dir="ltr">Sent: Monday, December 2, 2019 11:48:22 AM<br></div><div dir="ltr">Subject: Re: [ovirt-users] Re: [ANN] oVirt 4.3.7 Third Release Candidate is now available for testing<br></div><div dir="ltr"><br></div><div dir="ltr">Sorry about the late response.<br></div><div dir="ltr"><br></div><div dir="ltr">I looked at the logs. These errors are originating from posix-acl<br></div><div dir="ltr">translator -<br></div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr">*[2019-11-17 07:55:47.090065] E [MSGID: 115050]<br></div><div dir="ltr">[server-rpc-fops_v2.c:158:server4_lookup_cbk] 0-data_fast-server: 162496:<br></div><div dir="ltr">LOOKUP /.shard/5985adcb-0f4d-4317-8a26-1652973a2350.6<br></div><div dir="ltr">(be318638-e8a0-4c6d-977d-7a937aa84806/5985adcb-0f4d-4317-8a26-1652973a2350.6),<br></div><div dir="ltr">client:<br></div><div dir="ltr">CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0,<br></div><div dir="ltr">error-xlator: data_fast-access-control [Permission denied][2019-11-17<br></div><div dir="ltr">07:55:47.090174] I [MSGID: 139001]<br></div><div dir="ltr">[posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control:<br></div><div dir="ltr">client:<br></div><div dir="ltr">CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0,<br></div><div dir="ltr">gfid: be318638-e8a0-4c6d-977d-7a937aa84806,<br></div><div dir="ltr">req(uid:36,gid:36,perm:1,ngrps:3),<br></div><div dir="ltr">ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-)<br></div><div dir="ltr">[Permission denied][2019-11-17 07:55:47.090209] E [MSGID: 115050]<br></div><div dir="ltr">[server-rpc-fops_v2.c:158:server4_lookup_cbk] 0-data_fast-server: 162497:<br></div><div dir="ltr">LOOKUP /.shard/5985adcb-0f4d-4317-8a26-1652973a2350.7<br></div><div dir="ltr">(be318638-e8a0-4c6d-977d-7a937aa84806/5985adcb-0f4d-4317-8a26-1652973a2350.7),<br></div><div dir="ltr">client:<br></div><div dir="ltr">CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0,<br></div><div dir="ltr">error-xlator: data_fast-access-control [Permission denied][2019-11-17<br></div><div dir="ltr">07:55:47.090299] I [MSGID: 139001]<br></div><div dir="ltr">[posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control:<br></div><div dir="ltr">client:<br></div><div dir="ltr">CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0,<br></div><div dir="ltr">gfid: be318638-e8a0-4c6d-977d-7a937aa84806,<br></div><div dir="ltr">req(uid:36,gid:36,perm:1,ngrps:3),<br></div><div dir="ltr">ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-)<br></div><div dir="ltr">[Permission denied]*<br></div><div dir="ltr"><br></div><div dir="ltr">Jiffin/Raghavendra Talur,<br></div><div dir="ltr">Can you help?<br></div><div dir="ltr"><br></div><div dir="ltr">-Krutika<br></div><div dir="ltr"><br></div><div dir="ltr">On Wed, Nov 27, 2019 at 2:11 PM Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com" rel="nofollow" target="_blank">hunter86_bg@yahoo.com</a>><br></div><div dir="ltr">wrote:<br></div><div dir="ltr"><br></div><div dir="ltr">> Hi Nir,All,<br></div><div dir="ltr">><br></div><div dir="ltr">> it seems that 4.3.7 RC3 (and even RC4) are not the problem here(attached<br></div><div dir="ltr">> screenshot of oVirt running on v7 gluster).<br></div><div dir="ltr">> It seems strange that both my serious issues with oVirt are related to<br></div><div dir="ltr">> gluster issue (1st gluster v3 to v5 migration and now this one).<br></div><div dir="ltr">><br></div><div dir="ltr">> I have just updated to gluster v7.0 (Centos 7 repos), and rebooted all<br></div><div dir="ltr">> nodes.<br></div><div dir="ltr">> Now both Engine and all my VMs are back online - so if you hit issues with<br></div><div dir="ltr">> 6.6 , you should give a try to 7.0 (and even 7.1 is coming soon) before<br></div><div dir="ltr">> deciding to wipe everything.<br></div><div dir="ltr">><br></div><div dir="ltr">> @Krutika,<br></div><div dir="ltr">><br></div><div dir="ltr">> I guess you will ask for the logs, so let's switch to gluster-users about<br></div><div dir="ltr">> this one ?<br></div><div dir="ltr">><br></div><div dir="ltr">> Best Regards,<br></div><div dir="ltr">> Strahil Nikolov<br></div><div dir="ltr">><br></div><div dir="ltr">> В понеделник, 25 ноември 2019 г., 16:45:48 ч. Гринуич-5, Strahil Nikolov <<br></div><div dir="ltr">> <a href="mailto:hunter86_bg@yahoo.com" rel="nofollow" target="_blank">hunter86_bg@yahoo.com</a>> написа:<br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">> Hi Krutika,<br></div><div dir="ltr">><br></div><div dir="ltr">> I have enabled TRACE log level for the volume data_fast,<br></div><div dir="ltr">><br></div><div dir="ltr">> but the issue is not much clear:<br></div><div dir="ltr">> FUSE reports:<br></div><div dir="ltr">><br></div><div dir="ltr">> [2019-11-25 21:31:53.478130] I [MSGID: 133022]<br></div><div dir="ltr">> [shard.c:3674:shard_delete_shards] 0-data_fast-shard: Deleted shards of<br></div><div dir="ltr">> gfid=6d9ed2e5-d4f2-4749-839b-2f1<br></div><div dir="ltr">> 3a68ed472 from backend<br></div><div dir="ltr">> [2019-11-25 21:32:43.564694] W [MSGID: 114031]<br></div><div dir="ltr">> [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:<br></div><div dir="ltr">> remote operation failed. Path:<br></div><div dir="ltr">> /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79<br></div><div dir="ltr">> (00000000-0000-0000-0000-000000000000) [Permission denied]<br></div><div dir="ltr">> [2019-11-25 21:32:43.565653] W [MSGID: 114031]<br></div><div dir="ltr">> [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:<br></div><div dir="ltr">> remote operation failed. Path:<br></div><div dir="ltr">> /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79<br></div><div dir="ltr">> (00000000-0000-0000-0000-000000000000) [Permission denied]<br></div><div dir="ltr">> [2019-11-25 21:32:43.565689] W [MSGID: 114031]<br></div><div dir="ltr">> [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:<br></div><div dir="ltr">> remote operation failed. Path:<br></div><div dir="ltr">> /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79<br></div><div dir="ltr">> (00000000-0000-0000-0000-000000000000) [Permission denied]<br></div><div dir="ltr">> [2019-11-25 21:32:43.565770] E [MSGID: 133010]<br></div><div dir="ltr">> [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on<br></div><div dir="ltr">> shard 79 failed. Base file gfid = b0af2b81-22cf-482e-9b2f-c431b6449dae<br></div><div dir="ltr">> [Permission denied]<br></div><div dir="ltr">> [2019-11-25 21:32:43.565858] W [fuse-bridge.c:2830:fuse_readv_cbk]<br></div><div dir="ltr">> 0-glusterfs-fuse: 279: READ => -1 gfid=b0af2b81-22cf-482e-9b2f-c431b6449dae<br></div><div dir="ltr">> fd=0x7fbf40005ea8 (Permission denied)<br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">> While the BRICK logs on ovirt1/gluster1 report:<br></div><div dir="ltr">> 2019-11-25 21:32:43.564177] D [MSGID: 0] [io-threads.c:376:iot_schedule]<br></div><div dir="ltr">> 0-data_fast-io-threads: LOOKUP scheduled as fast priority fop<br></div><div dir="ltr">> [2019-11-25 21:32:43.564194] T [MSGID: 0]<br></div><div dir="ltr">> [defaults.c:2008:default_lookup_resume] 0-stack-trace: stack-address:<br></div><div dir="ltr">> 0x7fc02c00bbf8, winding from data_fast-io-threads to data_fast-upcall<br></div><div dir="ltr">> [2019-11-25 21:32:43.564206] T [MSGID: 0] [upcall.c:790:up_lookup]<br></div><div dir="ltr">> 0-stack-trace: stack-address: 0x7fc02c00bbf8, winding from data_fast-upcall<br></div><div dir="ltr">> to data_fast-leases<br></div><div dir="ltr">> [2019-11-25 21:32:43.564215] T [MSGID: 0] [defaults.c:2766:default_lookup]<br></div><div dir="ltr">> 0-stack-trace: stack-address: 0x7fc02c00bbf8, winding from data_fast-leases<br></div><div dir="ltr">> to data_fast-read-only<br></div><div dir="ltr">> [2019-11-25 21:32:43.564222] T [MSGID: 0] [defaults.c:2766:default_lookup]<br></div><div dir="ltr">> 0-stack-trace: stack-address: 0x7fc02c00bbf8, winding from<br></div><div dir="ltr">> data_fast-read-only to data_fast-worm<br></div><div dir="ltr">> [2019-11-25 21:32:43.564230] T [MSGID: 0] [defaults.c:2766:default_lookup]<br></div><div dir="ltr">> 0-stack-trace: stack-address: 0x7fc02c00bbf8, winding from data_fast-worm<br></div><div dir="ltr">> to data_fast-locks<br></div><div dir="ltr">> [2019-11-25 21:32:43.564241] T [MSGID: 0] [posix.c:2897:pl_lookup]<br></div><div dir="ltr">> 0-stack-trace: stack-address: 0x7fc02c00bbf8, winding from data_fast-locks<br></div><div dir="ltr">> to data_fast-access-control<br></div><div dir="ltr">> [2019-11-25 21:32:43.564254] I [MSGID: 139001]<br></div><div dir="ltr">> [posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control:<br></div><div dir="ltr">> client:<br></div><div dir="ltr">> CTX_ID:dae9ffad-6acd-4a43-9372-229a3018fde9-GRAPH_ID:0-PID:11468-HOST:ovirt2.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0,<br></div><div dir="ltr">> gfid: be318638-e8a0-4c6d-977d-7a937aa84806,<br></div><div dir="ltr">> req(uid:107,gid:107,perm:1,ngrps:4),<br></div><div dir="ltr">> ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-)<br></div><div dir="ltr">> [Permission denied]<br></div><div dir="ltr">> [2019-11-25 21:32:43.564268] D [MSGID: 0]<br></div><div dir="ltr">> [posix-acl.c:1057:posix_acl_lookup] 0-stack-trace: stack-address:<br></div><div dir="ltr">> 0x7fc02c00bbf8, data_fast-access-control returned -1 error: Permission<br></div><div dir="ltr">> denied [Permission denied]<br></div><div dir="ltr">> [2019-11-25 21:32:43.564279] D [MSGID: 0] [posix.c:2888:pl_lookup_cbk]<br></div><div dir="ltr">> 0-stack-trace: stack-address: 0x7fc02c00bbf8, data_fast-locks returned -1<br></div><div dir="ltr">> error: Permission denied [Permission denied]<br></div><div dir="ltr">> [2019-11-25 21:32:43.564289] D [MSGID: 0] [upcall.c:769:up_lookup_cbk]<br></div><div dir="ltr">> 0-stack-trace: stack-address: 0x7fc02c00bbf8, data_fast-upcall returned -1<br></div><div dir="ltr">> error: Permission denied [Permission denied]<br></div><div dir="ltr">> [2019-11-25 21:32:43.564302] D [MSGID: 0]<br></div><div dir="ltr">> [defaults.c:1349:default_lookup_cbk] 0-stack-trace: stack-address:<br></div><div dir="ltr">> 0x7fc02c00bbf8, data_fast-io-threads returned -1 error: Permission denied<br></div><div dir="ltr">> [Permission denied]<br></div><div dir="ltr">> [2019-11-25 21:32:43.564313] T [marker.c:2918:marker_lookup_cbk]<br></div><div dir="ltr">> 0-data_fast-marker: lookup failed with Permission denied<br></div><div dir="ltr">> [2019-11-25 21:32:43.564320] D [MSGID: 0]<br></div><div dir="ltr">> [marker.c:2955:marker_lookup_cbk] 0-stack-trace: stack-address:<br></div><div dir="ltr">> 0x7fc02c00bbf8, data_fast-marker returned -1 error: Permission denied<br></div><div dir="ltr">> [Permission denied]<br></div><div dir="ltr">> [2019-11-25 21:32:43.564334] D [MSGID: 0] [index.c:2070:index_lookup_cbk]<br></div><div dir="ltr">> 0-stack-trace: stack-address: 0x7fc02c00bbf8, data_fast-index returned -1<br></div><div dir="ltr">> error: Permission denied [Permission denied]<br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">> В понеделник, 25 ноември 2019 г., 23:10:41 ч. Гринуич+2, Strahil Nikolov <<br></div><div dir="ltr">> <a href="mailto:hunter86_bg@yahoo.com" rel="nofollow" target="_blank">hunter86_bg@yahoo.com</a>> написа:<br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">> Hi Krutika,<br></div><div dir="ltr">><br></div><div dir="ltr">> thanks for your assistance.<br></div><div dir="ltr">> Let me summarize some info about the volume:<br></div><div dir="ltr">><br></div><div dir="ltr">> Volume Name: data_fast<br></div><div dir="ltr">> Type: Replicate<br></div><div dir="ltr">> Volume ID: 378804bf-2975-44d8-84c2-b541aa87f9ef<br></div><div dir="ltr">> Status: Started<br></div><div dir="ltr">> Snapshot Count: 0<br></div><div dir="ltr">> Number of Bricks: 1 x (2 + 1) = 3<br></div><div dir="ltr">> Transport-type: tcp<br></div><div dir="ltr">> Bricks:<br></div><div dir="ltr">> Brick1: gluster1:/gluster_bricks/data_fast/data_fast<br></div><div dir="ltr">> Brick2: gluster2:/gluster_bricks/data_fast/data_fast<br></div><div dir="ltr">> Brick3: ovirt3:/gluster_bricks/data_fast/data_fast (arbiter)<br></div><div dir="ltr">> Options Reconfigured:<br></div><div dir="ltr">> performance.client-io-threads: on<br></div><div dir="ltr">> nfs.disable: on<br></div><div dir="ltr">> transport.address-family: inet<br></div><div dir="ltr">> performance.quick-read: off<br></div><div dir="ltr">> performance.read-ahead: off<br></div><div dir="ltr">> performance.io-cache: off<br></div><div dir="ltr">> performance.low-prio-threads: 32<br></div><div dir="ltr">> network.remote-dio: on<br></div><div dir="ltr">> cluster.eager-lock: enable<br></div><div dir="ltr">> cluster.quorum-type: auto<br></div><div dir="ltr">> cluster.server-quorum-type: server<br></div><div dir="ltr">> cluster.data-self-heal-algorithm: full<br></div><div dir="ltr">> cluster.locking-scheme: granular<br></div><div dir="ltr">> cluster.shd-max-threads: 8<br></div><div dir="ltr">> cluster.shd-wait-qlength: 10000<br></div><div dir="ltr">> features.shard: on<br></div><div dir="ltr">> user.cifs: off<br></div><div dir="ltr">> cluster.choose-local: on<br></div><div dir="ltr">> client.event-threads: 4<br></div><div dir="ltr">> server.event-threads: 4<br></div><div dir="ltr">> storage.owner-uid: 36<br></div><div dir="ltr">> storage.owner-gid: 36<br></div><div dir="ltr">> performance.strict-o-direct: on<br></div><div dir="ltr">> network.ping-timeout: 30<br></div><div dir="ltr">> cluster.granular-entry-heal: enable<br></div><div dir="ltr">> cluster.enable-shared-storage: enable<br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">> [<a href="mailto:root@ovirt1" rel="nofollow" target="_blank">root@ovirt1</a> ~]# gluster volume get engine all | grep shard<br></div><div dir="ltr">> features.shard on<br></div><div dir="ltr">> features.shard-block-size 64MB<br></div><div dir="ltr">> features.shard-lru-limit 16384<br></div><div dir="ltr">> features.shard-deletion-rate 100<br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">> On Sat, Nov 23, 2019 at 3:14 AM Nir Soffer <<a href="mailto:nsoffer@redhat.com" rel="nofollow" target="_blank">nsoffer@redhat.com</a>> wrote:<br></div><div dir="ltr">><br></div><div dir="ltr">> On Fri, Nov 22, 2019 at 10:41 PM Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com" rel="nofollow" target="_blank">hunter86_bg@yahoo.com</a>><br></div><div dir="ltr">> wrote:<br></div><div dir="ltr">><br></div><div dir="ltr">> On Thu, Nov 21, 2019 at 8:20 AM Sahina Bose <<a href="mailto:sabose@redhat.com" rel="nofollow" target="_blank">sabose@redhat.com</a>> wrote:<br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">> On Thu, Nov 21, 2019 at 6:03 AM Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com" rel="nofollow" target="_blank">hunter86_bg@yahoo.com</a>><br></div><div dir="ltr">> wrote:<br></div><div dir="ltr">><br></div><div dir="ltr">> Hi All,<br></div><div dir="ltr">><br></div><div dir="ltr">> another clue in the logs :<br></div><div dir="ltr">> [2019-11-21 00:29:50.536631] W [MSGID: 114031]<br></div><div dir="ltr">> [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:<br></div><div dir="ltr">> remote operation failed. Path:<br></div><div dir="ltr">> /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79<br></div><div dir="ltr">> (00000000-0000-0000-0000-000000000000) [Permission denied]<br></div><div dir="ltr">> [2019-11-21 00:29:50.536798] W [MSGID: 114031]<br></div><div dir="ltr">> [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:<br></div><div dir="ltr">> remote operation failed. Path:<br></div><div dir="ltr">> /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79<br></div><div dir="ltr">> (00000000-0000-0000-0000-000000000000) [Permission denied]<br></div><div dir="ltr">> [2019-11-21 00:29:50.536959] W [MSGID: 114031]<br></div><div dir="ltr">> [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:<br></div><div dir="ltr">> remote operation failed. Path:<br></div><div dir="ltr">> /.shard/b0af2b81-22cf-482e-9b2f-c431b6449dae.79<br></div><div dir="ltr">> (00000000-0000-0000-0000-000000000000) [Permission denied]<br></div><div dir="ltr">> [2019-11-21 00:29:50.537007] E [MSGID: 133010]<br></div><div dir="ltr">> [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on<br></div><div dir="ltr">> shard 79 failed. Base file gfid = b0af2b81-22cf-482e-9b2f-c431b6449dae<br></div><div dir="ltr">> [Permission denied]<br></div><div dir="ltr">> [2019-11-21 00:29:50.537066] W [fuse-bridge.c:2830:fuse_readv_cbk]<br></div><div dir="ltr">> 0-glusterfs-fuse: 12458: READ => -1<br></div><div dir="ltr">> gfid=b0af2b81-22cf-482e-9b2f-c431b6449dae fd=0x7fc63c00fe18 (Permission<br></div><div dir="ltr">> denied)<br></div><div dir="ltr">> [2019-11-21 00:30:01.177665] I [MSGID: 133022]<br></div><div dir="ltr">> [shard.c:3674:shard_delete_shards] 0-data_fast-shard: Deleted shards of<br></div><div dir="ltr">> gfid=eb103fbf-80dc-425d-882f-1e4efe510db5 from backend<br></div><div dir="ltr">> [2019-11-21 00:30:13.132756] W [MSGID: 114031]<br></div><div dir="ltr">> [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:<br></div><div dir="ltr">> remote operation failed. Path:<br></div><div dir="ltr">> /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79<br></div><div dir="ltr">> (00000000-0000-0000-0000-000000000000) [Permission denied]<br></div><div dir="ltr">> [2019-11-21 00:30:13.132824] W [MSGID: 114031]<br></div><div dir="ltr">> [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:<br></div><div dir="ltr">> remote operation failed. Path:<br></div><div dir="ltr">> /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79<br></div><div dir="ltr">> (00000000-0000-0000-0000-000000000000) [Permission denied]<br></div><div dir="ltr">> [2019-11-21 00:30:13.133217] W [MSGID: 114031]<br></div><div dir="ltr">> [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:<br></div><div dir="ltr">> remote operation failed. Path:<br></div><div dir="ltr">> /.shard/17c663c2-f582-455b-b806-3b9d01fb2c6c.79<br></div><div dir="ltr">> (00000000-0000-0000-0000-000000000000) [Permission denied]<br></div><div dir="ltr">> [2019-11-21 00:30:13.133238] E [MSGID: 133010]<br></div><div dir="ltr">> [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on<br></div><div dir="ltr">> shard 79 failed. Base file gfid = 17c663c2-f582-455b-b806-3b9d01fb2c6c<br></div><div dir="ltr">> [Permission denied]<br></div><div dir="ltr">> [2019-11-21 00:30:13.133264] W [fuse-bridge.c:2830:fuse_readv_cbk]<br></div><div dir="ltr">> 0-glusterfs-fuse: 12660: READ => -1<br></div><div dir="ltr">> gfid=17c663c2-f582-455b-b806-3b9d01fb2c6c fd=0x7fc63c007038 (Permission<br></div><div dir="ltr">> denied)<br></div><div dir="ltr">> [2019-11-21 00:30:38.489449] W [MSGID: 114031]<br></div><div dir="ltr">> [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0:<br></div><div dir="ltr">> remote operation failed. Path:<br></div><div dir="ltr">> /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6<br></div><div dir="ltr">> (00000000-0000-0000-0000-000000000000) [Permission denied]<br></div><div dir="ltr">> [2019-11-21 00:30:38.489520] W [MSGID: 114031]<br></div><div dir="ltr">> [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-1:<br></div><div dir="ltr">> remote operation failed. Path:<br></div><div dir="ltr">> /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6<br></div><div dir="ltr">> (00000000-0000-0000-0000-000000000000) [Permission denied]<br></div><div dir="ltr">> [2019-11-21 00:30:38.489669] W [MSGID: 114031]<br></div><div dir="ltr">> [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-2:<br></div><div dir="ltr">> remote operation failed. Path:<br></div><div dir="ltr">> /.shard/a10a5ae8-108b-4d78-9e65-cca188c27fc4.6<br></div><div dir="ltr">> (00000000-0000-0000-0000-000000000000) [Permission denied]<br></div><div dir="ltr">> [2019-11-21 00:30:38.489717] E [MSGID: 133010]<br></div><div dir="ltr">> [shard.c:2327:shard_common_lookup_shards_cbk] 0-data_fast-shard: Lookup on<br></div><div dir="ltr">> shard 6 failed. Base file gfid = a10a5ae8-108b-4d78-9e65-cca188c27fc4<br></div><div dir="ltr">> [Permission denied]<br></div><div dir="ltr">> [2019-11-21 00:30:38.489777] W [fuse-bridge.c:2830:fuse_readv_cbk]<br></div><div dir="ltr">> 0-glusterfs-fuse: 12928: READ => -1<br></div><div dir="ltr">> gfid=a10a5ae8-108b-4d78-9e65-cca188c27fc4 fd=0x7fc63c01a058 (Permission<br></div><div dir="ltr">> denied)<br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">> Anyone got an idea why is it happening?<br></div><div dir="ltr">> I checked user/group and selinux permissions - all OK<br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">> >Can you share the commands (and output) used to check this?<br></div><div dir="ltr">> I first thought that the file is cached in memory and that's why vdsm user<br></div><div dir="ltr">> can read the file , but the following shows opposite:<br></div><div dir="ltr">><br></div><div dir="ltr">> [<a href="mailto:root@ovirt1" rel="nofollow" target="_blank">root@ovirt1</a> 94f763e9-fd96-4bee-a6b2-31af841a918b]# ll<br></div><div dir="ltr">> total 562145<br></div><div dir="ltr">> -rw-rw----. 1 vdsm kvm 5368709120 Nov 12 23:29<br></div><div dir="ltr">> 5b1d3113-5cca-4582-9029-634b16338a2f<br></div><div dir="ltr">> -rw-rw----. 1 vdsm kvm 1048576 Nov 11 14:11<br></div><div dir="ltr">> 5b1d3113-5cca-4582-9029-634b16338a2f.lease<br></div><div dir="ltr">> -rw-r--r--. 1 vdsm kvm 313 Nov 11 14:11<br></div><div dir="ltr">> 5b1d3113-5cca-4582-9029-634b16338a2f.meta<br></div><div dir="ltr">> [<a href="mailto:root@ovirt1" rel="nofollow" target="_blank">root@ovirt1</a> 94f763e9-fd96-4bee-a6b2-31af841a918b]# pwd<br></div><div dir="ltr">><br></div><div dir="ltr">> /rhev/data-center/mnt/glusterSD/gluster1:_data__fast/396604d9-2a9e-49cd-9563-fdc79981f67b/images/94f763e9-fd96-4bee-a6b2-31af841a918b<br></div><div dir="ltr">> [<a href="mailto:root@ovirt1" rel="nofollow" target="_blank">root@ovirt1</a> 94f763e9-fd96-4bee-a6b2-31af841a918b]# echo 3 ><br></div><div dir="ltr">> /proc/sys/vm/drop_caches<br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">> I would use iflag=direct instead, no need to mess with caches. Vdsm always<br></div><div dir="ltr">> use direct I/O.<br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">> [<a href="mailto:root@ovirt1" rel="nofollow" target="_blank">root@ovirt1</a> 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd<br></div><div dir="ltr">> if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress<br></div><div dir="ltr">> dd: error reading ‘5b1d3113-5cca-4582-9029-634b16338a2f’: Permission denied<br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">> You got permissions denied...<br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">> 16+0 records in<br></div><div dir="ltr">> 16+0 records out<br></div><div dir="ltr">> 67108864 bytes (67 MB) copied, 0.198372 s, 338 MB/s<br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">> >Seems like it could read upto ~67MB successfully before it encountered<br></div><div dir="ltr">> 'Permission denied' errors. Assuming a shard-block-size >of 64MB, looks<br></div><div dir="ltr">> like all the shards under /.shard could not be accessed.<br></div><div dir="ltr">> >Could you share the following pieces of information:<br></div><div dir="ltr">> >1. brick logs of data_fast<br></div><div dir="ltr">> Attached in data_fast-brick-logs.tgz<br></div><div dir="ltr">><br></div><div dir="ltr">> >2. ls -la of .shard relative to the bricks (NOT the mount) on all the<br></div><div dir="ltr">> bricks of data_fast<br></div><div dir="ltr">> Not sure if I understood you correctly, so I ran "ls -lad<br></div><div dir="ltr">> /gluster_bricks/data_fast/data_fast/.shard". If it's not what you wanted to<br></div><div dir="ltr">> see - just correct me.<br></div><div dir="ltr">> I have run multiple "find" commands with "-exec chown vdsm:kvm {} \;" ,<br></div><div dir="ltr">> just to be sure that this is not happening.<br></div><div dir="ltr">> >3. and ls -la of all shards under .shard of data_fast (perhaps a handful<br></div><div dir="ltr">> of them have root permission assigned somehow which is causing access to be<br></div><div dir="ltr">> denied? Perhaps while resolving pending heals manually? )<br></div><div dir="ltr">> All shards seem to be owned by "vdsm:kvm" with 660.<br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">> -Krutika<br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">> And dd continue to read data?!<br></div><div dir="ltr">><br></div><div dir="ltr">> I have never seen anything like this.<br></div><div dir="ltr">><br></div><div dir="ltr">> It will be helpful to run this with strace:<br></div><div dir="ltr">><br></div><div dir="ltr">> strace -t -TT -o dd.strace dd if=vol-id of=/dev/null iflag=direct<br></div><div dir="ltr">> bs=8M status=progress<br></div><div dir="ltr">><br></div><div dir="ltr">> And share dd.strace.<br></div><div dir="ltr">><br></div><div dir="ltr">> Logs in /var/log/glusterfs/exportname.log will contain useful info for<br></div><div dir="ltr">> this test.<br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">> [<a href="mailto:root@ovirt1" rel="nofollow" target="_blank">root@ovirt1</a> 94f763e9-fd96-4bee-a6b2-31af841a918b]# dd<br></div><div dir="ltr">> if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress<br></div><div dir="ltr">> 5356126208 bytes (5.4 GB) copied, 12.061393 s, 444 MB/s<br></div><div dir="ltr">> 1280+0 records in<br></div><div dir="ltr">> 1280+0 records out<br></div><div dir="ltr">> 5368709120 bytes (5.4 GB) copied, 12.0876 s, 444 MB/s<br></div><div dir="ltr">> [<a href="mailto:root@ovirt1" rel="nofollow" target="_blank">root@ovirt1</a> 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd<br></div><div dir="ltr">> if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress<br></div><div dir="ltr">> 3598712832 bytes (3.6 GB) copied, 1.000540 s, 3.6 GB/s<br></div><div dir="ltr">> 1280+0 records in<br></div><div dir="ltr">> 1280+0 records out<br></div><div dir="ltr">> 5368709120 bytes (5.4 GB) copied, 1.47071 s, 3.7 GB/s<br></div><div dir="ltr">> [<a href="mailto:root@ovirt1" rel="nofollow" target="_blank">root@ovirt1</a> 94f763e9-fd96-4bee-a6b2-31af841a918b]# echo 3 ><br></div><div dir="ltr">> /proc/sys/vm/drop_caches<br></div><div dir="ltr">> [<a href="mailto:root@ovirt1" rel="nofollow" target="_blank">root@ovirt1</a> 94f763e9-fd96-4bee-a6b2-31af841a918b]# sudo -u vdsm dd<br></div><div dir="ltr">> if=5b1d3113-5cca-4582-9029-634b16338a2f of=/dev/null bs=4M status=progress<br></div><div dir="ltr">> 5171576832 bytes (5.2 GB) copied, 12.071837 s, 428 MB/s<br></div><div dir="ltr">> 1280+0 records in<br></div><div dir="ltr">> 1280+0 records out<br></div><div dir="ltr">> 5368709120 bytes (5.4 GB) copied, 12.4873 s, 430 MB/s<br></div><div dir="ltr">><br></div><div dir="ltr">> As you can see , once root user reads the file -> vdsm user can also do<br></div><div dir="ltr">> that.<br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">> Smells like issue on gluster side.<br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">> >I would try this on the hypervisor to check what vdsm/qemu see:<br></div><div dir="ltr">><br></div><div dir="ltr">> >$ ls -lahRZ /rhv/data-center/mnt/glusterSD/gluster-server:_path<br></div><div dir="ltr">> I'm attaching the output of the find I run, but this one should be enough:<br></div><div dir="ltr">> [<a href="mailto:root@ovirt1" rel="nofollow" target="_blank">root@ovirt1</a> ~]# find /rhev/data-center/mnt/glusterSD/*/[0-9]*/images/<br></div><div dir="ltr">> -not -user vdsm -print<br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">> A full output of ls -lahRZ, showing user, group, permissions bits, and<br></div><div dir="ltr">> selinux label<br></div><div dir="ltr">> of the entire tree will be more useful.<br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">> >Also, to make sure we don't have selinux issue on the hypervisor, you can<br></div><div dir="ltr">> change<br></div><div dir="ltr">> >selinux to permissive mode:<br></div><div dir="ltr">><br></div><div dir="ltr">> > setenforce 0<br></div><div dir="ltr">><br></div><div dir="ltr">> This is the first thing I did and the systems were still in permissive<br></div><div dir="ltr">> when I tried again.I'm 99.99% sure it's not selinux.<br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">> >And then try again. If this was selinux issue the permission denied issue<br></div><div dir="ltr">> will disappear.<br></div><div dir="ltr">> >If this is the case please provide the output of:<br></div><div dir="ltr">><br></div><div dir="ltr">> > ausearh -m AVC -ts today<br></div><div dir="ltr">><br></div><div dir="ltr">> >If the issue still exists, we eliminated selinux, and you can enable it<br></div><div dir="ltr">> again:<br></div><div dir="ltr">><br></div><div dir="ltr">> > setenforce 1<br></div><div dir="ltr">><br></div><div dir="ltr">> [<a href="mailto:root@ovirt3" rel="nofollow" target="_blank">root@ovirt3</a> ~]# ausearch -m AVC -ts today<br></div><div dir="ltr">> <no matches><br></div><div dir="ltr">> [<a href="mailto:root@ovirt2" rel="nofollow" target="_blank">root@ovirt2</a> ~]# ausearch -m AVC -ts today<br></div><div dir="ltr">> <no matches><br></div><div dir="ltr">> [<a href="mailto:root@ovirt1" rel="nofollow" target="_blank">root@ovirt1</a> ~]# ausearch -m AVC -ts today<br></div><div dir="ltr">> <no matches><br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">> So this is not selinux on the hypervisor. I wonder if it can be selinux on<br></div><div dir="ltr">> the gluster side?<br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">> I have a vague feeling that the issue is related to gluster v6.5 to 6.6<br></div><div dir="ltr">> upgrade which I several days before... So if any logs are needed (or debug<br></div><div dir="ltr">> enabled), just mention.<br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">> If this is the last change, and it worked before, most likely.<br></div><div dir="ltr">><br></div><div dir="ltr">> Nir<br></div><div dir="ltr">> _______________________________________________<br></div><div dir="ltr">> Users mailing list -- <a href="mailto:users@ovirt.org" rel="nofollow" target="_blank">users@ovirt.org</a><br></div><div dir="ltr">> To unsubscribe send an email to <a href="mailto:users-leave@ovirt.org" rel="nofollow" target="_blank">users-leave@ovirt.org</a><br></div><div dir="ltr">> Privacy Statement: <a href="https://www.ovirt.org/site/privacy-policy/" rel="nofollow" target="_blank">https://www.ovirt.org/site/privacy-policy/</a><br></div><div dir="ltr">> oVirt Code of Conduct:<br></div><div dir="ltr">> <a href="https://www.ovirt.org/community/about/community-guidelines/" rel="nofollow" target="_blank">https://www.ovirt.org/community/about/community-guidelines/</a><br></div><div dir="ltr">> List Archives:<br></div><div dir="ltr">> <a href="https://lists.ovirt.org/archives/list/users@ovirt.org/message/AKLLOJKG6NEJUB264RA5PLQMGWNG3GD3/" rel="nofollow" target="_blank">https://lists.ovirt.org/archives/list/users@ovirt.org/message/AKLLOJKG6NEJUB264RA5PLQMGWNG3GD3/</a><br></div><div dir="ltr">><br></div><div dir="ltr">><br></div><div dir="ltr">_______________________________________________<br></div><div dir="ltr">Users mailing list -- <a href="mailto:users@ovirt.org" rel="nofollow" target="_blank">users@ovirt.org</a><br></div><div dir="ltr">To unsubscribe send an email to <a href="mailto:users-leave@ovirt.org" rel="nofollow" target="_blank">users-leave@ovirt.org</a><br></div><div dir="ltr">Privacy Statement: <a href="https://www.ovirt.org/site/privacy-policy/" rel="nofollow" target="_blank">https://www.ovirt.org/site/privacy-policy/</a><br></div><div dir="ltr">oVirt Code of Conduct: <a href="https://www.ovirt.org/community/about/community-guidelines/" rel="nofollow" target="_blank">https://www.ovirt.org/community/about/community-guidelines/</a><br></div><div dir="ltr">List Archives: <a href="https://lists.ovirt.org/archives/list/users@ovirt.org/message/UJLBNDFKU27J24MPINAND5MJDDG6SG2A/" rel="nofollow" target="_blank">https://lists.ovirt.org/archives/list/users@ovirt.org/message/UJLBNDFKU27J24MPINAND5MJDDG6SG2A/</a><br></div></div>
</div>
</div></body></html>