[Gluster-users] [ovirt-users] Re: [ANN] oVirt 4.3.7 Third Release Candidate is now available for testing

[Strahil]

Hi Jiffin,

Thanks for the info.
As I'm now on Gluster v7 , I hope it won't happen again.
It's nice to know it got fixed.

Best Regards,
Strahil NikolovOn Dec 2, 2019 11:32, Jiffin Thottan <jthottan at redhat.com> wrote:
>
> Hi Krutika, 
>
> Apparently, in context acl info got corrupted see brick logs 
>
> [posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control: 
> client: 
> CTX_ID:dae9ffad-6acd-4a43-9372-229a3018fde9-GRAPH_ID:0-PID:11468-HOST:ovirt2.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0, 
> gfid: be318638-e8a0-4c6d-977d-7a937aa84806, 
> req(uid:107,gid:107,perm:1,ngrps:4), 
> ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-) 
> [Permission denied] 
>
> which resulted in the situation. There was one bug similar was reported https://bugzilla.redhat.com/show_bug.cgi?id=1668286 and 
>
> it got fixed in 6.6 release IMO https://review.gluster.org/#/c/glusterfs/+/23233/. But here he mentioned he saw the issue when 
>
> he upgraded from 6.5 to 6.6 
>
> One way to workaround is to perform a dummy setfacl(preferably using root) on the corrupted files which will forcefully fetch the acl 
>
> info again from backend and update the context. Another approach to restart brick process(kill and vol start force) 
>
> Regards, 
> Jiffin 
>
> ----- Original Message ----- 
> From: "Krutika Dhananjay" <kdhananj at redhat.com> 
> To: "Strahil Nikolov" <hunter86_bg at yahoo.com>, "Jiffin Thottan" <jthottan at redhat.com>, "raghavendra talur" <rtalur at redhat.com> 
> Cc: "Nir Soffer" <nsoffer at redhat.com>, "Rafi Kavungal Chundattu Parambil" <rkavunga at redhat.com>, "users" <users at ovirt.org>, "gluster-user" <gluster-users at gluster.org> 
> Sent: Monday, December 2, 2019 11:48:22 AM 
> Subject: Re: [ovirt-users] Re: [ANN] oVirt 4.3.7 Third Release Candidate is now available for testing 
>
> Sorry about the late response. 
>
> I looked at the logs. These errors are originating from posix-acl 
> translator - 
>
>
>
> *[2019-11-17 07:55:47.090065] E [MSGID: 115050] 
> [server-rpc-fops_v2.c:158:server4_lookup_cbk] 0-data_fast-server: 162496: 
> LOOKUP /.shard/5985adcb-0f4d-4317-8a26-1652973a2350.6 
> (be318638-e8a0-4c6d-977d-7a937aa84806/5985adcb-0f4d-4317-8a26-1652973a2350.6), 
> client: 
> CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0, 
> error-xlator: data_fast-access-control [Permission denied][2019-11-17 
> 07:55:47.090174] I [MSGID: 139001] 
> [posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control: 
> client: 
> CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0, 
> gfid: be318638-e8a0-4c6d-977d-7a937aa84806, 
> req(uid:36,gid:36,perm:1,ngrps:3), 
> ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-) 
> [Permission denied][2019-11-17 07:55:47.090209] E [MSGID: 115050] 
> [server-rpc-fops_v2.c:158:server4_lookup_cbk] 0-data_fast-server: 162497: 
> LOOKUP /.shard/5985adcb-0f4d-4317-8a26-1652973a2350.7 
> (be318638-e8a0-4c6d-977d-7a937aa84806/5985adcb-0f4d-4317-8a26-1652973a2350.7), 
> client: 
> CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0, 
> error-xlator: data_fast-access-control [Permission denied][2019-11-17 
> 07:55:47.090299] I [MSGID: 139001] 
> [posix-acl.c:263:posix_acl_log_permit_denied] 0-data_fast-access-control: 
> client: 
> CTX_ID:8bff2d95-4629-45cb-a7bf-2412e48896bc-GRAPH_ID:0-PID:13394-HOST:ovirt1.localdomain-PC_NAME:data_fast-client-0-RECON_NO:-0, 
> gfid: be318638-e8a0-4c6d-977d-7a937aa84806, 
> req(uid:36,gid:36,perm:1,ngrps:3), 
> ctx(uid:0,gid:0,in-groups:0,perm:000,updated-fop:INVALID, acl:-) 
> [Permission denied]* 
>
> Jiffin/Raghavendra Talur, 
> Can you help? 
>
> -Krutika 
>
> On Wed, Nov 27, 2019 at 2:11 PM Strahil Nikolov <hunter86_bg at yahoo.com> 
> wrote: 
>
> > Hi Nir,All, 
> > 
> > it seems that 4.3.7 RC3 (and even RC4) are not the problem here(attached 
> > screenshot of oVirt running on v7 gluster). 
> > It seems strange that both my serious issues with oVirt are related to 
> > gluster issue (1st gluster v3  to v5 migration and now this one). 
> > 
> > I have just updated to gluster v7.0 (Centos 7 repos), and rebooted all 
> > nodes. 
> > Now both Engine and all my VMs are back online - so if you hit issues with 
> > 6.6 , you should give a try to 7.0 (and even 7.1 is coming soon) before 
> > deciding to wipe everything. 
> > 
> > @Krutika, 
> > 
> > I guess you will ask for the logs, so let's switch to gluster-users about 
> > this one ? 
> > 
> > Best Regards, 
> > Strahil Nikolov 
> > 
> > В понеделник, 25 ноември 2019 г., 16:45:48 ч. Гринуич-5, Strahil Nikolov < 
> > hunter86_bg at yahoo.com> написа: 
> > 
> > 
> > Hi Krutika, 
> > 
> > I have enabled TRACE log level for the volume data_fast, 
> > 
> > but the issue is not much clear: 
> > FUSE reports: 
> > 
> > [2019-11-25 21:31:53.478130] I [MSGID: 133022] 
> > [shard.c:3674:shard_delete_shards] 0-data_fast-shard: Deleted shards of 
> > gfid=6d9ed2e5-d4f2-4749-839b-2f1 
> > 3a68ed472 from backend 
> > [2019-11-25 21:32:43.564694] W [MSGID: 114031] 
> > [client-rpc-fops_v2.c:2634:client4_0_lookup_cbk] 0-data_fast-client-0: 
> > remote operation failed. Path: