[Gluster-users] remote operation failed [Permission denied] every 10 minutes after upgrading from 5.10 to 7.0

[Amar Tumballi]

Answers inline.

On Wed, Dec 25, 2019 at 2:06 AM Artem Russakovskii <archon810 at gmail.com>
wrote:

> Hi everyone,
>
> I'm preparing the upgrade for our 5.10 gluster cluster running 1x4
> replicate volumes to 7.X and decided to upgrade our test cluster first.
>
> As soon as I upgraded to 7.0 (and now 7.1), I started seeing the following
> messages every 10 minutes in the log for one of the volumes:
>
> [2019-12-19 21:27:55.041949] W [MSGID: 114031]
>> [client-rpc-fops_v2.c:850:client4_0_setxattr_cbk] 0-dev_SNIP_data-client-0:
>> remote operation failed [Permission denied]
>> [2019-12-19 21:27:55.042002] W [MSGID: 114031]
>> [client-rpc-fops_v2.c:850:client4_0_setxattr_cbk] 0-dev_SNIP_data-client-2:
>> remote operation failed [Permission denied]
>> [2019-12-19 21:27:55.042013] W [MSGID: 114031]
>> [client-rpc-fops_v2.c:850:client4_0_setxattr_cbk] 0-dev_SNIP_data-client-1:
>> remote operation failed [Permission denied]
>> [2019-12-19 21:27:55.042634] E [MSGID: 148002]
>> [utime.c:146:gf_utime_set_mdata_setxattr_cbk] 0-dev_SNIP_data-utime: dict
>> set of key for set-ctime-mdata failed [Permission denied]
>> The message "W [MSGID: 114031]
>> [client-rpc-fops_v2.c:850:client4_0_setxattr_cbk] 0-dev_SNIP_data-client-0:
>> remote operation failed [Permission denied]" repeated 2 times between
>> [2019-12-19 21:27:55.041949] and [2019-12-19 21:27:55.047300]
>> The message "W [MSGID: 114031]
>> [client-rpc-fops_v2.c:850:client4_0_setxattr_cbk] 0-dev_SNIP_data-client-2:
>> remote operation failed [Permission denied]" repeated 2 times between
>> [2019-12-19 21:27:55.042002] and [2019-12-19 21:27:55.047312]
>> The message "W [MSGID: 114031]
>> [client-rpc-fops_v2.c:850:client4_0_setxattr_cbk] 0-dev_SNIP_data-client-1:
>> remote operation failed [Permission denied]" repeated 2 times between
>> [2019-12-19 21:27:55.042013] and [2019-12-19 21:27:55.047524]
>> The message "E [MSGID: 148002]
>> [utime.c:146:gf_utime_set_mdata_setxattr_cbk] 0-dev_SNIP_data-utime: dict
>> set of key for set-ctime-mdata failed [Permission denied]" repeated 2 times
>> between [2019-12-19 21:27:55.042634] and [2019-12-19 21:27:55.047887]
>>
>
>
>> [2019-12-19 21:37:55.541329] W [MSGID: 114031]
>> [client-rpc-fops_v2.c:850:client4_0_setxattr_cbk] 0-dev_SNIP_data-client-2:
>> remote operation failed [Permission denied]
>> [2019-12-19 21:37:55.541644] W [MSGID: 114031]
>> [client-rpc-fops_v2.c:850:client4_0_setxattr_cbk] 0-dev_SNIP_data-client-1:
>> remote operation failed [Permission denied]
>> [2019-12-19 21:37:55.541681] W [MSGID: 114031]
>> [client-rpc-fops_v2.c:850:client4_0_setxattr_cbk] 0-dev_SNIP_data-client-0:
>> remote operation failed [Permission denied]
>> [2019-12-19 21:37:55.542067] E [MSGID: 148002]
>> [utime.c:146:gf_utime_set_mdata_setxattr_cbk] 0-dev_SNIP_data-utime: dict
>> set of key for set-ctime-mdata failed [Permission denied]
>> The message "W [MSGID: 114031]
>> [client-rpc-fops_v2.c:850:client4_0_setxattr_cbk] 0-dev_SNIP_data-client-2:
>> remote operation failed [Permission denied]" repeated 3 times between
>> [2019-12-19 21:37:55.541329] and [2019-12-19 21:37:55.546695]
>> The message "W [MSGID: 114031]
>> [client-rpc-fops_v2.c:850:client4_0_setxattr_cbk] 0-dev_SNIP_data-client-1:
>> remote operation failed [Permission denied]" repeated 3 times between
>> [2019-12-19 21:37:55.541644] and [2019-12-19 21:37:55.546711]
>> The message "W [MSGID: 114031]
>> [client-rpc-fops_v2.c:850:client4_0_setxattr_cbk] 0-dev_SNIP_data-client-0:
>> remote operation failed [Permission denied]" repeated 3 times between
>> [2019-12-19 21:37:55.541681] and [2019-12-19 21:37:55.546761]
>> The message "E [MSGID: 148002]
>> [utime.c:146:gf_utime_set_mdata_setxattr_cbk] 0-dev_SNIP_data-utime: dict
>> set of key for set-ctime-mdata failed [Permission denied]" repeated 3 times
>> between [2019-12-19 21:37:55.542067] and [2019-12-19 21:37:55.547042]
>>
>
> etc.
>
> The questions are:
>
>    1. Is it a cause for concern? They weren't there before the upgrade.
>
> Looks like all these logs are related to ctime feature which is by default
enabled in 7.1 (and not in 5.x series)


>
>    1. How can I determine what's causing the errors?
>
>
Also check the server logs.


>
>    1. How can I fix them and prevent them from spamming the logs?
>
>
Check op-version of the cluster after upgrade. Disable 'ctime' option
(volume set <vol> ctime off).



> Thank you and happy holidays!
>

+1

Regards,
Amar
https://kadalu.io



>
> Sincerely,
> Artem
>
> --
> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
> <http://www.apkmirror.com/>, Illogical Robot LLC
> beerpla.net | @ArtemR <http://twitter.com/ArtemR>
> ________
>
> Community Meeting Calendar:
>
> APAC Schedule -
> Every 2nd and 4th Tuesday at 11:30 AM IST
> Bridge: https://bluejeans.com/441850968
>
> NA/EMEA Schedule -
> Every 1st and 3rd Tuesday at 01:00 PM EDT
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20191228/3e397a1c/attachment.html>