[Gluster-users] remote operation failed on ganesha gluster setup

[levin ng]

Hi,

We have two cluster running on ganesha NFS backed with glusterfs setup,
they are 9.1 and 9.2, and we see those operation error very often,
although the mount point is operational, but should I take any action to
those error? The mentioned shards are still present in the brick .shard
directory, but the head file are already gone. Does it mean it is failing
to remove the complete shard chain?

Thank you.
Regards,
Levin

First cluster

[2021-06-07 22:06:14.032672 +0000] E
[afr-transaction.c:2751:__afr_eager_lock_handle]
(-->/usr/lib64/glusterfs/9.2/xlator/cluster/replicate.so(+0x3dab0)
[0x7f65f6b09ab0]
-->/usr/lib64/glusterfs/9.2/xlator/cluster/replicate.so(+0x3d590)
[0x7f65f6b09590]
-->/usr/lib64/glusterfs/9.2/xlator/cluster/replicate.so(+0x3d4ef)
[0x7f65f6b094ef] ) 0-: Assertion failed: To attach gdb and coredump, Run
the script under "glusterfs/extras/debug/gfcore.py"
[2021-06-08 03:37:28.225245 +0000] W [MSGID: 114031]
[client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-0: remote
operation failed. [{path=/.shard/68867c67-597a-45a2-97ca-ffa0ff76440e.387},
{gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data
available}]
[2021-06-08 03:41:12.821056 +0000] W [MSGID: 114031]
[client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-0: remote
operation failed. [{path=/.shard/68867c67-597a-45a2-97ca-ffa0ff76440e.393},
{gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data
available}]
[2021-06-08 03:41:21.443070 +0000] W [MSGID: 114031]
[client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-1: remote
operation failed. [{path=/.shard/68867c67-597a-45a2-97ca-ffa0ff76440e.394},
{gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data
available}]
[2021-06-08 03:41:21.443117 +0000] W [MSGID: 114031]
[client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-2: remote
operation failed. [{path=/.shard/68867c67-597a-45a2-97ca-ffa0ff76440e.394},
{gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data
available}]
[2021-06-08 03:41:32.999235 +0000] W [MSGID: 114031]
[client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-0: remote
operation failed. [{path=/.shard/68867c67-597a-45a2-97ca-ffa0ff76440e.398},
{gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data
available}]
[2021-06-08 03:41:40.341562 +0000] W [MSGID: 114031]
[client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-1: remote
operation failed. [{path=/.shard/68867c67-597a-45a2-97ca-ffa0ff76440e.399},
{gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data
available}]
[2021-06-08 03:41:45.375038 +0000] W [MSGID: 114031]
[client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-1: remote
operation failed. [{path=/.shard/68867c67-597a-45a2-97ca-ffa0ff76440e.401},
{gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data
available}]
[2021-06-08 03:41:52.150104 +0000] W [MSGID: 114031]
[client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-1: remote
operation failed. [{path=/.shard/68867c67-597a-45a2-97ca-ffa0ff76440e.402},
{gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data
available}]
[2021-06-08 03:41:52.150292 +0000] W [MSGID: 114031]
[client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-0: remote
operation failed. [{path=/.shard/68867c67-597a-45a2-97ca-ffa0ff76440e.402},
{gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data
available}]
[2021-06-08 04:22:27.235500 +0000] W [MSGID: 114031]
[client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-2: remote
operation failed. [{path=/.shard/833296ac-34bc-44f9-97c9-0a0b4ee2ed52.10},
{gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data
available}]
[2021-06-08 04:22:27.235503 +0000] W [MSGID: 114031]
[client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-1: remote
operation failed. [{path=/.shard/833296ac-34bc-44f9-97c9-0a0b4ee2ed52.10},
{gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data
available}]
[2021-06-08 04:22:27.235683 +0000] W [MSGID: 114031]
[client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-0: remote
operation failed. [{path=/.shard/833296ac-34bc-44f9-97c9-0a0b4ee2ed52.10},
{gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data
available}]
[2021-06-08 04:24:06.220887 +0000] W [MSGID: 114031]
[client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-0: remote
operation failed. [{path=/.shard/833296ac-34bc-44f9-97c9-0a0b4ee2ed52.11},
{gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data
available}]
[2021-06-08 04:30:16.757007 +0000] W [MSGID: 114031]
[client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-0: remote
operation failed. [{path=/.shard/833296ac-34bc-44f9-97c9-0a0b4ee2ed52.21},
{gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data
available}]
[2021-06-08 04:30:27.199865 +0000] W [MSGID: 114031]
[client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-1: remote
operation failed. [{path=/.shard/833296ac-34bc-44f9-97c9-0a0b4ee2ed52.23},
{gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data
available}]
[2021-06-08 04:30:27.199866 +0000] W [MSGID: 114031]
[client-rpc-fops_v2.c:2625:client4_0_lookup_cbk] 0-sg1-client-0: remote
operation failed. [{path=/.shard/833296ac-34bc-44f9-97c9-0a0b4ee2ed52.23},
{gfid=00000000-0000-0000-0000-000000000000}, {errno=61}, {error=No data
available}]




Second cluster

[2021-06-06 10:00:10.940792 +0000] W [MSGID: 114031]
[client-rpc-fops_v2.c:456:client4_0_unlink_cbk] 0-hg1-client-0: remote
operation failed. [{errno=13}, {error=Permission denied}]
[2021-06-06 10:00:10.940908 +0000] W [MSGID: 114031]
[client-rpc-fops_v2.c:456:client4_0_unlink_cbk] 0-hg1-client-2: remote
operation failed. [{errno=13}, {error=Permission denied}]
[2021-06-06 10:00:10.940980 +0000] W [MSGID: 114031]
[client-rpc-fops_v2.c:456:client4_0_unlink_cbk] 0-hg1-client-1: remote
operation failed. [{errno=13}, {error=Permission denied}]
[2021-06-06 10:01:49.884528 +0000] I [MSGID: 133022]
[shard.c:3768:shard_delete_shards] 0-hg2-shard: Deleted shards of
gfid=752ef763-8529-404b-a817-f8cd7995f07d from backend
[2021-06-06 10:00:11.134620 +0000] W [MSGID: 114031]
[client-rpc-fops_v2.c:456:client4_0_unlink_cbk] 0-hg1-client-0: remote
operation failed. [{errno=13}, {error=Permission denied}]
[2021-06-06 10:00:11.134629 +0000] W [MSGID: 114031]
[client-rpc-fops_v2.c:456:client4_0_unlink_cbk] 0-hg1-client-1: remote
operation failed. [{errno=13}, {error=Permission denied}]
[2021-06-06 10:00:11.134735 +0000] W [MSGID: 114031]
[client-rpc-fops_v2.c:456:client4_0_unlink_cbk] 0-hg1-client-2: remote
operation failed. [{errno=13}, {error=Permission denied}]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20210608/e163b720/attachment.html>