[Gluster-users] Problems creating or renaming directories and files in gluster volume, via SAMBA

Fri Jul 5 15:06:13 UTC 2024

Hello all,

We have a distributed volume running in 7 hosts and 28 bricks.

We've been experiencing some strange behaviors over the time, some of them are solved when the gluster services are restarted, but other problems persist.

In the case below, and this one happens many times, some people access the gluster filesystem via Samba.

They have a mapped drive in their windows machine.

Most of the time things work as expected, but a recurrent issue is that sometimes when directories are created, they get a "permission denied" message. Apparently, the directories are created.

This sometimes happens also when they are renaming directories. And other times they get a message telling them that the files/directories do not exist.

There are some logs below.

[2024-06-18 14:02:52.931157 +0000] W [MSGID: 109037] [dht-common.c:4711:dht_getxattr_get_real_filename_cbk] 0-tank-dht: Failed to get real filename. [No such file or directory]

[2024-06-18 14:02:52.934395 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2097:client4_0_create_cbk] 0-tank-client-5: remote operation failed. [{path=/dir59/BallData.mat}, {errno=2}, {error=No such file or directory}]

[2024-06-18 14:04:52.221379 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2097:client4_0_create_cbk] 0-tank-client-5: remote operation failed. [{path=/dir59/BallData.mat}, {errno=2}, {error=No such file or directory}]

Here we have multiple attempts to rename a file

[2024-06-18 14:05:54.992761 +0000] I [MSGID: 109064] [dht-rename.c:1955:dht_rename] 0-tank-dht: renaming /dir59/BallData1.mat (ae1ddf78-5e3d-470c-8a4c-cedca17829b8) (hash=tank-client-4/cache=tank-client-4) => /dir59/BallData.mat ((null)) (hash=tank-client-5/cache=<nul>)

[2024-06-18 14:05:55.527004 +0000] I [MSGID: 109064] [dht-rename.c:1955:dht_rename] 0-tank-dht: renaming /dir59/BallData1.mat (ae1ddf78-5e3d-470c-8a4c-cedca17829b8) (hash=tank-client-4/cache=tank-client-4) => /dir59/BallData.mat ((null)) (hash=tank-client-5/cache=<nul>)

The message "I [MSGID: 109064] [dht-rename.c:1955:dht_rename] 0-tank-dht: renaming /dir59/BallData1.mat (ae1ddf78-5e3d-470c-8a4c-cedca17829b8) (hash=tank-client-4/cache=tank-client-4) => /dir59/BallData.mat ((null)) (hash=tank-client-5/cache=<nul>) " repeated 2 times between [2024-06-18 14:05:55.527004 +0000] and [2024-06-18 14:06:11.390068 +0000]

[2024-06-18 14:06:13.417710 +0000] I [MSGID: 109064] [dht-rename.c:1955:dht_rename] 0-tank-dht: renaming /dir59/BallData1.mat (ae1ddf78-5e3d-470c-8a4c-cedca17829b8) (hash=tank-client-4/cache=tank-client-4) => /dir59/BallData.mat ((null)) (hash=tank-client-5/cache=<nul>)

There are many messages in the logs like the ones below

[2024-06-16 12:28:07.720713 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib64/glusterfs/9.4/xlator/performance/md-cache.so(+0x11683) [0x7f998d68f683] -->/lib64/libglusterfs.so.0(dict_get_iatt+0x40) [0x7f999507bd10] -->/lib64/libglusterfs.so.0(dict_get_with_ref+0x85) [0x7f99950799b5] ) 0-dict: dict OR key (virt-gf-prestat) is NULL [Invalid argument]

[2024-06-16 12:28:29.099386 +0000] W [MSGID: 114061] [client-common.c:3327:client_pre_readdirp_v2] 0-tank-client-0: remote_fd is -1. EBADFD [{gfid=8def2ad4-72fd-49ed-af12-4b1f9878b2a4}, {errno=77}, {error=File descriptor in bad state}]

[2024-06-16 12:37:56.662296 +0000] I [MSGID: 109061] [dht-layout.c:641:dht_layout_normalize] 0-tank-dht: Found anomalies [{path=/dir59}, {gfid=3e58f9ed-8565-4ce2-bbbb-77b18b841929}, {holes=3}, {overlaps=0}]

[2024-06-16 12:37:56.663670 +0000] W [MSGID: 109005] [dht-selfheal.c:1064:dht_selfheal_dir_mkdir_cbk] 0-tank-dht: Healing of path failed [{path=/dir59}, {gfid=3e58f9ed-8565-4ce2-bbbb-77b18b841929}, {errno=2}, {error=No such file or directory}]

[2024-06-16 12:37:56.672373 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib64/glusterfs/9.4/xlator/cluster/distribute.so(+0x516c6) [0x7f998e3426c6] -->/lib64/libglusterfs.so.0(dict_get_uint32+0x3b) [0x7f999507a5bb] -->/lib64/libglusterfs.so.0(dict_get_with_ref+0x85) [0x7f99950799b5] ) 0-dict: dict OR key (trusted.glusterfs.dht.commithash) is NULL [Invalid argument]

Did any of you been through this before? The "permission denied" happens randomly.

About the renaming of files/directories, sometimes I think that the renames are being done, but the windows client obtains an error and is not informed of the success of the operation, and then, when they try again to rename, the original filename does not exist anymore, and they get "File not Found". But I'm not sure of this.

Thanks.

Carlo Rodrigues
Cloud & Infra Engineer

carlo.rodrigues at claranet.com
T  +351 21 319 92 00

[Logo  Description automatically generated]

Claranet Portugal
www.claranet.pt<http://www.claranet.pt/>

Todas as informações contidas nesta mensagem eletrónica da Claranet estão abrangidas pelo disclaimer disponível em https://www.claranet.pt/email-disclaimer
All the information contained within this electronic message from Claranet is covered by the disclaimer at https://www.claranet.pt/email-disclaimer-ENG

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20240705/927245e2/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 7171 bytes
Desc: image001.png
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20240705/927245e2/attachment.png>