<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Aptos",sans-serif;
mso-ligatures:standardcontextual;
mso-fareast-language:EN-US;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:11.0pt;
mso-fareast-language:EN-US;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-IE" link="#467886" vlink="#96607D" style="word-wrap:break-word">
<div class="WordSection1">
<p style="margin:0in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Hello all,<o:p></o:p></span></p>
<p style="margin:0in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">We have a distributed volume running in 7 hosts and 28 bricks.<o:p></o:p></span></p>
<p style="margin:0in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <o:p></o:p></span></p>
<p style="margin:0in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">We've been experiencing some strange behaviors over the time, some of them are solved when the gluster services are restarted, but other problems persist.<o:p></o:p></span></p>
<p style="margin:0in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <o:p></o:p></span></p>
<p style="margin:0in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">In the case below, and this one happens many times, some people access the gluster filesystem via Samba.<o:p></o:p></span></p>
<p style="margin:0in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">They have a mapped drive in their windows machine.<o:p></o:p></span></p>
<p style="margin:0in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <o:p></o:p></span></p>
<p style="margin:0in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Most of the time things work as expected, but a recurrent issue is that sometimes when directories are created, they get a "permission denied" message. Apparently, the directories
are created.<o:p></o:p></span></p>
<p style="margin:0in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">This sometimes happens also when they are renaming directories. And other times they get a message telling them that the files/directories do not exist.<o:p></o:p></span></p>
<p style="margin:0in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">There are some logs below.
<o:p></o:p></span></p>
<p style="margin:0in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <o:p></o:p></span></p>
<p style="margin:0in"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New"">[2024-06-18 14:02:52.931157 +0000] W [MSGID: 109037] [dht-common.c:4711:dht_getxattr_get_real_filename_cbk] 0-tank-dht: Failed to get real filename. [No such file or
directory]<o:p></o:p></span></p>
<p style="margin:0in"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New"">[2024-06-18 14:02:52.934395 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2097:client4_0_create_cbk] 0-tank-client-5: remote operation failed. [{path=/dir59/<b>BallData.mat</b>},
{errno=2}, {error=No such file or directory}]<o:p></o:p></span></p>
<p style="margin:0in"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p style="margin:0in"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New"">[2024-06-18 14:04:52.221379 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2097:client4_0_create_cbk] 0-tank-client-5: remote operation failed. [{path=/dir59/<b>BallData.mat</b>},
{errno=2}, {error=No such file or directory}]<o:p></o:p></span></p>
<p style="margin:0in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <o:p></o:p></span></p>
<p style="margin:0in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Here we have multiple attempts to rename a file <o:p></o:p></span></p>
<p style="margin:0in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <o:p></o:p></span></p>
<p style="margin:0in"><span style="font-size:10.0pt;font-family:"Courier New"">[2024-06-18 14:05:54.992761 +0000] I [MSGID: 109064] [dht-rename.c:1955:dht_rename] 0-tank-dht: renaming /dir59/<b>BallData1.mat</b> (ae1ddf78-5e3d-470c-8a4c-cedca17829b8) (hash=tank-client-4/cache=tank-client-4)
=> /dir59/<b>BallData.mat</b> ((null)) (hash=tank-client-5/cache=<nul>)<o:p></o:p></span></p>
<p style="margin:0in"><span style="font-size:10.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p style="margin:0in"><span style="font-size:10.0pt;font-family:"Courier New"">[2024-06-18 14:05:55.527004 +0000] I [MSGID: 109064] [dht-rename.c:1955:dht_rename] 0-tank-dht: renaming /dir59/<b>BallData1.mat</b> (ae1ddf78-5e3d-470c-8a4c-cedca17829b8) (hash=tank-client-4/cache=tank-client-4)
=> /dir59/<b>BallData.mat</b> ((null)) (hash=tank-client-5/cache=<nul>)<o:p></o:p></span></p>
<p style="margin:0in"><span style="font-size:10.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p style="margin:0in"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New"">The message "I [MSGID: 109064] [dht-rename.c:1955:dht_rename] 0-tank-dht: renaming /dir59/<b>BallData1.mat</b> (ae1ddf78-5e3d-470c-8a4c-cedca17829b8) (hash=tank-client-4/cache=tank-client-4)
=> /dir59/BallData.mat ((null)) (hash=tank-client-5/cache=<nul>) " repeated 2 times between [2024-06-18 14:05:55.527004 +0000] and [2024-06-18 14:06:11.390068 +0000]<o:p></o:p></span></p>
<p style="margin:0in"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p style="margin:0in"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New"">[2024-06-18 14:06:13.417710 +0000] I [MSGID: 109064] [dht-rename.c:1955:dht_rename] 0-tank-dht: renaming /dir59/<b>BallData1.mat</b> (ae1ddf78-5e3d-470c-8a4c-cedca17829b8)
(hash=tank-client-4/cache=tank-client-4) => /dir59/<b>BallData.mat</b> ((null)) (hash=tank-client-5/cache=<nul>)<o:p></o:p></span></p>
<p style="margin:0in"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p style="margin:0in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">There are many messages in the logs like the ones below<o:p></o:p></span></p>
<p style="margin:0in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <o:p></o:p></span></p>
<p style="margin:0in"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New"">[2024-06-16 12:28:07.720713 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib64/glusterfs/9.4/xlator/performance/md-cache.so(+0x11683) [0x7f998d68f683] -->/lib64/libglusterfs.so.0(dict_get_iatt+0x40)
[0x7f999507bd10] -->/lib64/libglusterfs.so.0(dict_get_with_ref+0x85) [0x7f99950799b5] ) 0-dict: dict OR key (virt-gf-prestat) is NULL [<b>Invalid argument</b>]<o:p></o:p></span></p>
<p style="margin:0in"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p style="margin:0in"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New"">[2024-06-16 12:28:29.099386 +0000] W [MSGID: 114061] [client-common.c:3327:client_pre_readdirp_v2] 0-tank-client-0: remote_fd is -1. EBADFD [{gfid=8def2ad4-72fd-49ed-af12-4b1f9878b2a4},
{errno=77}, {error=<b>File descriptor in bad state</b>}]<o:p></o:p></span></p>
<p style="margin:0in"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p style="margin:0in"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New"">[2024-06-16 12:37:56.662296 +0000] I [MSGID: 109061] [dht-layout.c:641:dht_layout_normalize] 0-tank-dht:
<b>Found anomalies</b> [{path=/dir59}, {gfid=3e58f9ed-8565-4ce2-bbbb-77b18b841929}, {holes=3}, {overlaps=0}]<o:p></o:p></span></p>
<p style="margin:0in"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p style="margin:0in"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New"">[2024-06-16 12:37:56.663670 +0000] W [MSGID: 109005] [dht-selfheal.c:1064:dht_selfheal_dir_mkdir_cbk] 0-tank-dht:
<b>Healing of path failed</b> [{path=/dir59}, {gfid=3e58f9ed-8565-4ce2-bbbb-77b18b841929}, {errno=2}, {<b>error=No such file or directory}</b>]<o:p></o:p></span></p>
<p style="margin:0in"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p style="margin:0in"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New"">[2024-06-16 12:37:56.672373 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib64/glusterfs/9.4/xlator/cluster/distribute.so(+0x516c6) [0x7f998e3426c6] -->/lib64/libglusterfs.so.0(dict_get_uint32+0x3b)
[0x7f999507a5bb] -->/lib64/libglusterfs.so.0(dict_get_with_ref+0x85) [0x7f99950799b5] ) 0-dict: dict OR key (trusted.glusterfs.dht.commithash)<b> is NULL [Invalid argument]</b><o:p></o:p></span></p>
<p style="margin:0in"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p style="margin:0in"><span lang="EN-US" style="font-size:10.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p style="margin:0in"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Did any of you been through this before? The "permission denied" happens randomly.<o:p></o:p></span></p>
<p style="margin:0in"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <o:p></o:p></span></p>
<p style="margin:0in"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">About the renaming of files/directories, sometimes I think that the renames are being done, but the windows client obtains an error and is not informed of the
success of the operation, and then, when they try again to rename, the original filename does not exist anymore, and they get "File not Found". But I'm not sure of this.<o:p></o:p></span></p>
<p style="margin:0in"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <o:p></o:p></span></p>
<p style="margin:0in"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Thanks.<o:p></o:p></span></p>
<p style="margin:0in"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="PT-BR" style="font-size:12.0pt;color:#262626;mso-fareast-language:EN-GB">Carlo Rodrigues<o:p></o:p></span></p>
<p class="MsoNormal"><b><span lang="PT-BR" style="font-size:12.0pt;color:#262626;mso-fareast-language:EN-GB">Cloud & Infra Engineer<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span lang="PT-BR" style="font-size:8.0pt;font-family:"Arial",sans-serif;mso-fareast-language:EN-GB"><o:p> </o:p></span></b></p>
<p class="MsoNormal"><b><u><span lang="PT-BR" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#E00000;mso-fareast-language:EN-GB">carlo.rodrigues@claranet.com<o:p></o:p></span></u></b></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif;mso-fareast-language:EN-GB">T +351 21 319 92 00<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif;mso-fareast-language:EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#666666"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.0pt;font-family:"Arial",sans-serif;mso-fareast-language:EN-GB"><img width="127" height="26" style="width:1.3273in;height:.2738in" id="Picture_x0020_2" src="cid:image001.png@01DACEF4.E8A0B800" alt="Logo
Description automatically generated"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:8.0pt;font-family:"Arial",sans-serif;mso-fareast-language:EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="PT" style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#262626;mso-fareast-language:EN-GB">Claranet Portugal<br>
</span><span lang="PT" style="font-size:12.0pt;mso-fareast-language:EN-IE"><a href="http://www.claranet.pt/"><b><span style="font-size:9.0pt;font-family:"Arial",sans-serif;color:#E00000">www.claranet.pt</span></b></a></span><span lang="PT" style="font-family:"Arial",sans-serif"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="PT" style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#262626;mso-fareast-language:EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="PT" style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#7F7F7F;mso-fareast-language:EN-IE">Todas as informações contidas nesta mensagem eletrónica da Claranet estão abrangidas pelo disclaimer disponível em
</span><span lang="PT" style="font-size:12.0pt;mso-fareast-language:EN-IE"><a href="https://www.claranet.pt/email-disclaimer"><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#E00000">https://www.claranet.pt/email-disclaimer</span></a></span><span lang="PT" style="font-family:"Arial",sans-serif;color:#E00000"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#7F7F7F;mso-fareast-language:EN-IE">All the information contained within this electronic message from Claranet is covered by the disclaimer at </span><span lang="PT" style="font-size:12.0pt;mso-fareast-language:EN-IE"><a href="https://www.claranet.pt/email-disclaimer-ENG"><span lang="EN-US" style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#E00000">https://www.claranet.pt/email-disclaimer-ENG</span></a></span><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#E00000;mso-fareast-language:EN-IE"><o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>