<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Lato;
panose-1:2 15 5 2 2 2 4 3 2 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Lato",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span lang="PT" style="font-size:10.0pt;font-family:"Lato",sans-serif">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="PT" style="font-size:10.0pt;font-family:"Lato",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">I have a replicate x3 volume with the following config:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">```<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Volume Name: gvol1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Type: Replicate<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Volume ID: 384acec2-5b5f-40da-bf0e-5c53d12b3ae2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Status: Started<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Snapshot Count: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Number of Bricks: 1 x 3 = 3<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Transport-type: tcp<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Bricks:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Brick1: vm0:/srv/brick1/gvol1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Brick2: vm1:/srv/brick1/gvol1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Brick3: vm2:/srv/brick1/gvol1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Options Reconfigured:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">storage.ctime: on<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">features.utime: on<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">storage.fips-mode-rchecksum: on<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">performance.client-io-threads: off<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">nfs.disable: on<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">transport.address-family: inet<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">```<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">This volume was actually created on v3.8, but as since been upgraded (version by version) to v4.1.4 and it’s working fine (for the most part):<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">```<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Client connections for volume gvol1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">----------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Brick : vm0:/srv/brick1/gvol1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Clients connected : 6<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Hostname BytesRead BytesWritten OpVersion<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">-------- --------- ------------ ---------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">10.X.0.5:49143 2096520 2480212 40100<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">10.X.0.6:49141 14000 12812 40100<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">10.X.0.4:49134 258324 333456 40100<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">10.X.0.4:49141 565372566 1643447105 40100<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">10.X.0.5:49145 491262003 291782440 40100<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">10.X.0.6:49139 482629418 328228888 40100<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">----------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Brick : vm1:/srv/brick1/gvol1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Clients connected : 6<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Hostname BytesRead BytesWritten OpVersion<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">-------- --------- ------------ ---------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">10.X.0.6:49146 658516 508904 40100<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">10.X.0.5:49133 4142848 7139858 40100<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">10.X.0.4:49138 4088 3696 40100<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">10.X.0.4:49140 471405874 284488736 40100<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">10.X.0.5:49140 585193563 1670630439 40100<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">10.X.0.6:49138 482407454 330274812 40100<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">----------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Brick : vm2:/srv/brick1/gvol1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Clients connected : 6<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Hostname BytesRead BytesWritten OpVersion<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">-------- --------- ------------ ---------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">10.X.0.6:49133 1789624 4340938 40100<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">10.X.0.5:49137 3010064 3005184 40100<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">10.X.0.4:49143 4268 3744 40100<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">10.X.0.4:49139 471328402 283798376 40100<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">10.X.0.5:49139 491404443 293342568 40100<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">10.X.0.6:49140 561683906 830511730 40100<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">----------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">```<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">I’m now getting a lot of errors on the brick log file, like:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">`The message "W [MSGID: 113117] [posix-metadata.c:627:posix_set_ctime] 0-gvol1-posix: posix set mdata failed, No ctime : /srv/brick1/gvol1/.glusterfs/18/d0/18d04927-1ec0-4779-8c5b-7ebb82e4a614
gfid:18d04927-1ec0-4779-8c5b-7ebb82e4a614 [Function not implemented]" repeated 2 times between [2018-09-21 08:21:52.480797] and [2018-09-21 08:22:07.529625]`<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">For different files but the most common is a file that the Node.js application that runs on top of the gluster via a fuse client (glusterfs) stats every 5s for changes,
<a href="https://nodejs.org/api/fs.html#fs_fs_stat_path_options_callback">https://nodejs.org/api/fs.html#fs_fs_stat_path_options_callback</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">I think this is also related to another issue, when reading the file it returns an empty result (not always), as the app reports:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">`2018-09-21 08:22:00 | [vm0] [nobody] sync hosts: invalid applications.json, response was empty.`<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Doing `gluster volume heal gvol1 info` yields 0 for all bricks.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Should I be concerned about the warning, is this a known issue? If not, what could be causing the empty file return sometimes? Could they be related?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">The application that is running on top of the cluster build and spawns other node.js applications with mostly small files, do you have any optimization tips for it?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">FYI it is a slightly modified version of
<a href="https://github.com/totaljs/superadmin">https://github.com/totaljs/superadmin</a> to run as a web farm.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Thank you so much for any help in advance,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Kind Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif">Pedro Maia Costa<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Lato",sans-serif"><o:p> </o:p></span></p>
</div>
</body>
</html>