<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"><html><head><meta content="text/html;charset=UTF-8" http-equiv="Content-Type"></head><body ><div style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 10pt;"><div>I see the volume tank is of type distribute only. If a brick disconnected, then the file present in that brick will not be available. Do you see any connection errors in the mount log? </div><div><br></div><div>---</div><div id="Zm-_Id_-Sgn" data-sigid="3848334000000010003" data-zbluepencil-ignore="true"><div>Aravinda<br></div><div>Kadalu Technologies</div></div><div><br></div><div class="zmail_extra_hr" style="border-top: 1px solid rgb(204, 204, 204); height: 0px; margin-top: 10px; margin-bottom: 10px; line-height: 0px;"><br></div><div class="zmail_extra" data-zbluepencil-ignore="true"><div><br></div><div id="Zm-_Id_-Sgn1">---- On Thu, 04 Jan 2024 01:24:55 +0530 <b>Carlo Rodrigues <carlo.rodrigues@pt.clara.net></b> wrote ---<br></div><div><br></div><blockquote id="blockquote_zmail" style="margin: 0px;"><div style="" class="zm_-3036306753179612113_parse_-5363691025001401990"><div class="x_1517879270WordSection1"><p class="" style="margin-top: 0px; margin-bottom: 0px;">Hello all,<br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"> <br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;">We're having problems with files that suddenly stop being seen on the fuse clients.<br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;">I couldn't yet find a way to reproduce this. It happens every once in a while.<br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"> <br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;">Sometimes you try to ls some file and it can't be found.<br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;">When you run ls on the parent directory, it is shown on the output, and, after that, you can access it.<br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;">I'm mentioning ls, but the problem also manifests when trying to access the file in any other way, not only when listing it with the ls command.<br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"> <br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;">Example: (The file names and location were edited for brevity)<br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"> <br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;">1. ls file returns "No such file or directory"<br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"> <br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">$ ls -l /home/RESEARCH/user.x/some_path/idtracker.log</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">ls: cannot access /home/RESEARCH/user.x/some_path/idtracker.log: No such file or directory</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"> <br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;">2. ls parent directory. The file is shown in the results.<br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"> <br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">$ ls -l /home/RESEARCH/user.x/some_path/</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">total 334500</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">-rw-r--r-- 1 user.x group_x 348521772 Jan 19 15:26 file_17.avi</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">-rw-r--r-- 1 user.x group_x 978252 Jan 19 15:26 file_17.csv</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="highlight" style="background-color:yellow"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size: 10pt; background-image: initial; background-position: initial; background-size: initial; background-repeat: initial; background-attachment: initial; background-origin: initial; background-clip: initial;">-rw-r--r-- 1 user.x group_x 1185 Jun 5 10:05 idtracker.log</span></span></span><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt"></span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">drwxr-xr-x 2 user.x group_x 4096 Jun 2 21:17 segm</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"> <br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;">3. try to ls file again. This time, it is found.<br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"> <br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">$ ls -l /home/RESEARCH/user.x/some_path/idtracker.log</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">-rw-r--r-- 1 user.x group_x 1185 Jun 5 10:05 /home/RESEARCH/user.x/some_path/idtracker.log</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"> <br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"> <br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;">The glusterfs filesystem is mounted like this<br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">storage:/tank on /nfs/tank type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"> <br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;">At the moment, <b>cluster.readdir-optimize</b> was <b>on</b>, <b>cluster.lookup-optimize</b> was <b>on</b> and <b>cluster.lookup-unhashed</b> was <b>off</b>.<br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"> <br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;">Now I've changed those 3 options to check if the problem still appears.<br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"> <br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;">Current configuration:<br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"> <br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">$ sudo gluster volume info tank</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt"> </span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Volume Name: tank</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Type: Distribute</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Volume ID: 2c3ca623-c1df-4b5f-b20e-5222d189f834</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Status: Started</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Snapshot Count: 0</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Number of Bricks: 24</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Transport-type: tcp</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Bricks:</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Brick1: swp-gluster-05:/tank/volume1/brick</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Brick2: swp-gluster-06:/tank/volume1/brick</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Brick3: swp-gluster-07:/tank/volume1/brick</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Brick4: swp-gluster-05:/tank/volume2/brick</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Brick5: swp-gluster-06:/tank/volume2/brick</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Brick6: swp-gluster-07:/tank/volume2/brick</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Brick7: swp-gluster-05:/tank/volume3/brick</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Brick8: swp-gluster-06:/tank/volume3/brick</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Brick9: swp-gluster-07:/tank/volume3/brick</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Brick10: swp-gluster-05:/tank/volume4/brick</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Brick11: swp-gluster-06:/tank/volume4/brick</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Brick12: swp-gluster-07:/tank/volume4/brick</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Brick13: swp-gluster-01:/tank/volume1/brick</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Brick14: swp-gluster-02:/tank/volume1/brick</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Brick15: swp-gluster-03:/tank/volume1/brick</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Brick16: swp-gluster-04:/tank/volume1/brick</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Brick17: swp-gluster-01:/tank/volume2/brick</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Brick18: swp-gluster-02:/tank/volume2/brick</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Brick19: swp-gluster-03:/tank/volume2/brick</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Brick20: swp-gluster-04:/tank/volume2/brick</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Brick21: swp-gluster-01:/tank/volume3/brick</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Brick22: swp-gluster-02:/tank/volume3/brick</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Brick23: swp-gluster-03:/tank/volume3/brick</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Brick24: swp-gluster-04:/tank/volume3/brick</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">Options Reconfigured:</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">cluster.lookup-unhashed: on</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">cluster.lookup-optimize: off</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">performance.aggregate-size: 1MB</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">performance.read-ahead-page-count: 16</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">performance.nl-cache-timeout: 600</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">performance.nl-cache: on</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">network.inode-lru-limit: 200000</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">performance.md-cache-timeout: 600</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">performance.cache-invalidation: on</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">performance.cache-samba-metadata: on</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">features.cache-invalidation-timeout: 600</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">features.cache-invalidation: on</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">performance.write-behind: off</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">performance.cache-size: 128MB</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">storage.fips-mode-rchecksum: on</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">transport.address-family: inet</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">nfs.disable: on</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">features.inode-quota: on</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">features.quota: on</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">server.event-threads: 32</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">client.event-threads: 16</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">cluster.readdir-optimize: off</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">performance.io-thread-count: 64</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">performance.readdir-ahead: on</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">performance.client-io-threads: on</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">performance.parallel-readdir: disable</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">performance.read-ahead: on</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">performance.stat-prefetch: on</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">performance.open-behind: off</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"><span class="font" style="font-family:"Courier New""><span class="size" style="font-size:10pt">features.quota-deem-statfs: on</span></span><br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"> <br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;">Has anyone gone through this before?<br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;">Thanks in advance.<br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;"> <br></p><p class="" style="margin-top: 0px; margin-bottom: 0px;">Carlo Rodrigues<br></p></div></div>________<br> <br> <br> <br>Community Meeting Calendar: <br> <br>Schedule - <br>Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC <br>Bridge: https://meet.google.com/cpu-eiue-hvk <br>Gluster-users mailing list <br>Gluster-users@gluster.org <br>https://lists.gluster.org/mailman/listinfo/gluster-users <br></blockquote></div></div><br></body></html>