<!DOCTYPE html><html><head><title></title><style type="text/css">p.MsoNormal,p.MsoNoSpacing{margin:0}
p.MsoNormal,p.MsoNoSpacing{margin:0}</style></head><body><div style="font-family:Arial;">Yes, I compared the client count like this:<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">gluster volume status <vol> clients |grep -B1 connected<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">I ran the find command on each client before and after shutting down the problematic daemon to determine any file count differences:<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">find /mount/point |wc -l<br></div><div style="font-family:Arial;"><br></div><div style="font-family:Arial;">After my last post I discovered that one of the clients had somehow been blocked by iptables from connecting to one of the bricks. So for an extended period any file creation from that one client was perpetuating an imbalance between bricks, causing different files to be visible for different clients. What baffles me is that gluster wouldn't automatically fix an imbalance between replicas like that.<br></div><div style="font-family:Arial;"><br></div><div>On Fri, Nov 20, 2020, at 1:24 AM, Benedikt Kaleß wrote:<br></div><blockquote type="cite" id="qt" style=""><p>Dear James,<br></p><p>we have exactly the same problems.<br></p><p>Could you describe what you did to discover which of your bricks
had the worst file count discrepancy and how you find out that all
clients matched after shutting down this daemon?<br></p><p><br></p><p>Best regards<br></p><p>Benedikt<br></p><div class="qt-moz-cite-prefix">Am 02.11.20 um 17:30 schrieb James H:<br></div><blockquote type="cite" cite="mid:CAEJhoTE6i6dph487mn37YxX9pTJOwR-5TxeV3qojFRz1r654mw@mail.gmail.com"><div dir="ltr"><span id="qt-m_-8688277350978558537gmail-docs-internal-guid-a486bb90-7fff-6dfc-2c04-017ed7510ebd"><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="color:rgb(0, 0, 0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap;"><span class="font" style="font-family:Arial;"><span class="size" style="font-size:11pt;">I found a solution after making a discovery. I logged into the brick with the worst file count discrepancy - odroid4 - and killed the gluster daemon there. All file counts across all clients then matched. So I started the daemon and ran this command to try to fix it up:</span></span></span><br></p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="color:rgb(0, 0, 0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap;"><span class="font" style="font-family:Arial;"><span class="size" style="font-size:11pt;"></span></span></span><br></p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="color:rgb(0, 0, 0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap;"><span class="font" style="font-family:Arial;"><span class="size" style="font-size:11pt;">gluster volume replace-brick gvol0 odroid4:/srv/gfs-brick/gvol0 odroid4:/srv/gfs-brick/gvol0_2 commit force</span></span></span><br></p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="color:rgb(0, 0, 0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap;"><span class="font" style="font-family:Arial;"><span class="size" style="font-size:11pt;"></span></span></span><br></p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span class="font" style="font-family:Arial;"><span class="colour" style="color:rgb(0, 0, 0);"><span style="white-space:pre-wrap;"><span class="size" style="font-size:14.6667px;">...and that fixed it. It's disconcerting that it's possible for Gluster to merrily hum along without any problems showing up in the various status summaries yet show vastly different directory listings to different clients. Is this a known problem or shall I open a bug report? Are there any particular error logs I should monitor to be alerted to this bad state?</span></span></span></span><br></p></span></div><div><br></div><div class="qt-gmail_quote"><div dir="ltr" class="qt-gmail_attr">On Thu, Oct 29, 2020 at 8:39
PM James H <<a href="mailto:stormdig@gmail.com" target="_blank">stormdig@gmail.com</a>>
wrote:<br></div><blockquote class="qt-gmail_quote" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204, 204, 204);padding-left:1ex;"><div dir="ltr"><div>Hi folks, I'm struggling to find a solution to missing
files on FUSE mounts. Which files are missing is different
on different clients. I can stat or ls the missing files
directly when called by filename but listing directories
won't show them.<br></div><div><br></div><div>So far I've:<br></div><div><ul><li>verified heal info shows no files in need of healing
and no split brain condition<br></li><li>verified the same number of clients are connected to
each brick <br></li><li>verified the file counts on the bricks match<br></li><li>upgraded Gluster server and clients from 3.x to 6.x
and 7.x<br></li><li>run a stat on all files<br></li><li>run a heal full<br></li><li>rebooted / remounted FUSE clients<br></li></ul></div><div>File count from running a 'find' command on FUSE mounts
on the bricks themselves. These counts should all be the
same:<br></div><div><b>38823 </b>fuse-odroid1-share2 <br></div><div><b>38823 </b>fuse-odroid2-share2<br></div><div><b>60962 </b>fuse-odroid3-share2<br></div><div><b>7202 </b>fuse-odroid4-share2<br></div><div><br></div><div>...and a FUSE mount on a seperate server:<br></div><div><b>38823 </b>fuse-phn2dsm-share2 <br></div><div><br></div><div>File count from running a 'find' command on brick
directories themselves:: <br></div><div><b>43382 </b>brick-odroid1-share2<br></div><div><b>43382 </b>brick-odroid2-share2<br></div><div><b>43382 </b>brick-arbiter-odroid3-share2<br></div><div><b>23075 </b>brick-odroid3-share2<br></div><div><b>23075 </b>brick-odroid4-share2<br></div><div><b>23075 </b>brick-arbiter-odroid2-share2<br></div><div><br></div><div>Here's some info about the setup:<br></div><div><br></div><div><div><b># gluster --version | head -1; cat /etc/lsb-release;
uname -r</b><br></div><div>glusterfs 7.8<br></div><div>DISTRIB_ID=Ubuntu<br></div><div>DISTRIB_RELEASE=18.04<br></div><div>DISTRIB_CODENAME=bionic<br></div><div>DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"<br></div><div>4.14.157-171<br></div><div><br></div><div><br></div><div><b># gluster volume info</b><br></div><div><div>Volume Name: gvol0<br></div><div>Type: Distributed-Replicate<br></div><div>Volume ID: 57e3a085-5fb7-417d-a71a-fed5cd0ae2d9<br></div><div>Status: Started<br></div><div>Snapshot Count: 0<br></div><div>Number of Bricks: 2 x (2 + 1) = 6<br></div><div>Transport-type: tcp<br></div><div>Bricks:<br></div><div>Brick1: odroid1:/srv/gfs-brick/gvol0<br></div><div>Brick2: odroid2:/srv/gfs-brick/gvol0<br></div><div>Brick3: odroid3:/srv/gfs-brick/gvol0-arbiter2 (arbiter)<br></div><div>Brick4: odroid3:/srv/gfs-brick/gvol0_2<br></div><div>Brick5: odroid4:/srv/gfs-brick/gvol0<br></div><div>Brick6: odroid2:/srv/gfs-brick/gvol0-arbiter2 (arbiter)<br></div><div>Options Reconfigured:<br></div><div>cluster.self-heal-daemon: enable<br></div><div>performance.readdir-ahead: yes<br></div><div>performance.cache-invalidation: on<br></div><div>performance.stat-prefetch: on<br></div><div>performance.quick-read: on<br></div><div>cluster.shd-max-threads: 4<br></div><div>performance.parallel-readdir: on<br></div><div>cluster.server-quorum-type: server<br></div><div>server.event-threads: 4<br></div><div>client.event-threads: 4<br></div><div>performance.nl-cache-timeout: 600<br></div><div>performance.nl-cache: on<br></div><div>network.inode-lru-limit: 200000<br></div><div>performance.md-cache-timeout: 600<br></div><div>performance.cache-samba-metadata: on<br></div><div>features.cache-invalidation-timeout: 600<br></div><div>features.cache-invalidation: on<br></div><div>storage.fips-mode-rchecksum: on<br></div><div>performance.client-io-threads: off<br></div><div>nfs.disable: on<br></div><div>transport.address-family: inet<br></div><div>features.bitrot: on<br></div><div>features.scrub: Active<br></div><div>features.scrub-throttle: lazy<br></div><div>features.scrub-freq: daily<br></div><div>cluster.min-free-disk: 10%<br></div></div></div><div><br></div><div><div><b># gluster volume status gvol0 detail</b><br></div><div>Status of volume: gvol0<br></div><div>------------------------------------------------------------------------------<br></div><div>Brick : Brick odroid1:/srv/gfs-brick/gvol0<br></div><div>TCP Port : 49152<br></div><div>RDMA Port : 0<br></div><div>Online : Y<br></div><div>Pid : 702<br></div><div>File System : xfs<br></div><div>Device : /dev/sda<br></div><div>Mount Options :
rw,noatime,nouuid,attr2,inode64,sunit=256,swidth=2560,noquota<br></div><div>Inode Size : 512<br></div><div>Disk Space Free : 983.4GB<br></div><div>Total Disk Space : 5.5TB<br></div><div>Inode Count : 586052224<br></div><div>Free Inodes : 585835873<br></div><div>------------------------------------------------------------------------------<br></div><div>Brick : Brick odroid2:/srv/gfs-brick/gvol0<br></div><div>TCP Port : 49152<br></div><div>RDMA Port : 0<br></div><div>Online : Y<br></div><div>Pid : 30206<br></div><div>File System : xfs<br></div><div>Device : /dev/sda<br></div><div>Mount Options :
rw,noatime,nouuid,attr2,inode64,sunit=256,swidth=2560,noquota<br></div><div>Inode Size : 512<br></div><div>Disk Space Free : 983.3GB<br></div><div>Total Disk Space : 5.5TB<br></div><div>Inode Count : 586052224<br></div><div>Free Inodes : 585711242<br></div><div>------------------------------------------------------------------------------<br></div><div>Brick : Brick
odroid3:/srv/gfs-brick/gvol0-arbiter2<br></div><div>TCP Port : 49152<br></div><div>RDMA Port : 0<br></div><div>Online : Y<br></div><div>Pid : 32449<br></div><div>File System : xfs<br></div><div>Device : /dev/sda<br></div><div>Mount Options :
rw,noatime,nouuid,attr2,inode64,sunit=256,swidth=2560,noquota<br></div><div>Inode Size : 512<br></div><div>Disk Space Free : 1.4TB<br></div><div>Total Disk Space : 2.7TB<br></div><div>Inode Count : 293026624<br></div><div>Free Inodes : 292378835<br></div><div>------------------------------------------------------------------------------<br></div><div>Brick : Brick
odroid3:/srv/gfs-brick/gvol0_2<br></div><div>TCP Port : 49153<br></div><div>RDMA Port : 0<br></div><div>Online : Y<br></div><div>Pid : 32474<br></div><div>File System : xfs<br></div><div>Device : /dev/sda<br></div><div>Mount Options :
rw,noatime,nouuid,attr2,inode64,sunit=256,swidth=2560,noquota<br></div><div>Inode Size : 512<br></div><div>Disk Space Free : 1.4TB<br></div><div>Total Disk Space : 2.7TB<br></div><div>Inode Count : 293026624<br></div><div>Free Inodes : 292378835<br></div><div>------------------------------------------------------------------------------<br></div><div>Brick : Brick odroid4:/srv/gfs-brick/gvol0<br></div><div>TCP Port : 49152<br></div><div>RDMA Port : 0<br></div><div>Online : Y<br></div><div>Pid : 23138<br></div><div>File System : xfs<br></div><div>Device : /dev/sda<br></div><div>Mount Options :
rw,noatime,nouuid,attr2,inode64,sunit=256,swidth=2560,noquota<br></div><div>Inode Size : 512<br></div><div>Disk Space Free : 1.4TB<br></div><div>Total Disk Space : 2.7TB<br></div><div>Inode Count : 293026624<br></div><div>Free Inodes : 292891910<br></div><div>------------------------------------------------------------------------------<br></div><div>Brick : Brick
odroid2:/srv/gfs-brick/gvol0-arbiter2<br></div><div>TCP Port : 49153<br></div><div>RDMA Port : 0<br></div><div>Online : Y<br></div><div>Pid : 30231<br></div><div>File System : xfs<br></div><div>Device : /dev/sda<br></div><div>Mount Options :
rw,noatime,nouuid,attr2,inode64,sunit=256,swidth=2560,noquota<br></div><div>Inode Size : 512<br></div><div>Disk Space Free : 983.3GB<br></div><div>Total Disk Space : 5.5TB<br></div><div>Inode Count : 586052224<br></div><div>Free Inodes : 585711242<br></div></div></div></blockquote></div><div><br></div><pre class="qt-moz-quote-pre">________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: <a class="qt-moz-txt-link-freetext" href="https://bluejeans.com/441850968">https://bluejeans.com/441850968</a>
Gluster-users mailing list
<a class="qt-moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="qt-moz-txt-link-freetext" href="https://lists.gluster.org/mailman/listinfo/gluster-users">https://lists.gluster.org/mailman/listinfo/gluster-users</a>
<br></pre></blockquote><pre class="qt-moz-signature" cols="72">--
forumZFD
Entschieden für Frieden|Committed to Peace
Benedikt Kaleß
Leiter Team IT|Head team IT
Forum Ziviler Friedensdienst e.V.|Forum Civil Peace Service
Am Kölner Brett 8 | 50825 Köln | Germany
Tel 0221 91273233 | Fax 0221 91273299 |
<a class="qt-moz-txt-link-freetext" href="http://www.forumZFD.de">http://www.forumZFD.de</a>
Vorstand nach § 26 BGB, einzelvertretungsberechtigt|Executive Board:
Oliver Knabe (Vorsitz|Chair), Sonja Wiekenberg-Mlalandle, Alexander Mauz
VR 17651 Amtsgericht Köln
Spenden|Donations: IBAN DE37 3702 0500 0008 2401 01 BIC BFSWDE33XXX<br></pre></blockquote><div style="font-family:Arial;"><br></div></body></html>