<div dir="ltr"><div><div><div>Greetings Thomas,<br><br></div>I&#39;ll try to assist in determining RC and resolving the issue.<br></div>The following will help me assisting you:<br></div><ol><li>Please create a an issue on GitHub with all the relevant details, it&#39;ll be easier to track</li><li>Please provide all client-side, brick-side and fix-layout logs</li></ol><div>With that information I could begin an initial assessment of the situation.<br><br></div><div>Thank you,<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Nov 8, 2020 at 5:35 PM Thomas Bätzler &lt;<a href="mailto:t.baetzler@bringe.com">t.baetzler@bringe.com</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
the other day we decided to expand our gluster storage by adding two <br>
bricks, going from a 3x2 to a 4x2 distributed-replicated setup. In order <br>
to get to this point we had done rolling upgrades from 3.something to <br>
5.13 to 7.8, all without issues. We ran into a spot of trouble during <br>
the fix-layout when both of the new nodes crashed in the space of two <br>
days. We rebooted them and the fix-layout process seemed to cope by <br>
starting itself again on these nodes.<br>
<br>
Now weird things are happening. We noticed that we can&#39;t access some <br>
files diretly anymore. Doing a stat on them returns a file not found. <br>
However, if we list the directory containing the files, the file is <br>
shown as present and subsequently it can be accessed on the client that <br>
ran the list, but not on other clients! Also, if we umount and remount, <br>
the file is inaccessible again.<br>
<br>
Does anybody have any ideas what&#39;s going on here? Is there any way to <br>
fix the volume without taking it offline for days? We have about 60T of <br>
data online and we need that data to be consistent and available.<br>
<br>
OS is Debian 10 with glusterfs-server 7.8-3.<br>
<br>
Volume configuration:<br>
<br>
Volume Name: hotcache<br>
Type: Distributed-Replicate<br>
Volume ID: 4c006efa-6fd6-4809-93b0-28dd33fee2d2<br>
Status: Started<br>
Snapshot Count: 0<br>
Number of Bricks: 4 x 2 = 8<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: hotcache1:/data/glusterfs/drive1/hotcache<br>
Brick2: hotcache2:/data/glusterfs/drive1/hotcache<br>
Brick3: hotcache3:/data/glusterfs/drive1/hotcache<br>
Brick4: hotcache4:/data/glusterfs/drive1/hotcache<br>
Brick5: hotcache5:/data/glusterfs/drive1/hotcache<br>
Brick6: hotcache6:/data/glusterfs/drive1/hotcache<br>
Brick7: hotcache7:/data/glusterfs/drive1/hotcache<br>
Brick8: hotcache8:/data/glusterfs/drive1/hotcache<br>
Options Reconfigured:<br>
performance.readdir-ahead: off<br>
diagnostics.client-log-level: INFO<br>
diagnostics.brick-log-level: INFO<br>
diagnostics.count-fop-hits: on<br>
diagnostics.latency-measurement: on<br>
server.statedump-path: /var/tmp<br>
diagnostics.brick-sys-log-level: ERROR<br>
storage.linux-aio: on<br>
performance.read-ahead: off<br>
performance.write-behind-window-size: 4MB<br>
performance.cache-max-file-size: 200kb<br>
nfs.disable: on<br>
performance.cache-refresh-timeout: 1<br>
performance.io-cache: on<br>
performance.stat-prefetch: off<br>
performance.quick-read: on<br>
performance.io-thread-count: 16<br>
auth.allow: *<br>
cluster.readdir-optimize: on<br>
performance.flush-behind: off<br>
transport.address-family: inet<br>
cluster.self-heal-daemon: enable<br>
<br>
TIA!<br>
<br>
Best regards,<br>
Thomas Bätzler<br>
-- <br>
BRINGE Informationstechnik GmbH<br>
Zur Seeplatte 12<br>
D-76228 Karlsruhe<br>
Germany<br>
<br>
Fon: +49 721 94246-0<br>
Fon: +49 171 5438457<br>
Fax: +49 721 94246-66<br>
Web: <a href="http://www.bringe.de/" rel="noreferrer" target="_blank">http://www.bringe.de/</a><br>
<br>
Geschäftsführer: Dipl.-Ing. (FH) Martin Bringe<br>
Ust.Id: DE812936645, HRB 108943 Mannheim                        <br>
<br>
________<br>
<br>
<br>
<br>
Community Meeting Calendar:<br>
<br>
Schedule -<br>
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" rel="noreferrer" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div style="margin-left:44px"><div style="margin-bottom:0px;margin-left:0px;padding-bottom:5px"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><font face="RedHatText, sans-serif"><span style="font-size:14px;text-transform:capitalize"><b>Barak Sason Rofman</b></span></font><br></div></div></div></div></div></div></div></div></div></div><div style="margin-left:44px"><div style="margin-bottom:0px;margin-left:0px;padding-bottom:5px"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><p style="margin:0px;font-size:14px;color:rgb(0,0,0);font-family:overpass,sans-serif;font-weight:bold;padding:0px;text-transform:uppercase"><span style="font-family:RedHatText,sans-serif;font-size:12px;text-transform:capitalize;font-weight:normal;color:rgb(34,34,34)">Gluster Storage </span><span style="font-family:RedHatText,sans-serif;font-size:12px;text-transform:capitalize;font-weight:normal;color:rgb(34,34,34)">Development</span></p></div></div></div></div></div></div></div></div></div></div><div style="margin-left:44px"><div style="margin-bottom:0px;margin-left:0px;padding-bottom:5px"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><p style="margin:0px 0px 4px;font-size:12px;font-family:RedHatText,sans-serif"><a href="https://www.redhat.com/" style="color:rgb(0,136,206);margin:0px" target="_blank">Red Hat Israel</a></p><div style="margin-bottom:4px"><p style="margin:0px;font-size:12px;font-family:RedHatText,sans-serif">34 Jerusalem rd. Ra&#39;anana, 43501</p></div><p style="margin:0px;font-size:12px;font-family:RedHatText,sans-serif"><span style="margin:0px;padding:0px">bsasonro<a href="mailto:adi@redhat.com" style="color:rgb(0,0,0);margin:0px" target="_blank">@redhat.com</a>   </span> T: <u>+972-9-7692304</u><br>M: <u>+972-52-4326355</u></p><div style="font-size:12px;font-family:RedHatText,sans-serif;margin-bottom:8px"><a href="https://twitter.com/redhat" title="twitter" style="color:rgb(0,0,0);background:transparent url(&quot;https://marketing-outfit-prod-images.s3-us-west-2.amazonaws.com/3780bd4ede961ef3cd4108b8c0e80186/web-icon-twitter.png&quot;) no-repeat scroll 0px 50%/13px;display:inline-block;line-height:20px;padding-left:13px" target="_blank"><span style="margin-left:2px">@RedHat</span></a>   <a href="https://www.linkedin.com/company/red-hat" title="LinkedIn" style="color:rgb(0,0,0);background:transparent url(&quot;https://marketing-outfit-prod-images.s3-us-west-2.amazonaws.com/8d3507e3c6b6c9ad10e301accf1a4af0/web-icon-linkedin.png&quot;) no-repeat scroll 0px 50%/12px;display:inline-block;line-height:20px;padding-left:12px;margin:0px 4px 0px 3px;padding-top:1px" target="_blank"><span style="margin-left:4px">Red Hat</span></a>  <a href="https://www.facebook.com/redhat.il/" title="Facebook" style="color:rgb(0,0,0);background:transparent url(&quot;https://marketing-outfit-prod-images.s3-us-west-2.amazonaws.com/220b85e2f100025e94cb1bcd993bd51d/web-icon-facebook.png&quot;) no-repeat scroll 0px 50%/11px;display:inline-block;line-height:20px;padding-left:13px" target="_blank">Red Hat</a></div><div style="margin-top:12px"><table border="0"><tbody><tr><td width="100px"><a href="https://red.ht/sig" style="color:rgb(17,85,204)" target="_blank"><img src="https://static.redhat.com/libs/redhat/brand-assets/latest/corp/logo.png" width="90" height="auto"></a></td></tr></tbody></table></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>