<div dir="ltr">Hi Jiri,<div><br></div><div>Unfortunately i don't know a solution to fix it, other than what i already mentioned, which doesn't seem to be applicable to your specific setup.</div><div>I don't think it's ovirt related (running ovirt my-self as well, but being stuck at 4.3 atm, since centos 7 is not supported for 4.4).</div><div>If memory serves me well, i believe i start seeing this issue after upgrading from glusterfs 3.12 to 4.x (I believe this whent together with upgrade from ovirt 4.1 to 4.2), then since is version i've observed this issue. currently running 7.9.</div><div>It would be nice to get to the bottom of this. I'm still not 100% sure it might even be a glusterfs issue, or something might be wrong with XFS or somewhere else in the IO stack. But I don't know what the next debugging steps could be.</div><div>Just as a side note i've also observed this issue on systems without LVM cache.</div><div><br></div><div>Cheers Olaf</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Op do 8 jul. 2021 om 16:53 schreef Jiří Sléžka <<a href="mailto:jiri.slezka@slu.cz">jiri.slezka@slu.cz</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Olaf,<br>
<br>
thanks for reply.<br>
<br>
On 7/8/21 3:29 PM, Olaf Buitelaar wrote:<br>
> Hi Jiri,<br>
> <br>
> your probleem looks pretty similar to mine, see; <br>
> <a href="https://lists.gluster.org/pipermail/gluster-users/2021-February/039134.html" rel="noreferrer" target="_blank">https://lists.gluster.org/pipermail/gluster-users/2021-February/039134.html</a> <br>
> <<a href="https://lists.gluster.org/pipermail/gluster-users/2021-February/039134.html" rel="noreferrer" target="_blank">https://lists.gluster.org/pipermail/gluster-users/2021-February/039134.html</a>><br>
> Any chance you also see the xfs errors in de brick logs?<br>
<br>
yes, I can see this log lines related to "health-check failed" items<br>
<br>
[root@ovirt-hci02 ~]# grep "aio_read" /var/log/glusterfs/bricks/*<br>
/var/log/glusterfs/bricks/gluster_bricks-vms2-vms2.log:[2021-07-07 <br>
07:13:37.408010] W [MSGID: 113075] <br>
[posix-helpers.c:2135:posix_fs_health_check] 0-vms-posix: <br>
aio_read_cmp_buf() on /gluster_bricks/vms2/vms2/.glusterfs/health_check <br>
returned ret is -1 error is Structure needs cleaning<br>
/var/log/glusterfs/bricks/gluster_bricks-vms2-vms2.log:[2021-07-07 <br>
16:11:14.518844] W [MSGID: 113075] <br>
[posix-helpers.c:2135:posix_fs_health_check] 0-vms-posix: <br>
aio_read_cmp_buf() on /gluster_bricks/vms2/vms2/.glusterfs/health_check <br>
returned ret is -1 error is Structure needs cleaning<br>
<br>
[root@ovirt-hci01 ~]# grep "aio_read" /var/log/glusterfs/bricks/*<br>
/var/log/glusterfs/bricks/gluster_bricks-engine-engine.log:[2021-07-05 <br>
13:15:51.982938] W [MSGID: 113075] <br>
[posix-helpers.c:2135:posix_fs_health_check] 0-engine-posix: <br>
aio_read_cmp_buf() on <br>
/gluster_bricks/engine/engine/.glusterfs/health_check returned ret is -1 <br>
error is Structure needs cleaning<br>
/var/log/glusterfs/bricks/gluster_bricks-vms2-vms2.log:[2021-07-05 <br>
01:53:35.768534] W [MSGID: 113075] <br>
[posix-helpers.c:2135:posix_fs_health_check] 0-vms-posix: <br>
aio_read_cmp_buf() on /gluster_bricks/vms2/vms2/.glusterfs/health_check <br>
returned ret is -1 error is Structure needs cleaning<br>
<br>
it looks very similar to your issue but in my case I don't use LVM cache <br>
and brick disks are JBOD (but connected through Broadcom / LSI MegaRAID <br>
SAS-3 3008 [Fury] (rev 02)).<br>
<br>
> For me the situation improved once i disabled brick multiplexing, but i <br>
> don't see that in your volume configuration.<br>
<br>
probably important is your note...<br>
<br>
> When i kill the brick process and start with "gluser v start x force" the<br>
> issue seems much more unlikely to occur, but when started from a fresh<br>
> reboot, or when killing the process and let it being started by glusterd<br>
> (e.g. service glusterd start) the error seems to arise after a couple of<br>
> minutes.<br>
<br>
...because in the ovirt list Jayme replied this<br>
<br>
<a href="https://lists.ovirt.org/archives/list/users@ovirt.org/message/BZRONK53OGWSOPUSGQ76GIXUM7J6HHMJ/" rel="noreferrer" target="_blank">https://lists.ovirt.org/archives/list/users@ovirt.org/message/BZRONK53OGWSOPUSGQ76GIXUM7J6HHMJ/</a><br>
<br>
and it looks to me like something you also observes.<br>
<br>
Cheers, Jiri<br>
<br>
> <br>
> Cheers Olaf<br>
> <br>
> Op do 8 jul. 2021 om 12:28 schreef Jiří Sléžka <<a href="mailto:jiri.slezka@slu.cz" target="_blank">jiri.slezka@slu.cz</a> <br>
> <mailto:<a href="mailto:jiri.slezka@slu.cz" target="_blank">jiri.slezka@slu.cz</a>>>:<br>
> <br>
> Hello gluster community,<br>
> <br>
> I am new to this list but using glusterfs for log time as our SDS<br>
> solution for storing 80+TiB of data. I'm also using glusterfs for small<br>
> 3 node HCI cluster with oVirt 4.4.6 and CentOS 8 (not stream yet).<br>
> Glusterfs version here is 8.5-2.el8.x86_64.<br>
> <br>
> For time to time (I belive) random brick on random host goes down<br>
> because health-check. It looks like<br>
> <br>
> [root@ovirt-hci02 ~]# grep "posix_health_check"<br>
> /var/log/glusterfs/bricks/*<br>
> /var/log/glusterfs/bricks/gluster_bricks-vms2-vms2.log:[2021-07-07<br>
> 07:13:37.408184] M [MSGID: 113075]<br>
> [posix-helpers.c:2214:posix_health_check_thread_proc] 0-vms-posix:<br>
> health-check failed, going down<br>
> /var/log/glusterfs/bricks/gluster_bricks-vms2-vms2.log:[2021-07-07<br>
> 07:13:37.408407] M [MSGID: 113075]<br>
> [posix-helpers.c:2232:posix_health_check_thread_proc] 0-vms-posix:<br>
> still<br>
> alive! -> SIGTERM<br>
> /var/log/glusterfs/bricks/gluster_bricks-vms2-vms2.log:[2021-07-07<br>
> 16:11:14.518971] M [MSGID: 113075]<br>
> [posix-helpers.c:2214:posix_health_check_thread_proc] 0-vms-posix:<br>
> health-check failed, going down<br>
> /var/log/glusterfs/bricks/gluster_bricks-vms2-vms2.log:[2021-07-07<br>
> 16:11:14.519200] M [MSGID: 113075]<br>
> [posix-helpers.c:2232:posix_health_check_thread_proc] 0-vms-posix:<br>
> still<br>
> alive! -> SIGTERM<br>
> <br>
> on other host<br>
> <br>
> [root@ovirt-hci01 ~]# grep "posix_health_check"<br>
> /var/log/glusterfs/bricks/*<br>
> /var/log/glusterfs/bricks/gluster_bricks-engine-engine.log:[2021-07-05<br>
> 13:15:51.983327] M [MSGID: 113075]<br>
> [posix-helpers.c:2214:posix_health_check_thread_proc] 0-engine-posix:<br>
> health-check failed, going down<br>
> /var/log/glusterfs/bricks/gluster_bricks-engine-engine.log:[2021-07-05<br>
> 13:15:51.983728] M [MSGID: 113075]<br>
> [posix-helpers.c:2232:posix_health_check_thread_proc] 0-engine-posix:<br>
> still alive! -> SIGTERM<br>
> /var/log/glusterfs/bricks/gluster_bricks-vms2-vms2.log:[2021-07-05<br>
> 01:53:35.769129] M [MSGID: 113075]<br>
> [posix-helpers.c:2214:posix_health_check_thread_proc] 0-vms-posix:<br>
> health-check failed, going down<br>
> /var/log/glusterfs/bricks/gluster_bricks-vms2-vms2.log:[2021-07-05<br>
> 01:53:35.769819] M [MSGID: 113075]<br>
> [posix-helpers.c:2232:posix_health_check_thread_proc] 0-vms-posix:<br>
> still<br>
> alive! -> SIGTERM<br>
> <br>
> I cannot link these errors to any storage/fs issue (in dmesg or<br>
> /var/log/messages), brick devices looks healthy (smartd).<br>
> <br>
> I can force start brick with<br>
> <br>
> gluster volume start vms|engine force<br>
> <br>
> and after some healing all works fine for few days<br>
> <br>
> Did anybody observe this behavior?<br>
> <br>
> vms volume has this structure (two bricks per host, each is separate<br>
> JBOD ssd disk), engine volume has one brick on each host...<br>
> <br>
> gluster volume info vms<br>
> <br>
> Volume Name: vms<br>
> Type: Distributed-Replicate<br>
> Volume ID: 52032ec6-99d4-4210-8fb8-ffbd7a1e0bf7<br>
> Status: Started<br>
> Snapshot Count: 0<br>
> Number of Bricks: 2 x 3 = 6<br>
> Transport-type: tcp<br>
> Bricks:<br>
> Brick1: 10.0.4.11:/gluster_bricks/vms/vms<br>
> Brick2: 10.0.4.13:/gluster_bricks/vms/vms<br>
> Brick3: 10.0.4.12:/gluster_bricks/vms/vms<br>
> Brick4: 10.0.4.11:/gluster_bricks/vms2/vms2<br>
> Brick5: 10.0.4.13:/gluster_bricks/vms2/vms2<br>
> Brick6: 10.0.4.12:/gluster_bricks/vms2/vms2<br>
> Options Reconfigured:<br>
> cluster.granular-entry-heal: enable<br>
> performance.stat-prefetch: off<br>
> cluster.eager-lock: enable<br>
> performance.io-cache: off<br>
> performance.read-ahead: off<br>
> performance.quick-read: off<br>
> user.cifs: off<br>
> network.ping-timeout: 30<br>
> network.remote-dio: off<br>
> performance.strict-o-direct: on<br>
> performance.low-prio-threads: 32<br>
> features.shard: on<br>
> storage.owner-gid: 36<br>
> storage.owner-uid: 36<br>
> transport.address-family: inet<br>
> storage.fips-mode-rchecksum: on<br>
> nfs.disable: on<br>
> performance.client-io-threads: off<br>
> <br>
> <br>
> Cheers,<br>
> <br>
> Jiri<br>
> <br>
> ________<br>
> <br>
> <br>
> <br>
> Community Meeting Calendar:<br>
> <br>
> Schedule -<br>
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
> Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" rel="noreferrer" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br>
> <<a href="https://meet.google.com/cpu-eiue-hvk" rel="noreferrer" target="_blank">https://meet.google.com/cpu-eiue-hvk</a>><br>
> Gluster-users mailing list<br>
> <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a> <mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>><br>
> <a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
> <<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a>><br>
> <br>
<br>
<br>
________<br>
<br>
<br>
<br>
Community Meeting Calendar:<br>
<br>
Schedule -<br>
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" rel="noreferrer" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div>