<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Dear Darrell,</p>
<p>Dear ...,<br>
<br>
Many thanks for the prompt reply. Here some of the additional
information requested (please feel free to ask for more if needed)<br>
<br>
<blockquote type="cite">CPU info:<br>
Hosts<br>
1. Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz (20 cores) hw RAID
1 (adaptec)<br>
2. Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz (20 cores) hw RAID
1 (adaptec)<br>
3. Intel(R) Xeon(R) Silver 4112 CPU @ 2.60GHz (8 cores) hw RAID
1 (adaptec)<br>
<br>
GlusterFS VMs<br>
1. 4 cores, 10 GB RAM, CPU model Broadwell<br>
2. 4 cores, 10 GB RAM, CPU model Broadwell<br>
3. 4 cores, 10 GB RAM, CPU model: host passthrough value<br>
<br>
Network info<br>
Physical connection between gluster nodes in a heartbeat network
that comprises a new Cisco switch.<br>
TCP connections with 1Gbit links<br>
Virtual default connectivity with virtio drivers for the NICs
and Macvtap connection to use the host´s connectivity.<br>
No errors or lost packages are recorded between the VMs or the
hosts. Quick iperf tests (between glusters and ctdbs-glusters
show now evident issues).<br>
</blockquote>
<blockquote type="cite">
<br>
Workload:<br>
An instantaneous from this moment which can be considered a peak
time is around 450 files open on the volume.<br>
In terms of "litteral" load we notice cpu peaks mostly related
to the shd process<br>
</blockquote>
<br>
All disks use virtIO drivers (virtIO disks).<br>
<br>
The file system on all nodes is XFS (not ZFS)<br>
<br>
Other than the clients on the gluster nodes themselves there are
clients on ctdbs that mount the gluster volume and then expose it
via smb to Windows clients (user profiles included for roaming
profiles).<br>
ctdbs reach the glusters through the heartbeat network<br>
<br>
We are considering to move the glusters to a network with existing
DNS capabilities in order to create a round-robin configuration by
assigning hosts by IP to a single hostname to use it then for the
mounts configuration of the ctdbs.<br>
The reasoning/hope behind that we would minimize access time and
sync issues.<br>
<br>
Thank you for the information about the "sharding" we will take
this into account and consider pros and cons in the current
situation, epsecially because turning back is not easy afterwards.
Also our main problem is mainly not with big files, but with a
large quantity of small files.<br>
<br>
We could gladly make use of some of the options you suggested
after we assess the situation again. We welcome any further
suggestion in the meantime.</p>
<p>Ilias<br>
</p>
<div class="moz-cite-prefix">Am 09.04.24 um 18:26 schrieb Darrell
Budic:<br>
</div>
<blockquote type="cite"
cite="mid:FDFC289F-5232-4D84-8CDB-BA10A3BAB2AB@onholyground.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
The big one I see of you is to investigate and enable sharding. It
can improve performance and makes it much easier to heal VM style
workloads. Be aware that once you turn it on, you can’t go back
easily, and you need to copy the VM disk images around to get them
to be sharded before it will show any real effect. A couple other
recommendations from my main volume (three dedicated host servers
with HDDs and SDD/NVM caching and log volumes o<span
style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">n ZFS </span>).
The cluster.shd-* entries are especially recommended. This is on
gluster 9.4 at the moment, so some of these won’t map exactly.
<div><br>
</div>
<div>
<p
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">Volume
Name: gv1</span></p>
<p
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">Type:
Replicate</span></p>
<p
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);">Number
of Bricks: 1 x 3 = 3</p>
<p
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">Transport-type:
tcp</span></p>
<p
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);">Options
Reconfigured:</p>
<p
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">cluster.read-hash-mode:
3</span></p>
<p
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">performance.client-io-threads:
on</span></p>
<p
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">performance.write-behind-window-size:
64MB</span></p>
<p
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">performance.cache-size:
1G</span></p>
<p
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);">nfs.disable:
on</p>
<p
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">performance.readdir-ahead:
on</span></p>
<p
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">performance.quick-read:
off</span></p>
<p
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">performance.read-ahead:
on</span></p>
<p
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">performance.io-cache:
off</span></p>
<p
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">performance.stat-prefetch:
on</span></p>
<p
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">cluster.eager-lock:
enable</span></p>
<p
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">network.remote-dio:
enable</span></p>
<p
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);">server.event-threads:
4</p>
<p
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">client.event-threads:
8</span></p>
<p
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">performance.io-thread-count:
64</span></p>
<p
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">performance.low-prio-threads:
32</span></p>
<p
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">features.shard:
on</span></p>
<p
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">features.shard-block-size:
64MB</span></p>
<p
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">cluster.locking-scheme:
granular</span></p>
<p
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);">cluster.data-self-heal-algorithm:
full</p>
<p
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">cluster.shd-max-threads:
8</span></p>
<p
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">cluster.shd-wait-qlength:
10240</span></p>
<p
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">cluster.choose-local:
false</span></p>
<p
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">cluster.granular-entry-heal:
enable</span></p>
<p
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><br>
</p>
<div>Otherwise, more details about your servers, CPU, RAM, and
Disks would be useful for suggestions, and details of your
network as well. And if you haven’t done kernel level tuning
on the servers, you should address that as well. These all
vary a lot by your work load and hardware setup, so there
aren’t many generic recommendations I can give other than to
make sure you tuned your tcp stack and enabled the none disk
elevator on SSDs or disks used by ZFS. </div>
<div><br>
</div>
<div>There’s a lot of tuning suggesting in the archives if you
go searching as well.</div>
<div><br>
</div>
<div> -Darrell</div>
<div><br>
</div>
<div><br>
<blockquote type="cite">
<div>On Apr 9, 2024, at 3:05 AM, Ilias Chasapakis forumZFD
<a class="moz-txt-link-rfc2396E" href="mailto:chasapakis@forumZFD.de"><chasapakis@forumZFD.de></a> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div>Dear all,<br>
<br>
we would like to describe the situation that we have and
that does not solve since a long time, that means after
many minor<br>
and major upgrades of GlusterFS<br>
<br>
We use a KVM environment for VMs for glusterfs and host
servers are updated regularly. Hosts are disomogeneous
hardware,<br>
but configured with same characteristics.<br>
<br>
The VMs have been also harmonized to use the virtio
drivers where available for devices and resources
reserved are the same<br>
on each host.<br>
<br>
Physical switch for hosts has been substituted with a
reliable one.<br>
<br>
Probing peers has been and is quite quick in the
heartbeat network and communication between the servers
for apparently has no issues on disruptions.<br>
<br>
And I say apparently because what we have is:<br>
<br>
- always pending failed heals that used to resolve by a
rotated reboot of the gluster vms (replica 3).
Restarting only<br>
glusterfs related services (daemon, events etc.) has no
effect, only reboot brings results<br>
- very often failed heals are directories<br>
<br>
We lately removed a brick that was on a vm on a host
that has been entirely substituted. Re-added the brick,
sync went on and<br>
all data was eventually synced and started with 0
pending failed heals. Now it develops failed heals too
like its fellow<br>
bricks. Please take into account we healed all the
failed entries (manually with various methods) before
adding the third brick.<br>
<br>
After some days of operating, the count of failed heals
rises again, not really fast but with new entries for
sure (which might solve<br>
with rotated reboots, or not).<br>
<br>
We have gluster clients also on ctdbs that connect to
the gluster and mount via glusterfs client. Windows
roaming profiles shared via smb become frequently
corrupted,(they are composed of a great number small
files and are though of big total dimension). Gluster
nodes are formatted with xfs.<br>
<br>
Also what we observer is that mounting with the vfs
option in smb on the ctdbs has some kind of delay. This
means that you can see the shared folder on for example<br>
a Windows client machine on a ctdb, but not on another
ctdb in the cluster and then after a while it appears
there too. And this frequently st<br>
<br>
<br>
This is an excerpt of entries on our shd logs:<br>
<br>
<blockquote type="cite">2024-04-08 10:13:26.213596
+0000] I [MSGID: 108026]
[afr-self-heal-entry.c:1080:afr_selfheal_entry_do]
0-gv-ho-replicate-0: performing full entry selfheal on
2c621415-6223-4b66-a4ca-3f6f267a448d<br>
[2024-04-08 10:14:08.135911 +0000] W [MSGID: 114031]
[client-rpc-fops_v2.c:2457:client4_0_link_cbk]
0-gv-ho-client-5: remote operation failed.
[{source=<gfid:91d83f0e-1864-4ff3-9174-b7c956e20596>},
{target=(null)}, {errno=116}, {error=Veraltete
Dateizugriffsnummer (file handle)}]<br>
[2024-04-08 10:15:59.135908 +0000] W [MSGID: 114061]
[client-common.c:2992:client_pre_readdir_v2]
0-gv-ho-client-5: remote_fd is -1. EBADFD
[{gfid=6b5e599e-c836-4ebe-b16a-8224425b88c7},
{errno=77}, {error=Die Dateizugriffsnummer ist in
schlechter Verfassung}]<br>
[2024-04-08 10:30:25.013592 +0000] I [MSGID: 108026]
[afr-self-heal-entry.c:1080:afr_selfheal_entry_do]
0-gv-ho-replicate-0: performing full entry selfheal on
24e82e12-5512-4679-9eb3-8bd098367db7<br>
[2024-04-08 10:33:17.613594 +0000] W [MSGID: 114031]
[client-rpc-fops_v2.c:2457:client4_0_link_cbk]
0-gv-ho-client-5: remote operation failed.
[{source=<gfid:ef9068fc-a329-4a21-88d2-265ecd3d208c>},
{target=(null)}, {errno=116}, {error=Veraltete
Dateizugriffsnummer (file handle)}]<br>
[2024-04-08 10:33:21.201359 +0000] W [MSGID: 114031]
[client-rpc-fops_v2.c:2457:client4_0_link_cbk]
0-gv-ho-client-5: remote operation failed. [{source=<br>
</blockquote>
<br>
How are he clients mapped to real hosts in order to know
on which one´s logs to look at?<br>
<br>
We would like to go by exclusion to finally eradicate
this, possibly in a conservative way (not rebuilding
everything) and we<br>
<br>
are becoming clueless as to where to look at as we also
tried various options settings regarding performance
etc.<br>
<br>
Here is the set on our main volume:<br>
<br>
<blockquote type="cite">cluster.lookup-unhashed
on (DEFAULT)<br>
cluster.lookup-optimize on (DEFAULT)<br>
cluster.min-free-disk 10% (DEFAULT)<br>
cluster.min-free-inodes 5% (DEFAULT)<br>
cluster.rebalance-stats off (DEFAULT)<br>
cluster.subvols-per-directory (null)
(DEFAULT)<br>
cluster.readdir-optimize off (DEFAULT)<br>
cluster.rsync-hash-regex (null)
(DEFAULT)<br>
cluster.extra-hash-regex (null)
(DEFAULT)<br>
cluster.dht-xattr-name
trusted.glusterfs.dht (DEFAULT)<br>
cluster.randomize-hash-range-by-gfid off (DEFAULT)<br>
cluster.rebal-throttle normal
(DEFAULT)<br>
cluster.lock-migration off<br>
cluster.force-migration off<br>
cluster.local-volume-name (null)
(DEFAULT)<br>
cluster.weighted-rebalance on (DEFAULT)<br>
cluster.switch-pattern (null)
(DEFAULT)<br>
cluster.entry-change-log on (DEFAULT)<br>
cluster.read-subvolume (null)
(DEFAULT)<br>
cluster.read-subvolume-index -1 (DEFAULT)<br>
cluster.read-hash-mode 1 (DEFAULT)<br>
cluster.background-self-heal-count 8 (DEFAULT)<br>
cluster.metadata-self-heal on<br>
cluster.data-self-heal on<br>
cluster.entry-self-heal on<br>
cluster.self-heal-daemon enable<br>
cluster.heal-timeout 600 (DEFAULT)<br>
cluster.self-heal-window-size 8 (DEFAULT)<br>
cluster.data-change-log on (DEFAULT)<br>
cluster.metadata-change-log on (DEFAULT)<br>
cluster.data-self-heal-algorithm (null)
(DEFAULT)<br>
cluster.eager-lock on (DEFAULT)<br>
disperse.eager-lock on (DEFAULT)<br>
disperse.other-eager-lock on (DEFAULT)<br>
disperse.eager-lock-timeout 1 (DEFAULT)<br>
disperse.other-eager-lock-timeout 1 (DEFAULT)<br>
cluster.quorum-type auto<br>
cluster.quorum-count 2<br>
cluster.choose-local true
(DEFAULT)<br>
cluster.self-heal-readdir-size 1KB (DEFAULT)<br>
cluster.post-op-delay-secs 1 (DEFAULT)<br>
cluster.ensure-durability on (DEFAULT)<br>
cluster.consistent-metadata no (DEFAULT)<br>
cluster.heal-wait-queue-length 128 (DEFAULT)<br>
cluster.favorite-child-policy none<br>
cluster.full-lock yes (DEFAULT)<br>
cluster.optimistic-change-log on (DEFAULT)<br>
diagnostics.latency-measurement off<br>
diagnostics.dump-fd-stats off (DEFAULT)<br>
diagnostics.count-fop-hits off<br>
diagnostics.brick-log-level INFO<br>
diagnostics.client-log-level INFO<br>
diagnostics.brick-sys-log-level CRITICAL
(DEFAULT)<br>
diagnostics.client-sys-log-level CRITICAL
(DEFAULT)<br>
diagnostics.brick-logger (null)
(DEFAULT)<br>
diagnostics.client-logger (null)
(DEFAULT)<br>
diagnostics.brick-log-format (null)
(DEFAULT)<br>
diagnostics.client-log-format (null)
(DEFAULT)<br>
diagnostics.brick-log-buf-size 5 (DEFAULT)<br>
diagnostics.client-log-buf-size 5 (DEFAULT)<br>
diagnostics.brick-log-flush-timeout 120 (DEFAULT)<br>
diagnostics.client-log-flush-timeout 120 (DEFAULT)<br>
diagnostics.stats-dump-interval 0 (DEFAULT)<br>
diagnostics.fop-sample-interval 0 (DEFAULT)<br>
diagnostics.stats-dump-format json
(DEFAULT)<br>
diagnostics.fop-sample-buf-size 65535
(DEFAULT)<br>
diagnostics.stats-dnscache-ttl-sec 86400
(DEFAULT)<br>
performance.cache-max-file-size 10<br>
performance.cache-min-file-size 0 (DEFAULT)<br>
performance.cache-refresh-timeout 1 (DEFAULT)<br>
performance.cache-priority (DEFAULT)<br>
performance.io-cache-size 32MB
(DEFAULT)<br>
performance.cache-size 32MB
(DEFAULT)<br>
performance.io-thread-count 16 (DEFAULT)<br>
performance.high-prio-threads 16 (DEFAULT)<br>
performance.normal-prio-threads 16 (DEFAULT)<br>
performance.low-prio-threads 16 (DEFAULT)<br>
performance.least-prio-threads 1 (DEFAULT)<br>
performance.enable-least-priority on (DEFAULT)<br>
performance.iot-watchdog-secs (null)
(DEFAULT)<br>
performance.iot-cleanup-disconnected-reqs off
(DEFAULT)<br>
performance.iot-pass-through false
(DEFAULT)<br>
performance.io-cache-pass-through false
(DEFAULT)<br>
performance.quick-read-cache-size 128MB
(DEFAULT)<br>
performance.cache-size 128MB
(DEFAULT)<br>
performance.quick-read-cache-timeout 1 (DEFAULT)<br>
performance.qr-cache-timeout 600<br>
performance.quick-read-cache-invalidation false
(DEFAULT)<br>
performance.ctime-invalidation false
(DEFAULT)<br>
performance.flush-behind on (DEFAULT)<br>
performance.nfs.flush-behind on (DEFAULT)<br>
performance.write-behind-window-size 4MB<br>
performance.resync-failed-syncs-after-fsync off
(DEFAULT)<br>
performance.nfs.write-behind-window-size 1MB (DEFAULT)<br>
performance.strict-o-direct off (DEFAULT)<br>
performance.nfs.strict-o-direct off (DEFAULT)<br>
performance.strict-write-ordering off (DEFAULT)<br>
performance.nfs.strict-write-ordering off (DEFAULT)<br>
performance.write-behind-trickling-writes on (DEFAULT)<br>
performance.aggregate-size 128KB
(DEFAULT)<br>
performance.nfs.write-behind-trickling-writes on
(DEFAULT)<br>
performance.lazy-open yes (DEFAULT)<br>
performance.read-after-open yes (DEFAULT)<br>
performance.open-behind-pass-through false
(DEFAULT)<br>
performance.read-ahead-page-count 4 (DEFAULT)<br>
performance.read-ahead-pass-through false
(DEFAULT)<br>
performance.readdir-ahead-pass-through false
(DEFAULT)<br>
performance.md-cache-pass-through false
(DEFAULT)<br>
performance.write-behind-pass-through false
(DEFAULT)<br>
performance.md-cache-timeout 600<br>
performance.cache-swift-metadata false
(DEFAULT)<br>
performance.cache-samba-metadata on<br>
performance.cache-capability-xattrs true
(DEFAULT)<br>
performance.cache-ima-xattrs true
(DEFAULT)<br>
performance.md-cache-statfs off (DEFAULT)<br>
performance.xattr-cache-list (DEFAULT)<br>
performance.nl-cache-pass-through false
(DEFAULT)<br>
network.frame-timeout 1800
(DEFAULT)<br>
network.ping-timeout 20<br>
network.tcp-window-size (null)
(DEFAULT)<br>
client.ssl off<br>
network.remote-dio disable
(DEFAULT)<br>
client.event-threads 4<br>
client.tcp-user-timeout 0<br>
client.keepalive-time 20<br>
client.keepalive-interval 2<br>
client.keepalive-count 9<br>
client.strict-locks off<br>
network.tcp-window-size (null)
(DEFAULT)<br>
network.inode-lru-limit 200000<br>
auth.allow *<br>
auth.reject (null)
(DEFAULT)<br>
transport.keepalive 1<br>
server.allow-insecure on (DEFAULT)<br>
server.root-squash off (DEFAULT)<br>
server.all-squash off (DEFAULT)<br>
server.anonuid 65534
(DEFAULT)<br>
server.anongid 65534
(DEFAULT)<br>
server.statedump-path
/var/run/gluster (DEFAULT)<br>
server.outstanding-rpc-limit 64 (DEFAULT)<br>
server.ssl off<br>
auth.ssl-allow *<br>
server.manage-gids off (DEFAULT)<br>
server.dynamic-auth on (DEFAULT)<br>
client.send-gids on (DEFAULT)<br>
server.gid-timeout 300 (DEFAULT)<br>
server.own-thread (null)
(DEFAULT)<br>
server.event-threads 4<br>
server.tcp-user-timeout 42 (DEFAULT)<br>
server.keepalive-time 20<br>
server.keepalive-interval 2<br>
server.keepalive-count 9<br>
transport.listen-backlog 1024<br>
ssl.own-cert (null)
(DEFAULT)<br>
ssl.private-key (null)
(DEFAULT)<br>
ssl.ca-list (null)
(DEFAULT)<br>
ssl.crl-path (null)
(DEFAULT)<br>
ssl.certificate-depth (null)
(DEFAULT)<br>
ssl.cipher-list (null)
(DEFAULT)<br>
ssl.dh-param (null)
(DEFAULT)<br>
ssl.ec-curve (null)
(DEFAULT)<br>
transport.address-family inet<br>
performance.write-behind off<br>
performance.read-ahead on<br>
performance.readdir-ahead on<br>
performance.io-cache off<br>
performance.open-behind on<br>
performance.quick-read on<br>
performance.nl-cache on<br>
performance.stat-prefetch on<br>
performance.client-io-threads off<br>
performance.nfs.write-behind on<br>
performance.nfs.read-ahead off<br>
performance.nfs.io-cache off<br>
performance.nfs.quick-read off<br>
performance.nfs.stat-prefetch off<br>
performance.nfs.io-threads off<br>
performance.force-readdirp true
(DEFAULT)<br>
performance.cache-invalidation on<br>
performance.global-cache-invalidation true
(DEFAULT)<br>
features.uss off<br>
features.snapshot-directory .snaps<br>
features.show-snapshot-directory off<br>
features.tag-namespaces off<br>
network.compression off<br>
network.compression.window-size -15 (DEFAULT)<br>
network.compression.mem-level 8 (DEFAULT)<br>
network.compression.min-size 0 (DEFAULT)<br>
network.compression.compression-level -1 (DEFAULT)<br>
network.compression.debug false
(DEFAULT)<br>
features.default-soft-limit 80% (DEFAULT)<br>
features.soft-timeout 60 (DEFAULT)<br>
features.hard-timeout 5 (DEFAULT)<br>
features.alert-time 86400
(DEFAULT)<br>
features.quota-deem-statfs off<br>
geo-replication.indexing off<br>
geo-replication.indexing off<br>
geo-replication.ignore-pid-check off<br>
geo-replication.ignore-pid-check off<br>
features.quota off<br>
features.inode-quota off<br>
features.bitrot disable<br>
debug.trace off<br>
debug.log-history no (DEFAULT)<br>
debug.log-file no (DEFAULT)<br>
debug.exclude-ops (null)
(DEFAULT)<br>
debug.include-ops (null)
(DEFAULT)<br>
debug.error-gen off<br>
debug.error-failure (null)
(DEFAULT)<br>
debug.error-number (null)
(DEFAULT)<br>
debug.random-failure off (DEFAULT)<br>
debug.error-fops (null)
(DEFAULT)<br>
nfs.disable on<br>
features.read-only off (DEFAULT)<br>
features.worm off<br>
features.worm-file-level off<br>
features.worm-files-deletable on<br>
features.default-retention-period 120 (DEFAULT)<br>
features.retention-mode relax
(DEFAULT)<br>
features.auto-commit-period 180 (DEFAULT)<br>
storage.linux-aio off (DEFAULT)<br>
storage.linux-io_uring off (DEFAULT)<br>
storage.batch-fsync-mode reverse-fsync
(DEFAULT)<br>
storage.batch-fsync-delay-usec 0 (DEFAULT)<br>
storage.owner-uid -1 (DEFAULT)<br>
storage.owner-gid -1 (DEFAULT)<br>
storage.node-uuid-pathinfo off (DEFAULT)<br>
storage.health-check-interval 30 (DEFAULT)<br>
storage.build-pgfid off (DEFAULT)<br>
storage.gfid2path on (DEFAULT)<br>
storage.gfid2path-separator : (DEFAULT)<br>
storage.reserve 1 (DEFAULT)<br>
storage.health-check-timeout 20 (DEFAULT)<br>
storage.fips-mode-rchecksum on<br>
storage.force-create-mode 0000
(DEFAULT)<br>
storage.force-directory-mode 0000
(DEFAULT)<br>
storage.create-mask 0777
(DEFAULT)<br>
storage.create-directory-mask 0777
(DEFAULT)<br>
storage.max-hardlinks 100 (DEFAULT)<br>
features.ctime on (DEFAULT)<br>
config.gfproxyd off<br>
cluster.server-quorum-type server<br>
cluster.server-quorum-ratio 51<br>
changelog.changelog off (DEFAULT)<br>
changelog.changelog-dir {{ brick.path
}}/.glusterfs/changelogs (DEFAULT)<br>
changelog.encoding ascii
(DEFAULT)<br>
changelog.rollover-time 15 (DEFAULT)<br>
changelog.fsync-interval 5 (DEFAULT)<br>
changelog.changelog-barrier-timeout 120<br>
changelog.capture-del-path off (DEFAULT)<br>
features.barrier disable<br>
features.barrier-timeout 120<br>
features.trash off (DEFAULT)<br>
features.trash-dir .trashcan
(DEFAULT)<br>
features.trash-eliminate-path (null)
(DEFAULT)<br>
features.trash-max-filesize 5MB (DEFAULT)<br>
features.trash-internal-op off (DEFAULT)<br>
cluster.enable-shared-storage disable<br>
locks.trace off (DEFAULT)<br>
locks.mandatory-locking off (DEFAULT)<br>
cluster.disperse-self-heal-daemon enable
(DEFAULT)<br>
cluster.quorum-reads no (DEFAULT)<br>
client.bind-insecure (null)
(DEFAULT)<br>
features.timeout 45 (DEFAULT)<br>
features.failover-hosts (null)
(DEFAULT)<br>
features.shard off<br>
features.shard-block-size 64MB
(DEFAULT)<br>
features.shard-lru-limit 16384
(DEFAULT)<br>
features.shard-deletion-rate 100 (DEFAULT)<br>
features.scrub-throttle lazy<br>
features.scrub-freq biweekly<br>
features.scrub false
(DEFAULT)<br>
features.expiry-time 120<br>
features.signer-threads 4<br>
features.cache-invalidation on<br>
features.cache-invalidation-timeout 600<br>
ganesha.enable off<br>
features.leases off<br>
features.lease-lock-recall-timeout 60 (DEFAULT)<br>
disperse.background-heals 8 (DEFAULT)<br>
disperse.heal-wait-qlength 128 (DEFAULT)<br>
cluster.heal-timeout 600 (DEFAULT)<br>
dht.force-readdirp on (DEFAULT)<br>
disperse.read-policy gfid-hash
(DEFAULT)<br>
cluster.shd-max-threads 4<br>
cluster.shd-wait-qlength 1024
(DEFAULT)<br>
cluster.locking-scheme full
(DEFAULT)<br>
cluster.granular-entry-heal no (DEFAULT)<br>
features.locks-revocation-secs 0 (DEFAULT)<br>
features.locks-revocation-clear-all false
(DEFAULT)<br>
features.locks-revocation-max-blocked 0 (DEFAULT)<br>
features.locks-monkey-unlocking false
(DEFAULT)<br>
features.locks-notify-contention yes (DEFAULT)<br>
features.locks-notify-contention-delay 5 (DEFAULT)<br>
disperse.shd-max-threads 1 (DEFAULT)<br>
disperse.shd-wait-qlength 4096<br>
disperse.cpu-extensions auto
(DEFAULT)<br>
disperse.self-heal-window-size 32 (DEFAULT)<br>
cluster.use-compound-fops off<br>
performance.parallel-readdir on<br>
performance.rda-request-size 131072<br>
performance.rda-low-wmark 4096
(DEFAULT)<br>
performance.rda-high-wmark 128KB
(DEFAULT)<br>
performance.rda-cache-limit 10MB<br>
performance.nl-cache-positive-entry false
(DEFAULT)<br>
performance.nl-cache-limit 10MB<br>
performance.nl-cache-timeout 600<br>
cluster.brick-multiplex disable<br>
cluster.brick-graceful-cleanup disable<br>
glusterd.vol_count_per_thread 100<br>
cluster.max-bricks-per-process 250<br>
disperse.optimistic-change-log on (DEFAULT)<br>
disperse.stripe-cache 4 (DEFAULT)<br>
cluster.halo-enabled False
(DEFAULT)<br>
cluster.halo-shd-max-latency 99999
(DEFAULT)<br>
cluster.halo-nfsd-max-latency 5 (DEFAULT)<br>
cluster.halo-max-latency 5 (DEFAULT)<br>
cluster.halo-max-replicas 99999
(DEFAULT)<br>
cluster.halo-min-replicas 2 (DEFAULT)<br>
features.selinux on<br>
cluster.daemon-log-level INFO<br>
debug.delay-gen off<br>
delay-gen.delay-percentage 10% (DEFAULT)<br>
delay-gen.delay-duration 100000
(DEFAULT)<br>
delay-gen.enable (DEFAULT)<br>
disperse.parallel-writes on (DEFAULT)<br>
disperse.quorum-count 0 (DEFAULT)<br>
features.sdfs off<br>
features.cloudsync off<br>
features.ctime on<br>
ctime.noatime on<br>
features.cloudsync-storetype (null)
(DEFAULT)<br>
features.enforce-mandatory-lock off<br>
config.global-threading off<br>
config.client-threads 16<br>
config.brick-threads 16<br>
features.cloudsync-remote-read off<br>
features.cloudsync-store-id (null)
(DEFAULT)<br>
features.cloudsync-product-id (null)
(DEFAULT)<br>
features.acl enable<br>
cluster.use-anonymous-inode yes<br>
rebalance.ensure-durability on (DEFAULT)<br>
</blockquote>
<br>
Again, sorry for the long post. We would be happy to
have this solved as we are excited using glusterfs and
we would like to go back to having a stable
configuration.<br>
<br>
We always appreciate the spirit of collaboration and
reciprocal help on this list.<br>
<br>
Best<br>
Ilias<br>
<br>
-- <br>
forumZFD<br>
Entschieden für Frieden | Committed to Peace<br>
<br>
Ilias Chasapakis<br>
Referent IT | IT Consultant<br>
<br>
Forum Ziviler Friedensdienst e.V. | Forum Civil Peace
Service<br>
Am Kölner Brett 8 | 50825 Köln | Germany<br>
<br>
Tel 0221 91273243 | Fax 0221 91273299 |
<a class="moz-txt-link-freetext" href="http://www.forumZFD.de">http://www.forumZFD.de</a><br>
<br>
Vorstand nach § 26 BGB,
einzelvertretungsberechtigt|Executive Board:<br>
Alexander Mauz, Sonja Wiekenberg-Mlalandle, Jens von
Bargen<br>
VR 17651 Amtsgericht Köln<br>
<br>
Spenden|Donations: IBAN DE90 4306 0967 4103 7264 00
BIC GENODEM1GLS<br>
<br>
________<br>
<br>
<br>
<br>
Community Meeting Calendar:<br>
<br>
Schedule -<br>
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
Bridge: <a class="moz-txt-link-freetext" href="https://meet.google.com/cpu-eiue-hvk">https://meet.google.com/cpu-eiue-hvk</a><br>
Gluster-users mailing list<br>
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a class="moz-txt-link-freetext" href="https://lists.gluster.org/mailman/listinfo/gluster-users">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
forumZFD
Entschieden für Frieden | Committed to Peace
Ilias Chasapakis
Referent IT | IT Consultant
Forum Ziviler Friedensdienst e.V. | Forum Civil Peace Service
Am Kölner Brett 8 | 50825 Köln | Germany
Tel 0221 91273243 | Fax 0221 91273299 | <a class="moz-txt-link-freetext" href="http://www.forumZFD.de">http://www.forumZFD.de</a>
Vorstand nach § 26 BGB, einzelvertretungsberechtigt|Executive Board:
Alexander Mauz, Sonja Wiekenberg-Mlalandle, Jens von Bargen
VR 17651 Amtsgericht Köln
Spenden|Donations: IBAN DE90 4306 0967 4103 7264 00 BIC GENODEM1GLS</pre>
</body>
</html>