<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Dear Darrell,</p>
<p>Thanks again for the suggestions. We used some of the options
suggested for our case and in the meantime we also set the
round-robin configuration in order to have a single pointer in the
samba share configuration that then picks one of the bricks
(single host name in our DNS each brick).</p>
<p>This has improved the situation and the bricks are developing
less entries. But less is not really "none" as unfortunately we
have a behaviour observed:</p>
<p>We very frequently have directories not healing and then the
files and dirs under this "base path".</p>
<p>Sometimes one brick reports "New Folder" in the path, and the
others have already a name for it.</p>
<p>So we see a pattern also with renamed directories. I think that
we have an issue with renaming taking place slowly on one brick
(and it is not necessary the same one) while the others have
already the correct name.</p>
<p>Rebooting solves this, or renaming the file on the volume (or in
case of new folder create a "New Folder" in the path the brick
which is wrong expects it), issue listing, (delete if "New
Folder") rename the folder with old name on the "wrong" brick,
listing, re-renaming and the entries are healed. At this point the
names match and everything is back. What is not fully
understandable for me (because I surely miss something
fundamental) is that the gfid should be the same everywhere and I
would not expect that a rename changes that. So what is seen as
mismatching is just the "name attribute", once this matches,
everything is back fine.</p>
<p>Sorry for the simplistic exposure of facts, but I hope you have
some suggestions anyway. Perhaps it would be better if I send a
post related to renamings. I saw folks setting open-behind to off,
tried that but no results. Could be also that we mount incorrectly
on our ctdbs? Again I apologize as I understand that the points of
failure can be many, but we are trying to do something with
data/logs and observations that we have available.</p>
<p>Best regards<br>
Ilias<br>
</p>
<div class="moz-cite-prefix">Am 10.04.24 um 18:40 schrieb Darrell
Budic:<br>
</div>
<blockquote type="cite"
cite="mid:38664E17-2F70-4DA0-8231-DF2D71092B29@onholyground.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
I would strongly recommend running the glusterfs servers directly
on bare metal instead of in VMs. Check out Ovirt, especially its
hybrid cluster model. While it’s not currently well maintained, it
works fine on your class of hardware and fully supports this model
of gluster on the bare metal and VMs running on the same hosts.
And we may see it get some more support after the VMWare buyout,
who knows?<br id="lineBreakAtBeginningOfMessage">
<div><br>
</div>
<div>Gluster isn’t known for small file performance, but hunt
through the archives for specific tuning hints. And if you’re
using it to host the VM image files, you’re making that problem
because the files shared by gluster are large. More cache and
write (behind) buffers can help, and 10G or better networking
would be something you want to do if you can afford it. Going to
2x1G LAGs can help a tiny bit, but you really want the lower
latency from a faster physical media if you can get it.</div>
<div><br>
</div>
<div>If you are not already using tuned to set virtual-guest
profiles on your VMs (and virtual-host on the hosts), I’d look
into that as well. Set the disk elevator to ’none’ on the VMs as
well.</div>
<div><br>
</div>
<div><br>
<blockquote type="cite">
<div>On Apr 10, 2024, at 10:07 AM, Ilias Chasapakis forumZFD
<a class="moz-txt-link-rfc2396E" href="mailto:chasapakis@forumZFD.de"><chasapakis@forumZFD.de></a> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<meta http-equiv="Content-Type"
content="text/html; charset=UTF-8">
<div>
<p>Dear Darrell,</p>
<p>Dear ...,<br>
<br>
Many thanks for the prompt reply. Here some of the
additional information requested (please feel free to
ask for more if needed)<br>
<br>
</p>
<blockquote type="cite">CPU info:<br>
Hosts<br>
1. Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz (20 cores)
hw RAID 1 (adaptec)<br>
2. Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz (20 cores)
hw RAID 1 (adaptec)<br>
3. Intel(R) Xeon(R) Silver 4112 CPU @ 2.60GHz (8 cores)
hw RAID 1 (adaptec)<br>
<br>
GlusterFS VMs<br>
1. 4 cores, 10 GB RAM, CPU model Broadwell<br>
2. 4 cores, 10 GB RAM, CPU model Broadwell<br>
3. 4 cores, 10 GB RAM, CPU model: host passthrough
value<br>
<br>
Network info<br>
Physical connection between gluster nodes in a heartbeat
network that comprises a new Cisco switch.<br>
TCP connections with 1Gbit links<br>
Virtual default connectivity with virtio drivers for the
NICs and Macvtap connection to use the host´s
connectivity.<br>
No errors or lost packages are recorded between the VMs
or the hosts. Quick iperf tests (between glusters and
ctdbs-glusters show now evident issues).<br>
</blockquote>
<blockquote type="cite"> <br>
Workload:<br>
An instantaneous from this moment which can be
considered a peak time is around 450 files open on the
volume.<br>
In terms of "litteral" load we notice cpu peaks mostly
related to the shd process<br>
</blockquote>
<br>
All disks use virtIO drivers (virtIO disks).<br>
<br>
The file system on all nodes is XFS (not ZFS)<br>
<br>
Other than the clients on the gluster nodes themselves
there are clients on ctdbs that mount the gluster volume
and then expose it via smb to Windows clients (user
profiles included for roaming profiles).<br>
ctdbs reach the glusters through the heartbeat network<br>
<br>
We are considering to move the glusters to a network with
existing DNS capabilities in order to create a round-robin
configuration by assigning hosts by IP to a single
hostname to use it then for the mounts configuration of
the ctdbs.<br>
The reasoning/hope behind that we would minimize access
time and sync issues.<br>
<br>
Thank you for the information about the "sharding" we will
take this into account and consider pros and cons in the
current situation, epsecially because turning back is not
easy afterwards. Also our main problem is mainly not with
big files, but with a large quantity of small files.<br>
<br>
We could gladly make use of some of the options you
suggested after we assess the situation again. We welcome
any further suggestion in the meantime.
<div><br class="webkit-block-placeholder">
</div>
<p>Ilias<br>
</p>
<div class="moz-cite-prefix">Am 09.04.24 um 18:26 schrieb
Darrell Budic:<br>
</div>
<blockquote type="cite"
cite="mid:FDFC289F-5232-4D84-8CDB-BA10A3BAB2AB@onholyground.com">
<meta http-equiv="content-type"
content="text/html; charset=UTF-8">
The big one I see of you is to investigate and enable
sharding. It can improve performance and makes it much
easier to heal VM style workloads. Be aware that once
you turn it on, you can’t go back easily, and you need
to copy the VM disk images around to get them to be
sharded before it will show any real effect. A couple
other recommendations from my main volume (three
dedicated host servers with HDDs and SDD/NVM caching and
log volumes o<span style="caret-color: rgb(0, 0, 0);">n
ZFS </span>). The cluster.shd-* entries are especially
recommended. This is on gluster 9.4 at the moment, so
some of these won’t map exactly.
<div><br>
</div>
<div>
<div
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">Volume Name: gv1</span></div>
<div
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">Type: Replicate</span></div>
<div
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);">Number
of Bricks: 1 x 3 = 3</div>
<div
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">Transport-type: tcp</span></div>
<div
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);">Options
Reconfigured:</div>
<div
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">cluster.read-hash-mode:
3</span></div>
<div
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">performance.client-io-threads:
on</span></div>
<div
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">performance.write-behind-window-size:
64MB</span></div>
<div
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">performance.cache-size:
1G</span></div>
<div
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);">nfs.disable:
on</div>
<div
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">performance.readdir-ahead:
on</span></div>
<div
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">performance.quick-read:
off</span></div>
<div
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">performance.read-ahead:
on</span></div>
<div
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">performance.io-cache:
off</span></div>
<div
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">performance.stat-prefetch:
on</span></div>
<div
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">cluster.eager-lock:
enable</span></div>
<div
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">network.remote-dio:
enable</span></div>
<div
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);">server.event-threads:
4</div>
<div
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">client.event-threads:
8</span></div>
<div
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">performance.io-thread-count:
64</span></div>
<div
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">performance.low-prio-threads:
32</span></div>
<div
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">features.shard: on</span></div>
<div
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">features.shard-block-size:
64MB</span></div>
<div
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">cluster.locking-scheme:
granular</span></div>
<div
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);">cluster.data-self-heal-algorithm:
full</div>
<div
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">cluster.shd-max-threads:
8</span></div>
<div
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">cluster.shd-wait-qlength:
10240</span></div>
<div
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">cluster.choose-local:
false</span></div>
<div
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><span
style="font-variant-ligatures: no-common-ligatures">cluster.granular-entry-heal:
enable</span></div>
<div
style="margin: 0px; font-stretch: normal; font-size: 15px; line-height: normal; font-family: "Fantasque Sans Mono"; font-size-adjust: none; font-kerning: auto; font-variant-alternates: normal; font-variant-ligatures: normal; font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-position: normal; font-feature-settings: normal; font-optical-sizing: auto; font-variation-settings: normal; color: rgb(40, 255, 0); background-color: rgba(255, 255, 255, 0.9);"><br>
</div>
<div>Otherwise, more details about your servers, CPU,
RAM, and Disks would be useful for suggestions, and
details of your network as well. And if you haven’t
done kernel level tuning on the servers, you should
address that as well. These all vary a lot by your
work load and hardware setup, so there aren’t many
generic recommendations I can give other than to
make sure you tuned your tcp stack and enabled the
none disk elevator on SSDs or disks used by ZFS. </div>
<div><br>
</div>
<div>There’s a lot of tuning suggesting in the
archives if you go searching as well.</div>
<div><br>
</div>
<div> -Darrell</div>
<div><br>
</div>
<div><br>
<blockquote type="cite">
<div>On Apr 9, 2024, at 3:05 AM, Ilias Chasapakis
forumZFD <a class="moz-txt-link-rfc2396E"
href="mailto:chasapakis@forumZFD.de"
moz-do-not-send="true"><chasapakis@forumZFD.de></a>
wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div>Dear all,<br>
<br>
we would like to describe the situation that
we have and that does not solve since a long
time, that means after many minor<br>
and major upgrades of GlusterFS<br>
<br>
We use a KVM environment for VMs for glusterfs
and host servers are updated regularly. Hosts
are disomogeneous hardware,<br>
but configured with same characteristics.<br>
<br>
The VMs have been also harmonized to use the
virtio drivers where available for devices and
resources reserved are the same<br>
on each host.<br>
<br>
Physical switch for hosts has been substituted
with a reliable one.<br>
<br>
Probing peers has been and is quite quick in
the heartbeat network and communication
between the servers for apparently has no
issues on disruptions.<br>
<br>
And I say apparently because what we have is:<br>
<br>
- always pending failed heals that used to
resolve by a rotated reboot of the gluster vms
(replica 3). Restarting only<br>
glusterfs related services (daemon, events
etc.) has no effect, only reboot brings
results<br>
- very often failed heals are directories<br>
<br>
We lately removed a brick that was on a vm on
a host that has been entirely substituted.
Re-added the brick, sync went on and<br>
all data was eventually synced and started
with 0 pending failed heals. Now it develops
failed heals too like its fellow<br>
bricks. Please take into account we healed all
the failed entries (manually with various
methods) before adding the third brick.<br>
<br>
After some days of operating, the count of
failed heals rises again, not really fast but
with new entries for sure (which might solve<br>
with rotated reboots, or not).<br>
<br>
We have gluster clients also on ctdbs that
connect to the gluster and mount via glusterfs
client. Windows roaming profiles shared via
smb become frequently corrupted,(they are
composed of a great number small files and are
though of big total dimension). Gluster nodes
are formatted with xfs.<br>
<br>
Also what we observer is that mounting with
the vfs option in smb on the ctdbs has some
kind of delay. This means that you can see the
shared folder on for example<br>
a Windows client machine on a ctdb, but not on
another ctdb in the cluster and then after a
while it appears there too. And this
frequently st<br>
<br>
<br>
This is an excerpt of entries on our shd logs:<br>
<br>
<blockquote type="cite">2024-04-08
10:13:26.213596 +0000] I [MSGID: 108026]
[afr-self-heal-entry.c:1080:afr_selfheal_entry_do]
0-gv-ho-replicate-0: performing full entry
selfheal on
2c621415-6223-4b66-a4ca-3f6f267a448d<br>
[2024-04-08 10:14:08.135911 +0000] W [MSGID:
114031]
[client-rpc-fops_v2.c:2457:client4_0_link_cbk]
0-gv-ho-client-5: remote operation failed.
[{source=<gfid:91d83f0e-1864-4ff3-9174-b7c956e20596>},
{target=(null)}, {errno=116},
{error=Veraltete Dateizugriffsnummer (file
handle)}]<br>
[2024-04-08 10:15:59.135908 +0000] W [MSGID:
114061]
[client-common.c:2992:client_pre_readdir_v2]
0-gv-ho-client-5: remote_fd is -1. EBADFD
[{gfid=6b5e599e-c836-4ebe-b16a-8224425b88c7},
{errno=77}, {error=Die Dateizugriffsnummer
ist in schlechter Verfassung}]<br>
[2024-04-08 10:30:25.013592 +0000] I [MSGID:
108026]
[afr-self-heal-entry.c:1080:afr_selfheal_entry_do]
0-gv-ho-replicate-0: performing full entry
selfheal on
24e82e12-5512-4679-9eb3-8bd098367db7<br>
[2024-04-08 10:33:17.613594 +0000] W [MSGID:
114031]
[client-rpc-fops_v2.c:2457:client4_0_link_cbk]
0-gv-ho-client-5: remote operation failed.
[{source=<gfid:ef9068fc-a329-4a21-88d2-265ecd3d208c>},
{target=(null)}, {errno=116},
{error=Veraltete Dateizugriffsnummer (file
handle)}]<br>
[2024-04-08 10:33:21.201359 +0000] W [MSGID:
114031]
[client-rpc-fops_v2.c:2457:client4_0_link_cbk]
0-gv-ho-client-5: remote operation failed.
[{source=<br>
</blockquote>
<br>
How are he clients mapped to real hosts in
order to know on which one´s logs to look at?<br>
<br>
We would like to go by exclusion to finally
eradicate this, possibly in a conservative way
(not rebuilding everything) and we<br>
<br>
are becoming clueless as to where to look at
as we also tried various options settings
regarding performance etc.<br>
<br>
Here is the set on our main volume:<br>
<br>
<blockquote type="cite">cluster.lookup-unhashed
on (DEFAULT)<br>
cluster.lookup-optimize on
(DEFAULT)<br>
cluster.min-free-disk 10%
(DEFAULT)<br>
cluster.min-free-inodes 5%
(DEFAULT)<br>
cluster.rebalance-stats off
(DEFAULT)<br>
cluster.subvols-per-directory
(null) (DEFAULT)<br>
cluster.readdir-optimize off
(DEFAULT)<br>
cluster.rsync-hash-regex
(null) (DEFAULT)<br>
cluster.extra-hash-regex
(null) (DEFAULT)<br>
cluster.dht-xattr-name
trusted.glusterfs.dht (DEFAULT)<br>
cluster.randomize-hash-range-by-gfid off
(DEFAULT)<br>
cluster.rebal-throttle
normal (DEFAULT)<br>
cluster.lock-migration off<br>
cluster.force-migration off<br>
cluster.local-volume-name
(null) (DEFAULT)<br>
cluster.weighted-rebalance on
(DEFAULT)<br>
cluster.switch-pattern
(null) (DEFAULT)<br>
cluster.entry-change-log on
(DEFAULT)<br>
cluster.read-subvolume
(null) (DEFAULT)<br>
cluster.read-subvolume-index -1
(DEFAULT)<br>
cluster.read-hash-mode 1
(DEFAULT)<br>
cluster.background-self-heal-count 8
(DEFAULT)<br>
cluster.metadata-self-heal on<br>
cluster.data-self-heal on<br>
cluster.entry-self-heal on<br>
cluster.self-heal-daemon enable<br>
cluster.heal-timeout 600
(DEFAULT)<br>
cluster.self-heal-window-size 8
(DEFAULT)<br>
cluster.data-change-log on
(DEFAULT)<br>
cluster.metadata-change-log on
(DEFAULT)<br>
cluster.data-self-heal-algorithm
(null) (DEFAULT)<br>
cluster.eager-lock on
(DEFAULT)<br>
disperse.eager-lock on
(DEFAULT)<br>
disperse.other-eager-lock on
(DEFAULT)<br>
disperse.eager-lock-timeout 1
(DEFAULT)<br>
disperse.other-eager-lock-timeout 1
(DEFAULT)<br>
cluster.quorum-type auto<br>
cluster.quorum-count 2<br>
cluster.choose-local
true (DEFAULT)<br>
cluster.self-heal-readdir-size 1KB
(DEFAULT)<br>
cluster.post-op-delay-secs 1
(DEFAULT)<br>
cluster.ensure-durability on
(DEFAULT)<br>
cluster.consistent-metadata no
(DEFAULT)<br>
cluster.heal-wait-queue-length 128
(DEFAULT)<br>
cluster.favorite-child-policy none<br>
cluster.full-lock yes
(DEFAULT)<br>
cluster.optimistic-change-log on
(DEFAULT)<br>
diagnostics.latency-measurement off<br>
diagnostics.dump-fd-stats off
(DEFAULT)<br>
diagnostics.count-fop-hits off<br>
diagnostics.brick-log-level INFO<br>
diagnostics.client-log-level INFO<br>
diagnostics.brick-sys-log-level
CRITICAL (DEFAULT)<br>
diagnostics.client-sys-log-level
CRITICAL (DEFAULT)<br>
diagnostics.brick-logger
(null) (DEFAULT)<br>
diagnostics.client-logger
(null) (DEFAULT)<br>
diagnostics.brick-log-format
(null) (DEFAULT)<br>
diagnostics.client-log-format
(null) (DEFAULT)<br>
diagnostics.brick-log-buf-size 5
(DEFAULT)<br>
diagnostics.client-log-buf-size 5
(DEFAULT)<br>
diagnostics.brick-log-flush-timeout 120
(DEFAULT)<br>
diagnostics.client-log-flush-timeout 120
(DEFAULT)<br>
diagnostics.stats-dump-interval 0
(DEFAULT)<br>
diagnostics.fop-sample-interval 0
(DEFAULT)<br>
diagnostics.stats-dump-format
json (DEFAULT)<br>
diagnostics.fop-sample-buf-size
65535 (DEFAULT)<br>
diagnostics.stats-dnscache-ttl-sec
86400 (DEFAULT)<br>
performance.cache-max-file-size 10<br>
performance.cache-min-file-size 0
(DEFAULT)<br>
performance.cache-refresh-timeout 1
(DEFAULT)<br>
performance.cache-priority (DEFAULT)<br>
performance.io-cache-size
32MB (DEFAULT)<br>
performance.cache-size
32MB (DEFAULT)<br>
performance.io-thread-count 16
(DEFAULT)<br>
performance.high-prio-threads 16
(DEFAULT)<br>
performance.normal-prio-threads 16
(DEFAULT)<br>
performance.low-prio-threads 16
(DEFAULT)<br>
performance.least-prio-threads 1
(DEFAULT)<br>
performance.enable-least-priority on
(DEFAULT)<br>
performance.iot-watchdog-secs
(null) (DEFAULT)<br>
performance.iot-cleanup-disconnected-reqs
off (DEFAULT)<br>
performance.iot-pass-through
false (DEFAULT)<br>
performance.io-cache-pass-through
false (DEFAULT)<br>
performance.quick-read-cache-size
128MB (DEFAULT)<br>
performance.cache-size
128MB (DEFAULT)<br>
performance.quick-read-cache-timeout 1
(DEFAULT)<br>
performance.qr-cache-timeout 600<br>
performance.quick-read-cache-invalidation
false (DEFAULT)<br>
performance.ctime-invalidation
false (DEFAULT)<br>
performance.flush-behind on
(DEFAULT)<br>
performance.nfs.flush-behind on
(DEFAULT)<br>
performance.write-behind-window-size 4MB<br>
performance.resync-failed-syncs-after-fsync
off (DEFAULT)<br>
performance.nfs.write-behind-window-size 1MB
(DEFAULT)<br>
performance.strict-o-direct off
(DEFAULT)<br>
performance.nfs.strict-o-direct off
(DEFAULT)<br>
performance.strict-write-ordering off
(DEFAULT)<br>
performance.nfs.strict-write-ordering off
(DEFAULT)<br>
performance.write-behind-trickling-writes on
(DEFAULT)<br>
performance.aggregate-size
128KB (DEFAULT)<br>
performance.nfs.write-behind-trickling-writes on (DEFAULT)<br>
performance.lazy-open yes
(DEFAULT)<br>
performance.read-after-open yes
(DEFAULT)<br>
performance.open-behind-pass-through
false (DEFAULT)<br>
performance.read-ahead-page-count 4
(DEFAULT)<br>
performance.read-ahead-pass-through
false (DEFAULT)<br>
performance.readdir-ahead-pass-through
false (DEFAULT)<br>
performance.md-cache-pass-through
false (DEFAULT)<br>
performance.write-behind-pass-through
false (DEFAULT)<br>
performance.md-cache-timeout 600<br>
performance.cache-swift-metadata
false (DEFAULT)<br>
performance.cache-samba-metadata on<br>
performance.cache-capability-xattrs
true (DEFAULT)<br>
performance.cache-ima-xattrs
true (DEFAULT)<br>
performance.md-cache-statfs off
(DEFAULT)<br>
performance.xattr-cache-list (DEFAULT)<br>
performance.nl-cache-pass-through
false (DEFAULT)<br>
network.frame-timeout
1800 (DEFAULT)<br>
network.ping-timeout 20<br>
network.tcp-window-size
(null) (DEFAULT)<br>
client.ssl off<br>
network.remote-dio
disable (DEFAULT)<br>
client.event-threads 4<br>
client.tcp-user-timeout 0<br>
client.keepalive-time 20<br>
client.keepalive-interval 2<br>
client.keepalive-count 9<br>
client.strict-locks off<br>
network.tcp-window-size
(null) (DEFAULT)<br>
network.inode-lru-limit 200000<br>
auth.allow *<br>
auth.reject
(null) (DEFAULT)<br>
transport.keepalive 1<br>
server.allow-insecure on
(DEFAULT)<br>
server.root-squash off
(DEFAULT)<br>
server.all-squash off
(DEFAULT)<br>
server.anonuid
65534 (DEFAULT)<br>
server.anongid
65534 (DEFAULT)<br>
server.statedump-path
/var/run/gluster (DEFAULT)<br>
server.outstanding-rpc-limit 64
(DEFAULT)<br>
server.ssl off<br>
auth.ssl-allow *<br>
server.manage-gids off
(DEFAULT)<br>
server.dynamic-auth on
(DEFAULT)<br>
client.send-gids on
(DEFAULT)<br>
server.gid-timeout 300
(DEFAULT)<br>
server.own-thread
(null) (DEFAULT)<br>
server.event-threads 4<br>
server.tcp-user-timeout 42
(DEFAULT)<br>
server.keepalive-time 20<br>
server.keepalive-interval 2<br>
server.keepalive-count 9<br>
transport.listen-backlog 1024<br>
ssl.own-cert
(null) (DEFAULT)<br>
ssl.private-key
(null) (DEFAULT)<br>
ssl.ca-list
(null) (DEFAULT)<br>
ssl.crl-path
(null) (DEFAULT)<br>
ssl.certificate-depth
(null) (DEFAULT)<br>
ssl.cipher-list
(null) (DEFAULT)<br>
ssl.dh-param
(null) (DEFAULT)<br>
ssl.ec-curve
(null) (DEFAULT)<br>
transport.address-family inet<br>
performance.write-behind off<br>
performance.read-ahead on<br>
performance.readdir-ahead on<br>
performance.io-cache off<br>
performance.open-behind on<br>
performance.quick-read on<br>
performance.nl-cache on<br>
performance.stat-prefetch on<br>
performance.client-io-threads off<br>
performance.nfs.write-behind on<br>
performance.nfs.read-ahead off<br>
performance.nfs.io-cache off<br>
performance.nfs.quick-read off<br>
performance.nfs.stat-prefetch off<br>
performance.nfs.io-threads off<br>
performance.force-readdirp
true (DEFAULT)<br>
performance.cache-invalidation on<br>
performance.global-cache-invalidation
true (DEFAULT)<br>
features.uss off<br>
features.snapshot-directory .snaps<br>
features.show-snapshot-directory off<br>
features.tag-namespaces off<br>
network.compression off<br>
network.compression.window-size -15
(DEFAULT)<br>
network.compression.mem-level 8
(DEFAULT)<br>
network.compression.min-size 0
(DEFAULT)<br>
network.compression.compression-level -1
(DEFAULT)<br>
network.compression.debug
false (DEFAULT)<br>
features.default-soft-limit 80%
(DEFAULT)<br>
features.soft-timeout 60
(DEFAULT)<br>
features.hard-timeout 5
(DEFAULT)<br>
features.alert-time
86400 (DEFAULT)<br>
features.quota-deem-statfs off<br>
geo-replication.indexing off<br>
geo-replication.indexing off<br>
geo-replication.ignore-pid-check off<br>
geo-replication.ignore-pid-check off<br>
features.quota off<br>
features.inode-quota off<br>
features.bitrot disable<br>
debug.trace off<br>
debug.log-history no
(DEFAULT)<br>
debug.log-file no
(DEFAULT)<br>
debug.exclude-ops
(null) (DEFAULT)<br>
debug.include-ops
(null) (DEFAULT)<br>
debug.error-gen off<br>
debug.error-failure
(null) (DEFAULT)<br>
debug.error-number
(null) (DEFAULT)<br>
debug.random-failure off
(DEFAULT)<br>
debug.error-fops
(null) (DEFAULT)<br>
nfs.disable on<br>
features.read-only off
(DEFAULT)<br>
features.worm off<br>
features.worm-file-level off<br>
features.worm-files-deletable on<br>
features.default-retention-period 120
(DEFAULT)<br>
features.retention-mode
relax (DEFAULT)<br>
features.auto-commit-period 180
(DEFAULT)<br>
storage.linux-aio off
(DEFAULT)<br>
storage.linux-io_uring off
(DEFAULT)<br>
storage.batch-fsync-mode
reverse-fsync (DEFAULT)<br>
storage.batch-fsync-delay-usec 0
(DEFAULT)<br>
storage.owner-uid -1
(DEFAULT)<br>
storage.owner-gid -1
(DEFAULT)<br>
storage.node-uuid-pathinfo off
(DEFAULT)<br>
storage.health-check-interval 30
(DEFAULT)<br>
storage.build-pgfid off
(DEFAULT)<br>
storage.gfid2path on
(DEFAULT)<br>
storage.gfid2path-separator :
(DEFAULT)<br>
storage.reserve 1
(DEFAULT)<br>
storage.health-check-timeout 20
(DEFAULT)<br>
storage.fips-mode-rchecksum on<br>
storage.force-create-mode
0000 (DEFAULT)<br>
storage.force-directory-mode
0000 (DEFAULT)<br>
storage.create-mask
0777 (DEFAULT)<br>
storage.create-directory-mask
0777 (DEFAULT)<br>
storage.max-hardlinks 100
(DEFAULT)<br>
features.ctime on
(DEFAULT)<br>
config.gfproxyd off<br>
cluster.server-quorum-type server<br>
cluster.server-quorum-ratio 51<br>
changelog.changelog off
(DEFAULT)<br>
changelog.changelog-dir {{
brick.path }}/.glusterfs/changelogs
(DEFAULT)<br>
changelog.encoding
ascii (DEFAULT)<br>
changelog.rollover-time 15
(DEFAULT)<br>
changelog.fsync-interval 5
(DEFAULT)<br>
changelog.changelog-barrier-timeout 120<br>
changelog.capture-del-path off
(DEFAULT)<br>
features.barrier disable<br>
features.barrier-timeout 120<br>
features.trash off
(DEFAULT)<br>
features.trash-dir
.trashcan (DEFAULT)<br>
features.trash-eliminate-path
(null) (DEFAULT)<br>
features.trash-max-filesize 5MB
(DEFAULT)<br>
features.trash-internal-op off
(DEFAULT)<br>
cluster.enable-shared-storage disable<br>
locks.trace off
(DEFAULT)<br>
locks.mandatory-locking off
(DEFAULT)<br>
cluster.disperse-self-heal-daemon
enable (DEFAULT)<br>
cluster.quorum-reads no
(DEFAULT)<br>
client.bind-insecure
(null) (DEFAULT)<br>
features.timeout 45
(DEFAULT)<br>
features.failover-hosts
(null) (DEFAULT)<br>
features.shard off<br>
features.shard-block-size
64MB (DEFAULT)<br>
features.shard-lru-limit
16384 (DEFAULT)<br>
features.shard-deletion-rate 100
(DEFAULT)<br>
features.scrub-throttle lazy<br>
features.scrub-freq biweekly<br>
features.scrub
false (DEFAULT)<br>
features.expiry-time 120<br>
features.signer-threads 4<br>
features.cache-invalidation on<br>
features.cache-invalidation-timeout 600<br>
ganesha.enable off<br>
features.leases off<br>
features.lease-lock-recall-timeout 60
(DEFAULT)<br>
disperse.background-heals 8
(DEFAULT)<br>
disperse.heal-wait-qlength 128
(DEFAULT)<br>
cluster.heal-timeout 600
(DEFAULT)<br>
dht.force-readdirp on
(DEFAULT)<br>
disperse.read-policy
gfid-hash (DEFAULT)<br>
cluster.shd-max-threads 4<br>
cluster.shd-wait-qlength
1024 (DEFAULT)<br>
cluster.locking-scheme
full (DEFAULT)<br>
cluster.granular-entry-heal no
(DEFAULT)<br>
features.locks-revocation-secs 0
(DEFAULT)<br>
features.locks-revocation-clear-all
false (DEFAULT)<br>
features.locks-revocation-max-blocked 0
(DEFAULT)<br>
features.locks-monkey-unlocking
false (DEFAULT)<br>
features.locks-notify-contention yes
(DEFAULT)<br>
features.locks-notify-contention-delay 5
(DEFAULT)<br>
disperse.shd-max-threads 1
(DEFAULT)<br>
disperse.shd-wait-qlength 4096<br>
disperse.cpu-extensions
auto (DEFAULT)<br>
disperse.self-heal-window-size 32
(DEFAULT)<br>
cluster.use-compound-fops off<br>
performance.parallel-readdir on<br>
performance.rda-request-size 131072<br>
performance.rda-low-wmark
4096 (DEFAULT)<br>
performance.rda-high-wmark
128KB (DEFAULT)<br>
performance.rda-cache-limit 10MB<br>
performance.nl-cache-positive-entry
false (DEFAULT)<br>
performance.nl-cache-limit 10MB<br>
performance.nl-cache-timeout 600<br>
cluster.brick-multiplex disable<br>
cluster.brick-graceful-cleanup disable<br>
glusterd.vol_count_per_thread 100<br>
cluster.max-bricks-per-process 250<br>
disperse.optimistic-change-log on
(DEFAULT)<br>
disperse.stripe-cache 4
(DEFAULT)<br>
cluster.halo-enabled
False (DEFAULT)<br>
cluster.halo-shd-max-latency
99999 (DEFAULT)<br>
cluster.halo-nfsd-max-latency 5
(DEFAULT)<br>
cluster.halo-max-latency 5
(DEFAULT)<br>
cluster.halo-max-replicas
99999 (DEFAULT)<br>
cluster.halo-min-replicas 2
(DEFAULT)<br>
features.selinux on<br>
cluster.daemon-log-level INFO<br>
debug.delay-gen off<br>
delay-gen.delay-percentage 10%
(DEFAULT)<br>
delay-gen.delay-duration
100000 (DEFAULT)<br>
delay-gen.enable (DEFAULT)<br>
disperse.parallel-writes on
(DEFAULT)<br>
disperse.quorum-count 0
(DEFAULT)<br>
features.sdfs off<br>
features.cloudsync off<br>
features.ctime on<br>
ctime.noatime on<br>
features.cloudsync-storetype
(null) (DEFAULT)<br>
features.enforce-mandatory-lock off<br>
config.global-threading off<br>
config.client-threads 16<br>
config.brick-threads 16<br>
features.cloudsync-remote-read off<br>
features.cloudsync-store-id
(null) (DEFAULT)<br>
features.cloudsync-product-id
(null) (DEFAULT)<br>
features.acl enable<br>
cluster.use-anonymous-inode yes<br>
rebalance.ensure-durability on
(DEFAULT)<br>
</blockquote>
<br>
Again, sorry for the long post. We would be
happy to have this solved as we are excited
using glusterfs and we would like to go back
to having a stable configuration.<br>
<br>
We always appreciate the spirit of
collaboration and reciprocal help on this
list.<br>
<br>
Best<br>
Ilias<br>
<br>
-- <br>
forumZFD<br>
Entschieden für Frieden | Committed to Peace<br>
<br>
Ilias Chasapakis<br>
Referent IT | IT Consultant<br>
<br>
Forum Ziviler Friedensdienst e.V. | Forum
Civil Peace Service<br>
Am Kölner Brett 8 | 50825 Köln | Germany<br>
<br>
Tel 0221 91273243 | Fax 0221 91273299 | <a
class="moz-txt-link-freetext"
href="http://www.forumzfd.de/"
moz-do-not-send="true">http://www.forumZFD.de</a><br>
<br>
Vorstand nach § 26 BGB,
einzelvertretungsberechtigt|Executive Board:<br>
Alexander Mauz, Sonja Wiekenberg-Mlalandle,
Jens von Bargen<br>
VR 17651 Amtsgericht Köln<br>
<br>
Spenden|Donations: IBAN DE90 4306 0967 4103
7264 00 BIC GENODEM1GLS<br>
<br>
________<br>
<br>
<br>
<br>
Community Meeting Calendar:<br>
<br>
Schedule -<br>
Every 2nd and 4th Tuesday at 14:30 IST / 09:00
UTC<br>
Bridge: <a class="moz-txt-link-freetext"
href="https://meet.google.com/cpu-eiue-hvk"
moz-do-not-send="true">https://meet.google.com/cpu-eiue-hvk</a><br>
Gluster-users mailing list<br>
<a
class="moz-txt-link-abbreviated moz-txt-link-freetext"
href="mailto:Gluster-users@gluster.org"
moz-do-not-send="true">Gluster-users@gluster.org</a><br>
<a class="moz-txt-link-freetext"
href="https://lists.gluster.org/mailman/listinfo/gluster-users"
moz-do-not-send="true">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
forumZFD
Entschieden für Frieden | Committed to Peace
Ilias Chasapakis
Referent IT | IT Consultant
Forum Ziviler Friedensdienst e.V. | Forum Civil Peace Service
Am Kölner Brett 8 | 50825 Köln | Germany
Tel 0221 91273243 | Fax 0221 91273299 | <a class="moz-txt-link-freetext"
href="http://www.forumzfd.de/" moz-do-not-send="true">http://www.forumZFD.de</a>
Vorstand nach § 26 BGB, einzelvertretungsberechtigt|Executive Board:
Alexander Mauz, Sonja Wiekenberg-Mlalandle, Jens von Bargen
VR 17651 Amtsgericht Köln
Spenden|Donations: IBAN DE90 4306 0967 4103 7264 00 BIC GENODEM1GLS</pre>
</div>
________<br>
<br>
<br>
<br>
Community Meeting Calendar:<br>
<br>
Schedule -<br>
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
Bridge: <a class="moz-txt-link-freetext" href="https://meet.google.com/cpu-eiue-hvk">https://meet.google.com/cpu-eiue-hvk</a><br>
Gluster-users mailing list<br>
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a class="moz-txt-link-freetext" href="https://lists.gluster.org/mailman/listinfo/gluster-users">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</div>
</blockquote>
</div>
<br>
</blockquote>
<pre class="moz-signature" cols="72">--
forumZFD
Entschieden für Frieden | Committed to Peace
Ilias Chasapakis
Referent IT | IT Consultant
Forum Ziviler Friedensdienst e.V. | Forum Civil Peace Service
Am Kölner Brett 8 | 50825 Köln | Germany
Tel 0221 91273243 | Fax 0221 91273299 | <a class="moz-txt-link-freetext" href="http://www.forumZFD.de">http://www.forumZFD.de</a>
Vorstand nach § 26 BGB, einzelvertretungsberechtigt|Executive Board:
Alexander Mauz, Sonja Wiekenberg-Mlalandle, Jens von Bargen
VR 17651 Amtsgericht Köln
Spenden|Donations: IBAN DE90 4306 0967 4103 7264 00 BIC GENODEM1GLS</pre>
</body>
</html>