<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Quick top-post reply.</p>
<p>We host websites on Gluster. It took us a bit of doing to get
working quickly but here's roughly how we do it. I'm not saying
this is *the* way to do, just a way that works for us:</p>
<p>1. Gluster volume with sharding enabled for storing filesystem
files as a backing store for iSCSI.</p>
<p>2. Gluster servers also provide TGT iSCSI LUNS for multipath
access.</p>
<p>3. Cluster of NFS servers connecting to iSCSI LUNS with
multipathd. We run this using Corosync to manage
redundancy/failover. Make sure you get this bit right. It's
probably the hardest bit. It's probably also the bit most likely
ripe for improvement.<br>
</p>
<p>4. VMs running web servers access the NFS data for websites. Each
server runs an FS-Cache volume for the NFS mounted volumes.</p>
<p>5. OPCache/FPM for the dynamic stuff (assuming PHP here).<br>
</p>
<p>Your underlying storage needs to be relatively fast/capable of
course.</p>
<p>We have our VM volumes on sharded Gluster too.<br>
</p>
<p>My theory in setting it up this way was to avoid the small-file
scenario that Gluster is not very good at for one it is good at,
sharded big files.</p>
<p>Anyway, hope that helps a little.</p>
<p>Ronny<br>
</p>
<p><br>
</p>
<div class="moz-cite-prefix">Jaco Kroon wrote on 14/12/2022 11:28:<br>
</div>
<blockquote type="cite"
cite="mid:45e9d695-f64c-0947-bd6a-8daa8688add7@uls.co.za">Hi All,
<br>
<br>
We've got a glusterfs cluster that houses some php web sites.
<br>
<br>
This is generally considered a bad idea and we can see why.
<br>
<br>
With performance.nl-cache on it actually turns out to be very
reasonable, however, with this turned of performance is roughly 5x
worse. meaning a request that would take sub 500ms now takes
2500ms. In other cases we see far, far worse cases, eg, with
nl-cache takes ~1500ms, without takes ~30s (20x worse).
<br>
<br>
So why not use nl-cache? Well, it results in readdir reporting
files which then fails to open with ENOENT. The cache also never
clears even though the configuration says nl-cache entries should
only be cached for 60s. Even for "ls -lah" in affected folders
you'll notice ???? mark entries for attributes on files. If this
recovers in a reasonable time (say, a few seconds, sure).
<br>
<br>
# gluster volume info
<br>
Type: Replicate
<br>
Volume ID: cbe08331-8b83-41ac-b56d-88ef30c0f5c7
<br>
Status: Started
<br>
Snapshot Count: 0
<br>
Number of Bricks: 1 x 2 = 2
<br>
Transport-type: tcp
<br>
Options Reconfigured:
<br>
performance.nl-cache: on
<br>
cluster.readdir-optimize: on
<br>
config.client-threads: 2
<br>
config.brick-threads: 4
<br>
config.global-threading: on
<br>
performance.iot-pass-through: on
<br>
storage.fips-mode-rchecksum: on
<br>
cluster.granular-entry-heal: enable
<br>
cluster.data-self-heal-algorithm: full
<br>
cluster.locking-scheme: granular
<br>
client.event-threads: 2
<br>
server.event-threads: 2
<br>
transport.address-family: inet
<br>
nfs.disable: on
<br>
cluster.metadata-self-heal: off
<br>
cluster.entry-self-heal: off
<br>
cluster.data-self-heal: off
<br>
cluster.self-heal-daemon: on
<br>
server.allow-insecure: on
<br>
features.ctime: off
<br>
performance.io-cache: on
<br>
performance.cache-invalidation: on
<br>
features.cache-invalidation: on
<br>
performance.qr-cache-timeout: 600
<br>
features.cache-invalidation-timeout: 600
<br>
performance.io-cache-size: 128MB
<br>
performance.cache-size: 128MB
<br>
<br>
Are there any other recommendations short of abandon all hope of
redundancy and to revert to a single-server setup (for the web
code at least). Currently the cost of the redundancy seems to
outweigh the benefit.
<br>
<br>
Glusterfs version 10.2. With patch for --inode-table-size, mounts
happen with:
<br>
<br>
/usr/sbin/glusterfs --acl --reader-thread-count=2
--lru-limit=524288 --inode-table-size=524288 --invalidate-limit=16
--background-qlen=32 --fuse-mountopts=nodev,nosuid,noexec,noatime
--process-name fuse --volfile-server=127.0.0.1
--volfile-id=gv_home --fuse-mountopts=nodev,nosuid,noexec,noatime
/home
<br>
<br>
Kind Regards,
<br>
Jaco
<br>
<br>
________
<br>
<br>
<br>
<br>
Community Meeting Calendar:
<br>
<br>
Schedule -
<br>
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
<br>
Bridge: <a class="moz-txt-link-freetext" href="https://meet.google.com/cpu-eiue-hvk">https://meet.google.com/cpu-eiue-hvk</a>
<br>
Gluster-users mailing list
<br>
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<br>
<a class="moz-txt-link-freetext" href="https://lists.gluster.org/mailman/listinfo/gluster-users">https://lists.gluster.org/mailman/listinfo/gluster-users</a>
<br>
</blockquote>
<div class="moz-signature">-- <br>
<pre>Ronny Adsetts
Technical Director
Amazing Internet Ltd, London
t: +44 20 8977 8943
w: <a class="moz-txt-link-abbreviated" href="http://www.amazinginternet.com">www.amazinginternet.com</a>
Registered office: 85 Waldegrave Park, Twickenham, TW1 4TJ
Registered in England. Company No. 4042957
</pre>
</div>
</body>
</html>