<div dir="ltr"><div><div><div><div>Hi Alex,<br></div> I don't use Gluster for storing large amounts of small files, however from what I've read, that does appear to its big achilles heel.<br></div>Personally, if you're not looking to scale out to a lot more servers, I'd go with Ceph or DRBD. Gluster's best features are in its scalability.<br></div>Also, it's worth pointing out that in any setup, you've got to be careful with 2 node configurations as they're highly vulnerable to split-brain scenarios.<br><br></div><div>Given the relatively small size of your data, caching tweaks & an arbiter may well save you here, however I don't use enough of its caching features to be able to give advice on it.<br></div><div><br></div>D<br></div><div class="gmail_extra"><br><div class="gmail_quote">On 3 February 2017 at 08:28, Alex Sudakar <span dir="ltr"><<a href="mailto:alex.sudakar@gmail.com" target="_blank">alex.sudakar@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi. I'm looking for a clustered filesystem for a very simple<br>
scenario. I've set up Gluster but my tests have shown quite a<br>
performance penalty when compared to using a local XFS filesystem.<br>
This no doubt reflects the reality of moving to a proper distributed<br>
filesystem, but I'd like to quickly check that I haven't missed<br>
something obvious that might improve performance.<br>
<br>
I plan to have two Amazon AWS EC2 instances (virtual machines) both<br>
accessing the same filesystem for read/writes. Access will be almost<br>
entirely reads, with the occasional modification, deletion or creation<br>
of files. Ideally I wanted all those reads going straight to the<br>
local XFS filesystem and just the writes incurring a distributed<br>
performance penalty. :-)<br>
<br>
So I've set up two VMs with Centos 7.2 and Gluster 3.8.8, each machine<br>
running as a combined Gluster server and client. One brick on each<br>
machine, one volume in a 1 x 2 replica configuration.<br>
<br>
Everything works, it's just the performance penalty which is a surprise. :-)<br>
<br>
My test directory has 9,066 files and directories; 7,987 actual files.<br>
Total size is 63MB data, 85MB allocated; an average size of 8KB data<br>
per file. The brick's files have a total of 117MB allocated, with the<br>
extra 32MB working out pretty much to be exactly the sum of the extra<br>
4KB extents that would have been allocated for the XFS attributes per<br>
file - the VMs were installed with the default 256 byte inode size for<br>
the local filesystem, and from what I've read Gluster will force the<br>
filesystem to allocate an extent for its attributes. 'xfs_bmap' on a<br>
few files shows this is the case.<br>
<br>
A simple 'cat' of every file when laid out in 'native' directories on<br>
the XFS filesystem takes about 3 seconds. A cat of all the files in<br>
the brick's directory on the same filesystem takes about 6.4 seconds,<br>
which I figure is due to the extra I/O for the inode metadata extents<br>
(although not quite certain; the additional extents added about 40%<br>
extra to the disk block allocation, so I'm unsure as to why the time<br>
increase was 100%).<br>
<br>
Doing the same test through the glusterfs mount takes about 25<br>
seconds; roughly four times longer than reading those same files<br>
directly from the brick itself.<br>
<br>
It took 30 seconds until I applied the 'md-cache' settings (for those<br>
variables that still exist in 3.8.8) mentioned in this very helpful<br>
article:<br>
<br>
<a href="http://blog.gluster.org/category/performance/" rel="noreferrer" target="_blank">http://blog.gluster.org/<wbr>category/performance/</a><br>
<br>
So use of the md-cache in a 'cold run' shaved off 5 seconds - due to<br>
common directory LOOKUP operations being cached I guess.<br>
<br>
Output of a 'volume info' is as follows:<br>
<br>
Volume Name: g1<br>
Type: Replicate<br>
Volume ID: bac6cd70-ca0d-4173-9122-<wbr>644051444fe5<br>
Status: Started<br>
Snapshot Count: 0<br>
Number of Bricks: 1 x 2 = 2<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: serverA:/data/brick1<br>
Brick2: serverC:/data/brick1<br>
Options Reconfigured:<br>
transport.address-family: inet<br>
performance.readdir-ahead: on<br>
nfs.disable: on<br>
cluster.self-heal-daemon: enable<br>
features.cache-invalidation: on<br>
features.cache-invalidation-<wbr>timeout: 600<br>
performance.stat-prefetch: on<br>
performance.md-cache-timeout: 60<br>
network.inode-lru-limit: 90000<br>
<br>
The article suggests a value of 600 for<br>
features.cache-invalidation-<wbr>timeout but my Gluster version only<br>
permits a maximum value of 60.<br>
<br>
Network speed between the two VMs is about 120 MBytes/sec - the two<br>
VMs inhabit the same Amazon Virtual Private Cloud - so I don't think<br>
bandwidth is a factor.<br>
<br>
The 400% slowdown is no doubt the penalty incurred in moving to a<br>
proper distributed filesystem. That article and other web pages I've<br>
read all say that each open of a file results in synchronous LOOKUP<br>
operations on all the replicas, so I'm guessing it just takes that<br>
much time for everything to happen before a file can be opened.<br>
Gluster profiling shows that there are 11,198 LOOKUP operations on the<br>
test cat of the 7,987 files.<br>
<br>
As a Gluster newbie I'd appreciate some quick advice if possible -<br>
<br>
1. Is this sort of performance hit - on directories of small files -<br>
typical for such a simple Gluster configuration?<br>
<br>
2. Is there anything I can do to speed things up? :-)<br>
<br>
3. Repeating the 'cat' test immediately after the first test run saw<br>
the time dive from 25 seconds down to 4 seconds. Before I'd set those<br>
md-cache variables it had taken 17 seconds, due, I assume, to the<br>
actual file data being cached in the Linux buffer cache. So those<br>
md-cache settings really did make a change - taking off another 13<br>
seconds - once everything was cached.<br>
<br>
Flushing/invalidating the Linux memory cache made the next test go<br>
back to the 25 seconds. So it seems to me that the md-cache must hold<br>
its contents in the Linux memory buffers cache ... which surprised me,<br>
because I thought a user-space system like Gluster would have the<br>
cache within the daemons or maybe a shared memory segment, nothing<br>
that would be affected by clearing the Linux buffer cache. I was<br>
expecting a run after invalidating the linux cache would take<br>
something between 4 seconds and 25 seconds, with the md-cache still<br>
primed but the file data expired.<br>
<br>
Just out of curiosity in how the md-cache is implemented ... why does<br>
clearing the Linux buffers seem to affect it?<br>
<br>
4. The documentation says that Geo Gluster does 'asynchronous<br>
replication', which is something that would really help, but that it's<br>
'master/slave', so I'm assuming that Geo Gluster won't fulfill my<br>
requirements of both servers being able to occasionally<br>
write/modify/delete files?<br>
<br>
5. In my brick directory I have a '.trashcan' subdirectory - which is<br>
documented - but also a '.glusterfs' directory, which seems to have<br>
lots of magical files in some sort of housekeeping structure.<br>
Surprisingly the total amount of data under .glusterfs is greater than<br>
the total size of the actual files in my test directory. I haven't<br>
seen a description of what .glusterfs is used for ... are they vital<br>
to the operation of Gluster, or can they be deleted? Just curious.<br>
At once stage I had 1.1 GB of files in my volume, which expanded to be<br>
1.5GB in the brick (due to the metadata extents) and a whopping 1.6GB<br>
of extra data materialized under the .glusterfs directory!<br>
<br>
6. Since I'm using Centos I try to stick with things that are<br>
available through the Red Hat repository channel ... so in my looking<br>
for distributed filesystems I saw mention of Ceph. Because I wanted<br>
only a simple replicated filesystem it seemed to me that Ceph - being<br>
based/focused on 'object' storage? - wouldn't be as good a fit as<br>
Gluster. Evil question to a Gluster mailing list - will Ceph give me<br>
any significantly better performance in reading small files?<br>
<br>
I've tried to investigate and find out what I can but I could be<br>
missing something really obvious in my ignorance, so I would<br>
appreciate any quick tips/answers from the experts. Thanks!<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br>
</blockquote></div><br></div>