<div dir="ltr">Hi <span style="font-size:12.8px">Xavi</span><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">At this time I'm using 'plain' bricks with XFS. I'll be moving to LVM cached bricks. </span></div><div><span style="font-size:12.8px">There is no RAID for data bricks, but I'll be using hardware RAID10 for SSD cache disks (I can use 'writeback' cache in this case).</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">'small file performance' is the main reason I'm looking at different options, i.e. using formated sparse files. </span></div><div><span style="font-size:12.8px">I spent considerable amount of time tuning 10GB/kernel/gluster to reduce latency - the small file performance improved ~50% but it's still no good enough, especially when I need to use Gluster for /home folders.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">I understand limitations and single point of failure in case with sparse files. I'm considering different options to provide HA (pacemaker/corosync, keepalived or using VMs - RHEV - to deliver storage). </span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Thank you for your reply.</span></div><div><span style="font-size:12.8px"><br></span></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Sep 26, 2017 at 3:55 AM, Xavi Hernandez <span dir="ltr"><<a href="mailto:jahernan@redhat.com" target="_blank">jahernan@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Dmitri,<span class=""><br>
<br>
On 22/09/17 17:07, Dmitri Chebotarov wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Hello<br>
<br>
I'm running some tests to compare performance between Gluster FUSE mount and formated sparse files (located on the same Gluster FUSE mount).<br>
<br>
The Gluster volume is EC (same for both tests).<br>
<br>
I'm seeing HUGE difference and trying to figure out why.<br>
</blockquote>
<br></span>
Could you explain what hardware configuration are you using ?<br>
<br>
Do you have a plain disk for each brick formatted in XFS, or do you have some RAID configuration ?<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
<br>
Here is an example:<br>
<br>
GlusterFUSE mount:<br>
<br>
# cd /mnt/glusterfs<br>
# rm -f testfile1 ; dd if=/dev/zero of=testfile1 bs=1G count=1<br>
1+0 records in<br>
1+0 records out<br></span>
1073741824 bytes (1.1 GB) copied, 9.74757 s, *110 MB/s*<span class=""><br>
<br>
Sparse file (located on GlusterFUSE mount):<br>
<br>
# truncate -l 100GB /mnt/glusterfs/xfs-100G.img<br>
# mkfs.xfs /mnt/glusterfs/xfs-100G.img<br>
# mount -o loop /mnt/glusterfs/xfs-100G.img /m<wbr>nt/xfs-100G<br>
# cd /mnt/xfs-100G<br>
# rm -f testfile1 ; dd if=/dev/zero of=testfile1 bs=1G count=1<br>
1+0 records in<br>
1+0 records out<br></span>
1073741824 bytes (1.1 GB) copied, 1.20576 s, *891 MB/s*<span class=""><br>
<br>
The same goes for working with small files (i.e. code file, make, etc) with the same data located on FUSE mount vs formated sparse file on the same FUSE mount.<br>
<br>
What would explain such difference?<br>
</span></blockquote>
<br>
First of all, doing tests with relatively small files tends to be misleading because of caching capacity of the operating system (to minimize that, you can add 'conv=fsync' option to dd). You should do tests with file sizes bigger than the amount of physical memory on servers. This way you minimize cache effects and see the real sustained performance.<br>
<br>
A second important point to note is that gluster is a distributed file system that can be accessed simultaneously by more than one client. This means that consistency must be assured in all cases, which makes things go to bricks sooner than local filesystems normally do.<br>
<br>
In your case, all data saved to the fuse volume will most probably be present on bricks once the dd command completes. On the other side, the test through the formatted sparse file, most probably, is keeping most of the data in the cache of the client machine.<br>
<br>
Note that using the formatted sparse file makes it possible a better use of local cache, improving (relatively) small file access, but on the other side, this filesystem can only be used from a single client (single mount). If this client fails for some reason, you will loose access to your data.<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
How does Gluster work with sparse files in general? I may move some of the data on gluster volumes to formated sparse files..<br>
</blockquote>
<br></span>
Gluster works fine with sparse files. However you should consider the previous points before choosing the formatted sparse files option. I guess that the sustained throughput will be very similar for bigger files.<br>
<br>
Regards,<br>
<br>
Xavi<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Thank you.<br>
<br>
<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
<br>
</blockquote>
</blockquote></div><br></div>