<div dir="ltr">Hi Ben<div><br></div><div>Thank you. </div><div>I just ran some tests with the same data on EC and R3 volumes (same hardware).</div><div>R3 is a lot faster....</div><div><br></div><div><div>EC</div><div><br></div><div>untar 48.879s</div><div>find 2.993s</div><div>rm 11.244s</div><div><br></div><div>R3</div><div><br></div><div>untar 10.938s</div><div>find 0.722s</div><div>rm 4.144s</div></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Sep 27, 2017 at 3:12 AM, Ben Turner <span dir="ltr"><<a href="mailto:bturner@redhat.com" target="_blank">bturner@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Have you done any testing with replica 2/3? IIRC my replica 2/3 tests out performed EC on smallfile workloads, it may be worth looking into if you can't get EC up to where you need it to be.<br>
<span class="HOEnZb"><font color="#888888"><br>
-b<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
----- Original Message -----<br>
> From: "Dmitri Chebotarov" <<a href="mailto:4dimach@gmail.com">4dimach@gmail.com</a>><br>
> Cc: "gluster-users" <<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>><br>
> Sent: Tuesday, September 26, 2017 9:57:55 AM<br>
> Subject: Re: [Gluster-users] sparse files on EC volume<br>
><br>
> Hi Xavi<br>
><br>
> At this time I'm using 'plain' bricks with XFS. I'll be moving to LVM cached<br>
> bricks.<br>
> There is no RAID for data bricks, but I'll be using hardware RAID10 for SSD<br>
> cache disks (I can use 'writeback' cache in this case).<br>
><br>
> 'small file performance' is the main reason I'm looking at different options,<br>
> i.e. using formated sparse files.<br>
> I spent considerable amount of time tuning 10GB/kernel/gluster to reduce<br>
> latency - the small file performance improved ~50% but it's still no good<br>
> enough, especially when I need to use Gluster for /home folders.<br>
><br>
> I understand limitations and single point of failure in case with sparse<br>
> files. I'm considering different options to provide HA (pacemaker/corosync,<br>
> keepalived or using VMs - RHEV - to deliver storage).<br>
><br>
> Thank you for your reply.<br>
><br>
><br>
> On Tue, Sep 26, 2017 at 3:55 AM, Xavi Hernandez < <a href="mailto:jahernan@redhat.com">jahernan@redhat.com</a> ><br>
> wrote:<br>
><br>
><br>
> Hi Dmitri,<br>
><br>
> On 22/09/17 17:07, Dmitri Chebotarov wrote:<br>
><br>
><br>
><br>
> Hello<br>
><br>
> I'm running some tests to compare performance between Gluster FUSE mount and<br>
> formated sparse files (located on the same Gluster FUSE mount).<br>
><br>
> The Gluster volume is EC (same for both tests).<br>
><br>
> I'm seeing HUGE difference and trying to figure out why.<br>
><br>
> Could you explain what hardware configuration are you using ?<br>
><br>
> Do you have a plain disk for each brick formatted in XFS, or do you have some<br>
> RAID configuration ?<br>
><br>
><br>
><br>
><br>
> Here is an example:<br>
><br>
> GlusterFUSE mount:<br>
><br>
> # cd /mnt/glusterfs<br>
> # rm -f testfile1 ; dd if=/dev/zero of=testfile1 bs=1G count=1<br>
> 1+0 records in<br>
> 1+0 records out<br>
> 1073741824 bytes (1.1 GB) copied, 9.74757 s, *110 MB/s*<br>
><br>
> Sparse file (located on GlusterFUSE mount):<br>
><br>
> # truncate -l 100GB /mnt/glusterfs/xfs-100G.img<br>
> # mkfs.xfs /mnt/glusterfs/xfs-100G.img<br>
> # mount -o loop /mnt/glusterfs/xfs-100G.img /mnt/xfs-100G<br>
> # cd /mnt/xfs-100G<br>
> # rm -f testfile1 ; dd if=/dev/zero of=testfile1 bs=1G count=1<br>
> 1+0 records in<br>
> 1+0 records out<br>
> 1073741824 bytes (1.1 GB) copied, 1.20576 s, *891 MB/s*<br>
><br>
> The same goes for working with small files (i.e. code file, make, etc) with<br>
> the same data located on FUSE mount vs formated sparse file on the same FUSE<br>
> mount.<br>
><br>
> What would explain such difference?<br>
><br>
> First of all, doing tests with relatively small files tends to be misleading<br>
> because of caching capacity of the operating system (to minimize that, you<br>
> can add 'conv=fsync' option to dd). You should do tests with file sizes<br>
> bigger than the amount of physical memory on servers. This way you minimize<br>
> cache effects and see the real sustained performance.<br>
><br>
> A second important point to note is that gluster is a distributed file system<br>
> that can be accessed simultaneously by more than one client. This means that<br>
> consistency must be assured in all cases, which makes things go to bricks<br>
> sooner than local filesystems normally do.<br>
><br>
> In your case, all data saved to the fuse volume will most probably be present<br>
> on bricks once the dd command completes. On the other side, the test through<br>
> the formatted sparse file, most probably, is keeping most of the data in the<br>
> cache of the client machine.<br>
><br>
> Note that using the formatted sparse file makes it possible a better use of<br>
> local cache, improving (relatively) small file access, but on the other<br>
> side, this filesystem can only be used from a single client (single mount).<br>
> If this client fails for some reason, you will loose access to your data.<br>
><br>
><br>
><br>
><br>
> How does Gluster work with sparse files in general? I may move some of the<br>
> data on gluster volumes to formated sparse files..<br>
><br>
> Gluster works fine with sparse files. However you should consider the<br>
> previous points before choosing the formatted sparse files option. I guess<br>
> that the sustained throughput will be very similar for bigger files.<br>
><br>
> Regards,<br>
><br>
> Xavi<br>
><br>
><br>
><br>
><br>
> Thank you.<br>
><br>
><br>
> ______________________________<wbr>_________________<br>
> Gluster-users mailing list<br>
> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> <a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br>
><br>
><br>
><br>
> ______________________________<wbr>_________________<br>
> Gluster-users mailing list<br>
> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> <a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br>
</div></div></blockquote></div><br></div>