[Gluster-users] GlusterFS with FUSE slow vs ZFS volume

[Kiran Patil]

Hi,

We are using fio(https://github.com/axboe/fio) for load/stress testing.

We have not done performance check on a single node.

I will try to verify it.

Thanks,
Kiran.

On Thu, Feb 5, 2015 at 4:47 PM, Pranith Kumar Karampuri
<pkarampu at redhat.com> wrote:
> +Kiran Patil may know about this.
>
> Pranith
> On 02/03/2015 12:56 AM, ML mail wrote:
>>
>> Hello,
>>
>> I am testing GlusterFS for the first time and have installed the latest
>> GlusterFS 3.5 stable version on Debian 7 on brand new SuperMicro hardware
>> with ZFS instead of hardware RAID. My ZFS pool is a RAIDZ-2 with 6 SATA
>> disks of 2 TB each.
>>
>> After setting up a first and single test brick on my currently single test
>> node I wanted first to see how much slower will GlusterFS be compared to
>> writting directly to the ZFS volume. For that purpose I have mounted my
>> GlusterFS volume locally on the same server using FUSE.
>>
>> For my tests I have used bonnie++ with the command "bonnie++ -n16 -b" and
>> I must say I am quite shocked to see that with this current setup GlusterFS
>> slows down the whole file system with a factor of approximately 6 to 8. For
>> example:
>>
>> ZFS volume
>>
>> Sequential output by block (read): 936 MB/sec
>> Sequential input by block (write): 1520 MB/sec
>>
>>
>> GlusterFS on top of same ZFS volume mounted with FUSE
>> Sequential output by block (read): 114 MB/sec
>> Sequential input by block (write): 312 MB/sec
>>
>>
>> Now I was wondering if such a performance drop on a single GlusterFS node
>> is expected? If not is it maybe ZFS which is messing up things?
>>
>> bonnie++ took 3 minutes to rune on the ZFS volume and 18 minutes on the
>> GlusterFS mount. I have copied the bonnie++ results below just in case in
>> CVS format:
>>
>>
>> 1.96,1.96,ZFS,1,1422907597,31960M,,170,99,936956,94,484417,74,463,99,1520120,98,815.4,41,16,,,,,3376,26,+++++,+++,3109,22,3261,21,+++++,+++,3305,20,66881us,15214us,84887us,23648us,53641us,93322us,39607us,363us,298ms,136ms,18us,176ms
>>
>> 1.96,1.96,GFS,1,1422897979,31960M,,16,17,114223,20,92610,20,+++++,+++,312557,14,444.5,6,16,,,,,385,3,5724,5,916,4,357,3,2044,4,750,4,550ms,9715us,23094us,3334us,125ms,90070us,154ms,8609us,17570us,67180us,4116us,7879us
>>
>> Maybe they are a few performance tuning trick that I am not aware of?
>>
>> Let me know if I should provide any more information. In advance thanks
>> for your comments.
>>
>> Best regards
>> ML
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>
>