[Gluster-users] GlusterFS with FUSE slow vs ZFS volume

[ML mail]

Hello,

I am testing GlusterFS for the first time and have installed the latest GlusterFS 3.5 stable version on Debian 7 on brand new SuperMicro hardware with ZFS instead of hardware RAID. My ZFS pool is a RAIDZ-2 with 6 SATA disks of 2 TB each.

After setting up a first and single test brick on my currently single test node I wanted first to see how much slower will GlusterFS be compared to writting directly to the ZFS volume. For that purpose I have mounted my GlusterFS volume locally on the same server using FUSE.

For my tests I have used bonnie++ with the command "bonnie++ -n16 -b" and I must say I am quite shocked to see that with this current setup GlusterFS slows down the whole file system with a factor of approximately 6 to 8. For example:

ZFS volume

Sequential output by block (read): 936 MB/sec
Sequential input by block (write): 1520 MB/sec


GlusterFS on top of same ZFS volume mounted with FUSE
Sequential output by block (read): 114 MB/sec
Sequential input by block (write): 312 MB/sec


Now I was wondering if such a performance drop on a single GlusterFS node is expected? If not is it maybe ZFS which is messing up things?

bonnie++ took 3 minutes to rune on the ZFS volume and 18 minutes on the GlusterFS mount. I have copied the bonnie++ results below just in case in CVS format:

1.96,1.96,ZFS,1,1422907597,31960M,,170,99,936956,94,484417,74,463,99,1520120,98,815.4,41,16,,,,,3376,26,+++++,+++,3109,22,3261,21,+++++,+++,3305,20,66881us,15214us,84887us,23648us,53641us,93322us,39607us,363us,298ms,136ms,18us,176ms
1.96,1.96,GFS,1,1422897979,31960M,,16,17,114223,20,92610,20,+++++,+++,312557,14,444.5,6,16,,,,,385,3,5724,5,916,4,357,3,2044,4,750,4,550ms,9715us,23094us,3334us,125ms,90070us,154ms,8609us,17570us,67180us,4116us,7879us

Maybe they are a few performance tuning trick that I am not aware of?

Let me know if I should provide any more information. In advance thanks for your comments.

Best regards
ML