[Gluster-users] Write speed issues with 16MB files and using Gluster fuse mount vs NFS

Wed Jan 27 16:51:31 UTC 2016

Hi all,

We were initially planning on using NFS mounts for our gluster deployment to reduce the amount of client-side changes we had to make to swap out our existing NFS solution.   I ran into an HA issue with NFS (see my other post from this morning), so started looking into the Fuse client as an alternative.  What I found is that I can't seem to get above ~30MB/s with a gluster mount transferring 16MB files, even though a NFS mount form the same client to the same volume reaches ~165MB/s.   This seems like a much more drastic difference that I've seen others mention, but I don't have a good feel for how to troubleshoot this.  Any advice?

Below shows what I'm seeing.     nfs03-poc is the client, and gfs-slc02.mgmt and gfs-slc03.mgmt are the gluster nodes serving the data via RRDNS at gfs-slc.mgmt.   Please let me know if any other info would be helpful.

--

On the client side:

[root at nfs03-poc fio]# cat GlusterConfig.fio
[global]
description=fio load testing...simulating disk activity for writing the WAL file
blocksize=8192
filesize=16m
fsync=0
iomem=malloc
loops=1
numjobs=1
prio=0
runtime=1m
rw=write
size=200M
startdelay=0
time_based

[wal01]
directory=/mnt/gluster/fio_test
ioengine=sync
nrfiles=1

[wal02]
directory=/mnt/gluster/fio_test
ioengine=sync
nrfiles=1

[wal03]
directory=/mnt/gluster/fio_test
ioengine=sync
nrfiles=1

..
..
..

[wal18]
directory=/mnt/gluster/fio_test
ioengine=sync
nrfiles=1

[wal19]
directory=/mnt/gluster/fio_test
ioengine=sync
nrfiles=1

[wal20]
directory=/mnt/gluster/fio_test
ioengine=sync
nrfiles=1

[root at nfs03-poc fio]# iperf -c gfs-slc02.mgmt
------------------------------------------------------------
Client connecting to gfs-slc02.mgmt, TCP port 5001
TCP window size: 19.3 KByte (default)
------------------------------------------------------------
[  3] local 10.124.10.18 port 58877 connected with 10.124.0.91 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  9.96 GBytes  8.56 Gbits/sec

[root at nfs03-poc fio]# iperf -c gfs-slc03.mgmt
------------------------------------------------------------
Client connecting to gfs-slc03.mgmt, TCP port 5001
TCP window size: 19.3 KByte (default)
------------------------------------------------------------
[  3] local 10.124.10.18 port 46043 connected with 10.124.0.88 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  8.69 GBytes  7.47 Gbits/sec

[root at nfs03-poc fio]# umount /mnt/gluster/
[root at nfs03-poc fio]# mount -t nfs -o async gfs-slc.mgmt:/dr_opt_nwea /mnt/gluster/
[root at nfs03-poc fio]# mount | grep gluster
gfs-slc.mgmt:/dr_opt_nwea on /mnt/gluster type nfs (rw,addr=10.124.0.151)
[root at nfs03-poc fio]# fio GlusterConfig.fio -output=Gluster1.txt
[root at nfs03-poc fio]# tail -n 1 Gluster1.txt.0% done] [0K/176.6M/0K /s] [0 /22.6K/0  iops] [eta 00m:00s]
  WRITE: io=9749.1MB, aggrb=166113KB/s, minb=7034KB/s, maxb=9929KB/s, mint=60015msec, maxt=60103msec

[root at nfs03-poc fio]# umount /mnt/gluster/
[root at nfs03-poc fio]# mount -t glusterfs -o direct-io-mode=disable gfs-slc.mgmt:/dr_opt_nwea /mnt/gluster
[root at nfs03-poc fio]# mount | grep gluster
gfs-slc.mgmt:/dr_opt_nwea on /mnt/gluster type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)
[root at nfs03-poc fio]# fio GlusterConfig.fio -output=Gluster2.txt
[root at nfs03-poc fio]# tail -n 1 Gluster2.txt.0% done] [0K/33088K/0K /s] [0 /4136 /0  iops] [eta 00m:00s]
  WRITE: io=1756.8MB, aggrb=29975KB/s, minb=1498KB/s, maxb=1499KB/s, mint=60001msec, maxt=60012msec

On the gluster server side:

[root at gfs-slc02.mgmt ~]$ gluster --version
glusterfs 3.7.5 built on Oct  7 2015 16:27:05

[root at gfs-slc02.mgmt ~]$ gluster volume info dr_opt_nwea
Volume Name: dr_opt_nwea
Type: Replicate
Volume ID: 6ecf855e-e7e0-4479-bde9-f32e5e820365
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: gfs-slc02.mgmt:/data/glusterfs/dr_opt_nwea_brick1/brick1
Brick2: gfs-slc03.mgmt:/data/glusterfs/dr_opt_nwea_brick1/brick1
Options Reconfigured:
performance.readdir-ahead: on
nfs.export-volumes: on
nfs.addr-namelookup: Off
nfs.disable: off
network.ping-timeout: 5
cluster.server-quorum-type: server
cluster.server-quorum-ratio: 51%

[root at gfs-slc02.mgmt ~]$ dd if=/dev/zero of=/data/glusterfs/dr_opt_nwea_brick1/brick1/ddtest bs=4k count=10000
10000+0 records in
10000+0 records out
40960000 bytes (41 MB) copied, 0.0311661 s, 1.3 GB/s

[root at gfs-slc03.mgmt ~]$ dd if=/dev/zero of=/data/glusterfs/dr_opt_nwea_brick1/brick1/ddtest bs=4k count=10000
10000+0 records in
10000+0 records out
40960000 bytes (41 MB) copied, 0.0445559 s, 919 MB/s

---

I've tried with the following volume settings as well, which makes little to no difference on the gluster mount test:

performance.write-behind-window-size: 1MB
performance.io-cache: on
performance.client-io-threads: on
performance.cache-size: 512MB
performance.io-thread-count: 64

Ideas?

Thanks,
Kris

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160127/3bf5ce43/attachment.html>