[Gluster-users] Performance gap between clients

[Pavel Szalbot]

Hello everybody,

I am experiencing peculiar performance difference on my client nodes.

One node is blank Ubuntu (Xenial), second is also Xenial with a web server
(nginx) that serves media files stored on disk image that is on Gluster
volume.
Both clients are 3.8.5, 10Gbe NICs used for Gluster network, 32 GB RAM on
both.
Gluster servers are 64GB RAM, 6 SSDs on each.
Switching is done on Juniper EX4550, load is very low, MTU 9000 (almost no
difference to 1500).

I get about 300MB/s on the node with nginx and only 160MB/s on the second
one.

[global]
filename=/mnt/gluster_vol/fio
ioengine=libaio
direct=1
bs=256k
rw=read
iodepth=1
numjobs=1
size=8192m

I did check packet drops, if 10Gbe is actually used, vmstat for iowait,
traffic distribution on server nodes, sysctl -a diff, iperf between
clients, servers, client-server, and probably a dozen of other things. I
tried to install nginx on the "blank" client with but it did not make any
difference. Slower node actually has more RAM available.

Do you have any ideas what could cause this?

Volume options:
cluster.self-heal-daemon: enable
nfs.disable: on
performance.readdir-ahead: on
performance.cache-size: 1GB
performance.client-io-threads: on
performance.io-thread-count: 64
performance.read-ahead: off

-ps
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20161023/c78cf4e8/attachment.html>