[Gluster-users] Gluster-users Digest, Vol 57, Issue 32

[Robert Hajime Lanning]

On 01/15/13 07:06, glusterzhxue wrote:
> About the following issue, anyone could help us? Why using VM mounted by
> NFS V3(which is also provided by gluster) could approach full
> bandwidth(100MB/s),while VM mounted by gluster only reach half of
> bandwidth (50MB/s).
> The following is the mount command:
> 1)gluster native client:
> mount -t glusterfs server1:/test-volume /var/lib/nova/instance/
> 2) NFS V3:
> mount -t nfs -o vers=3 server1:/test-volume /var/lib/nova/instance/
> /var/lib/nova/instance/ is the file directory storing VMs
> Thanks.
> ------------------------------------------------------------------------
> zhxue

What is your volume layout?
The output of "gluster volume info test-volume"

What type of network hardware do you have?
What type of ethernet ports on the host running the VMs?
What type of ethernet ports on the hosts containing the storage (the 
gluster servers)?
How are they configured?

My configuration has 4 1Gb ports LACP bonded on the VM host and single 
10Gb ports on the the storage nodes (gluster servers.)

Our volume is: (4 bricks per gluster server)
Volume Name: vol01
Type: Distributed-Replicate
Volume ID: 5ebcaed8-dd9b-4c48-be75-46457fe2e978
Status: Started
Number of Bricks: 24 x 2 = 48
Transport-type: tcp

With the iozone testing on a FUSE mount, I am currently doing, I get:

~100MB/s writes if the replicated bricks chosen hash out in the LACP 
bonding to the different 1Gb ethernet ports.

~50MB/s writes if the replicated bricks chosen hash out in the LACP 
bonding to the same 1Gb ethernet ports.

~20MB/s writes for the test using the O_SYNC option for iozone, no 
matter what 1Gb ethernet ports end up being used.

Reads are always ~100MB/s as it comes from one brick.

If you have a replica volume, then mount using FUSE, the writes are 
duplicated on the client.  If you mount via NFS, the writes are 
duplicated on the server.

Also of note, with the iozone going full speed, the FUSE client takes 
50% of a CPU core.

-- 
Mr. Flibble
King of the Potato People