[Gluster-users] Poor Gluster performance

[Ben Turner]

----- Original Message -----
> From: "Lars Hanke" <debian at lhanke.de>
> To: "Ben Turner" <bturner at redhat.com>
> Cc: gluster-users at gluster.org
> Sent: Wednesday, February 18, 2015 5:09:19 PM
> Subject: Re: [Gluster-users] Poor Gluster performance
> 
> Am 18.02.2015 um 22:05 schrieb Ben Turner:
> > ----- Original Message -----
> >> From: "Lars Hanke" <debian at lhanke.de>
> >> To: gluster-users at gluster.org
> >> Sent: Wednesday, February 18, 2015 3:01:54 PM
> >> Subject: [Gluster-users] Poor Gluster performance
> >>
> >> I set up a distributed, replicated volume consisting of just 2 bricks on
> >> two physical nodes. The nodes are peered using a dedicated GB ethernet
> >> and can be accessed from the clients using a separate GB ethernet NIC.
> >>
> >> Doing a simple dd performance test I see about 11 MB/s for read and
> >> write. Running a local setup, i.e. both bricks on the same machine and
> >> local mount, I saw even 500 MB/s. So network sould be the limiting
> >> factor. But using NFS or CIFS on the same network I see 110 MB/s.
> >>
> >> Is gluster 10 times slower than NFS?
> >
> > Something is going on there.  On my gigabit setups I see 100-120 MB / sec
> > writes for pure distribute and about 45-55 MB / sec with replica 2.  What
> > block size are you using?  I could see that if you were writing something
> > like 4k or under but 64k and up you should be getting about what I said.
> > Can you tell me more about your test?
> 
> Block size is 50M:
> 
> root at gladsheim:/# mount -t glusterfs node2:/test ~/mnt
> root at gladsheim:/# dd if=/dev/zero of=~/mnt/testfile.null bs=50M count=10
> 10+0 records in
> 10+0 records out
> 524288000 bytes (524 MB) copied, 46.6079 s, 11.2 MB/s
> root at gladsheim:/# dd if=~/mnt/testfile.null of=/dev/null bs=50M count=10
> 10+0 records in
> 10+0 records out
> 524288000 bytes (524 MB) copied, 45.7487 s, 11.5 MB/s

This looks like the NICs may only be negotiating to 100Mb(max theoretical of 12.5 MB / sec), can you check ethtool on all of your NICs?  Also I like to run iperf between servers and clients and servers and servers before I do anything with gluster, if you aren't getting ~line speed with iperf gluster wont be able to either.  Double check you NICs and your backend and see if you can spot the bottleneck at either of those layers.

-b
 
> It doesn't depend on whether I use node1 or node2 for the mount.
> 
> BTW: does the cut of the bandwidth to half in replicated mode mean that
> the client writes to both nodes, i.e. doubles the network load on the
> client side network? I hoped that replication would be run on the server
> side network.
> 
> Regards,
>   - lars.
> 
>