[Gluster-users] Glusterfs with Infiniband tips

[Andrei Mikhailovsky]

Hello guys, 

I was wondering if someone could share their glusterfs volume and system settings if you are running glusterfs with infiniband networking. In particular I am interested in using the glusterfs + infiniband + kvm for virtualisation. However, any other implementation would also be useful for me. 

I've tried various versions of glusterfs (versions 3.2, 3.3 and 3.4beta) over the past year. My experience with gluster + infiniband is below. The underlying fs is a zfs pool with compression=lz4 made of 8 disks in raid 10 equivalent setup. Filesystem itself can do over 900MB/s with single thread and around 1050MB/s with multiple threads based on dd with bs=128K with 100GB data file generated from /dev/urandom. 

1. volume transport set to RDMA is very very unstable. disconnecting all the time even with minimal load. I've found that when it does work the speeds are very fast - was getting around 1.4 - 1.6GB/s with 16 threads on 40gbit/s QDR link. 


2. volume transport set to tcp works over IPoIB interface, but i've not managed to get a good level of performance with real data. I was maxing about 220MB/s with a single thread and around 550MB/s with 12 or more threads. Benchmarking with a huge file made from /dev/zero gives me around 800-900MB/s with multiple threads and around 750MB/s single thread. Not sure why would glusterfs has a good level of performance when reading from RAM but is only working at 1/2 of the disk speed when reading real data from the disk? 

3. Glusterfs with NFS also gives me crappy results. Running NFS directly from zfs pool gives me a far better performance. 

4. Tried various performance related options for glusterfs, but with only small performance increase (( 

5. Clients perform horribly when additing new bricks to the cluser. By that I mean over 2 hours to run "time ls -lhR /glustervolume" which contains just 10 files. Basically, mounted fs is completely unusable during this time! 

6. Virtual machines with volumes stored on the glusterfs mounted filesystem have an extremely slow performance. I've not managed to get speeds over 50MB/s using cache=none option. 

If any of you guys managed to get a decent level of performance I would be grateful if you share your experience and any tips. 

Many thanks 

Andrei 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130512/b187b34d/attachment.html>