[Gluster-users] Performance

[Joe Landman]

On 04/19/2011 08:52 PM, Mohit Anchlia wrote:
> I am getting miserable performance in LAN setup with 6 servers with
> distributed and 2 replicas with 1GigE. I am using native glsuter
> clients (mount -t glusterfs server1:/vol /mnt) I am only able to get
> 20Mb/s. This is some of the output from sar:
>
> Each server has 4 10K SAS drives RAID0. This is a new setup and I
> expected to get much much higher performance. Can someone please help
> with recommendations?

It seems like I just handled a case like this a few months ago ...

What does your IO workload actually look like?  Much more interested in 
iostat like output (though dstat also works very well for bandwidth 
heavy loads).

Gluster isn't going to do well with small IO operations without serious 
caching (NFS client).  Despite the fact that these are RAID0 across 4x 
10kRPM SAS drives, this is *not* a high performance IO system in most 
senses of the definition.  The design of the storage should be driven by 
the application and anticipated workloads.

More to the point, what specifically are your goals in terms of 
throughput/bandwidth ... what will your storage loads look like?

What RAID cards are you using (if any)?  If software raid, could you 
report output of

	mdadm --detail /dev/MD

where /dev/MD is your MD raid device.  Which SAS 10k disks are you 
using?  How are they connected to the machine if not through a RAID 
card?  Is the RAID0 a hardware or software RAID?

> !sar
> sar -B 1 100
>
>
> 05:44:38 PM  pgpgin/s pgpgout/s   fault/s  majflt/s
> 05:44:39 PM      0.00      0.00   1413.00      0.00
> 05:44:40 PM      0.00  29896.00     29.00      0.00
> 05:44:41 PM      0.00   4510.89   1523.76      0.00
> 05:44:42 PM      0.00     16.16     20.20      0.00
> 05:44:43 PM      0.00     12.00     16.00      0.00
> 05:44:44 PM      0.00    102.97     15.84      0.00
> 05:44:45 PM      0.00  21100.00     14.00      0.00
> 05:44:46 PM      0.00   8092.00     19.00      0.00

sar isn't too useful for figuring out whats going on in the io channel. 
iostat is much better.  dstat, atop, vmstat are all specifically good 
tools.  If you want too much data (e.g. it gathers everything of value), 
use collectl, with a 1 second interval, and the right options.


-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615