[Gluster-users] gluster client performance

[Pavan T C]

On Tuesday 26 July 2011 09:24 PM, John Lalande wrote:
> Thanks for your help, Pavan!
>>
>> Hi John,
>>
>> I would need some more information about your setup to estimate the
>> performance you should get with your gluster setup.
>>
>> 1. Can you provide the details of how disks are connected to the
>> storage boxes ? Is it via FC ? What raid configuration is it using (if
>> at all any) ?
> The disks are 2TB near-line SAS direct attached via a PERC H700
> controller (the Dell PowerEdge R515 has 12 3.5" drive bays). They are in
> a RAID6 config, exported as a single volume, that's split into 3
> equal-size partitions (due to ext4's (well, e2fsprogs') 16 TB limit).
>>
>> 2. What is the disk bandwidth you are getting on the local filesystem
>> on a given storage node ? I mean, pick any of the 10 storage servers
>> dedicated for Gluster Storage and perform a dd as below:
> Seeing an average of 740 MB/s write, 971 GB/s read.

I presume you did this in one of the /data-brick*/export directories ?
Command output with the command line would have been clearer, but thats 
fine.

>
>>
>> 3. What is the IB bandwidth that you are getting between the compute
>> node and the glusterfs storage node? You can run the tool "rdma_bw" to
>> get the details:
> 30407: Bandwidth peak (#0 to #976): 2594.58 MB/sec
> 30407: Bandwidth average: 2593.62 MB/sec
> 30407: Service Demand peak (#0 to #976): 978 cycles/KB
> 30407: Service Demand Avg : 978 cycles/KB

This looks like a DDR connection. "ibv_devinfo -v" will tell a better 
story about the line width and speed of your infiniband connection.
QDR should have a much higher bandwidth.

But that still does not explain why you should get as low as 50 MB/s for 
a single stream single client write when the backend can support direct 
IO throughput of more than 700 MB/s.

On the server, can you collect:

# iostat -xcdh 2 > iostat.log.brickXX

for the duration of the dd command ?

and

# strace -f -o stracelog.server -tt -T -e trace=write,writev -p 
<glusterfsd.pid>
(again for the duration of the dd command)

With the above, I want to measure the delay between the writes coming in 
from the client. iostat will describe the IO scenario on the server.
Once the exercise is done, please attach the iostat.log.brickXX and 
stracelog.server.

Pavan

>
>
> Here's our gluster config:
>
> # gluster volume info data
>
> Volume Name: data
> Type: Distribute
> Status: Started
> Number of Bricks: 30
> Transport-type: rdma
> Bricks:
> Brick1: data-3-1-infiniband.infiniband:/data-brick1/export
> Brick2: data-3-3-infiniband.infiniband:/data-brick1/export
> Brick3: data-3-5-infiniband.infiniband:/data-brick1/export
> Brick4: data-3-7-infiniband.infiniband:/data-brick1/export
> Brick5: data-3-9-infiniband.infiniband:/data-brick1/export
> Brick6: data-3-11-infiniband.infiniband:/data-brick1/export
> Brick7: data-3-13-infiniband.infiniband:/data-brick1/export
> Brick8: data-3-15-infiniband.infiniband:/data-brick1/export
> Brick9: data-3-17-infiniband.infiniband:/data-brick1/export
> Brick10: data-3-19-infiniband.infiniband:/data-brick1/export
> Brick11: data-3-1-infiniband.infiniband:/data-brick2/export
> Brick12: data-3-3-infiniband.infiniband:/data-brick2/export
> Brick13: data-3-5-infiniband.infiniband:/data-brick2/export
> Brick14: data-3-7-infiniband.infiniband:/data-brick2/export
> Brick15: data-3-9-infiniband.infiniband:/data-brick2/export
> Brick16: data-3-11-infiniband.infiniband:/data-brick2/export
> Brick17: data-3-13-infiniband.infiniband:/data-brick2/export
> Brick18: data-3-15-infiniband.infiniband:/data-brick2/export
> Brick19: data-3-17-infiniband.infiniband:/data-brick2/export
> Brick20: data-3-19-infiniband.infiniband:/data-brick2/export
> Brick21: data-3-1-infiniband.infiniband:/data-brick3/export
> Brick22: data-3-3-infiniband.infiniband:/data-brick3/export
> Brick23: data-3-5-infiniband.infiniband:/data-brick3/export
> Brick24: data-3-7-infiniband.infiniband:/data-brick3/export
> Brick25: data-3-9-infiniband.infiniband:/data-brick3/export
> Brick26: data-3-11-infiniband.infiniband:/data-brick3/export
> Brick27: data-3-13-infiniband.infiniband:/data-brick3/export
> Brick28: data-3-15-infiniband.infiniband:/data-brick3/export
> Brick29: data-3-17-infiniband.infiniband:/data-brick3/export
> Brick30: data-3-19-infiniband.infiniband:/data-brick3/export
> Options Reconfigured:
> nfs.disable: on
>