[Gluster-users] Performance

[Joe Landman]

On 04/20/2011 02:29 PM, Mohit Anchlia wrote:
> Please find
>
>
> [root at dsdb1 ~]# cat /proc/sys/vm/drop_caches
> 3
> [root at dsdb1 ~]# dd if=/dev/zero of=/data/big.file bs=128k count=80k oflag=direct
>
> 81920+0 records in
> 81920+0 records out
> 10737418240 bytes (11 GB) copied, 521.553 seconds, 20.6 MB/s

Suddenly this makes a great deal more sense.

> [root at dsdb1 ~]#
> [root at dsdb1 ~]# dd if=/dev/zero of=/data/big.file bs=128k count=80k iflag=direct
> dd: opening `/dev/zero': Invalid argument
> [root at dsdb1 ~]# dd of=/dev/null if=/data/big.file bs=128k iflag=direct
> 81920+0 records in
> 81920+0 records out
> 10737418240 bytes (11 GB) copied, 37.854 seconds, 284 MB/s
> [root at dsdb1 ~]#

About what I expected.

Ok.  Uncached OS writes get you to 20MB/s.  Which is about what you are 
seeing with the fuse mount and a dd.  So I think we understand the write 
side.

The read side is about where I expected (lower actually, but not by 
enough that I am concerned).

You can try changing bs=2M count=6k on both to see the effect of larger 
blocks.  You should get some improvement.

I think we need to dig into the details of that RAID0 construction now. 
  This might be something better done offlist (unless everyone wants to 
see the gory details of digging into the hardware side).

My current thought is that this is a hardware issue, and not a gluster 
issue per se, but that there are possibilities for improving performance 
on the gluster side of the equation.

Short version:  PERC is not fast (never has been), and it is often a bad 
choice for high performance.  You are often better off building an MD 
RAID using the software tools in Linux, it will be faster.  Think of 
PERC as an HBA with some modicum of built in RAID capability.  You don't 
really want to use that capability if possible, but you do want to use 
the HBA.

Longer version:  Likely a striping issue, or a caching issue (need to 
see battery state, cache size, etc.), not to mention the slow chip.  Are 
the disk write caches off or on (guessing off which is the right thing 
to do for some workloads but it does impact performance).  Also, the 
RAID cpu in PERC (its a rebadged LSI) is very low performance in 
general, and specifically not terribly good even at RAID0.  These are 
direct writes, skipping OS cache.  They will let you see how fast the 
underlying hardware is, and if it can handle the amount of data you want 
to shove onto disks.

Here is my desktop:

root at metal:/local2/home/landman# dd if=/dev/zero of=/local2/big.file 
bs=128k count=80k oflag=direct
81920+0 records in
81920+0 records out
10737418240 bytes (11 GB) copied, 64.7407 s, 166 MB/s

root at metal:/local2/home/landman# dd if=/dev/zero of=/local2/big.file 
bs=2M count=6k oflag=direct
6144+0 records in
6144+0 records out
12884901888 bytes (13 GB) copied, 86.0184 s, 150 MB/s



and a server in the lab

[root at jr5-1 ~]# dd if=/dev/zero of=/data/big.file bs=128k count=80k 
oflag=direct
81920+0 records in
81920+0 records out
10737418240 bytes (11 GB) copied, 11.0948 seconds, 968 MB/s

[root at jr5-1 ~]# dd if=/dev/zero of=/data/big.file bs=2M count=6k 
oflag=direct
6144+0 records in
6144+0 records out
12884901888 bytes (13 GB) copied, 5.11935 seconds, 2.5 GB/s


Gluster will not be faster than the bare metal (silicon).  It may hide 
some of the issues with caching.  But it is bounded by how fast you can 
push to or pull bits from the media.

In an "optimal" config, the 4x SAS 10k RPM drives should be able to 
sustain ~600 MB/s write.  Reality will be less than this, guessing 
250-400 MB/s in most cases.  This is still pretty low in performance.

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615