[Gluster-users] Performance

[Hiren Joshi]

> -----Original Message-----
> From: Mark Mielke [mailto:mark at mark.mielke.cc] 
> Sent: 12 August 2009 16:35
> To: Hiren Joshi
> Cc: gluster-users at gluster.org
> Subject: Re: [Gluster-users] Performance
> 
> On 08/12/2009 11:24 AM, Hiren Joshi wrote:
> > Hello all,
> >
> > I have 2 servers both exporting 36 partitions. The client 
> mirrors the 36
> > partitions on each server and puts them all into a DHT.
> >
> > dd if=/dev/zero of=/home/webspace_glust/zeros bs=1024 count=1024000
> > Takes 8 minutes, compared to 30 seconds vi NFS, now granted 
> I'm using
> > mostly default settings but a transfer rate of 2.1Mb/s vs 
> about 40Mb/s
> > seems low.
> >
> > What am I doing wrong? Where should I start with the 
> performance tuning?
> 
> How about start with a 1:1 comparison? Unless is somehow doing 
> mirroring, you should remove the mirroring from your solution before 
> comparing.


Fair point, I've put on a single brick (no distribution no mirror) and I
get:
time dd if=/dev/zero of=/home/webspace_glust/zeros bs=1024 count=1024000
1024000+0 records in
1024000+0 records out
1048576000 bytes (1.0 GB) copied, 109.289 seconds, 9.6 MB/s

real    1m49.291s
user    0m0.156s
sys     0m2.701s

Compared to:
time dd if=/dev/zero of=/home/webspace_nfs/josh/zeros bs=1024
count=1024000
1024000+0 records in
1024000+0 records out
1048576000 bytes (1.0 GB) copied, 29.4801 seconds, 35.6 MB/s

real    0m29.526s
user    0m0.486s
sys     0m1.927s


For NFS.

> 
> Are you using striping? If you are not using striping, then the file 
> "/home/webspace_glust" is going to be assigned to a single partition, 
> and you would only be using a mirror of one partition.

No striping, thanks, I didn't consider this, so I tried:
time for x in `seq 1 1 1000`; do dd if=/dev/zero
of=/home/webspace_nfs/zeros$x bs=1024 count=1024; echo $x; done
real    0m45.292s
user    0m0.760s
sys     0m3.209s
(about 20-40MB/s)

and
time for x in `seq 1 1 1000`; do dd if=/dev/zero
of=/home/webspace_glust/zeros$x bs=1024 count=1024; echo $x; done
real    1m58.931s
user    0m0.432s
sys     0m6.412s
(about 8-10MB/s)

> 
> 36 partitions on each server - the word "partition" is ambiguous. Are 
> they 36 separate drives? Or multiple partitions on the same drive. If 
> multiple partitions on the same drive, this would be a bad 
> idea, as it 
> would require the disk head to move back and forth between the 
> partitions, significantly increasing the latency, and therefore 
> significantly reducing the performance. If each partition is 
> on its own 
> drive, you still won't see benefit unless you have many clients 
> concurrently changing many different files. In your above case, it's 
> touching a single file in sequence, and having a cluster is 
> costing you 
> rather than benefitting you.

We went with 36 partitions (on a single raid 6 drive) incase we got file
system corruption, it would take less time to fsck a 100G partition than
a 3.6TB one. Would a 3.6TB single disk be better?


> 
> As for the 2.1Mb/s vs 40Mb/s, I have no clue. I'm new to 
> Gluster myself, 
> and I have yet to install it on my node cluster (10 servers) 
> and perform 
> timing myself.
> 
> Cheers,
> mark
> 
> -- 
> Mark Mielke<mark at mielke.cc>
> 
>