[Gluster-users] Performance

Fri Apr 22 00:49:02 UTC 2011

After lot of digging today finaly figured out that it's not really
using PERC controller but some Fusion MPT. Then it wasn't clear which
tool it supports. Finally I installed lsiutil and was able to change
the cache size.

[root at dsdb1 ~]# lspci|grep LSI
02:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E
PCI-Express Fusion-MPT SAS (rev 08)
[root at dsdb1 ~]# dd if=/dev/zero of=/data/big.file bs=128k count=40k oflag=direct
1024+0 records in
1024+0 records out
134217728 bytes (134 MB) copied, 0.742517 seconds, 181 MB/s

I compared this with SW RAID mdadm that I created yesterday on one of
the servers and I get around 300MB/s. I will test out first with what
we have before destroying and testing with mdadm.

Thanks for your help that led me to this path. Another question I had
was when creating mdadm RAID does it make sense to use multipathing?

On Thu, Apr 21, 2011 at 10:30 AM, Joe Landman
<landman at scalableinformatics.com> wrote:
> In the lab working on fixing a unit will be about an hour
> Please pardon brevity and spelling errors, sent from my BlackBerry
>
> -----Original Message-----
> From: Mohit Anchlia <mohitanchlia at gmail.com>
> Date: Thu, 21 Apr 2011 10:29:52
> To: <landman at scalableinformatics.com>; <gluster-users at gluster.org>
> Subject: Re: [Gluster-users] Performance
>
> Hi Joe,
>
> When you get chance can you please at my mail? It will be helpful to
> get your advise.
>
>
> On Wed, Apr 20, 2011 at 6:16 PM, Mohit Anchlia <mohitanchlia at gmail.com> wrote:
>> Yes indeed. As soon as I sent you last email I realized that and set
>> the write back option. Now I get 130MB/s better but still nowhere
>> close to 600MB/s as advertised or what others say one should see.
>>
>> What are your recommendations about HW choice? What is more
>> preferrable and better?
>>
>> Another question do I need to set WCE on all the disks first before
>> creating RAID0? Or can I do that after creating RAID0? I tried to set
>> WCE on existing RAID0 but it fails "change_mode_page: failed fetching
>> page: Caching (SBC)".
>>
>>  sdparm --set=WCE --save /dev/sda
>>    /dev/sda: FUJITSU   MBD2300RC         D809
>>
>> dd if=/dev/zero of=/dev/sda bs=128k count=80k oflag=direct
>> 81920+0 records in
>> 81920+0 records out
>> 10737418240 bytes (11 GB) copied, 82.8041 seconds, 130 MB/s
>>
>> Thanks for your help as always.
>>
>> On Wed, Apr 20, 2011 at 5:53 PM, Joe Landman
>> <landman at scalableinformatics.com> wrote:
>>> On 04/20/2011 07:50 PM, Mohit Anchlia wrote:
>>>>
>>>> I did that but it looks the same. I did get an error even though it
>>>> says write-caching is on.
>>>>
>>>> [root at dslg1 ~]# hdparm -W1 /dev/sda
>>>>
>>>> /dev/sda:
>>>>  setting drive write-caching to 1 (on)
>>>>  HDIO_DRIVE_CMD(setcache) failed: Invalid argument
>>>> [root at dslg1 ~]# hdparm /dev/sda
>>>
>>> You might need sdparm
>>>
>>>        sdparm -a /dev/sda | grep WCE
>>>
>>> With WCE on I see
>>>
>>>        [root at smash ~]# sdparm -a /dev/sda | grep WCE
>>>          WCE         1
>>>
>>> and with it off, I see
>>>
>>>        [root at smash ~]# hdparm -W0 /dev/sda
>>>
>>>        /dev/sda:
>>>         setting drive write-caching to 0 (off)
>>>         write-caching =  0 (off)
>>>
>>>        [root at smash ~]# sdparm -a /dev/sda | grep WCE
>>>          WCE         0
>>>
>>> You might need to change WCE using
>>>
>>>        sdparm --set=WCE -a /dev/sda
>>>
>>> or similar ...
>>>
>>>>
>>>> /dev/sda:
>>>>  readonly     =  0 (off)
>>>>  readahead    = 256 (on)
>>>>  geometry     = 36472/255/63, sectors = 585937500, start = 0
>>>> [root at dslg1 ~]# [A
>>>> [root at dslg1 ~]# dd if=/dev/zero of=/dev/sda bs=128k count=1k oflag=direct
>>>> 1024+0 records in
>>>> 1024+0 records out
>>>> 134217728 bytes (134 MB) copied, 8.10005 seconds, 16.6 MB/s
>>>>
>>>>
>>>> On Wed, Apr 20, 2011 at 5:45 PM, Joe Landman
>>>> <landman at scalableinformatics.com>  wrote:
>>>>>
>>>>> On 04/20/2011 07:28 PM, Mohit Anchlia wrote:
>>>>>>
>>>>>> dd of=/dev/null if=/dev/sda bs=128k count=80k iflag=direct
>>>>>> 81920+0 records in
>>>>>> 81920+0 records out
>>>>>> 10737418240 bytes (11 GB) copied, 83.8293 seconds, 128 MB/s
>>>>>
>>>>> Ok, this is closer to what I was expecting (really ~150 MB/s would make
>>>>> more
>>>>> sense to me, but I can live with 128 MB/s).
>>>>>
>>>>> The write speed is definitely problematic.  I am wondering if write cache
>>>>> is
>>>>> off, and other features are turned off in strange ways.
>>>>>
>>>>> This is a 2 year old SATA disk
>>>>>
>>>>> [root at smash ~]# dd if=/dev/zero of=/dev/sda2 bs=128k oflag=direct
>>>>> dd: writing `/dev/sda2': No space left on device
>>>>> 16379+0 records in
>>>>> 16378+0 records out
>>>>> 2146798080 bytes (2.1 GB) copied, 20.8322 s, 103 MB/s
>>>>>
>>>>> Write cache is enabled.  Turning write cache off (might not be so
>>>>> relevant
>>>>> for a RAID0),
>>>>>
>>>>> [root at smash ~]# hdparm -W /dev/sda
>>>>>
>>>>> /dev/sda:
>>>>>  write-caching =  1 (on)
>>>>> [root at smash ~]# hdparm -W0 /dev/sda
>>>>>
>>>>> /dev/sda:
>>>>>  setting drive write-caching to 0 (off)
>>>>>  write-caching =  0 (off)
>>>>>
>>>>> [root at smash ~]# dd if=/dev/zero of=/dev/sda2 bs=128k oflag=direct
>>>>> dd: writing `/dev/sda2': No space left on device
>>>>> 16379+0 records in
>>>>> 16378+0 records out
>>>>> 2146798080 bytes (2.1 GB) copied, 155.636 s, 13.8 MB/s
>>>>>
>>>>> See if you can do an
>>>>>
>>>>>        hdparm -W1 /dev/sda
>>>>>
>>>>> and see if it has any impact on the write speed.  If you are using a
>>>>> RAID0,
>>>>> safety isn't so much on your mind anyway, so you can see if you can
>>>>> adjust
>>>>> your cache settings.  If this doesn't work, you might need to get to the
>>>>> console and tell it to allow caching.
>>>>>
>>>>>
>>>>> --
>>>>> Joseph Landman, Ph.D
>>>>> Founder and CEO
>>>>> Scalable Informatics, Inc.
>>>>> email: landman at scalableinformatics.com
>>>>> web  : http://scalableinformatics.com
>>>>>       http://scalableinformatics.com/sicluster
>>>>> phone: +1 734 786 8423 x121
>>>>> fax  : +1 866 888 3112
>>>>> cell : +1 734 612 4615
>>>>>
>>>
>>>
>>> --
>>> Joseph Landman, Ph.D
>>> Founder and CEO
>>> Scalable Informatics, Inc.
>>> email: landman at scalableinformatics.com
>>> web  : http://scalableinformatics.com
>>>       http://scalableinformatics.com/sicluster
>>> phone: +1 734 786 8423 x121
>>> fax  : +1 866 888 3112
>>> cell : +1 734 612 4615
>>>
>>
>