[Gluster-users] Performance

[Joe Landman]

On 04/26/2011 05:48 PM, Mohit Anchlia wrote:
> I am not sure how valid this performance url is
>
> http://www.gluster.com/community/documentation/index.php/Guide_to_Optimizing_GlusterFS
>
> Does it make sense to separate out the journal and create mkfs -I 256?
>
> Also, if I already have a file system on a different partition can I
> still use it to store journal from other partition without corrupting
> the file system?

Journals are small write heavy.  You really want a raw device for them. 
  You do not want file system caching underneath them.

Raw partition for an external journal is best.  Also, understand that 
ext* suffers badly under intense parallel loads.  Keep that in mind as 
you make your file system choice.

>
> On Thu, Apr 21, 2011 at 7:23 PM, Joe Landman
> <landman at scalableinformatics.com>  wrote:
>> On 04/21/2011 08:49 PM, Mohit Anchlia wrote:
>>>
>>> After lot of digging today finaly figured out that it's not really
>>> using PERC controller but some Fusion MPT. Then it wasn't clear which
>>
>> PERC is a rebadged LSI based on the 1068E chip.
>>
>>> tool it supports. Finally I installed lsiutil and was able to change
>>> the cache size.
>>>
>>> [root at dsdb1 ~]# lspci|grep LSI
>>> 02:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E
>>> PCI-Express Fusion-MPT SAS (rev 08)
>>
>>   This looks like PERC.  These are roughly equivalent to the LSI 3081 series.
>>   These are not fast units.  There is a variant of this that does RAID6, its
>> usually available as a software update or plugin module (button?) to this.
>>   I might be thinking of the 1078 chip though.
>>
>>   Regardless, these are fairly old designs.
>>
>>
>>> [root at dsdb1 ~]# dd if=/dev/zero of=/data/big.file bs=128k count=40k
>>> oflag=direct
>>> 1024+0 records in
>>> 1024+0 records out
>>> 134217728 bytes (134 MB) copied, 0.742517 seconds, 181 MB/s
>>>
>>> I compared this with SW RAID mdadm that I created yesterday on one of
>>> the servers and I get around 300MB/s. I will test out first with what
>>> we have before destroying and testing with mdadm.
>>
>> So the software RAID is giving you 300 MB/s and the hardware 'RAID' is
>> giving you ~181 MB/s?  Seems a pretty simple choice :)
>>
>> BTW: The 300MB/s could also be a limitation of the PCIe channel interconnect
>> (or worse, if they hung the chip off a PCIx bridge).  The motherboard
>> vendors are generally loathe to put more than a few PCIe lanes for handling
>> SATA, Networking, etc.  So typically you wind up with very low powered
>> 'RAID' and 'SATA/SAS' on the motherboard, connected by PCIe x2 or x4 at
>> most.  A number of motherboards have NICs that are served by a single PCIe
>> x1 link.
>>
>>> Thanks for your help that led me to this path. Another question I had
>>> was when creating mdadm RAID does it make sense to use multipathing?
>>
>> Well, for a shared backend over a fabric, I'd say possibly.  For an internal
>> connected set, I'd say no.  Given what you are doing with Gluster, I'd say
>> that the additional expense/pain of setting up a multipath scenario probably
>> isn't worth it.
>>
>> Gluster lets you get many of these benefits at a higher level in the stack.
>>   Which to a degree, and in some use cases, obviates the need for
>> multipathing at a lower level.  I'd still suggest real RAID at the lower
>> level (RAID6, and sometimes RAID10 make the most sense) for the backing
>> store.
>>
>>
>> --
>> Joseph Landman, Ph.D
>> Founder and CEO
>> Scalable Informatics, Inc.
>> email: landman at scalableinformatics.com
>> web  : http://scalableinformatics.com
>>        http://scalableinformatics.com/sicluster
>> phone: +1 734 786 8423 x121
>> fax  : +1 866 888 3112
>> cell : +1 734 612 4615
>>


-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615