[Gluster-users] Performance

Todd Daugherty todd at fotokem.hu
Fri Jun 11 05:03:57 UTC 2010


I agree it is a matter of tuning. So what do we tune? I am not stuck
on 2.0.9, there just was no performance benefit for my data set.
(Millions of 8-50 megabyte files) My test system has 12 Gigabytes of
RAM which is why I used a size of 16 Gigabytes so that cache is not a
factor in the test. But away what did you learn from these results?
Read performance is the most important to me. (That is because the
write performance is pretty good already)

Thanks again.

Todd


 iozone -a -i0 -i1 -s 8192m -r 64k iozone.$$.tmp
        Iozone: Performance Test of File I/O
                Version $Revision: 3.283 $
                Compiled for 64 bit mode.
                Build: linux

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
                     Erik Habbinga, Kris Strecker, Walter Wong.

        Run began: Fri Jun 11 06:48:48 2010

        Auto Mode
        File size set to 8388608 KB
        Record Size 64 KB
        Command line used: iozone -a -i0 -i1 -s 8192m -r 64k iozone.1059.tmp
        Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                            random
random    bkwd  record  stride
              KB  reclen   write rewrite    read    reread    read
write    read rewrite    read   fwrite frewrite   fread  freread
         8388608      64  394906  728411   266523   255494

iozone test complete.

iozone -a -i0 -i1 -s 8192m -r 128k iozone.$$.tmp
        Iozone: Performance Test of File I/O
                Version $Revision: 3.283 $
                Compiled for 64 bit mode.
                Build: linux

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
                     Erik Habbinga, Kris Strecker, Walter Wong.

        Run began: Fri Jun 11 06:53:03 2010

        Auto Mode
        File size set to 8388608 KB
        Record Size 128 KB
        Command line used: iozone -a -i0 -i1 -s 8192m -r 128k iozone.1059.tmp
        Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                            random
random    bkwd  record  stride
              KB  reclen   write rewrite    read    reread    read
write    read rewrite    read   fwrite frewrite   fread  freread
         8388608     128  406930  746811   233122   225809

iozone test complete.

iozone -a -i0 -i1 -s 8192m -r 1m iozone.$$.tmp
        Iozone: Performance Test of File I/O
                Version $Revision: 3.283 $
                Compiled for 64 bit mode.
                Build: linux

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
                     Erik Habbinga, Kris Strecker, Walter Wong.

        Run began: Fri Jun 11 06:55:56 2010

        Auto Mode
        File size set to 8388608 KB
        Record Size 1024 KB
        Command line used: iozone -a -i0 -i1 -s 8192m -r 1m iozone.1059.tmp
        Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                            random
random    bkwd  record  stride
              KB  reclen   write rewrite    read    reread    read
write    read rewrite    read   fwrite frewrite   fread  freread
         8388608    1024  459811  717837   231110   219715

iozone test complete.


glusterfs with /dev/ram0 (cluster)
 iozone -a -i0 -i1 -s 8192m -r 64k iozone.$$.tmp
        Iozone: Performance Test of File I/O
                Version $Revision: 3.283 $
                Compiled for 64 bit mode.
                Build: linux

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy,
                     Erik Habbinga, Kris Strecker, Walter Wong.

        Run began: Fri Jun 11 06:47:11 2010

        Auto Mode
        File size set to 8388608 KB
        Record Size 64 KB
        Command line used: iozone -a -i0 -i1 -s 8192m -r 64k iozone.1059.tmp
        Output is in Kbytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                            random
random    bkwd  record  stride
              KB  reclen   write rewrite    read    reread    read
write    read rewrite    read   fwrite frewrite   fread  freread
         8388608      64 1038493 1171185   511793   508550

iozone test complete.



On Fri, Jun 11, 2010 at 6:25 AM, Harshavardhana <harsha at gluster.com> wrote:
> On 06/10/2010 08:52 PM, Todd Daugherty wrote:
>>
>> I am not sure what you are saying? The default record size of
>> ib_write_bw is 64K. But here is 128K.
>>
>> Do you think that the QDR Infiniband is not doing 3.3 gigabytes per
>> seconds? And that is why I am only getting 500 megabytes per seconds
>> via GlusterFS? You think that it is something wrong with the QDR
>> setup? What are the actual commands that you want me run on the
>> machines? I will run them.
>>
>> Do you have a single stream GlusterFS that can go OVER 3.3 gigabytes
>> per second? What setup is it?
>>
>
> Excuse me for not being clear
>
> you need to run "iozone" with higher record sizes. Your earlier command
> suggested you used "16k" for creating a 8GB file.
>
> 16k chunk writes over 2.0.9 version has lot of performance penalty, i think
> that is the reason why you are seeing such a slowness.
>
> Please run the below commands
>
> "iozone -a -i0 -i1 -s 8192m -r 64k iozone.$$.tmp"
> "iozone -a -i0 -i1 -s 8192m -r 128k iozone.$$.tmp"
> "iozone -a -i0 -i1 -s 8192m -r 1m iozone.$$.tmp"
>
> Since your infiniband fabric looks quite alright and also your backend disk
> performing at 1GB/sec.  We should be near to 700-850MB/sec over glusterfs
> for writes as well as reads, if not then its a tuning or configuration
> issue.
>
> Thanks
>
> --
> Harshavardhana
> Gluster Inc - http://www.gluster.com
> +1(408)-770-1887, Ext-113
> +1(408)-480-1730
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>



More information about the Gluster-users mailing list