[Gluster-users] 3.2.2 Performance Issue

[Joey McDonald]

Hello again,

For testing purposes, I have deleted my original distributed/replicated
volume that was seeing write's at around 10 MB/s and created one on the same
cluster that is only distributed.

Now, I'm seeing the kind of performance I was hoping to see from
distributed/replicated:

[root at vm-container-0-0 ~]# gluster volume info pifs

Volume Name: pifs
Type: Distribute
Status: Started
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: vm-container-0-0:/gluster
Brick2: vm-container-0-1:/gluster
Brick3: vm-container-0-2:/gluster
Brick4: vm-container-0-3:/gluster

[root at vm-container-0-0 ~]# mount | grep pifs
glusterfs#127.0.0.1:pifs on /pifs type fuse
(rw,allow_other,default_permissions,max_read=131072)

[root at vm-container-0-0 ~]# dd if=/dev/zero of=/pifs/dd_test.img bs=1M
count=2000
2000+0 records in
2000+0 records out
2097152000 bytes (2.1 GB) copied, 22.2793 seconds, 94.1 MB/s


An order of magnitude slower with replication. What's going on I wonder?
Thanks for any suggestions.

  --joey



On Wed, Aug 10, 2011 at 10:05 AM, Joey McDonald <joey at scare.org> wrote:

>
>
> On Tue, Aug 9, 2011 at 11:19 PM, Mohit Anchlia <mohitanchlia at gmail.com>wrote:
>
>> And can you also give the mount options of gluster fs?
>>
>>
> Sure:
>
> [root at vm-container-0-0 ~]# tail -2 /etc/fstab
> /dev/sdb1               /gluster                ext3    defaults        0 1
> 127.0.0.1:pifs          /pifs                   glusterfs defaults,_netdev
> 0 0
>
> I'm not running VM's off the gluster share but that's the general idea of
> where I'd like to get with this platform. That's obviously going to require
> some different mount options but I'd like to get this performance issue
> resolved before I start in on that.
>
>
>    --joey
>
>
>
>
>> On Tue, Aug 9, 2011 at 4:41 PM, Joey McDonald <joey at scare.org> wrote:
>> >
>> >
>> > On Tue, Aug 9, 2011 at 5:40 PM, Joey McDonald <joey at scare.org> wrote:
>> >>
>> >> Hi Pavan,
>> >> Thanks for your quick reply, comments inline:
>> >>
>> >>>
>> >>> 1. Are these baremetal systems or are they Virtual machines ?
>> >>
>> >> Bare metal systems.
>> >>
>> >>>
>> >>> 2. What is the amount of RAM of each of these systems ?
>> >>
>> >> They all have 4194304 kB of memory.
>> >>
>> >>>
>> >>> 3. How many CPUs do they have ?
>> >>
>> >> They each have 8 procs.
>> >>
>> >>>
>> >>> 4. Can you also perform the dd on /gluster as opposed to /root to
>> check
>> >>> the backend performance ?
>> >>
>> >> Sure, here is that output:
>> >>
>> >> [root at vm-container-0-0 ~]# dd if=/dev/zero of=/gluster/dd_test.img
>> bs=1M
>> >> count=2000
>> >> 2000+0 records in
>> >> 2000+0 records out
>> >> 2097152000 bytes (2.1 GB) copied, 6.65193 seconds, 315 MB/s
>> >>
>> >>>
>> >>> 5. What is your disk backend ? Is it direct attached or is it an array
>> ?
>> >>
>> >> Direct attached, /gluster is /dev/sdb1, 1TB SATA drive (as is
>> /dev/sda):
>> >> [root at vm-container-0-0 ~]# hdparm -i /dev/sdb
>> >> /dev/sdb:
>> >>  Model=WDC WD1002FBYS-02A6B0                   , FwRev=03.00C06,
>> SerialNo=
>> >>     WD-WMATV5311442
>> >>  Config={ HardSect NotMFM HdSw>15uSec SpinMotCtl Fixed DTR>5Mbs
>> FmtGapReq
>> >> }
>> >>  RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=50
>> >>  BuffType=unknown, BuffSize=32767kB, MaxMultSect=16, MultSect=?0?
>> >>  CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=268435455
>> >>  IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
>> >>  PIO modes:  pio0 pio3 pio4
>> >>  DMA modes:  mdma0 mdma1 mdma2
>> >>  UDMA modes: udma0 udma1 udma2
>> >>  AdvancedPM=no WriteCache=enabled
>> >>  Drive conforms to: Unspecified:  ATA/ATAPI-1 ATA/ATAPI-2 ATA/ATAPI-3
>> >> ATA/ATAPI-4 ATA/ATAPI-5 ATA/ATAPI-6 ATA/ATAPI-7
>> >>  * signifies the current active mode
>> >>
>> >>>
>> >>> 6. What is the backend filesystem ?
>> >>
>> >> ext3
>> >>
>> >>>
>> >>> 7. Can you run a simple scp of about 10M between any two of these
>> systems
>> >>> and report the speed ?
>> >>
>> >>  Sure, output:
>> >> [root at vm-container-0-1 ~]# scp vm-container-0-0:/gluster/dd_test.img .
>> >> Warning: Permanently added 'vm-container-0-0' (RSA) to the list of
>> known
>> >> hosts.
>> >> root at vm-container-0-0's password:
>> >> dd_test.img
>> >>                                                               100%
>> 2000MB
>> >>  39.2MB/s   00:51
>> >>
>> >>    --joey
>> >>
>> >>
>> >
>> >
>> > _______________________________________________
>> > Gluster-users mailing list
>> > Gluster-users at gluster.org
>> > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>> >
>> >
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20110810/6d9162c8/attachment.html>