[Gluster-users] Gluster speed sooo slow
Ivan Dimitrov
dobber at amln.net
Mon Aug 13 07:32:55 UTC 2012
There is a big difference with working with small files (around 16kb)
and big files (2mb). Performance is much better with big files. Witch is
too bad for me ;(
On 8/11/12 2:15 AM, Gandalf Corvotempesta wrote:
> What do you mean with "small files"? 16k ? 160k? 16mb?
> Do you know any workaround or any other software for this?
>
> Mee too i'm trying to create a clustered storage for many
> small file
>
> 2012/8/10 Philip Poten <philip.poten at gmail.com
> <mailto:philip.poten at gmail.com>>
>
> Hi Ivan,
>
> that's because Gluster has really bad "many small files" performance
> due to it's architecture.
>
> On all stat() calls (which rsync is doing plenty of), all replicas are
> being checked for integrity.
>
> regards,
> Philip
>
> 2012/8/10 Ivan Dimitrov <dobber at amln.net <mailto:dobber at amln.net>>:
> > So I stopped a node to check the BIOS and after it went up, the
> rebalance
> > kicked in. I was looking for those kind of speeds on a normal
> write. The
> > rebalance is much faster than my rsync/cp.
> >
> >
> https://dl.dropbox.com/u/282332/Screen%20Shot%202012-08-10%20at%202.04.09%20PM.png
> >
> > Best Regards
> > Ivan Dimitrov
> >
> >
> > On 8/10/12 1:23 PM, Ivan Dimitrov wrote:
> >>
> >> Hello
> >> What am I doing wrong?!?
> >>
> >> I have a test setup with 4 identical servers with 2 disks each in
> >> distribute-replicate 2. All servers are connected to a GB switch.
> >>
> >> I am experiencing really slow speeds at anything I do. Slow
> write, slow
> >> read, not to mention random write/reads.
> >>
> >> Here is an example:
> >> random-files is a directory with 32768 files with average size
> 16kb.
> >> [root at gltclient]:~# rsync -a /root/speedtest/random-files/
> >> /home/gltvolume/
> >> ^^ This will take more than 3 hours.
> >>
> >> On any of the servers if I do "iostat" the disks are not loaded
> at all:
> >>
> >>
> https://dl.dropbox.com/u/282332/Screen%20Shot%202012-08-10%20at%201.08.54%20PM.png
> >>
> >> This is similar result for all servers.
> >>
> >> Here is an example of simple "ls" command on the content.
> >> [root at gltclient]:~# unalias ls
> >> [root at gltclient]:~# /usr/bin/time -f "%e seconds" ls
> /home/gltvolume/ | wc
> >> -l
> >> 2.81 seconds
> >> 5393
> >>
> >> almost 3 seconds to display 5000 files?!?! When they are
> 32,000, the ls
> >> will take around 35-45 seconds.
> >>
> >> This directory is on local disk:
> >> [root at gltclient]:~# /usr/bin/time -f "%e seconds" ls
> >> /root/speedtest/random-files/ | wc -l
> >> 1.45 seconds
> >> 32768
> >>
> >> [root at gltclient]:~# /usr/bin/time -f "%e seconds" cat
> /home/gltvolume/*
> >> >/dev/null
> >> 190.50 seconds
> >>
> >> [root at gltclient]:~# /usr/bin/time -f "%e seconds" du -sh
> /home/gltvolume/
> >> 126M /home/gltvolume/
> >> 75.23 seconds
> >>
> >>
> >> Here is the volume information.
> >>
> >> [root at glt1]:~# gluster volume info
> >>
> >> Volume Name: gltvolume
> >> Type: Distributed-Replicate
> >> Volume ID: 16edd852-8d23-41da-924d-710b753bb374
> >> Status: Started
> >> Number of Bricks: 4 x 2 = 8
> >> Transport-type: tcp
> >> Bricks:
> >> Brick1: 1.1.74.246:/home/sda3
> >> Brick2: glt2.network.net:/home/sda3
> >> Brick3: 1.1.74.246:/home/sdb1
> >> Brick4: glt2.network.net:/home/sdb1
> >> Brick5: glt3.network.net:/home/sda3
> >> Brick6: gltclient.network.net:/home/sda3
> >> Brick7: glt3.network.net:/home/sdb1
> >> Brick8: gltclient.network.net:/home/sdb1
> >> Options Reconfigured:
> >> performance.io-thread-count: 32
> >> performance.cache-size: 256MB
> >> cluster.self-heal-daemon: on
> >>
> >>
> >> [root at glt1]:~# gluster volume status all detail
> >> Status of volume: gltvolume
> >>
> >>
> ------------------------------------------------------------------------------
> >> Brick : Brick 1.1.74.246:/home/sda3
> >> Port : 24009
> >> Online : Y
> >> Pid : 1479
> >> File System : ext4
> >> Device : /dev/sda3
> >> Mount Options : rw,noatime
> >> Inode Size : 256
> >> Disk Space Free : 179.3GB
> >> Total Disk Space : 179.7GB
> >> Inode Count : 11968512
> >> Free Inodes : 11901550
> >>
> >>
> ------------------------------------------------------------------------------
> >> Brick : Brick glt2.network.net:/home/sda3
> >> Port : 24009
> >> Online : Y
> >> Pid : 1589
> >> File System : ext4
> >> Device : /dev/sda3
> >> Mount Options : rw,noatime
> >> Inode Size : 256
> >> Disk Space Free : 179.3GB
> >> Total Disk Space : 179.7GB
> >> Inode Count : 11968512
> >> Free Inodes : 11901550
> >>
> >>
> ------------------------------------------------------------------------------
> >> Brick : Brick 1.1.74.246:/home/sdb1
> >> Port : 24010
> >> Online : Y
> >> Pid : 1485
> >> File System : ext4
> >> Device : /dev/sdb1
> >> Mount Options : rw,noatime
> >> Inode Size : 256
> >> Disk Space Free : 228.8GB
> >> Total Disk Space : 229.2GB
> >> Inode Count : 15269888
> >> Free Inodes : 15202933
> >>
> >>
> ------------------------------------------------------------------------------
> >> Brick : Brick glt2.network.net:/home/sdb1
> >> Port : 24010
> >> Online : Y
> >> Pid : 1595
> >> File System : ext4
> >> Device : /dev/sdb1
> >> Mount Options : rw,noatime
> >> Inode Size : 256
> >> Disk Space Free : 228.8GB
> >> Total Disk Space : 229.2GB
> >> Inode Count : 15269888
> >> Free Inodes : 15202933
> >>
> >>
> ------------------------------------------------------------------------------
> >> Brick : Brick glt3.network.net:/home/sda3
> >> Port : 24009
> >> Online : Y
> >> Pid : 28963
> >> File System : ext4
> >> Device : /dev/sda3
> >> Mount Options : rw,noatime
> >> Inode Size : 256
> >> Disk Space Free : 179.3GB
> >> Total Disk Space : 179.7GB
> >> Inode Count : 11968512
> >> Free Inodes : 11906058
> >>
> >>
> ------------------------------------------------------------------------------
> >> Brick : Brick gltclient.network.net:/home/sda3
> >> Port : 24009
> >> Online : Y
> >> Pid : 3145
> >> File System : ext4
> >> Device : /dev/sda3
> >> Mount Options : rw,noatime
> >> Inode Size : 256
> >> Disk Space Free : 179.3GB
> >> Total Disk Space : 179.7GB
> >> Inode Count : 11968512
> >> Free Inodes : 11906058
> >>
> >>
> ------------------------------------------------------------------------------
> >> Brick : Brick glt3.network.net:/home/sdb1
> >> Port : 24010
> >> Online : Y
> >> Pid : 28969
> >> File System : ext4
> >> Device : /dev/sdb1
> >> Mount Options : rw,noatime
> >> Inode Size : 256
> >> Disk Space Free : 228.8GB
> >> Total Disk Space : 229.2GB
> >> Inode Count : 15269888
> >> Free Inodes : 15207375
> >>
> >>
> ------------------------------------------------------------------------------
> >> Brick : Brick gltclient.network.net:/home/sdb1
> >> Port : 24010
> >> Online : Y
> >> Pid : 3151
> >> File System : ext4
> >> Device : /dev/sdb1
> >> Mount Options : rw,noatime
> >> Inode Size : 256
> >> Disk Space Free : 228.8GB
> >> Total Disk Space : 229.2GB
> >> Inode Count : 15269888
> >> Free Inodes : 15207375
> >> _______________________________________________
> >> Gluster-users mailing list
> >> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
> >> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
> >>
> >
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
> > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120813/8f3303d5/attachment.html>
More information about the Gluster-users
mailing list