[Gluster-users] Gluster speed sooo slow
Philip Poten
philip.poten at gmail.com
Fri Aug 10 14:27:51 UTC 2012
Hi Ivan,
that's because Gluster has really bad "many small files" performance
due to it's architecture.
On all stat() calls (which rsync is doing plenty of), all replicas are
being checked for integrity.
regards,
Philip
2012/8/10 Ivan Dimitrov <dobber at amln.net>:
> So I stopped a node to check the BIOS and after it went up, the rebalance
> kicked in. I was looking for those kind of speeds on a normal write. The
> rebalance is much faster than my rsync/cp.
>
> https://dl.dropbox.com/u/282332/Screen%20Shot%202012-08-10%20at%202.04.09%20PM.png
>
> Best Regards
> Ivan Dimitrov
>
>
> On 8/10/12 1:23 PM, Ivan Dimitrov wrote:
>>
>> Hello
>> What am I doing wrong?!?
>>
>> I have a test setup with 4 identical servers with 2 disks each in
>> distribute-replicate 2. All servers are connected to a GB switch.
>>
>> I am experiencing really slow speeds at anything I do. Slow write, slow
>> read, not to mention random write/reads.
>>
>> Here is an example:
>> random-files is a directory with 32768 files with average size 16kb.
>> [root at gltclient]:~# rsync -a /root/speedtest/random-files/
>> /home/gltvolume/
>> ^^ This will take more than 3 hours.
>>
>> On any of the servers if I do "iostat" the disks are not loaded at all:
>>
>> https://dl.dropbox.com/u/282332/Screen%20Shot%202012-08-10%20at%201.08.54%20PM.png
>>
>> This is similar result for all servers.
>>
>> Here is an example of simple "ls" command on the content.
>> [root at gltclient]:~# unalias ls
>> [root at gltclient]:~# /usr/bin/time -f "%e seconds" ls /home/gltvolume/ | wc
>> -l
>> 2.81 seconds
>> 5393
>>
>> almost 3 seconds to display 5000 files?!?! When they are 32,000, the ls
>> will take around 35-45 seconds.
>>
>> This directory is on local disk:
>> [root at gltclient]:~# /usr/bin/time -f "%e seconds" ls
>> /root/speedtest/random-files/ | wc -l
>> 1.45 seconds
>> 32768
>>
>> [root at gltclient]:~# /usr/bin/time -f "%e seconds" cat /home/gltvolume/*
>> >/dev/null
>> 190.50 seconds
>>
>> [root at gltclient]:~# /usr/bin/time -f "%e seconds" du -sh /home/gltvolume/
>> 126M /home/gltvolume/
>> 75.23 seconds
>>
>>
>> Here is the volume information.
>>
>> [root at glt1]:~# gluster volume info
>>
>> Volume Name: gltvolume
>> Type: Distributed-Replicate
>> Volume ID: 16edd852-8d23-41da-924d-710b753bb374
>> Status: Started
>> Number of Bricks: 4 x 2 = 8
>> Transport-type: tcp
>> Bricks:
>> Brick1: 1.1.74.246:/home/sda3
>> Brick2: glt2.network.net:/home/sda3
>> Brick3: 1.1.74.246:/home/sdb1
>> Brick4: glt2.network.net:/home/sdb1
>> Brick5: glt3.network.net:/home/sda3
>> Brick6: gltclient.network.net:/home/sda3
>> Brick7: glt3.network.net:/home/sdb1
>> Brick8: gltclient.network.net:/home/sdb1
>> Options Reconfigured:
>> performance.io-thread-count: 32
>> performance.cache-size: 256MB
>> cluster.self-heal-daemon: on
>>
>>
>> [root at glt1]:~# gluster volume status all detail
>> Status of volume: gltvolume
>>
>> ------------------------------------------------------------------------------
>> Brick : Brick 1.1.74.246:/home/sda3
>> Port : 24009
>> Online : Y
>> Pid : 1479
>> File System : ext4
>> Device : /dev/sda3
>> Mount Options : rw,noatime
>> Inode Size : 256
>> Disk Space Free : 179.3GB
>> Total Disk Space : 179.7GB
>> Inode Count : 11968512
>> Free Inodes : 11901550
>>
>> ------------------------------------------------------------------------------
>> Brick : Brick glt2.network.net:/home/sda3
>> Port : 24009
>> Online : Y
>> Pid : 1589
>> File System : ext4
>> Device : /dev/sda3
>> Mount Options : rw,noatime
>> Inode Size : 256
>> Disk Space Free : 179.3GB
>> Total Disk Space : 179.7GB
>> Inode Count : 11968512
>> Free Inodes : 11901550
>>
>> ------------------------------------------------------------------------------
>> Brick : Brick 1.1.74.246:/home/sdb1
>> Port : 24010
>> Online : Y
>> Pid : 1485
>> File System : ext4
>> Device : /dev/sdb1
>> Mount Options : rw,noatime
>> Inode Size : 256
>> Disk Space Free : 228.8GB
>> Total Disk Space : 229.2GB
>> Inode Count : 15269888
>> Free Inodes : 15202933
>>
>> ------------------------------------------------------------------------------
>> Brick : Brick glt2.network.net:/home/sdb1
>> Port : 24010
>> Online : Y
>> Pid : 1595
>> File System : ext4
>> Device : /dev/sdb1
>> Mount Options : rw,noatime
>> Inode Size : 256
>> Disk Space Free : 228.8GB
>> Total Disk Space : 229.2GB
>> Inode Count : 15269888
>> Free Inodes : 15202933
>>
>> ------------------------------------------------------------------------------
>> Brick : Brick glt3.network.net:/home/sda3
>> Port : 24009
>> Online : Y
>> Pid : 28963
>> File System : ext4
>> Device : /dev/sda3
>> Mount Options : rw,noatime
>> Inode Size : 256
>> Disk Space Free : 179.3GB
>> Total Disk Space : 179.7GB
>> Inode Count : 11968512
>> Free Inodes : 11906058
>>
>> ------------------------------------------------------------------------------
>> Brick : Brick gltclient.network.net:/home/sda3
>> Port : 24009
>> Online : Y
>> Pid : 3145
>> File System : ext4
>> Device : /dev/sda3
>> Mount Options : rw,noatime
>> Inode Size : 256
>> Disk Space Free : 179.3GB
>> Total Disk Space : 179.7GB
>> Inode Count : 11968512
>> Free Inodes : 11906058
>>
>> ------------------------------------------------------------------------------
>> Brick : Brick glt3.network.net:/home/sdb1
>> Port : 24010
>> Online : Y
>> Pid : 28969
>> File System : ext4
>> Device : /dev/sdb1
>> Mount Options : rw,noatime
>> Inode Size : 256
>> Disk Space Free : 228.8GB
>> Total Disk Space : 229.2GB
>> Inode Count : 15269888
>> Free Inodes : 15207375
>>
>> ------------------------------------------------------------------------------
>> Brick : Brick gltclient.network.net:/home/sdb1
>> Port : 24010
>> Online : Y
>> Pid : 3151
>> File System : ext4
>> Device : /dev/sdb1
>> Mount Options : rw,noatime
>> Inode Size : 256
>> Disk Space Free : 228.8GB
>> Total Disk Space : 229.2GB
>> Inode Count : 15269888
>> Free Inodes : 15207375
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
More information about the Gluster-users
mailing list