[Gluster-users] Gluster speed sooo slow

[Ivan Dimitrov]

Hello
What am I doing wrong?!?

I have a test setup with 4 identical servers with 2 disks each in 
distribute-replicate 2. All servers are connected to a GB switch.

I am experiencing really slow speeds at anything I do. Slow write, slow 
read, not to mention random write/reads.

Here is an example:
random-files is a directory with 32768 files with average size 16kb.
[root at gltclient]:~# rsync -a /root/speedtest/random-files/ /home/gltvolume/
^^ This will take more than 3 hours.

On any of the servers if I do "iostat" the disks are not loaded at all:
https://dl.dropbox.com/u/282332/Screen%20Shot%202012-08-10%20at%201.08.54%20PM.png

This is similar result for all servers.

Here is an example of simple "ls" command on the content.
[root at gltclient]:~# unalias ls
[root at gltclient]:~# /usr/bin/time -f "%e seconds" ls /home/gltvolume/ | 
wc -l
2.81 seconds
5393

almost 3 seconds to display 5000 files?!?! When they are 32,000, the ls 
will take around 35-45 seconds.

This directory is on local disk:
[root at gltclient]:~# /usr/bin/time -f "%e seconds" ls 
/root/speedtest/random-files/ | wc -l
1.45 seconds
32768

[root at gltclient]:~# /usr/bin/time -f "%e seconds" cat /home/gltvolume/* 
 >/dev/null
190.50 seconds

[root at gltclient]:~# /usr/bin/time -f "%e seconds" du -sh /home/gltvolume/
126M    /home/gltvolume/
75.23 seconds


Here is the volume information.

[root at glt1]:~# gluster volume info

Volume Name: gltvolume
Type: Distributed-Replicate
Volume ID: 16edd852-8d23-41da-924d-710b753bb374
Status: Started
Number of Bricks: 4 x 2 = 8
Transport-type: tcp
Bricks:
Brick1: 1.1.74.246:/home/sda3
Brick2: glt2.network.net:/home/sda3
Brick3: 1.1.74.246:/home/sdb1
Brick4: glt2.network.net:/home/sdb1
Brick5: glt3.network.net:/home/sda3
Brick6: gltclient.network.net:/home/sda3
Brick7: glt3.network.net:/home/sdb1
Brick8: gltclient.network.net:/home/sdb1
Options Reconfigured:
performance.io-thread-count: 32
performance.cache-size: 256MB
cluster.self-heal-daemon: on


[root at glt1]:~# gluster volume status all detail
Status of volume: gltvolume
------------------------------------------------------------------------------
Brick                : Brick 1.1.74.246:/home/sda3
Port                 : 24009
Online               : Y
Pid                  : 1479
File System          : ext4
Device               : /dev/sda3
Mount Options        : rw,noatime
Inode Size           : 256
Disk Space Free      : 179.3GB
Total Disk Space     : 179.7GB
Inode Count          : 11968512
Free Inodes          : 11901550
------------------------------------------------------------------------------
Brick                : Brick glt2.network.net:/home/sda3
Port                 : 24009
Online               : Y
Pid                  : 1589
File System          : ext4
Device               : /dev/sda3
Mount Options        : rw,noatime
Inode Size           : 256
Disk Space Free      : 179.3GB
Total Disk Space     : 179.7GB
Inode Count          : 11968512
Free Inodes          : 11901550
------------------------------------------------------------------------------
Brick                : Brick 1.1.74.246:/home/sdb1
Port                 : 24010
Online               : Y
Pid                  : 1485
File System          : ext4
Device               : /dev/sdb1
Mount Options        : rw,noatime
Inode Size           : 256
Disk Space Free      : 228.8GB
Total Disk Space     : 229.2GB
Inode Count          : 15269888
Free Inodes          : 15202933
------------------------------------------------------------------------------
Brick                : Brick glt2.network.net:/home/sdb1
Port                 : 24010
Online               : Y
Pid                  : 1595
File System          : ext4
Device               : /dev/sdb1
Mount Options        : rw,noatime
Inode Size           : 256
Disk Space Free      : 228.8GB
Total Disk Space     : 229.2GB
Inode Count          : 15269888
Free Inodes          : 15202933
------------------------------------------------------------------------------
Brick                : Brick glt3.network.net:/home/sda3
Port                 : 24009
Online               : Y
Pid                  : 28963
File System          : ext4
Device               : /dev/sda3
Mount Options        : rw,noatime
Inode Size           : 256
Disk Space Free      : 179.3GB
Total Disk Space     : 179.7GB
Inode Count          : 11968512
Free Inodes          : 11906058
------------------------------------------------------------------------------
Brick                : Brick gltclient.network.net:/home/sda3
Port                 : 24009
Online               : Y
Pid                  : 3145
File System          : ext4
Device               : /dev/sda3
Mount Options        : rw,noatime
Inode Size           : 256
Disk Space Free      : 179.3GB
Total Disk Space     : 179.7GB
Inode Count          : 11968512
Free Inodes          : 11906058
------------------------------------------------------------------------------
Brick                : Brick glt3.network.net:/home/sdb1
Port                 : 24010
Online               : Y
Pid                  : 28969
File System          : ext4
Device               : /dev/sdb1
Mount Options        : rw,noatime
Inode Size           : 256
Disk Space Free      : 228.8GB
Total Disk Space     : 229.2GB
Inode Count          : 15269888
Free Inodes          : 15207375
------------------------------------------------------------------------------
Brick                : Brick gltclient.network.net:/home/sdb1
Port                 : 24010
Online               : Y
Pid                  : 3151
File System          : ext4
Device               : /dev/sdb1
Mount Options        : rw,noatime
Inode Size           : 256
Disk Space Free      : 228.8GB
Total Disk Space     : 229.2GB
Inode Count          : 15269888
Free Inodes          : 15207375