[Gluster-users] GlusterFS Hardware Recommendations

[Andy Pace]

Hello all, new to the list here. Figured I'd hop in to get some insight/opinion on the hardware requirements of Gluster in our environment.

We've been testing Gluster as our shared storage technology for our new Cloud product we're going to be launching. The primary role of the Gluster infrastructure is to house several hundred (potentially thousands) of Xen sparse image files.

In our lab, we have the following servers/specs setup:

Bricks1-4:
Dual Xeon 5410 (8 cores)
16G RAM
4X1TB 7200RPM HDD @ 3ware RAID5
Bond0 - 2Gbps

Cloud1 (client)
Q8200 2.33GHz
8G RAM
1X1TB 7200RPM HDD
Eth0 - 1Gbps

It's all primarily SuperMicro chassis and motherboards, Seagate drives and standard RAM.

The Gluster client is configured to distribute and mirror across both pairs of bricks -a typical setup for users like us that need the scalability and redundancy. Until yesterday, things were working great. We were getting line speed writes (55MB/s, because gluster has to write to 2 bricks), and amazing reads. However, after running 5 concurrent iozone benchmarks, we noticed the bricks becoming loaded, primarily in CPU, and some IO:

Cloud1  saw peak of 92% cpu utilization with pdflush (buffer cache manager) reaching
up to 12% of cpu

top - 13:53:12 up 21:44,  2 users,  load average: 1.72, 1.28, 0.72
Tasks: 102 total,   2 running, 100 sleeping,   0 stopped,   0 zombie
Cpu(s):  7.7%us, 10.1%sy,  0.0%ni, 80.6%id,  0.0%wa,  0.0%hi,  1.2%si,  0.5%st
Mem:   8120388k total,  8078832k used,    41556k free,    30760k buffers
Swap:  4200988k total,      752k used,  4200236k free,  7606560k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 2304 root      20   0  245m 7032 1388 R   91  0.1   8:27.03 glusterfs
 4409 root      20   0 46812  16m  172 S    1  0.2   0:01.86 iozone
 1159 root      15  -5     0    0    0 S    1  0.0   0:00.12 kjournald
 4410 root      20   0 46812  16m  172 S    1  0.2   0:02.22 iozone
 4411 root      20   0 46812  16m  172 S    1  0.2   0:01.32 iozone
 4412 root      20   0 46812  16m  172 S    1  0.2   0:01.96 iozone
 4413 root      20   0 46812  16m  172 S    1  0.2   0:02.00 iozone
    1 root      20   0 10312  752  620 S    0  0.0   0:00.16 init
    2 root      15  -5     0    0    0 S    0  0.0   0:00.00 kthreadd
    3 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/0

Brick1 saw peak of 14% cpu utilization

top - 13:52:39 up 8 days,  5:10,  1 user,  load average: 0.07, 0.05, 0.01
Tasks: 151 total,   2 running, 149 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.5%us,  0.7%sy,  0.0%ni, 98.1%id,  0.0%wa,  0.1%hi,  0.6%si,  0.0%st
Mem:  16439672k total,  2705356k used, 13734316k free,   228624k buffers
Swap:  8388600k total,      156k used,  8388444k free,  2255200k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 3413 root      15   0  187m 9332 1172 R 10.0  0.1  25:28.51 glusterfsd
    1 root      15   0 10348  692  580 S  0.0  0.0   0:02.15 init
    2 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 migration/0

brick2 saw peak of 37 % cpu utilization

top - 13:55:29 up 8 days,  5:54,  2 users,  load average: 0.21, 0.14, 0.05
Tasks: 152 total,   1 running, 151 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.0%us,  1.1%sy,  0.0%ni, 97.2%id,  0.0%wa,  0.1%hi,  0.6%si,  0.0%st
Mem:  16439672k total,  2698776k used, 13740896k free,   257288k buffers
Swap:  8388600k total,      152k used,  8388448k free,  2251084k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 3404 root      15   0  187m 4996 1156 S 16.6  0.0  36:50.78 glusterfsd
10547 root      15   0     0    0    0 S  1.0  0.0   0:11.93 pdflush
    1 root      15   0 10348  692  580 S  0.0  0.0   0:02.18 init

brick3

top - 21:44:22 up 7 days, 23:36,  3 users,  load average: 0.74, 0.39, 0.19
Tasks: 110 total,   2 running, 108 sleeping,   0 stopped,   0 zombie
Cpu(s):  3.7%us,  4.1%sy,  0.0%ni, 89.0%id,  0.0%wa,  0.4%hi,  2.8%si,  0.0%st
Mem:  16439808k total, 14291352k used,  2148456k free,   191444k buffers
Swap:  8388600k total,      120k used,  8388480k free, 13721644k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 2953 root      15   0  187m 8380 1176 R 34.3  0.1  12:46.74 glusterfsd
    1 root      15   0 10348  692  580 S  0.0  0.0   0:01.75 init

brick4

top - 13:56:29 up 8 days,  3:57,  2 users,  load average: 0.89, 0.62, 0.35
Tasks: 153 total,   1 running, 152 sleeping,   0 stopped,   0 zombie
Cpu(s):  2.3%us,  2.9%sy,  0.0%ni, 92.8%id,  0.0%wa,  0.3%hi,  1.7%si,  0.0%st
Mem:  16439672k total, 14405528k used,  2034144k free,   193340k buffers
Swap:  8388600k total,      144k used,  8388456k free, 13782972k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 3318 root      15   0  253m 7812 1168 S 44.6  0.0  16:36.88 glusterfsd
30901 root      15   0 12740 1108  800 R  0.3  0.0   0:00.01 top
    1 root      15   0 10348  692  580 S  0.0  0.0   0:02.32 init


Command line used: /usr/sbin/iozone -R -l 5 -u 5 -r 4k -s 2048m -F /distributed/f1 /distributed/f2 /distributed/f3 /distributed/f4 /distributed/f5 -G

This obviously is very concerning, that 1 client can induce this sort of load across our gluster infrastructure. So, I'm thinking we get 4 of the following:

Dual Xeon 5520
32G RAM (DDR3)
14X1TB 7200RPM HDD, 12 @ RAID10, 2 @ RAID1 (for the operating system)
MaxIQ or CacheCade SSD addon depending on whether we go with Adaptec or LSI
6X1Gige bonded to 6Gbps

Is this overkill? Here are some things to keep in mind:


*         The majority of our customers will be hosting web applications, blogs, forums, primarily database driven - lots of reads, some writes

*         Each client server only has a 1Gbps connection to the brick(s)

*         We obviously are trying to get the most bang for our buck, but not trying to spend $40k on something that's overpowered

Is that iozone benchmark too intense for what we're ACTUALLY going to be seeing from our Xen instances?
Is there an advantage to using more bricks at half the horsepower?
At 6Gbps to the bricks, does RAID10 make more sense, because of its exceptional performance?
Has anyone on the list setup something like this before? If so, mind sharing your wisdom?

Thanks in advance everyone :)

Just for reference, here are the iozone benchmarking results:

Run began: Wed Jun 16 13:48:41 2010

    Excel chart generation enabled
    Record Size 4 KB
    File size set to 2097152 KB
    Using msync(MS_SYNC) on mmap files
    Command line used: /usr/sbin/iozone -R -l 5 -u 5 -r 4k -s 2048m -F /distributed/f1 /distributed/f2 /distributed/f3 /distributed/f4 /distributed/f5 -G
    Output is in Kbytes/sec
    Time Resolution = 0.000001 seconds.
    Processor cache size set to 1024 Kbytes.
    Processor cache line size set to 32 bytes.
    File stride size set to 17 * record size.
    Min process = 5
    Max process = 5
    Throughput test with 5 processes
    Each process writes a 2097152 Kbyte file in 4 Kbyte records

    Children see throughput for  5 initial writers     =   34936.65 KB/sec
    Parent sees throughput for  5 initial writers     =   31595.96 KB/sec
    Min throughput per process             =    6713.67 KB/sec
    Max throughput per process             =    7407.69 KB/sec
    Avg throughput per process             =    6987.33 KB/sec
    Min xfer                     = 1900672.00 KB

    Children see throughput for  5 rewriters     =   34336.49 KB/sec
    Parent sees throughput for  5 rewriters     =   33737.35 KB/sec
    Min throughput per process             =    6506.53 KB/sec
    Max throughput per process             =    7430.44 KB/sec
    Avg throughput per process             =    6867.30 KB/sec
    Min xfer                     = 1836416.00 KB

    Children see throughput for  5 readers         =  111977.31 KB/sec
    Parent sees throughput for  5 readers         =  111846.84 KB/sec
    Min throughput per process             =   20259.48 KB/sec
    Max throughput per process             =   25610.23 KB/sec
    Avg throughput per process             =   22395.46 KB/sec
    Min xfer                     = 1658992.00 KB

    Children see throughput for 5 re-readers     =  111582.38 KB/sec
    Parent sees throughput for 5 re-readers     =  111420.54 KB/sec
    Min throughput per process             =   20841.22 KB/sec
    Max throughput per process             =   25012.94 KB/sec
    Avg throughput per process             =   22316.48 KB/sec
    Min xfer                     = 1747440.00 KB

    Children see throughput for 5 reverse readers     =  110576.31 KB/sec
    Parent sees throughput for 5 reverse readers     =  110356.95 KB/sec
    Min throughput per process             =   18543.04 KB/sec
    Max throughput per process             =   23964.38 KB/sec
    Avg throughput per process             =   22115.26 KB/sec
    Min xfer                     = 1622784.00 KB

    Children see throughput for 5 stride readers     =    6513.98 KB/sec
    Parent sees throughput for 5 stride readers     =    6513.14 KB/sec
    Min throughput per process             =    1189.43 KB/sec
    Max throughput per process             =    1497.43 KB/sec
    Avg throughput per process             =    1302.80 KB/sec
    Min xfer                     = 1665796.00 KB

    Children see throughput for 5 random readers     =    3460.22 KB/sec
    Parent sees throughput for 5 random readers     =    3460.08 KB/sec
    Min throughput per process             =     622.45 KB/sec
    Max throughput per process             =     799.28 KB/sec
    Avg throughput per process             =     692.04 KB/sec
    Min xfer                     = 1633196.00 KB