[Gluster-devel] mainline-2.5 performance a known issue?

[Dale Dude]

Dell 2950, sas, 4gig ram.

/volume1 = local JFS mount
/volumes = glusterfs mount over /volume1
/mnt/test = local nfs mount over /volume1

*/volumes# tiotest -t 10*
Tiotest results for 10 concurrent io threads:
,----------------------------------------------------------------------.
| Item                  | Time     | Rate         | Usr CPU  | Sys CPU |
+-----------------------+----------+--------------+----------+---------+
| Write         100 MBs |    3.7 s |  27.301 MB/s |  13.7 %  | 120.1 % |
| Random Write   39 MBs |    4.9 s |   7.938 MB/s |   0.0 %  |  48.4 % |
| Read          100 MBs |    0.8 s | 124.976 MB/s |   0.0 %  | 356.2 % |
| Random Read    39 MBs |    3.0 s |  13.191 MB/s |  10.1 %  |  72.9 % |
`----------------------------------------------------------------------'
Tiotest latency results:
,-------------------------------------------------------------------------.
| Item         | Average latency | Maximum latency | % >2 sec | % >10 sec |
+--------------+-----------------+-----------------+----------+-----------+
| Write        |        0.660 ms |       35.745 ms |  0.00000 |   0.00000 |
| Random Write |        1.835 ms |       28.604 ms |  0.00000 |   0.00000 |
| Read         |        0.276 ms |       73.803 ms |  0.00000 |   0.00000 |
| Random Read  |        2.650 ms |       43.676 ms |  0.00000 |   0.00000 |
|--------------+-----------------+-----------------+----------+-----------|
| Total        |        0.966 ms |       73.803 ms |  0.00000 |   0.00000 |
`--------------+-----------------+-----------------+----------+-----------'

=======================

*/volume1# tiotest -t 10 *
Tiotest results for 10 concurrent io threads:
,----------------------------------------------------------------------.
| Item                  | Time     | Rate         | Usr CPU  | Sys CPU |
+-----------------------+----------+--------------+----------+---------+
| Write         100 MBs |    3.0 s |  33.092 MB/s |   3.3 %  | 287.9 % |
| Random Write   39 MBs |    2.5 s |  15.801 MB/s |   4.0 %  |  67.6 % |
| Read          100 MBs |    0.1 s | 803.884 MB/s |  32.2 %  | 353.7 % |
| Random Read    39 MBs |    0.0 s | 795.781 MB/s |   0.0 %  | 326.0 % |
`----------------------------------------------------------------------'
Tiotest latency results:
,-------------------------------------------------------------------------.
| Item         | Average latency | Maximum latency | % >2 sec | % >10 sec |
+--------------+-----------------+-----------------+----------+-----------+
| Write        |        0.064 ms |      151.144 ms |  0.00000 |   0.00000 |
| Random Write |        0.008 ms |        0.311 ms |  0.00000 |   0.00000 |
| Read         |        0.006 ms |        0.032 ms |  0.00000 |   0.00000 |
| Random Read  |        0.006 ms |        0.053 ms |  0.00000 |   0.00000 |
|--------------+-----------------+-----------------+----------+-----------|
| Total        |        0.027 ms |      151.144 ms |  0.00000 |   0.00000 |
`--------------+-----------------+-----------------+----------+-----------'

======================

*/mnt/test# tiotest -t 10*
Tiotest results for 10 concurrent io threads:
,----------------------------------------------------------------------.
| Item                  | Time     | Rate         | Usr CPU  | Sys CPU |
+-----------------------+----------+--------------+----------+---------+
| Write         100 MBs |    0.8 s | 127.386 MB/s |   0.0 %  | 382.2 % |
| Random Write   39 MBs |    0.8 s |  46.980 MB/s |   0.0 %  |  75.8 % |
| Read          100 MBs |    0.3 s | 326.097 MB/s |  35.9 %  | 776.1 % |
| Random Read    39 MBs |    0.1 s | 761.467 MB/s |   0.0 %  | 389.9 % |
`----------------------------------------------------------------------'
Tiotest latency results:
,-------------------------------------------------------------------------.
| Item         | Average latency | Maximum latency | % >2 sec | % >10 sec |
+--------------+-----------------+-----------------+----------+-----------+
| Write        |        0.134 ms |      123.788 ms |  0.00000 |   0.00000 |
| Random Write |        0.342 ms |       89.255 ms |  0.00000 |   0.00000 |
| Read         |        0.092 ms |      155.992 ms |  0.00000 |   0.00000 |
| Random Read  |        0.006 ms |        0.060 ms |  0.00000 |   0.00000 |
|--------------+-----------------+-----------------+----------+-----------|
| Total        |        0.130 ms |      155.992 ms |  0.00000 |   0.00000 |
`--------------+-----------------+-----------------+----------+-----------'

==========================

client.vol:
volume server1
         type protocol/client
         option transport-type tcp/client     # for TCP/IP transport
         option remote-host 127.0.0.1     # IP address of the remote brick
         option remote-subvolume volumenamespace
end-volume

volume server1vol1
         type protocol/client
         option transport-type tcp/client     # for TCP/IP transport
         option remote-host 127.0.0.1     # IP address of the remote brick
         option remote-subvolume clusterfs1
end-volume

###################

volume bricks
  type cluster/unify
  option namespace server1
  option readdir-force-success on  # ignore failed mounts
  subvolumes server1vol1

  option scheduler rr
  option rr.limits.min-free-disk 5 #%
end-volume

volume writebehind   #writebehind improves write performance a lot
  type performance/write-behind
  option aggregate-size 131072 # in bytes
  subvolumes bricks
end-volume


volume readahead
  type performance/read-ahead
  option page-size 65536     # unit in bytes
  option page-count 16       # cache per file  = (page-count x page-size)
  subvolumes writebehind
end-volume

volume iocache
  type performance/io-cache
  option page-size 128KB
  option page-count 128
  subvolumes readahead
end-volume

============================
server.vol:
volume volume1
  type storage/posix
  option directory /volume1
end-volume

volume clusterfs1
   type performance/io-threads
   option thread-count 10
   subvolumes volume1
end-volume

#######

volume volumenamespace
  type storage/posix
  option directory /volume.namespace
end-volume

###

volume clusterfs
  type protocol/server
  option transport-type tcp/server
  subvolumes clusterfs1 volumenamespace
  option auth.ip.clusterfs1.allow *
  option auth.ip.volumenamespace.allow *
end-volume