[Gluster-users] help me saturate my reads with a stripe of 4 machines

[Erick Tryzelaar]

I'm having trouble getting stripe to perform as well as I'd expect.
I've managed to get it to work pretty well with 2 machines in a stripe
set with these files:

server.vol:
volume posix
  type storage/posix
  option directory /tmp/gluster
end-volume

volume locks
  type features/locks
  subvolumes posix
end-volume

volume io-threads
  type performance/io-threads
  option thread-count 16
  subvolumes locks
end-volume

volume read-ahead
  type performance/read-ahead
  option page-count 16
  subvolumes io-threads
end-volume

volume write-behind
  type performance/write-behind
  option cache-size 1MB
  subvolumes read-ahead
end-volume

volume server
  type protocol/server
  option transport-type tcp
  option auth.addr.write-behind.allow *
  subvolumes write-behind
end-volume

client.vol:
volume machine01
  type protocol/client
  option transport-type tcp
  option remote-host machine01
  option remote-subvolume write-behind
end-volume

volume machine02
  type protocol/client
  option transport-type tcp
  option remote-host machine02
  option remote-subvolume write-behind
end-volume

volume stripe
  type cluster/stripe
  option block-size 1MB
  subvolumes machine01 machine02
end-volume

volume read-ahead
  type performance/read-ahead
  option page-count 16
  subvolumes stripe
end-volume

volume write-behind
  type performance/write-behind
  option cache-size 1MB
  subvolumes read-ahead
end-volume


I can saturate my link when catting a 4gb file to /dev/null at around
112MB/s. I'm having trouble, however, getting this to happen when I've
got 4 machines in the stripe. I can only sustain about a 60-80MB/s
read, though I manage to do a 112MB/s write. Anyone see anything
obvious? Here's my client.vol file, and the server.vol is the same as
above:


volume machine01
  type protocol/client
  option transport-type tcp
  option remote-host machine01
  option remote-subvolume write-behind
end-volume

volume machine02
  type protocol/client
  option transport-type tcp
  option remote-host machine02
  option remote-subvolume write-behind
end-volume

volume machine03
  type protocol/client
  option transport-type tcp
  option remote-host machine03
  option remote-subvolume write-behind
end-volume

volume machine04
  type protocol/client
  option transport-type tcp
  option remote-host machine04
  option remote-subvolume write-behind
end-volume

volume stripe
  type cluster/stripe
  option block-size 1MB
  subvolumes machine01 machine02 machine03 machine04
end-volume

volume read-ahead
  type performance/read-ahead
  option page-count 16
  subvolumes stripe
end-volume

volume write-behind
  type performance/write-behind
  option cache-size 1MB
  subvolumes read-ahead
end-volume