[Gluster-devel] how best to set up for performance?

[Niall Dalton]

On Mar 16, 2008, at 12:46 PM, Anand Avati wrote:
>  You might want to try specifying
>
> 'option window-size 2097152' in all your protocol (client and  
> server) volumes.
>
> Also, you might also want read-ahead on the server (over iothreads)  
> with the same page-size as the client, but higher page-count.



Thanks for all the help. I turned on the window-size option but  
specifying in the client volumes slowed things right down. Setting it  
only in the servers showed a speedup.

I also discovered that in striped mode I was basically running at  
exactly the same speed as distinct gluster fs volumes on each storage  
server. So, splitting things into 2 mount points on my client, each  
against one of the servers, doubled my aggregate performance to around  
~650MB/s.
I estimate I should be able to double performance again if I could  
figure out how these various settings are all interacting.

Any other suggestions appreciated - or if someone from zresearch would  
like to log in and take a look I'd be delighted to set that up.

niall

server-jr1
------------
volume posix
   type storage/posix
   option directory /big
end-volume

volume brick
   type performance/io-threads
   option thread-count 8
   option cache-size 12288MB
   subvolumes posix
end-volume

volume readahead-brick
   type performance/read-ahead
   option page-size 1M
   option page-count 512
   subvolumes brick
end-volume

volume server
   type protocol/server
   subvolumes brick
   option transport-type tcp/server     # For TCP/IP transport
   option auth.ip.brick.allow *
   subvolumes readahead-brick
   option window-size 2097152
end-volume


server-jr2
------------
volume posix
   type storage/posix
   option directory /big
end-volume

volume brick
   type performance/io-threads
   option thread-count 8
   option cache-size 12288MB
   subvolumes posix
end-volume

volume readahead-brick
   type performance/read-ahead
   option page-size 1MB
   option page-count 512
   subvolumes brick
end-volume

volume server
   type protocol/server
   subvolumes brick
   option transport-type tcp/server     # For TCP/IP transport
   option auth.ip.brick.allow *
   option window-size 2097152
   subvolumes readahead-brick
end-volume


client-jr1:
------------
volume jr1
   type protocol/client
   option transport-type tcp/client
   option remote-host 192.168.3.2
   option remote-subvolume brick
end-volume

volume readahead-jr1
   type performance/read-ahead
   option page-size 1MB
   option page-count 64
   subvolumes jr1
end-volume

volume iot
  type performance/io-threads
  option thread-count 4
  option cache-size 1024MB
  subvolumes readahead-jr1
end-volume

volume writebehind
   type performance/write-behind
   subvolumes iot
end-volume

volume readahead
   type performance/read-ahead
   option page-size 1MB
   option page-count 64
   subvolumes writebehind
end-volume

client-jr2
------------
volume jr2
   type protocol/client
   option transport-type tcp/client
   option remote-host 192.168.2.2
   option remote-subvolume brick
end-volume

volume readahead-jr2
   type performance/read-ahead
   option page-size 1MB
   option page-count 64
   subvolumes jr2
end-volume

volume iot
  type performance/io-threads
  option thread-count 4
  option cache-size 1024MB
  subvolumes readahead-jr2
end-volume

volume readahead-jr2
   type performance/read-ahead
   option page-size 1MB
   option page-count 64
   subvolumes iot
end-volume

volume writebehind
   type performance/write-behind
   subvolumes iot
end-volume

volume readahead
   type performance/read-ahead
   option page-size 1MB
   option page-count 64
   subvolumes writebehind
end-volume