[Gluster-users] tuning

[Papp Tamas]

On 2011-07-06 23:58, Papp Tamas wrote:
> hi!
>
> I'm almost absolutely new to glusterfs.
>
> Until now we used Storage Platform (3.0.5).
> Today we installed Ubuntu 11.04 and glusterfs 3.2.1.
>
> $ cat w-vol-fuse.vol
> volume w-vol-client-0
>     type protocol/client
>     option remote-host gl0
>     option remote-subvolume /mnt/brick1
>     option transport-type tcp
> end-volume
>
> volume w-vol-client-1
>     type protocol/client
>     option remote-host gl1
>     option remote-subvolume /mnt/brick1
>     option transport-type tcp
> end-volume
>
> volume w-vol-client-2
>     type protocol/client
>     option remote-host gl2
>     option remote-subvolume /mnt/brick1
>     option transport-type tcp
> end-volume
>
> volume w-vol-client-3
>     type protocol/client
>     option remote-host gl3
>     option remote-subvolume /mnt/brick1
>     option transport-type tcp
> end-volume
>
> volume w-vol-client-4
>     type protocol/client
>     option remote-host gl4
>     option remote-subvolume /mnt/brick1
>     option transport-type tcp
> end-volume
>
> volume w-vol-dht
>     type cluster/distribute
>     subvolumes w-vol-client-0 w-vol-client-1 w-vol-client-2 
> w-vol-client-3 w-vol-client-4
> end-volume
>
> volume w-vol-write-behind
>     type performance/write-behind
>     option cache-size 4MB
>     subvolumes w-vol-dht
> end-volume
>
> volume w-vol-read-ahead
>     type performance/read-ahead
>     subvolumes w-vol-write-behind
> end-volume
>
> volume w-vol-io-cache
>     type performance/io-cache
>     option cache-size 128MB
>     subvolumes w-vol-read-ahead
> end-volume
>
> volume w-vol-quick-read
>     type performance/quick-read
>     option cache-size 128MB
>     subvolumes w-vol-io-cache
> end-volume
>
> volume w-vol-stat-prefetch
>     type performance/stat-prefetch
>     subvolumes w-vol-quick-read
> end-volume
>
> volume w-vol
>     type debug/io-stats
>     option latency-measurement off
>     option count-fop-hits off
>     subvolumes w-vol-stat-prefetch
> end-volume
>
>
> $ cat w-vol.gl0.mnt-brick1.vol
> volume w-vol-posix
>     type storage/posix
>     option directory /mnt/brick1
> end-volume
>
> volume w-vol-access-control
>     type features/access-control
>     subvolumes w-vol-posix
> end-volume
>
> volume w-vol-locks
>     type features/locks
>     subvolumes w-vol-access-control
> end-volume
>
> volume w-vol-io-threads
>     type performance/io-threads
>     subvolumes w-vol-locks
> end-volume
>
> volume w-vol-marker
>     type features/marker
>     option volume-uuid ad362448-7ef0-49ae-b13c-74cb82ce9be5
>     option timestamp-file /etc/glusterd/vols/w-vol/marker.tstamp
>     option xtime off
>     option quota off
>     subvolumes w-vol-io-threads
> end-volume
>
> volume /mnt/brick1
>     type debug/io-stats
>     option latency-measurement off
>     option count-fop-hits off
>     subvolumes w-vol-marker
> end-volume
>
> volume w-vol-server
>     type protocol/server
>     option transport-type tcp
>     option auth.addr./mnt/brick1.allow *
>     subvolumes /mnt/brick1
> end-volume
>
>
> There is 5 nodes. 3 have 8 disks in RAID 6 (supermicro server, are 
> controller), 2 have 8 disks in raid5+spare (DL180).
> Filesystem of datas was created via this command (of course a bit 
> different on HPs):
>
> mkfs.xfs -b size=4096 -d sunit=256,swidth=1536 -L gluster /dev/sda4
>
>
> The performance is far away that was before. I tried to modify
>
> performance.write-behind-window-size 4MB
> gluster volume set w-vol performance.cache-size 128MB
> gluster volume set w-vol nfs.disable on
>
> echo 512 > /sys/block/sda/queue/nr_requests
> blockdev --setra 16384 /dev/sda
> sysctl -w vm.swappiness=5
> sysctl -w vm.dirty_background_ratio=3
> sysctl -w vm.dirty_ratio=40
> sysctl -w kernel.sysrq=0
>
> Nothing really helped.
> Can somebody give some instructions?

Some more information.

On the node:

$ dd if=/dev/zero of=adsdfgrr bs=128K count=100k oflag=direct
102400+0 records in
102400+0 records out
13421772800 bytes (13 GB) copied, 27.4022 s, 490 MB/s

The same on the cluster volume is ~50-60 MB/s.

Network layer is GE, nodes are connected with two NICs in bonding.

I am absolutely desparated. Is it Ubuntu? Would be better with Fedora? 
Or does the Storage Platform run on an optimized kernel or something 
like that?


Thank you,

tamas