[Gluster-users] Gluster samba vfs read performance slow

kane stef_9k at 163.com
Wed Sep 18 05:34:48 UTC 2013 

Hi Anand,

	I use 2 gluster server , this is my volume info:
Volume Name: soul
Type: Distribute
Volume ID: 58f049d0-a38a-4ebe-94c0-086d492bdfa6
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: 192.168.101.133:/dcsdata/d0
Brick2: 192.168.101.134:/dcsdata/d0

each brick use a raid 5 logic disk with 8*2TSATA hdd.

smb.conf:
[gvol]
        comment = For samba export of volume  test
        vfs objects = glusterfs
        glusterfs:volfile_server = localhost
        glusterfs:volume = soul
        path = /
        read only = no
        guest ok = yes

this my testparm result:
[global]
	workgroup = MYGROUP
	server string = DCS Samba Server
	log file = /var/log/samba/log.vfs
	max log size = 500000
	max xmit = 262144
	socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=262144 SO_SNDBUF=262144
	stat cache = No
	kernel oplocks = No
	idmap config * : backend = tdb
	aio read size = 262144
	aio write size = 262144
	aio write behind = true
	cups options = raw

in client mount the smb share with cifs to dir /mnt/vfs, 
then use iozone executed in the cifs mount dir "/mnt/vfs":
$ ./iozone -s 10G -r 128k -i0 -i1 -t 4
	File size set to 10485760 KB
	Record Size 128 KB
	Command line used: ./iozone -s 10G -r 128k -i0 -i1 -t 4
	Output is in Kbytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 Kbytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
	Throughput test with 4 processes
	Each process writes a 10485760 Kbyte file in 128 Kbyte records

	Children see throughput for  4 initial writers 	=  534315.84 KB/sec
	Parent sees throughput for  4 initial writers 	=  519428.83 KB/sec
	Min throughput per process 			=  133154.69 KB/sec 
	Max throughput per process 			=  134341.05 KB/sec
	Avg throughput per process 			=  133578.96 KB/sec
	Min xfer 					= 10391296.00 KB

	Children see throughput for  4 rewriters 	=  536634.88 KB/sec
	Parent sees throughput for  4 rewriters 	=  522618.54 KB/sec
	Min throughput per process 			=  133408.80 KB/sec 
	Max throughput per process 			=  134721.36 KB/sec
	Avg throughput per process 			=  134158.72 KB/sec
	Min xfer 					= 10384384.00 KB

	Children see throughput for  4 readers 		=   77403.54 KB/sec
	Parent sees throughput for  4 readers 		=   77402.86 KB/sec
	Min throughput per process 			=   19349.42 KB/sec 
	Max throughput per process 			=   19353.42 KB/sec
	Avg throughput per process 			=   19350.88 KB/sec
	Min xfer 					= 10483712.00 KB

	Children see throughput for 4 re-readers 	=   77424.40 KB/sec
	Parent sees throughput for 4 re-readers 	=   77423.89 KB/sec
	Min throughput per process 			=   19354.75 KB/sec 
	Max throughput per process 			=   19358.50 KB/sec
	Avg throughput per process 			=   19356.10 KB/sec
	Min xfer 					= 10483840.00 KB

then the use the same command test in the dir mounted with glister fuse:
	File size set to 10485760 KB
	Record Size 128 KB
	Command line used: ./iozone -s 10G -r 128k -i0 -i1 -t 4
	Output is in Kbytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 Kbytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
	Throughput test with 4 processes
	Each process writes a 10485760 Kbyte file in 128 Kbyte records

	Children see throughput for  4 initial writers 	=  887534.72 KB/sec
	Parent sees throughput for  4 initial writers 	=  848830.39 KB/sec
	Min throughput per process 			=  220140.91 KB/sec 
	Max throughput per process 			=  223690.45 KB/sec
	Avg throughput per process 			=  221883.68 KB/sec
	Min xfer 					= 10319360.00 KB

	Children see throughput for  4 rewriters 	=  892774.92 KB/sec
	Parent sees throughput for  4 rewriters 	=  871186.83 KB/sec
	Min throughput per process 			=  222326.44 KB/sec 
	Max throughput per process 			=  223970.17 KB/sec
	Avg throughput per process 			=  223193.73 KB/sec
	Min xfer 					= 10431360.00 KB

	Children see throughput for  4 readers 		=  605889.12 KB/sec
	Parent sees throughput for  4 readers 		=  601767.96 KB/sec
	Min throughput per process 			=  143133.14 KB/sec 
	Max throughput per process 			=  159550.88 KB/sec
	Avg throughput per process 			=  151472.28 KB/sec
	Min xfer 					= 9406848.00 KB

it shows much higher perf.

any places i did wrong?


thank you
-Kane

在 2013-9-18，下午1:19，Anand Avati <avati at gluster.org> 写道：

> How are you testing this? What tool are you using?
> 
> Avati
> 
> 
> On Tue, Sep 17, 2013 at 9:02 PM, kane <stef_9k at 163.com> wrote:
> Hi Vijay
> 
>         I used the code in https://github.com/gluster/glusterfs.git with the lasted commit:
> commit de2a8d303311bd600cb93a775bc79a0edea1ee1a
> Author: Anand Avati <avati at redhat.com>
> Date:   Tue Sep 17 16:45:03 2013 -0700
> 
>     Revert "cluster/distribute: Rebalance should also verify free inodes"
> 
>     This reverts commit 215fea41a96479312a5ab8783c13b30ab9fe00fa
> 
>     Realized soon after merging, ….
> 
> which include the patch you mentioned last time improve read perf, written by Anand.
> 
> but the read perf was still slow:
> write: 500MB/s
> read: 77MB/s
> 
> while via fuse :
> write 800MB/s
> read 600MB/s
> 
> any advises?
> 
> 
> Thank you.
> -Kane
> 
> 在 2013-9-13，下午10:37，kane <stef_9k at 163.com> 写道：
> 
> > Hi Vijay，
> >
> >       thank you for post this message, i will try it soon
> >
> > -kane
> >
> >
> >
> > 在 2013-9-13，下午9:21，Vijay Bellur <vbellur at redhat.com> 写道：
> >
> >> On 09/13/2013 06:10 PM, kane wrote:
> >>> Hi
> >>>
> >>> We use gluster samba vfs test io,but the read performance via vfs is
> >>> half of write perfomance,
> >>> but via fuse the read and write performance is almost the same.
> >>>
> >>> this is our smb.conf:
> >>> [global]
> >>>        workgroup = MYGROUP
> >>>        server string = DCS Samba Server
> >>>        log file = /var/log/samba/log.vfs
> >>>        max log size = 500000
> >>> #       use sendfile = true
> >>>        aio read size = 262144
> >>>        aio write size = 262144
> >>>        aio write behind = true
> >>>        min receivefile size = 262144
> >>>        write cache size = 268435456
> >>>        security = user
> >>>        passdb backend = tdbsam
> >>>        load printers = yes
> >>>        cups options = raw
> >>>        read raw = yes
> >>>        write raw = yes
> >>>        max xmit = 262144
> >>>        socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=262144
> >>> SO_SNDBUF=262144
> >>>        kernel oplocks = no
> >>>        stat cache = no
> >>>
> >>> any advises helpful?
> >>>
> >>
> >> This patch has shown improvement in read performance with libgfapi:
> >>
> >> http://review.gluster.org/#/c/5897/
> >>
> >> Would it be possible for you to try this patch and check if it improves performance in your case?
> >>
> >> -Vijay
> >>
> >
> 
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130918/fabbf07e/attachment.html>

More information about the Gluster-users mailing list