[Gluster-users] Gluster samba vfs read performance slow
Anand Avati
avati at redhat.com
Wed Sep 18 05:38:14 UTC 2013
On 9/17/13 10:34 PM, kane wrote:
> Hi Anand,
>
> I use 2 gluster server , this is my volume info:
> Volume Name: soul
> Type: Distribute
> Volume ID: 58f049d0-a38a-4ebe-94c0-086d492bdfa6
> Status: Started
> Number of Bricks: 2
> Transport-type: tcp
> Bricks:
> Brick1: 192.168.101.133:/dcsdata/d0
> Brick2: 192.168.101.134:/dcsdata/d0
>
> each brick use a raid 5 logic disk with 8*2TSATA hdd.
>
> smb.conf:
> [gvol]
> comment = For samba export of volume test
> vfs objects = glusterfs
> glusterfs:volfile_server = localhost
> glusterfs:volume = soul
> path = /
> read only = no
> guest ok = yes
>
> this my testparm result:
> [global]
> workgroup = MYGROUP
> server string = DCS Samba Server
> log file = /var/log/samba/log.vfs
> max log size = 500000
> max xmit = 262144
> socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=262144
> SO_SNDBUF=262144
> stat cache = No
> kernel oplocks = No
> idmap config * : backend = tdb
> aio read size = 262144
> aio write size = 262144
> aio write behind = true
> cups options = raw
>
> in client mount the smb share with cifs to dir /mnt/vfs,
> then use iozone executed in the cifs mount dir "/mnt/vfs":
> $ ./iozone -s 10G -r 128k -i0 -i1 -t 4
> File size set to 10485760 KB
> Record Size 128 KB
> Command line used: ./iozone -s 10G -r 128k -i0 -i1 -t 4
> Output is in Kbytes/sec
> Time Resolution = 0.000001 seconds.
> Processor cache size set to 1024 Kbytes.
> Processor cache line size set to 32 bytes.
> File stride size set to 17 * record size.
> Throughput test with 4 processes
> Each process writes a 10485760 Kbyte file in 128 Kbyte records
>
> Children see throughput for 4 initial writers = 534315.84 KB/sec
> Parent sees throughput for 4 initial writers = 519428.83 KB/sec
> Min throughput per process = 133154.69 KB/sec
> Max throughput per process = 134341.05 KB/sec
> Avg throughput per process = 133578.96 KB/sec
> Min xfer = 10391296.00 KB
>
> Children see throughput for 4 rewriters = 536634.88 KB/sec
> Parent sees throughput for 4 rewriters = 522618.54 KB/sec
> Min throughput per process = 133408.80 KB/sec
> Max throughput per process = 134721.36 KB/sec
> Avg throughput per process = 134158.72 KB/sec
> Min xfer = 10384384.00 KB
>
> Children see throughput for 4 readers = 77403.54 KB/sec
> Parent sees throughput for 4 readers = 77402.86 KB/sec
> Min throughput per process = 19349.42 KB/sec
> Max throughput per process = 19353.42 KB/sec
> Avg throughput per process = 19350.88 KB/sec
> Min xfer = 10483712.00 KB
>
> Children see throughput for 4 re-readers = 77424.40 KB/sec
> Parent sees throughput for 4 re-readers = 77423.89 KB/sec
> Min throughput per process = 19354.75 KB/sec
> Max throughput per process = 19358.50 KB/sec
> Avg throughput per process = 19356.10 KB/sec
> Min xfer = 10483840.00 KB
>
> then the use the same command test in the dir mounted with glister fuse:
> File size set to 10485760 KB
> Record Size 128 KB
> Command line used: ./iozone -s 10G -r 128k -i0 -i1 -t 4
> Output is in Kbytes/sec
> Time Resolution = 0.000001 seconds.
> Processor cache size set to 1024 Kbytes.
> Processor cache line size set to 32 bytes.
> File stride size set to 17 * record size.
> Throughput test with 4 processes
> Each process writes a 10485760 Kbyte file in 128 Kbyte records
>
> Children see throughput for 4 initial writers = 887534.72 KB/sec
> Parent sees throughput for 4 initial writers = 848830.39 KB/sec
> Min throughput per process = 220140.91 KB/sec
> Max throughput per process = 223690.45 KB/sec
> Avg throughput per process = 221883.68 KB/sec
> Min xfer = 10319360.00 KB
>
> Children see throughput for 4 rewriters = 892774.92 KB/sec
> Parent sees throughput for 4 rewriters = 871186.83 KB/sec
> Min throughput per process = 222326.44 KB/sec
> Max throughput per process = 223970.17 KB/sec
> Avg throughput per process = 223193.73 KB/sec
> Min xfer = 10431360.00 KB
>
> Children see throughput for 4 readers = 605889.12 KB/sec
> Parent sees throughput for 4 readers = 601767.96 KB/sec
> Min throughput per process = 143133.14 KB/sec
> Max throughput per process = 159550.88 KB/sec
> Avg throughput per process = 151472.28 KB/sec
> Min xfer = 9406848.00 KB
>
> it shows much higher perf.
>
> any places i did wrong?
>
>
> thank you
> -Kane
>
> 在 2013-9-18，下午1:19，Anand Avati <avati at gluster.org
> <mailto:avati at gluster.org>> 写道：
>
>> How are you testing this? What tool are you using?
>>
>> Avati
>>
>>
>> On Tue, Sep 17, 2013 at 9:02 PM, kane <stef_9k at 163.com
>> <mailto:stef_9k at 163.com>> wrote:
>>
>> Hi Vijay
>>
>> I used the code in
>> https://github.com/gluster/glusterfs.git with the lasted commit:
>> commit de2a8d303311bd600cb93a775bc79a0edea1ee1a
>> Author: Anand Avati <avati at redhat.com <mailto:avati at redhat.com>>
>> Date: Tue Sep 17 16:45:03 2013 -0700
>>
>> Revert "cluster/distribute: Rebalance should also verify free
>> inodes"
>>
>> This reverts commit 215fea41a96479312a5ab8783c13b30ab9fe00fa
>>
>> Realized soon after merging, ….
>>
>> which include the patch you mentioned last time improve read perf,
>> written by Anand.
>>
>> but the read perf was still slow:
>> write: 500MB/s
>> read: 77MB/s
>>
>> while via fuse :
>> write 800MB/s
>> read 600MB/s
>>
>> any advises?
>>
>>
>> Thank you.
>> -Kane
>>
>> 在 2013-9-13，下午10:37，kane <stef_9k at 163.com
>> <mailto:stef_9k at 163.com>> 写道：
>>
>> > Hi Vijay，
>> >
>> > thank you for post this message, i will try it soon
>> >
>> > -kane
>> >
>> >
>> >
>> > 在 2013-9-13，下午9:21，Vijay Bellur <vbellur at redhat.com
>> <mailto:vbellur at redhat.com>> 写道：
>> >
>> >> On 09/13/2013 06:10 PM, kane wrote:
>> >>> Hi
>> >>>
>> >>> We use gluster samba vfs test io,but the read performance via
>> vfs is
>> >>> half of write perfomance,
>> >>> but via fuse the read and write performance is almost the same.
>> >>>
>> >>> this is our smb.conf:
>> >>> [global]
>> >>> workgroup = MYGROUP
>> >>> server string = DCS Samba Server
>> >>> log file = /var/log/samba/log.vfs
>> >>> max log size = 500000
>> >>> # use sendfile = true
>> >>> aio read size = 262144
>> >>> aio write size = 262144
>> >>> aio write behind = true
>> >>> min receivefile size = 262144
>> >>> write cache size = 268435456
>> >>> security = user
>> >>> passdb backend = tdbsam
>> >>> load printers = yes
>> >>> cups options = raw
>> >>> read raw = yes
>> >>> write raw = yes
>> >>> max xmit = 262144
>> >>> socket options = TCP_NODELAY IPTOS_LOWDELAY
>> SO_RCVBUF=262144
>> >>> SO_SNDBUF=262144
>> >>> kernel oplocks = no
>> >>> stat cache = no
>> >>>
>> >>> any advises helpful?
>> >>>
>> >>
>> >> This patch has shown improvement in read performance with libgfapi:
>> >>
>> >> http://review.gluster.org/#/c/5897/
>> >>
>> >> Would it be possible for you to try this patch and check if it
>> improves performance in your case?
>> >>
>> >> -Vijay
>> >>
>> >
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>>
>
Please add 'kernel oplocks = no' in the [gvol] section and try again.
Avati
More information about the Gluster-users
mailing list