[Gluster-users] Gluster samba vfs read performance slow
kane
stef_9k at 163.com
Wed Sep 18 08:27:28 UTC 2013
I compared the profile dumps while write and read is separately running;
writing:
------------------------------------------------
Interval 58 Stats:
Block Size: 65536b+ 131072b+
No. of Reads: 0 0
No. of Writes: 27120 10500
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
100.00 133.51 us 36.00 us 1339.00 us 37619 WRITE
Duration: 12 seconds
Data Read: 0 bytes
Data Written: 3153854464 bytes
------------------------------------------------
read:
------------------------------------------------
Interval 63 Stats:
Block Size: 131072b+
No. of Reads: 3529
No. of Writes: 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.54 87.86 us 68.00 us 127.00 us 42 FSTAT
99.46 193.68 us 89.00 us 2121.00 us 3529 READ
Duration: 12 seconds
Data Read: 462553088 bytes
Data Written: 0 bytes
------------------------------------------------
two server brick avg dumps:
================================
Brick: 192.168.101.133:/dcsdata/d0
----------------------------------
Cumulative Stats:
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 0 0 0
No. of Writes: 2 1 1
Block Size: 65536b+ 131072b+ 262144b+
No. of Reads: 0 1613832 0
No. of Writes: 2282474 1148962 227
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 14 FORGET
0.00 0.00 us 0.00 us 0.00 us 39 RELEASE
0.00 0.00 us 0.00 us 0.00 us 114 RELEASEDIR
0.00 84.50 us 54.00 us 115.00 us 2 OPENDIR
0.00 79.00 us 52.00 us 127.00 us 4 OPEN
0.00 47.00 us 14.00 us 130.00 us 8 FLUSH
0.00 342.00 us 311.00 us 373.00 us 2 CREATE
0.00 104.77 us 26.00 us 281.00 us 13 STATFS
0.01 131.75 us 35.00 us 285.00 us 93 LOOKUP
0.02 7446.00 us 104.00 us 29191.00 us 4 READDIRP
0.07 2784.89 us 49.00 us 49224.00 us 36 GETXATTR
0.20 64.49 us 29.00 us 164.00 us 4506 FSTAT
1.07 399482.25 us 361616.00 us 450370.00 us 4 UNLINK
42.87 167.36 us 56.00 us 44827.00 us 381080 READ
55.76 71.51 us 35.00 us 7032.00 us 1159912 WRITE
Duration: 22156 seconds
Data Read: 211528187904 bytes
Data Written: 300276908032 bytes
Interval 71 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 1 RELEASEDIR
0.18 54.00 us 54.00 us 54.00 us 1 OPENDIR
1.05 107.33 us 40.00 us 217.00 us 3 STATFS
2.90 126.57 us 81.00 us 256.00 us 7 LOOKUP
95.88 14669.00 us 147.00 us 29191.00 us 2 READDIRP
Duration: 581 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: 192.168.101.134:/dcsdata/d0
----------------------------------
Cumulative Stats:
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 0 0 0
No. of Writes: 2 3 24
Block Size: 65536b+ 131072b+ 262144b+
No. of Reads: 22 1563063 0
No. of Writes: 1522412 1525007 184
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 14 FORGET
0.00 0.00 us 0.00 us 0.00 us 39 RELEASE
0.00 0.00 us 0.00 us 0.00 us 114 RELEASEDIR
0.00 116.50 us 111.00 us 122.00 us 2 OPENDIR
0.00 69.25 us 23.00 us 95.00 us 8 FLUSH
0.00 418.00 us 285.00 us 551.00 us 2 CREATE
0.00 239.25 us 101.00 us 396.00 us 4 READDIRP
0.00 93.00 us 39.00 us 249.00 us 13 STATFS
0.01 142.89 us 78.00 us 241.00 us 87 LOOKUP
0.09 48402.25 us 114.00 us 99173.00 us 4 OPEN
0.19 10974.42 us 60.00 us 345979.00 us 36 GETXATTR
0.20 94.33 us 41.00 us 200.00 us 4387 FSTAT
0.85 440436.25 us 381525.00 us 582989.00 us 4 UNLINK
35.80 193.96 us 57.00 us 23312.00 us 380869 READ
62.86 134.89 us 29.00 us 9976.00 us 961593 WRITE
Duration: 22155 seconds
Data Read: 204875400152 bytes
Data Written: 299728837956 bytes
================================
Kane
在 2013-9-18，下午2:45，Anand Avati <avati at gluster.org> 写道：
> Can you get the volume profile dumps for both the runs and compare them?
>
> Avati
>
>
>
> On Tue, Sep 17, 2013 at 10:46 PM, kane <stef_9k at 163.com> wrote:
> I have already used "kernel oplocks = no" in the smb.conf, next is my original smb.conf file global settings:
> [global]
> workgroup = MYGROUP
> server string = DCS Samba Server
> log file = /var/log/samba/log.vfs
> max log size = 500000
> aio read size = 262144
> aio write size = 262144
> aio write behind = true
> security = user
> passdb backend = tdbsam
> load printers = yes
> cups options = raw
> read raw = yes
> write raw = yes
> max xmit = 262144
> socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=262144 SO_SNDBUF=262144
> # max protocol = SMB2
> kernel oplocks = no
> stat cache = no
>
> thank you
> -Kane
> 在 2013-9-18，下午1:38，Anand Avati <avati at redhat.com> 写道：
>
> > On 9/17/13 10:34 PM, kane wrote:
> >> Hi Anand,
> >>
> >> I use 2 gluster server , this is my volume info:
> >> Volume Name: soul
> >> Type: Distribute
> >> Volume ID: 58f049d0-a38a-4ebe-94c0-086d492bdfa6
> >> Status: Started
> >> Number of Bricks: 2
> >> Transport-type: tcp
> >> Bricks:
> >> Brick1: 192.168.101.133:/dcsdata/d0
> >> Brick2: 192.168.101.134:/dcsdata/d0
> >>
> >> each brick use a raid 5 logic disk with 8*2TSATA hdd.
> >>
> >> smb.conf:
> >> [gvol]
> >> comment = For samba export of volume test
> >> vfs objects = glusterfs
> >> glusterfs:volfile_server = localhost
> >> glusterfs:volume = soul
> >> path = /
> >> read only = no
> >> guest ok = yes
> >>
> >> this my testparm result:
> >> [global]
> >> workgroup = MYGROUP
> >> server string = DCS Samba Server
> >> log file = /var/log/samba/log.vfs
> >> max log size = 500000
> >> max xmit = 262144
> >> socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=262144
> >> SO_SNDBUF=262144
> >> stat cache = No
> >> kernel oplocks = No
> >> idmap config * : backend = tdb
> >> aio read size = 262144
> >> aio write size = 262144
> >> aio write behind = true
> >> cups options = raw
> >>
> >> in client mount the smb share with cifs to dir /mnt/vfs,
> >> then use iozone executed in the cifs mount dir "/mnt/vfs":
> >> $ ./iozone -s 10G -r 128k -i0 -i1 -t 4
> >> File size set to 10485760 KB
> >> Record Size 128 KB
> >> Command line used: ./iozone -s 10G -r 128k -i0 -i1 -t 4
> >> Output is in Kbytes/sec
> >> Time Resolution = 0.000001 seconds.
> >> Processor cache size set to 1024 Kbytes.
> >> Processor cache line size set to 32 bytes.
> >> File stride size set to 17 * record size.
> >> Throughput test with 4 processes
> >> Each process writes a 10485760 Kbyte file in 128 Kbyte records
> >>
> >> Children see throughput for 4 initial writers = 534315.84 KB/sec
> >> Parent sees throughput for 4 initial writers = 519428.83 KB/sec
> >> Min throughput per process = 133154.69 KB/sec
> >> Max throughput per process = 134341.05 KB/sec
> >> Avg throughput per process = 133578.96 KB/sec
> >> Min xfer = 10391296.00 KB
> >>
> >> Children see throughput for 4 rewriters = 536634.88 KB/sec
> >> Parent sees throughput for 4 rewriters = 522618.54 KB/sec
> >> Min throughput per process = 133408.80 KB/sec
> >> Max throughput per process = 134721.36 KB/sec
> >> Avg throughput per process = 134158.72 KB/sec
> >> Min xfer = 10384384.00 KB
> >>
> >> Children see throughput for 4 readers = 77403.54 KB/sec
> >> Parent sees throughput for 4 readers = 77402.86 KB/sec
> >> Min throughput per process = 19349.42 KB/sec
> >> Max throughput per process = 19353.42 KB/sec
> >> Avg throughput per process = 19350.88 KB/sec
> >> Min xfer = 10483712.00 KB
> >>
> >> Children see throughput for 4 re-readers = 77424.40 KB/sec
> >> Parent sees throughput for 4 re-readers = 77423.89 KB/sec
> >> Min throughput per process = 19354.75 KB/sec
> >> Max throughput per process = 19358.50 KB/sec
> >> Avg throughput per process = 19356.10 KB/sec
> >> Min xfer = 10483840.00 KB
> >>
> >> then the use the same command test in the dir mounted with glister fuse:
> >> File size set to 10485760 KB
> >> Record Size 128 KB
> >> Command line used: ./iozone -s 10G -r 128k -i0 -i1 -t 4
> >> Output is in Kbytes/sec
> >> Time Resolution = 0.000001 seconds.
> >> Processor cache size set to 1024 Kbytes.
> >> Processor cache line size set to 32 bytes.
> >> File stride size set to 17 * record size.
> >> Throughput test with 4 processes
> >> Each process writes a 10485760 Kbyte file in 128 Kbyte records
> >>
> >> Children see throughput for 4 initial writers = 887534.72 KB/sec
> >> Parent sees throughput for 4 initial writers = 848830.39 KB/sec
> >> Min throughput per process = 220140.91 KB/sec
> >> Max throughput per process = 223690.45 KB/sec
> >> Avg throughput per process = 221883.68 KB/sec
> >> Min xfer = 10319360.00 KB
> >>
> >> Children see throughput for 4 rewriters = 892774.92 KB/sec
> >> Parent sees throughput for 4 rewriters = 871186.83 KB/sec
> >> Min throughput per process = 222326.44 KB/sec
> >> Max throughput per process = 223970.17 KB/sec
> >> Avg throughput per process = 223193.73 KB/sec
> >> Min xfer = 10431360.00 KB
> >>
> >> Children see throughput for 4 readers = 605889.12 KB/sec
> >> Parent sees throughput for 4 readers = 601767.96 KB/sec
> >> Min throughput per process = 143133.14 KB/sec
> >> Max throughput per process = 159550.88 KB/sec
> >> Avg throughput per process = 151472.28 KB/sec
> >> Min xfer = 9406848.00 KB
> >>
> >> it shows much higher perf.
> >>
> >> any places i did wrong?
> >>
> >>
> >> thank you
> >> -Kane
> >>
> >> 在 2013-9-18，下午1:19，Anand Avati <avati at gluster.org
> >> <mailto:avati at gluster.org>> 写道：
> >>
> >>> How are you testing this? What tool are you using?
> >>>
> >>> Avati
> >>>
> >>>
> >>> On Tue, Sep 17, 2013 at 9:02 PM, kane <stef_9k at 163.com
> >>> <mailto:stef_9k at 163.com>> wrote:
> >>>
> >>> Hi Vijay
> >>>
> >>> I used the code in
> >>> https://github.com/gluster/glusterfs.git with the lasted commit:
> >>> commit de2a8d303311bd600cb93a775bc79a0edea1ee1a
> >>> Author: Anand Avati <avati at redhat.com <mailto:avati at redhat.com>>
> >>> Date: Tue Sep 17 16:45:03 2013 -0700
> >>>
> >>> Revert "cluster/distribute: Rebalance should also verify free
> >>> inodes"
> >>>
> >>> This reverts commit 215fea41a96479312a5ab8783c13b30ab9fe00fa
> >>>
> >>> Realized soon after merging, ….
> >>>
> >>> which include the patch you mentioned last time improve read perf,
> >>> written by Anand.
> >>>
> >>> but the read perf was still slow:
> >>> write: 500MB/s
> >>> read: 77MB/s
> >>>
> >>> while via fuse :
> >>> write 800MB/s
> >>> read 600MB/s
> >>>
> >>> any advises?
> >>>
> >>>
> >>> Thank you.
> >>> -Kane
> >>>
> >>> 在 2013-9-13，下午10:37，kane <stef_9k at 163.com
> >>> <mailto:stef_9k at 163.com>> 写道：
> >>>
> >>>> Hi Vijay，
> >>>>
> >>>> thank you for post this message, i will try it soon
> >>>>
> >>>> -kane
> >>>>
> >>>>
> >>>>
> >>>> 在 2013-9-13，下午9:21，Vijay Bellur <vbellur at redhat.com
> >>> <mailto:vbellur at redhat.com>> 写道：
> >>>>
> >>>>> On 09/13/2013 06:10 PM, kane wrote:
> >>>>>> Hi
> >>>>>>
> >>>>>> We use gluster samba vfs test io,but the read performance via
> >>> vfs is
> >>>>>> half of write perfomance,
> >>>>>> but via fuse the read and write performance is almost the same.
> >>>>>>
> >>>>>> this is our smb.conf:
> >>>>>> [global]
> >>>>>> workgroup = MYGROUP
> >>>>>> server string = DCS Samba Server
> >>>>>> log file = /var/log/samba/log.vfs
> >>>>>> max log size = 500000
> >>>>>> # use sendfile = true
> >>>>>> aio read size = 262144
> >>>>>> aio write size = 262144
> >>>>>> aio write behind = true
> >>>>>> min receivefile size = 262144
> >>>>>> write cache size = 268435456
> >>>>>> security = user
> >>>>>> passdb backend = tdbsam
> >>>>>> load printers = yes
> >>>>>> cups options = raw
> >>>>>> read raw = yes
> >>>>>> write raw = yes
> >>>>>> max xmit = 262144
> >>>>>> socket options = TCP_NODELAY IPTOS_LOWDELAY
> >>> SO_RCVBUF=262144
> >>>>>> SO_SNDBUF=262144
> >>>>>> kernel oplocks = no
> >>>>>> stat cache = no
> >>>>>>
> >>>>>> any advises helpful?
> >>>>>>
> >>>>>
> >>>>> This patch has shown improvement in read performance with libgfapi:
> >>>>>
> >>>>> http://review.gluster.org/#/c/5897/
> >>>>>
> >>>>> Would it be possible for you to try this patch and check if it
> >>> improves performance in your case?
> >>>>>
> >>>>> -Vijay
> >>>>>
> >>>>
> >>>
> >>>
> >>> _______________________________________________
> >>> Gluster-users mailing list
> >>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
> >>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
> >>>
> >>>
> >>
> >
> > Please add 'kernel oplocks = no' in the [gvol] section and try again.
> >
> > Avati
> >
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130918/ab613dc7/attachment.html>
More information about the Gluster-users
mailing list