[Gluster-users] Gluster 3.4 Samba VFS writes slow in Win 7 clients
Anand Avati
anand.avati at gmail.com
Thu Aug 22 18:27:03 UTC 2013
I guess the reason for this big difference in write v/s rewrite in win7
performance could be, the fact that we don't implement fallocate() method
in vfs_glusterfs. If the client (e.g win7) requests guaranteed reservation
of space before initiating the write, then Samba falls back to the crude
method of writing zeroes to the file for the full size even before
accepting the first write call from the client. I am assuming the linux
cifs client does not care about reservation of space and therefore Samba
does not execute the zero-writing fallacote fallback.
Avati
On Thu, Aug 22, 2013 at 11:12 AM, Lalatendu Mohanty <lmohanty at redhat.com>wrote:
> On 08/22/2013 02:14 PM, kane wrote:
>
> Hi Raghavendra Talur,
>
> 1. I found that use the smb.conf test with iozone, it shows some
> difference in results:
> smb.conf:
> -----------
> [root at localhost ~]# testparm
> Load smb config files from /etc/samba/smb.conf
> rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)
> Processing section "[homes]"
> Processing section "[printers]"
> Processing section "[cifs]"
> Processing section "[raw]"
> Processing section "[gvol]"
> Loaded services file OK.
> Server role: ROLE_STANDALONE
> Press enter to see a dump of your service definitions
>
> [global]
> workgroup = MYGROUP
> server string = DCS Samba Server
> log file = /var/log/samba/log.vfs
> max log size = 500000
> max protocol = SMB2
> min receivefile size = 262144
> max xmit = 262144
> socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=262144
> SO_SNDBUF=262144
> idmap config * : backend = tdb
> aio read size = 262144
> aio write size = 262144
> aio write behind = true
> write cache size = 268435456
> cups options = raw
> …….
>
> [raw]
> path = /dcsdata/d0
> read only = No
> guest ok = Yes
>
> [gvol]
> comment = For samba export of volume test
> path = /
> read only = No
> guest ok = Yes
> vfs objects = glusterfs
> glusterfs:volume = soul
> glusterfs:volfile_server = localhost
> -----------
> iozone test with cmd : iozone -s 10G -r 1m -i0 -t 4
> -----------
> Run began: Thu Aug 22 16:11:40 2013
>
> File size set to 10485760 KB
> Record Size 1024 KB
> Command line used: iozone -s 10G -r 1m -i0 -t 4
> Output is in Kbytes/sec
> Time Resolution = 0.000000 seconds.
> Processor cache size set to 1024 Kbytes.
> Processor cache line size set to 32 bytes.
> File stride size set to 17 * record size.
> Throughput test with 4 processes
> Each process writes a 10485760 Kbyte file in 1024 Kbyte records
>
> Children see throughput for 4 initial writers = 147008.14
> KB/sec
> Parent sees throughput for 4 initial writers = 146846.43 KB/sec
> Min throughput per process = 36750.59 KB/sec
> Max throughput per process = 36754.97 KB/sec
> Avg throughput per process = 36752.04 KB/sec
> Min xfer = 10484736.00 KB
>
> Children see throughput for 4 rewriters = 147494.85
> KB/sec
> Parent sees throughput for 4 rewriters = 147310.95 KB/sec
> Min throughput per process = 36871.96 KB/sec
> Max throughput per process = 36877.09 KB/sec
> Avg throughput per process = 36873.71 KB/sec
> Min xfer = 10484736.00 KB
>
> iozone test complete.
> -----------
>
> The results of rewrite show some difference, with your recommend
> smb.conf, the rewite and write diff in iozone docs :
>
> Write: This test measures the performance of writing a new file. When a
> new file is written not only does the data need to be stored but also the
> overhead information for keeping track of where the data is located on the
> storage media. This overhead is called the “metadata” It consists of the
> directory information, the space allocation and any other data associated
> with a file that is not part of the data contained in the file. It is
> normal for the initial write performance to be lower than the performance
> of re- writing a file due to this overhead information.
>
> Re-write: This test measures the performance of writing a file that
> already exists. When a file is written that already exists the work
> required is less as the metadata already exists. It is normal for the
> rewrite performance to be higher than the performance of writing a new
> file.
> but use iozone test with 4 threads, the rewrite performs much better than
> write,
> i thought rewite:180MB/s vs write:150MB/s is reasonable, but
> rewrite:400MB/s vs 140MB/s is out of my expectation.
>
>
>
> Kane,
>
> When we are using same samba share on windows and Linux, the only thing
> different is "unix extensions" support in Linux which Linux uses by
> default. However I don't think "unix extension" has anything to do with
> performance of writes and rewrites. I only guess is the samba client in
> Linux is works well with samba than the Microsoft's smb client which is in
> windows 7.
>
> "unix extensions" parameter controls whether Samba implements the CIFS
> UNIX extensions. These extensions enable Samba to better serve UNIX CIFS
> clients by supporting features such as symbolic links, hard links, etc...
> These extensions require a similarly enabled client, and are of no current
> use to Windows clients.
>
> To disable unix extension put following in global section of smb.conf and
> restart smb service*
> unix extensions = no*
>
> thanks
> -kane
>
> 在 2013-8-22，下午4:06，kane <stef_9k at 163.com> 写道：
>
> Hi Raghavendra Talur,
>
> 1. My samba version is:
> [root at localhost ~]# smbd -V
> Version 3.6.9-151.el6
>
> 2. Sorry in the first mail list, I forgot to tell, when use win7 client
> mount the server raw xfs backend with a raid5 disk,
> it shows good write performance with same smb.conf in samba vfs glusterfs
> 3.4 test show next in point 3:
> $ ./iozone -s 10G -r 128k -i0 -t 4
> ----------------
> Run began: Thu Aug 22 15:59:11 2013
>
> File size set to 10485760 KB
> Record Size 1024 KB
> Command line used: iozone -s 10G -r 1m -i0 -t 4
> Output is in Kbytes/sec
> Time Resolution = -0.000000 seconds.
> Processor cache size set to 1024 Kbytes.
> Processor cache line size set to 32 bytes.
> File stride size set to 17 * record size.
> Throughput test with 4 processes
> Each process writes a 10485760 Kbyte file in 1024 Kbyte records
>
> Children see throughput for 4 initial writers = 566996.86
> KB/sec
> Parent sees throughput for 4 initial writers = 566831.18 KB/sec
> Min throughput per process = 141741.52 KB/sec
> Max throughput per process = 141764.00 KB/sec
> Avg throughput per process = 141749.21 KB/sec
> Min xfer = 10482688.00 KB
>
> Children see throughput for 4 rewriters = 432868.28
> KB/sec
> Parent sees throughput for 4 rewriters = 420648.01 KB/sec
> Min throughput per process = 108115.68 KB/sec
> Max throughput per process = 108383.86 KB/sec
> Avg throughput per process = 108217.07 KB/sec
> Min xfer = 10460160.00 KB
>
>
>
> iozone test complete.
> ----------------
>
> 3. With your recommended conf added in smb.conf, this is testparm result:
> [root at localhost ~]# testparm
> Load smb config files from /etc/samba/smb.conf
> rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)
> Processing section "[homes]"
> Processing section "[printers]"
> Processing section "[cifs]"
> Processing section "[raw]"
> Processing section "[gvol]"
> Loaded services file OK.
> Server role: ROLE_STANDALONE
> Press enter to see a dump of your service definitions
>
> [global]
> workgroup = MYGROUP
> server string = DCS Samba Server
> log file = /var/log/samba/log.vfs
> max log size = 500000
> max protocol = SMB2
> max xmit = 262144
> socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=262144
> SO_SNDBUF=262144
> stat cache = No
> kernel oplocks = No
> idmap config * : backend = tdb
> aio read size = 262144
> aio write size = 262144
> aio write behind = true
> write cache size = 268435456
> cups options = raw
> ……
>
> [cifs]
> path = /mnt/fuse
> read only = No
> guest ok = Yes
>
> [raw]
> path = /dcsdata/d0
> read only = No
> guest ok = Yes
>
> [gvol]
> comment = For samba export of volume test
> path = /
> read only = No
> guest ok = Yes
> vfs objects = glusterfs
> glusterfs:volume = soul
> glusterfs:volfile_server = localhost
>
> the iozone test result with cmd: iozone -s 10G -r 1m -i0 -t 4
> -------------
> Run began: Thu Aug 22 15:47:31 2013
>
> File size set to 10485760 KB
> Record Size 1024 KB
> Command line used: iozone -s 10G -r 1m -i0 -t 4
> Output is in Kbytes/sec
> Time Resolution = -0.000000 seconds.
> Processor cache size set to 1024 Kbytes.
> Processor cache line size set to 32 bytes.
> File stride size set to 17 * record size.
> Throughput test with 4 processes
> Each process writes a 10485760 Kbyte file in 1024 Kbyte records
>
> Children see throughput for 4 initial writers = 135588.82
> KB/sec
> Parent sees throughput for 4 initial writers = 135549.95 KB/sec
> Min throughput per process = 33895.92 KB/sec
> Max throughput per process = 33900.02 KB/sec
> Avg throughput per process = 33897.20 KB/sec
> Min xfer = 10484736.00 KB
>
> Children see throughput for 4 rewriters = 397494.38
> KB/sec
> Parent sees throughput for 4 rewriters = 387431.63 KB/sec
> Min throughput per process = 99280.98 KB/sec
> Max throughput per process = 99538.40 KB/sec
> Avg throughput per process = 99373.59 KB/sec
> Min xfer = 10459136.00 KB
> --------------
>
>
>
>
> 在 2013-8-22，下午3:31，RAGHAVENDRA TALUR <raghavendra.talur at gmail.com> 写道：
>
> Hi Kane,
>
> 1. Which version of samba are you running?
>
> 2. Can you re-run the test after adding the following lines to
> smb.conf's global section and tell if it helps?
> kernel oplocks = no
> stat cache = no
>
> Thanks,
> Raghavendra Talur
>
>
> On Wed, Aug 21, 2013 at 3:48 PM, kane <stef_9k at 163.com> wrote:
>
>> Hi Lala, thank you for reply this issue.
>>
>> this is our smb.conf:
>> --------
>> [global]
>> workgroup = MYGROUP
>> server string = DCS Samba Server
>> log file = /var/log/samba/log.vfs
>> max log size = 500000
>> # log level = 10
>> # max xmit = 65535
>> # getwd cache = yes
>> # use sendfile = yes
>> # strict sync = no
>> # sync always = no
>> # large readwrite = yes
>> aio read size = 262144
>> aio write size = 262144
>> aio write behind = true
>> # min receivefile size = 262144
>> write cache size = 268435456
>> # oplocks = yes
>> security = user
>> passdb backend = tdbsam
>> load printers = yes
>> cups options = raw
>> read raw = yes
>> write raw = yes
>> max xmit = 262144
>> read size = 262144
>> socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=262144
>> SO_SNDBUF=262144
>> max protocol = SMB2
>>
>> [homes]
>> comment = Home Directories
>> browseable = no
>> writable = yes
>>
>>
>> [printers]
>> comment = All Printers
>> path = /var/spool/samba
>> browseable = no
>> guest ok = no
>> writable = no
>> printable = yes
>>
>> [cifs]
>> path = /mnt/fuse
>> guest ok = yes
>> writable = yes
>>
>> [raw]
>> path = /dcsdata/d0
>> guest ok = yes
>> writable = yes
>>
>> [gvol]
>> comment = For samba export of volume test
>> vfs objects = glusterfs
>> glusterfs:volfile_server = localhost
>> glusterfs:volume = soul
>> path = /
>> read only = no
>> guest ok = yes
>> --------
>>
>> our win 7 client hardware:
>> Intel® Xeon® E31230 @ 3.20GHz
>> 8GB RAM
>>
>> linux client hardware:
>> Intel(R) Xeon(R) CPU X3430 @ 2.40GHz
>> 16GB RAM
>>
>> pretty thanks
>>
>> -kane
>>
>> 在 2013-8-21，下午4:53，Lalatendu Mohanty <lmohanty at redhat.com> 写道：
>>
>> On 08/21/2013 01:32 PM, kane wrote:
>>
>> Hello：
>>
>> We have used glusterfs3.4 with the lasted samba-glusterfs-vfs lib to
>> test samba performance in windows client.
>>
>> two glusterfs server nodes export share with name of "gvol":
>> hardwares:
>> brick use a raid 5 logic disk with 8 * 2T SATA HDDs
>> 10G network connection
>>
>> one linux client mount the "gvol" with cmd:
>> [root at localhost current]# mount.cifs //192.168.100.133/gvol /mnt/vfs -o
>> user=kane,pass=123456
>>
>> then i use iozone to test the write performance in mount dir "/mnt/vfs":
>> [root at localhost current]# ./iozone -s 10G -r 128k -i0 -t 4
>> …..
>> File size set to 10485760 KB
>> Record Size 128 KB
>> Command line used: ./iozone -s 10G -r 128k -i0 -t 4
>> Output is in Kbytes/sec
>> Time Resolution = 0.000001 seconds.
>> Processor cache size set to 1024 Kbytes.
>> Processor cache line size set to 32 bytes.
>> File stride size set to 17 * record size.
>> Throughput test with 4 processes
>> Each process writes a 10485760 Kbyte file in 128 Kbyte records
>>
>> Children see throughput for 4 initial writers = 487376.67 KB/sec
>> Parent sees throughput for 4 initial writers = 486184.67 KB/sec
>> Min throughput per process = 121699.91 KB/sec
>> Max throughput per process = 122005.73 KB/sec
>> Avg throughput per process = 121844.17 KB/sec
>> Min xfer = 10459520.00 KB
>>
>> Children see throughput for 4 rewriters = 491416.41 KB/sec
>> Parent sees throughput for 4 rewriters = 490298.11 KB/sec
>> Min throughput per process = 122808.87 KB/sec
>> Max throughput per process = 122937.74 KB/sec
>> Avg throughput per process = 122854.10 KB/sec
>> Min xfer = 10474880.00 KB
>>
>> linux client mount with cifs , write performance reach 480MB/s per
>> client;
>>
>> but when i use win7 client mount the "gvol" with cmd:
>> net use Z: \\192.168.100.133\gvol 123456 /user:kane
>>
>> then also use iozone test in dir Z, even with write block 1Mbyte :
>> File size set to 10485760 KB
>> Record Size 1024 KB
>> Command line used: iozone -s 10G -r 1m -i0 -t 4
>> Output is in Kbytes/sec
>> Time Resolution = -0.000000 seconds.
>> Processor cache size set to 1024 Kbytes.
>> Processor cache line size set to 32 bytes.
>> File stride size set to 17 * record size.
>> Throughput test with 4 processes
>> Each process writes a 10485760 Kbyte file in 1024 Kbyte records
>>
>> Children see throughput for 4 initial writers = 148164.82
>> KB/sec
>> Parent sees throughput for 4 initial writers = 148015.48
>> KB/sec
>> Min throughput per process = 37039.91
>> KB/sec
>> Max throughput per process = 37044.45
>> KB/sec
>> Avg throughput per process = 37041.21
>> KB/sec
>> Min xfer = 10484736.00 KB
>>
>> Children see throughput for 4 rewriters = 147642.12
>> KB/sec
>> Parent sees throughput for 4 rewriters = 147472.16
>> KB/sec
>> Min throughput per process = 36909.13
>> KB/sec
>> Max throughput per process = 36913.29
>> KB/sec
>> Avg throughput per process = 36910.53
>> KB/sec
>> Min xfer = 10484736.00 KB
>>
>> iozone test complete.
>>
>> then reach 140MB/s
>>
>> so , anyone meet with this problem.Is there win7 clinet to reconfigure
>> to perform well?
>>
>> Thanks!
>>
>> kane
>> ----------------------------------------------------------------
>> Email: kai.zhou at soulinfo.com
>> 电话: 0510-85385788-616
>>
>>
>>
>> Hi kane,
>>
>> I do run IOs using win7 client with glusterfs3.4 , but I never compared
>> the performance with Linux cifs mount. I don't think we need to do any
>> special configuration on Windows side. I hope your Linux and Windows client
>> have similar configuration i.e. RAM, cache, disk type etc. However I am
>> curious to know if your setup uses the vfs plug-in correctly. We can
>> confirm that looking at smb.conf entry for the gluster volume which should
>> have been created by "gluster start command" automatically .
>>
>> e.g: entry in smb.conf for one of volume "smbvol" of mine looks like below
>>
>> [gluster-smbvol]
>> comment = For samba share of volume smbvol
>> vfs objects = glusterfs
>> glusterfs:volume = smbvol
>> path = /
>> read only = no
>> guest ok = yes
>>
>> Kindly copy the entries in smb.conf for your gluster volume in this
>> email.
>> -Lala
>>
>>
>> _______________________________________________
>> Gluster-users mailing listGluster-users at gluster.orghttp://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>>
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>
>
>
> --
> *Raghavendra Talur *
>
>
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130822/c300139f/attachment.html>
More information about the Gluster-users
mailing list