[Gluster-users] low samba performance with glusterfs backend

[nuaa_liuben]

Hello folks,

We test samba performance with local ext4 and glusterfs backends, it shows performance is very different.
The samba server has 4 1Gbps NICs and bond with mode 6, backend storage is raid0 with 12 SAS disks.
A LUN is created over all disks, make as EXT4 file system, and used as glusterfs brick.

On the samba server, use dd test local ext4 and glusterfs, write bandwidth are 477MB/s and 357MB/s, details as follow.
When mount cifs on Centos 6.0, write bandwidth downgrade to 184MB/s and 117MB/s, very different.
Using 4 win7 clients to test with SANergy/Iometer, cifs over ext4 aggregate throughput can get about 391 MB/s.
However, cifs over glusterfs aggregate throughput only can get about 180 MB/s.

Why CIFS performance is so slow over glusterfs backend? 
Anybody meet such issues and how to resolve(samba and glusterfs parameters)? 
thanks a lot in advance.

#test local ext4
[root at pana53 5f4554cf-3b56-43e5-847e-cb409b0edc30]# dd if=/dev/zero of=dd.dat bs=1MB count=10k
10240+0 records in
10240+0 records out
10240000000 bytes (10 GB) copied, 21.4764 s, 477 MB/s

#test local glusterfs
[root at pana53 61659a17-69bd-4704-81fe-0853f1c891cd]# dd if=/dev/zero of=dd.dat bs=1MB count=10k       
10240+0 records in
10240+0 records out
10240000000 bytes (10 GB) copied, 28.7148 s, 357 MB/s

#test cifs over ext4
[root at pana53 ext4-share]# dd if=/dev/zero of=dd.dat bs=1MB count=10k
10240+0 records in
10240+0 records out
10240000000 bytes (10 GB) copied, 55.5549 s, 184 MB/s

#test cifs over glusterfs
[root at pana53 glusterfs-share]# dd if=/dev/zero of=dd.dat bs=1MB count=10k
10240+0 records in
10240+0 records out
10240000000 bytes (10 GB) copied, 87.4899 s, 117 MB/s


here is smb.conf:
[global]
        server string = samba server
        map to guest = Bad Password
        log file = /var/log/samba/log.%m
        max log size = 50
        max protocol = SMB2
        socket options = TCP_NODELAY IPTOS_LOWDELAY
        idmap config * : backend = tdb
        aio read size = 16384
        aio write size = 16384
        use sendfile = Yes
        posix locking = No

[ext4-share]
        comment = None
        path = /data/5f4554cf-3b56-43e5-847e-cb409b0edc30
        read only = No
        guest ok = Yes

[glusterfs-share]
        comment = None
        path = /reexport/61659a17-69bd-4704-81fe-0853f1c891cd
        read only = No
        guest ok = Yes 

here is gluster volume info:
[root at pana53 glusterfs-share]# gluster volume info 
Volume Name: vol1
Type: Distribute
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: pana53:/data/5f4554cf-3b56-43e5-847e-cb409b0edc30
Options Reconfigured:
auth.allow: *,192.168.1.*
features.quota: on
nfs.disable: on

BR,
liuben



nuaa_liuben at sina.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20121013/507da560/attachment.html>