[Gluster-devel] Fwd: Gluster - Performance issue while copying bulk files/folders

[Srikanth Mampilakal]

Hi,


I have production gluster file service used as a shared storage where the
content management system uses it as document root. I have run in to a
performance issue with the gluster/fuse client.

Looking for your thoughts and experience in resolving Gluster performance
issues:

Gluster Infrastructure

Gluster version :GlusterFS 3.7.6

2 gluster nodes of the same config below

Redhat EL7.0-64
Memory : 4GB
Processor : 2 x 2.0 Ghz
Network : 100 Mbps
File Storage Volume : NETAPP Storage LUN with 2.0 IOPS/GB

Gluster Volume information:

[root at GlusterFileServe1 ~]# gluster volume info

Volume Name: prodcmsroot
Type: Replicate
Volume ID: f1284bf0-1939-46f9-a672-a7716e362947
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: Server1:/glusterfs/brick1/prodcmsroot
Brick2: Server2:/glusterfs/brick1/prodcmsroot
Options Reconfigured:
performance.io-thread-count: 64
performance.cache-size: 1073741824
performance.readdir-ahead: on
performance.write-behind-window-size: 524288

[root at GlusterFileServe1  ~]#

The replication between Gluster node are quick and consistent.

The apache webservers are accessing the Gluster volume using native gluster
fuse client and located in the same VLAN as the Gluster Server.

GlusterFileServe1:/prodcmsroot  /mnt/glusterfs glusterfs
direct-io-mode=disable,defaults,_netdev 0 0

The server utilization (memory,cpu,network and disk 1/0) is relatively low

I am experiencing very slow performance while copying multiple file/folders
(approx 75 MB) and it takes atleast approx 35 min. Even copy a folder (with
multiple files/subfolders) within the Gluster volume take the same time.

However, if I do dd to check the copy speed, I get the below result.

[root at ClientServer ~]#  time sh -c "dd if=/dev/zero
of=/mnt/testmount/test.tmp bs=4k count=20000 && sync"
20000+0 records in
20000+0 records out
81920000 bytes (82 MB) copied, 17.1357 s, 4.8 MB/s

real    0m17.337s
user    0m0.031s
sys     0m0.317s


Anyone experience the same kind of performance issue, please let me know
your thoughts.

Cheers
Srikanth
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20151209/0f1b4336/attachment.html>