[Gluster-users] Gluster - Performance issue while copying bulk files/folders

[Anuradha Talur]

Response inline.

----- Original Message -----
> From: "Srikanth Mampilakal" <shrikanth1980 at gmail.com>
> To: gluster-users at gluster.org
> Sent: Thursday, December 10, 2015 7:59:04 PM
> Subject: Re: [Gluster-users] Gluster - Performance issue while copying bulk	files/folders
> 
> 
> 
> Hi members,
> 
> Really appreciate if you can share your thoughts or any feedback for
> resolving the slow copy issue
> 
> Regards
> Srikanth
> On 10-Dec-2015 2:12 AM, "Srikanth Mampilakal" < srikanth.mampilakal at gmail.com
> > wrote:
> 
> 
> 
> Hi,
> 
> 
> I have production gluster file service used as a shared storage where the
> content management system uses it as document root. I have run in to a
> performance issue with the gluster/fuse client.
> 
> Looking for your thoughts and experience in resolving Gluster performance
> issues:
> 
> Gluster Infrastructure
> 
> Gluster version :GlusterFS 3.7.6
> 
> 2 gluster nodes of the same config below
> 
> Redhat EL7.0-64
> Memory : 4GB
> Processor : 2 x 2.0 Ghz
> Network : 100 Mbps
> File Storage Volume : NETAPP Storage LUN with 2.0 IOPS/GB
> 
> Gluster Volume information:
> 
> [root at GlusterFileServe1 ~]# gluster volume info
> 
> Volume Name: prodcmsroot
> Type: Replicate
> Volume ID: f1284bf0-1939-46f9-a672-a7716e362947
> Status: Started
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: Server1:/glusterfs/brick1/prodcmsroot
> Brick2: Server2:/glusterfs/brick1/prodcmsroot
> Options Reconfigured:
> performance.io-thread-count: 64
> performance.cache-size: 1073741824
> performance.readdir-ahead: on
> performance.write-behind-window-size: 524288
> 
> [root at GlusterFileServe1 ~]#
> 
> The replication between Gluster node are quick and consistent.
> 
> The apache webservers are accessing the Gluster volume using native gluster
> fuse client and located in the same VLAN as the Gluster Server.
> 
> GlusterFileServe1:/prodcmsroot /mnt/glusterfs glusterfs
> direct-io-mode=disable,defaults,_netdev 0 0
> 
> The server utilization (memory,cpu,network and disk 1/0) is relatively low
> 
> I am experiencing very slow performance while copying multiple file/folders
> (approx 75 MB) and it takes atleast approx 35 min. Even copy a folder (with
> multiple files/subfolders) within the Gluster volume take the same time.
> 
> However, if I do dd to check the copy speed, I get the below result.
> 
> [root at ClientServer ~]# time sh -c "dd if=/dev/zero of=/mnt/testmount/test.tmp
> bs=4k count=20000 && sync"
> 20000+0 records in
> 20000+0 records out
> 81920000 bytes (82 MB) copied, 17.1357 s, 4.8 MB/s
> 
> real 0m17.337s
> user 0m0.031s
> sys 0m0.317s
> 
> 
> Anyone experience the same kind of performance issue, please let me know your
> thoughts.
> 
Hi Srikanth,
 
Could you please provide the following information so that the reason behind
slow copy can be deduced?

1) Profile info of the volume when you copy dirs/files into glusterfs.
2) Profile info of the volume when you copy dirs/files within glusterfs.

The following steps should help you with profile info:
1) gluster volume profile <VOLNAME> start
2) Perform copy operations
3) gluster volume profile <VOLNAME> info (you will get stats of the FOPs at this point)
4) gluster volume profile <VOLNAME> stop

Please follow steps 1 through 4 twice. Once for copy into glusterfs and once for copy
within.

> Cheers
> Srikanth
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

-- 
Thanks,
Anuradha.