<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"><html>
<head>
<meta name="Generator" content="Zarafa WebAccess v7.1.14-51822">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>RE: [Gluster-users] Extremely slow du</title>
<style type="text/css">
body
{
font-family: Arial, Verdana, Sans-Serif ! important;
font-size: 12px;
padding: 5px 5px 5px 5px;
margin: 0px;
border-style: none;
background-color: #ffffff;
}
p, ul, li
{
margin-top: 0px;
margin-bottom: 0px;
}
</style>
</head>
<body>
<p>Hi,</p><p> </p><p> </p><p>I also noticed disappearing of files with the combination of certain settings. If you use cluster.readdir-optimize but not some of the other settings, they don't disappear.</p><p> </p><p>Unfortunately can't remember which setting was conflicting...</p><p> </p><p> </p><p>Performance wise I don't see difference between 3.10 and 3.11 over here. Didn't test recently with 3.8.</p><p><br /><br />Regards</p><p>Jo</p><p> </p><p><br /> </p><blockquote style="border-left: 2px solid #325FBA; padding-left: 5px;margin-left:5px;">-----Original message-----<br /><strong>From:</strong> Vijay Bellur <vbellur@redhat.com><br /><strong>Sent:</strong> Tue 11-07-2017 17:22<br /><strong>Subject:</strong> Re: [Gluster-users] Extremely slow du<br /><strong>To:</strong> mohammad kashif <kashif.alig@gmail.com>; Raghavendra Gowdappa <rgowdapp@redhat.com>; Poornima Gurusiddaiah <pgurusid@redhat.com>; <br /><strong>CC:</strong> gluster-users Discussion List <Gluster-users@gluster.org>; <br />Hi Kashif,<br /><br />Thank you for your feedback! Do you have some data on the nature of <br />performance improvement observed with 3.11 in the new setup?<br /><br />Adding Raghavendra and Poornima for validation of configuration and help <br />with identifying why certain files disappeared from the mount point <br />after enabling readdir-optimize.<br /><br />Regards,<br />Vijay<br /><br />On 07/11/2017 11:06 AM, mohammad kashif wrote:<br />> Hi Vijay and Experts<br />><br />> I didn't want to experiment with my production setup so started a<br />> parallel system with two server and around 80TB storage. First<br />> configured with gluster 3.8 and had the same lookup performance issue.<br />> Then upgraded to 3.11 as you suggested and it made huge improvement in<br />> lookup time. I also did some more optimization as suggested in other<br />> threads.<br />> Now I am going to update my production server. I am planning to use<br />> following optimization option, it would be very useful if you can point<br />> out any inconsistency or suggest some other options. My production setup<br />> has 5 servers consisting of 400TB storage and around 80 million files<br />> of varying lengths.<br />><br />> Options Reconfigured:<br />> server.event-threads: 4<br />> client.event-threads: 4<br />> cluster.lookup-optimize: on<br />> cluster.readdir-optimize: off<br />> performance.client-io-threads: on<br />> performance.cache-size: 1GB<br />> performance.parallel-readdir: on<br />> performance.md-cache-timeout: 600<br />> performance.cache-invalidation: on<br />> performance.stat-prefetch: on<br />> features.cache-invalidation-timeout: 600<br />> features.cache-invalidation: on<br />> nfs.disable: on<br />> performance.readdir-ahead: on<br />> transport.address-family: inet<br />> auth.allow: 163.1.136.*<br />> diagnostics.latency-measurement: on<br />> diagnostics.count-fop-hits: on<br />><br />> I found that setting cluster.readdir-optimize to 'on' made some files<br />> disappear from client !<br />><br />> Thanks<br />><br />> Kashif<br />><br />><br />><br />> On Sun, Jun 18, 2017 at 4:57 PM, Vijay Bellur <vbellur@redhat.com<br />> <mailto:vbellur@redhat.com>> wrote:<br />><br />> Hi Mohammad,<br />><br />> A lot of time is being spent in addressing metadata calls as<br />> expected. Can you consider testing out with 3.11 with md-cache [1]<br />> and readdirp [2] improvements?<br />><br />> Adding Poornima and Raghavendra who worked on these enhancements to<br />> help out further.<br />><br />> Thanks,<br />> Vijay<br />><br />> [1] https://gluster.readthedocs.io/en/latest/release-notes/3.9.0/<br />> <https://gluster.readthedocs.io/en/latest/release-notes/3.9.0/><br />><br />> [2] https://github.com/gluster/glusterfs/issues/166<br />> <https://github.com/gluster/glusterfs/issues/166><br />><br />> On Fri, Jun 16, 2017 at 2:49 PM, mohammad kashif<br />> <kashif.alig@gmail.com <mailto:kashif.alig@gmail.com>> wrote:<br />><br />> Hi Vijay<br />><br />> Did you manage to look into the gluster profile logs ?<br />><br />> Thanks<br />><br />> Kashif<br />><br />> On Mon, Jun 12, 2017 at 11:40 AM, mohammad kashif<br />> <kashif.alig@gmail.com <mailto:kashif.alig@gmail.com>> wrote:<br />><br />> Hi Vijay<br />><br />> I have enabled client profiling and used this script<br />> https://github.com/bengland2/gluster-profile-analysis/blob/master/gvp-client.sh<br />> <https://github.com/bengland2/gluster-profile-analysis/blob/master/gvp-client.sh><br />> to extract data. I am attaching output files. I don't have<br />> any reference data to compare with my output. Hopefully you<br />> can make some sense out of it.<br />><br />> On Sat, Jun 10, 2017 at 10:47 AM, Vijay Bellur<br />> <vbellur@redhat.com <mailto:vbellur@redhat.com>> wrote:<br />><br />> Would it be possible for you to turn on client profiling<br />> and then run du? Instructions for turning on client<br />> profiling can be found at [1]. Providing the client<br />> profile information can help us figure out where the<br />> latency could be stemming from.<br />><br />> Regards,<br />> Vijay<br />><br />> [1] https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Performance%20Testing/#client-side-profiling<br />> <https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Performance%20Testing/#client-side-profiling><br />><br />> On Fri, Jun 9, 2017 at 7:22 PM, mohammad kashif<br />> <kashif.alig@gmail.com <mailto:kashif.alig@gmail.com>><br />> wrote:<br />><br />> Hi Vijay<br />><br />> Thanks for your quick response. I am using gluster<br />> 3.8.11 on Centos 7 servers<br />> glusterfs-3.8.11-1.el7.x86_64<br />><br />> clients are centos 6 but I tested with a centos 7<br />> client as well and results didn't change<br />><br />> gluster volume info Volume Name: atlasglust<br />> Type: Distribute<br />> Volume ID: fbf0ebb8-deab-4388-9d8a-f722618a624b<br />> Status: Started<br />> Snapshot Count: 0<br />> Number of Bricks: 5<br />> Transport-type: tcp<br />> Bricks:<br />> Brick1: pplxgluster01.x.y.z:/glusteratlas/brick001/gv0<br />> Brick2: pplxgluster02..x.y.z:/glusteratlas/brick002/gv0<br />> Brick3: pplxgluster03.x.y.z:/glusteratlas/brick003/gv0<br />> Brick4: pplxgluster04.x.y.z:/glusteratlas/brick004/gv0<br />> Brick5: pplxgluster05.x.y.z:/glusteratlas/brick005/gv0<br />> Options Reconfigured:<br />> nfs.disable: on<br />> performance.readdir-ahead: on<br />> transport.address-family: inet<br />> auth.allow: x.y.z<br />><br />> I am not using directory quota.<br />><br />> Please let me know if you require some more info<br />><br />> Thanks<br />><br />> Kashif<br />><br />><br />><br />> On Fri, Jun 9, 2017 at 2:34 PM, Vijay Bellur<br />> <vbellur@redhat.com <mailto:vbellur@redhat.com>> wrote:<br />><br />> Can you please provide more details about your<br />> volume configuration and the version of gluster<br />> that you are using?<br />><br />> Regards,<br />> Vijay<br />><br />> On Fri, Jun 9, 2017 at 5:35 PM, mohammad kashif<br />> <kashif.alig@gmail.com<br />> <mailto:kashif.alig@gmail.com>> wrote:<br />><br />> Hi<br />><br />> I have just moved our 400 TB HPC storage<br />> from lustre to gluster. It is part of a<br />> research institute and users have very small<br />> files to big files ( few KB to 20GB) . Our<br />> setup consists of 5 servers, each with 96TB<br />> RAID 6 disks. All servers are connected<br />> through 10G ethernet but not all clients.<br />> Gluster volumes are distributed without any<br />> replication. There are approximately 80<br />> million files in file system.<br />> I am mounting using glusterfs on clients.<br />><br />> I have copied everything from lustre to<br />> gluster but old file system exist so I can<br />> compare.<br />><br />> The problem, I am facing is extremely slow<br />> du on even a small directory. Also the time<br />> taken is substantially different each time.<br />> I tried du from same client on a particular<br />> directory twice and got these results.<br />><br />> time du -sh /data/aa/bb/cc<br />> 3.7G /data/aa/bb/cc<br />> real 7m29.243s<br />> user 0m1.448s<br />> sys 0m7.067s<br />><br />> time du -sh /data/aa/bb/cc<br />> 3.7G /data/aa/bb/cc<br />> real 16m43.735s<br />> user 0m1.097s<br />> sys 0m5.802s<br />><br />> 16m and 7m is too long for a 3.7 G<br />> directory. I must mention that the directory<br />> contains huge number of files (208736)<br />><br />> but running du on same directory on old data<br />> gives this result<br />><br />> time du -sh /olddata/aa/bb/cc<br />> 4.0G /olddata/aa/bb/cc<br />> real 3m1.255s<br />> user 0m0.755s<br />> sys 0m38.099s<br />><br />> much better if I run same command again<br />><br />> time du -sh /olddata/aa/bb/cc<br />> 4.0G /olddata/aa/bb/cc<br />> real 0m8.309s<br />> user 0m0.313s<br />> sys 0m7.755s<br />><br />> Is there anything I can do to improve this<br />> performance? I would also like hear from<br />> some one who is running same kind of setup.<br />><br />> Thanks<br />><br />> Kashif<br />><br />><br />><br />> _______________________________________________<br />> Gluster-users mailing list<br />> Gluster-users@gluster.org<br />> <mailto:Gluster-users@gluster.org><br />> http://lists.gluster.org/mailman/listinfo/gluster-users<br />> <http://lists.gluster.org/mailman/listinfo/gluster-users><br />><br />><br />><br />><br />><br />><br />><br />><br /><br />_______________________________________________<br />Gluster-users mailing list<br />Gluster-users@gluster.org<br />http://lists.gluster.org/mailman/listinfo/gluster-users<br /></blockquote>
</body>
</html>