[Gluster-devel] md-cache changes and impact on tiering

[Dan Lambright]

----- Original Message -----
> From: "Poornima Gurusiddaiah" <pgurusid at redhat.com>
> To: "Dan Lambright" <dlambrig at redhat.com>, "Nithya Balachandran" <nbalacha at redhat.com>
> Cc: "Gluster Devel" <gluster-devel at gluster.org>
> Sent: Tuesday, August 23, 2016 12:56:38 AM
> Subject: md-cache changes and impact on tiering
> 
> Hi,
> 
> The basic patches for md-cache and integrating it with cache-invalidation is
> merged in master. You could try master build and enable the following
> settings, to see if there is any impact on tiering performance at all:
> 
> # gluster volume set <volname> performance.stat-prefetch on
> # gluster volume set <volname> features.cache-invalidation on
> # gluster volume set <volname> performance.cache-samba-metadata on
> # gluster volume set <volname> performance.md-cache-timeout 600
> # gluster volume set <volname> features.cache-invalidation-timeout 600

On the tests I run, this cut the number of LOOKUPs by about three orders of magnitude. Each saved lookup reduces a round trip over the network.

I'm running a "small file" performance test. It creates 16K 64 byte files in a seven level directory. It then reads each file twice. 

Configuration is HOT: 2 x 2 ramdisk COLD: 2 x (8 + 4) disk, network is 10000Mb/s 9000 mtu. The number of lookups is a factor of the number of directories and subvolumes. On each I/O the file is re-opened and each directory is laboriously rechecked for existence/permission. 

Without using md-cache, these lookups used to be further propagated across each subvolume by DHT to obtain the entire layout. So it would be something like order of 16K*7*26 round trips across the network. 

The counts are all visible with gluster profile. 


> 
> Note: It has to be executed in the same order.
> 
> Tracker bug: https://bugzilla.redhat.com/show_bug.cgi?id=1211863
> Patches:
> http://review.gluster.org/#/q/status:open+project:glusterfs+branch:master+topic:bug-1211863
> 
> Thanks,
> Poornima
>