[Gluster-users] Cache performance degrades over time

[Matt]

Hello list,

I have about a half a dozen nginx servers sitting in front of Gluster 
(3.4.6, I know it's old) serving a mix videos and images. It's a 
moderate amount of traffic, each of two geo-repped sites will do 2-4 
gigs/second throughout the day.

Here's the problem. Reads from gluster, due to the way nginx buffers 
the video, can far exceed what's being served out to the internet. 
Serving 1 gig of video may read 3 gigs from Gluster.

I can fix this by setting the performance cache on the volume to a 
pretty large size, right now it's at 2 gigs. This works great, gluster 
uses 1.5 - 2 gigs of RAM and the in/out bandwidth on the nginx machines 
becomes a healthy 1:1 or better.

For a few days. Over time, as the machines vfs cache fills, gluster 
starts to use less RAM, and that ratio gets worse. Rebooting the nginx 
boxes (or I presume, simply dropping their caches) fixes it immediately.

I'm going to try increasing vfs.cache_pressure on the nginx boxes, as 
this doc recommends:

https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Linux%20Kernel%20Tuning/#vmvfs95cache95pressure

Does that make sense to tune this on the clients? Is Gluster competing 
with the kernel cache? That's sort of my understanding but I can't find 
a clear explanation.

Other recommendations would be welcome, though tweaking the direct-io 
options is unfortunately not an option in my setup.

-Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20161018/5f30cc76/attachment.html>