[Gluster-users] Gluster NFS performance issue upgrading from 3.2.5 to 3.2.6/3.3.0

[Simon Detheridge]

Hi,

I have a situation where I'm mounting a gluster volume on several web servers via NFS. The web servers run Rails applications off the gluster NFS mounts. The whole thing is running on EC2.

On 3.2.5, starting a Rails application on the web server was sluggish but acceptable. However, after upgrading to 3.2.6 the length of time taken to start a Rails application has increased by over 10 times, to something that's not really suitable for a production environment. The situation still occurs with 3.3.0 as well.

If I attach strace to the rails process as it's starting up, I see that it's looking for a very large number of nonexistent files. I think this is something that Rails does that can't be helped - it checks to see if a file is there for many things, and does something accordingly if it does.

Has something changed that could negatively affect the length of time it takes to stat a nonexistent file over a NFS mount to a gluster volume, between 3.2.5 and 3.2.6? Is there any way I can get the old behaviour without downgrading?

-- I don't currently have proof that it's the nonexistent files that's causing the problem, but it seems highly likely as performance for the other tasks that the servers carry out appears unaffected.

Sorry this is slightly vague. I can run some more tests/benchmarks to try and figure out what's going on in more detail, but thought I would ask here first in case this is related to a known issue.

Thanks,
Simon

-- 
Simon Detheridge - CTO, Widgit Software
26 Queen Street, Cubbington, CV32 7NA - Tel: +44 (0)1926 333680