[Gluster-devel] Memory usage of glusterfsd increases when directories are created

[Chaloulos, Klearchos (Nokia - GR/Athens)]

Hello,

We have found that the memory usage of glusterfsd is increasing when directories are created. Environment details can seen in the attached environment.txt file, and the translator graph can be seen in the attached file translator_graph.txt. In summary we have a setup of two gluster bricks, using replicate mode, and one third gluster peer for quorum. We use glusterfs version 3.6.9. We build glusterfs directly from source code.

The test script creates many directories on the /mnt/export volume from the storage client side. It creates directories of the path "/mnt/export/dir.XXXXX", so all directories are under the same folder. In each directory a file of a random size between 1024 and 11204 is created.
The number of directories created varied from 10000 to 200000 for different runs. Before each run all directories were deleted.

A graph of the memory usage can be seen here: http://imgur.com/z8LOskQ

In the X-axis the number of directories is seen. The number varies because before each run all directories are deleted. So first 80000 directories were created, then they were deleted and 160000 were created then deleted and 200000 were created, then deleted and 40000 were created etc. Due to the amount of data the resolution of the X-axis is not fine enough to show the exact variation, but the general idea is that thousands of directories were created and deleted.

The Y-axis shows the RSS memory usage of the glusterfsd process of this volume on the two storage nodes (SN-0 and SN-1), as measured by 'pidstat -urdh -p <pid>'.
We can see that the memory usage is continually increasing.


1)     Is this increase normal expected? I can imagine that glusterfsd would allocated more memory if more files or directories are created, but is there an upper limit, or will it increase forever?

2)     Can we control the memory allocation by any configuration parameters?

3)     Is there a particular translator that controls this memory allocation?

We did another test, where only files were created. In this case the memory usage reached an upper limit. Graph is available here: http://imgur.com/zDtkbml

So if only files are created there is an upper limit in memory allocation. Is the case of directory creation different and why? Or is this a memory leak? Has anyone else seen this behavior?

Best regards,
Klearchos
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20160511/b80161fb/attachment-0001.html>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: environment.txt
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20160511/b80161fb/attachment-0002.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: translator_graph.txt
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20160511/b80161fb/attachment-0003.txt>