[Gluster-users] Gluster memory useage

Sun May 11 19:59:17 UTC 2014

Hello All,

I'm having some trouble with the memory allocation of my glusterfsd processes. The server keeps running out of memory, no matter how much I add (it's a vm) and I just don't understand how gluster allocates memory to its processes.

The situation is as follows: rhel 6.4, 16 GB ram, gluster 3.4.2-1
I have one big lun of 20 TB that contains all my bricks. I started with 4 GB memory, but saw it was getting full, so I stepped it up to 8GB and later 16. The weird thing is VMware claims only 50% of the memory is actively used so I guess this is all due to caching?
I tried the flushcaches thing, but I don't see to free up much.

If I check my memory consumption I see this:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
26800 root      20   0 2593m 1.9g 1088 S 17.3 12.5   5261:43 glusterfsd
26914 root      20   0 2541m 1.9g 1072 S 15.9 12.2   1081:04 glusterfsd
33299 root      20   0 2938m 1.9g 1056 S 15.9 12.1   1787:40 glusterfsd
26872 root      20   0 2093m 1.3g 1080 S 15.6  8.6   3391:02 glusterfsd
26995 root      20   0 1828m 1.3g 1072 S  8.0  8.1 782:29.85 glusterfsd
26934 root      20   0 1891m 1.0g  376 S  0.0  6.6 795:44.19 glusterfsd
37651 root      20   0 1309m 1.0g  332 S  0.0  6.6   0:03.52 glusterfs
27015 root      20   0 1374m 780m  388 S  0.0  4.9 580:29.19 glusterfsd
29712 root      20   0  989m 504m 1076 S 20.2  3.2 227:47.87 glusterfsd
8635 root      20   0 2415m 186m  392 S  0.0  1.2 594:02.47 glusterfsd
31859 root      20   0  643m 174m  392 S  0.0  1.1  99:06.60 glusterfsd
27308 root      20   0  440m 169m 1296 S 10.0  1.1 646:30.02 glusterfs
27458 root      20   0  415m 149m 1280 S 12.3  0.9 781:36.04 glusterfs
27358 root      20   0  420m 143m 1272 S 10.3  0.9 553:04.97 glusterfs
7453 root      20   0  609m 133m  872 S  0.0  0.8  27:49.83 glusterfsd
27508 root      20   0  410m 117m 1272 S  9.6  0.7 560:42.51 glusterfs
8608 root      20   0 2418m 111m  372 S  0.0  0.7 785:53.31 glusterfsd
27408 root      20   0  373m  96m 1272 S  9.6  0.6 431:32.28 glusterfs
27558 root      20   0  368m  87m  344 S  0.0  0.6 318:37.92 glusterfs
40715 root      20   0  203m  86m 1036 S  0.3  0.5   3:41.78 puppetd
30940 root      20   0  674m  78m  960 S  0.0  0.5  40:18.17 glusterfsd
19363 root      20   0 1437m  74m  728 S  0.0  0.5  60:12.02 glusterfsd
8773 root      20   0  685m  71m  956 S  0.0  0.4  19:40.05 glusterfsd
  789 root      20   0  625m  67m  900 S  0.0  0.4  22:45.64 glusterfsd
1599 root      20   0 1291m  41m  956 S  0.0  0.3  13:03.23 glusterfsd
27808 root      20   0  482m  36m  344 S  0.0  0.2  42:40.67 glusterfs
47625 root      20   0  677m  31m  900 S  0.0  0.2   4:09.29 glusterfsd
30539 root      20   0 1381m  27m  852 S  0.0  0.2  20:31.90 glusterfsd
Etc...

Right now I'm at:
             total       used       free     shared    buffers     cached
Mem:      16326684   16176692     149992          0      14284      59520
-/+ buffers/cache:   16102888     223796
Swap:      4194296    4194252         44

... the lack of free swap worries me a lot

Is it normal for some volumes to consume that much memory? How does gluster decide how much memory it should allocate? It there any way to influence this?
>From time to time bricks switch themselves offline. Is this due to the lack of free memory?

Any help to shed some light on this situation would be most welcome as I have no idea where to start.....
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140511/c7d42a30/attachment.html>