[Gluster-users] maxing out cpu

[Dave Hall]

Hi all,

I have been fighting with Gluster on and off over the last month or so.
I am ready to wave the white flag!

Environment:

* Host: 64bit KVM virtual machines 4 virtual CPUs and 4G RAM
  - Phyiscal Hosts are Dell R900 4 Dual Core Xeons & 32G RAM

* Distro: Ubuntu 9.10 (amd64)

* Gluster Version: 2.0.9 self compiled debs

* Storage: ~250G (small fries)

I initially tried using the stock Ubuntu 2.0.2 debs, but they had the
same problem as described below.

Each of the 3 client nodes are web heads.  The gluster server is also
running on each of the nodes, mounting an ext4 fs and exporting it.  Our
plan was to load balance across the 3 nodes using ha-proxy, with
memcached sessions and have uploads and config changes replicated across
the nodes using GlusterFS.

Everything looks great on paper - except when one of the nodes decided
to max a CPU.  The trigger for the node freaking out seems to be quite
random.  Some times it is when running "find /path/to/gluster-mount
-type f" other times it can be when the server is idling.  I think the
largest file we have on the file system is a few hundred Kb.

I am not sure what information you need, but here the config files,
which I hope will help.

$ cat /etc/glusterfs/glusterfsd.vol 
volume posix
  type storage/posix
  option directory /srv/glusterfs/export
end-volume

volume locks
  type features/locks
  subvolumes posix
end-volume

volume brick
  type performance/io-threads
  option thread-count 4
  subvolumes locks
end-volume

volume server
  type protocol/server
  option transport-type tcp
  option transport.socket.bind-address 192.168.XXX.123
  option auth.addr.brick.allow 192.168.XXX.122,192.168.XXX.123,192.168.XXX.124
  option auth.addr.locks.allow 192.168.XXX.*
  option auth.addr.posix.allow 192.168.XXX.*
  subvolumes brick
end-volume


$ cat /etc/glusterfs/glusterfs.vol 
# Note the 3 node is missing because I removed it earlier today
volume my-storage-c
  type protocol/client
  option transport-type tcp
  option remote-host 192.168.201.123
  option remote-subvolume brick
end-volume

volume my-storage-d
  type protocol/client
  option transport-type tcp
  option remote-host 192.168.201.124
  option remote-subvolume brick
end-volume

volume replication
  type cluster/replicate
  subvolumes my-storage-c my-storage-d
  option read-subvolume my-storage-c
end-volume

volume writebehind
  type performance/write-behind
  option window-size 1MB
  subvolumes replication
end-volume

volume cache
  type performance/io-cache
  option cache-size 256MB
  subvolumes writebehind
end-volume

selective content of /etc/fstab

/dev/mapper/my--ui--c-glusterfs--export /srv/glusterfs/export ext4     noatime,nodev,nosuid 0       2
/etc/glusterfs/glusterfs.vol  /path/to/mount  glusterfs  defaults  0  0

If you need more information let me know and I'll try to supply it.  I
welcome any suggestions/assistance.

Cheers


Dave