[Gluster-users] GlusterFS as webroot

[Björn Puttmann]

Dear gluster-users,

I am having issues using gluster as filesystem for some of our webservers. For machines with a moderate load, the system works fine
and response times of the webservers is ok.
But one site has a bit more load (ca. 7000 hits/sec) and when using gluserfs as webroot on this machine, the server will spike
to a load of > 100 (Server has 8GB Ram, 4 Cores Xeon X5460  @3.16GHz) and eventually becomes unresponsive.
The webserver runs CentOS-6.6 x86_64, apache-2.4 and php-5.6. We used a NetApp NFS mount as webroot before without problems.

We are running glusterfs-server-3.5.1 on a zfs filesystem. The cluster consists of two HP DL 160G5 machine with eight Xeon E5462 @2.80GHz 
and 24GB RAM.

What we can see on the gluster servers is that after some time the kswapd process will spike up to 100% CPU usage and will than
stay at a high percentage. A restart via /etc/init.d/glusterd will solve this for some time after which it will eventually spike again.

Has anybody here some experience with using glusterfs on zfs as fileserver for webroots? If so, has anyone an idea as to why we see the kswapd spiking?

Any help/input would be greatly appreciated ;)

We tweaked zfs a bit by adding:
options zfs zfs_arc_min=8589934592 zfs_arc_max=19327352832 zfs_prefetch_disable=1 zfs_txg_timeout=5
to /etc/modprobe.d/zfs.conf

The config of the zfs tank is:
NAME                  PROPERTY              VALUE                  SOURCE
tank/customer/webroot  type                  filesystem             -
tank/customer/webroot  creation              Fri Sep 12  9:34 2014  -
tank/customer/webroot  used                  70.1G                  -
tank/customer/webroot  available             29.9G                  -
tank/customer/webroot  referenced            65.7G                  -
tank/customer/webroot  compressratio         1.13x                  -
tank/customer/webroot  mounted               yes                    -
tank/customer/webroot  quota                 100G                   local
tank/customer/webroot  reservation           none                   default
tank/customer/webroot  recordsize            128K                   default
tank/customer/webroot  mountpoint            /tank/buhv/SmartSuite  default
tank/customer/webroot  sharenfs              off                    default
tank/customer/webroot  checksum              on                     default
tank/customer/webroot  compression           on                     local
tank/customer/webroot  atime                 off                    local
tank/customer/webroot  devices               on                     default
tank/customer/webroot  exec                  on                     default
tank/customer/webroot  setuid                on                     default
tank/customer/webroot  readonly              off                    default
tank/customer/webroot  zoned                 off                    default
tank/customer/webroot  snapdir               hidden                 default
tank/customer/webroot  aclinherit            restricted             default
tank/customer/webroot  canmount              on                     default
tank/customer/webroot  xattr                 sa                     inherited from tank
tank/customer/webroot  copies                1                      default
tank/customer/webroot  version               5                      -
tank/customer/webroot  utf8only              off                    -
tank/customer/webroot  normalization         none                   -
tank/customer/webroot  casesensitivity       sensitive              -
tank/customer/webroot  vscan                 off                    default
tank/customer/webroot  nbmand                off                    default
tank/customer/webroot  sharesmb              off                    default
tank/customer/webroot  refquota              none                   default
tank/customer/webroot  refreservation        none                   default
tank/customer/webroot  primarycache          all                    default
tank/customer/webroot  secondarycache        all                    default
tank/customer/webroot  usedbysnapshots       4.37G                  -
tank/customer/webroot  usedbydataset         65.7G                  -
tank/customer/webroot  usedbychildren        0                      -
tank/customer/webroot  usedbyrefreservation  0                      -
tank/customer/webroot  logbias               latency                default
tank/customer/webroot  dedup                 off                    default
tank/customer/webroot  mlslabel              none                   default
tank/customer/webroot  sync                  disabled               inherited from tank
tank/customer/webroot  refcompressratio      1.11x                  -
tank/customer/webroot  written               76.1M                  -
tank/customer/webroot  logicalused           72.8G                  -
tank/customer/webroot  logicalreferenced     68.7G                  -
tank/customer/webroot  snapdev               hidden                 default
tank/customer/webroot  acltype               off                    default
tank/customer/webroot  context               none                   default
tank/customer/webroot  fscontext             none                   default
tank/customer/webroot  defcontext            none                   default
tank/customer/webroot  rootcontext           none                   default
tank/customer/webroot  relatime              off                    default

The config of the gluster volume:
Volume Name: webroot
Type: Replicate
Volume ID: bb7e80cd-a945-4c22-a3ec-5840cf442cd6
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: gluster-01:/tank/customer/webroot/brick
Brick2: gluster-02:/tank/customer/webroot/brick
Options Reconfigured:
nfs.rpc-auth-allow: xx.xx.xx.xx
auth.allow: xx.xx.xx.xx
nfs.enable-ino32: on

Thanks and take care,
Björn
--
dbap GmbH - one-stop e-business
phone +49 251 609979-0 / fax +49 251 609979-99
Martin-Luther-King Weg 42-44 - 48155 Muenster, Germany

http://www.dbap.de


dbap GmbH, Sitz: Münster 
HRB 5891, Amtsgericht Münster 
Geschäftsführer: Björn Dolkemeier, Stefan Goebel