[Gluster-users] Dubious write performance on simple "nfs" setup

[Peter Gervai]

Hello,

Simple setup: one server one client. Client is using 2.6.26 (debian)
kernel but with the gluster-provided fuse module. (Without it the
performance is non-existant.)

# server confy
volume stor
  type storage/posix
  option directory /srv/glusterfs/
end-volume

volume locks
  type features/posix-locks
  option mandatory-locks on
  subvolumes stor
end-volume

volume readahead
  type performance/read-ahead
  option page-count 2 # 4                       # def 2
  option force-atime-update off         # def off
  subvolumes locks
end-volume

volume cache
  type performance/io-cache
  option cache-size 128MB                       # default 32MB
  option page-size 512KB                        # def 128KB
  option cache-timeout 2                # def 1
  subvolumes readahead
end-volume

volume threads
  type performance/io-threads
  option thread-count 16                # default 16
  subvolumes cache
end-volume

-----------------------------------------
# gluster client confy
volume remote
  type protocol/client
  option transport-type tcp
  option remote-host 1.1.1.1
  option remote-subvolume brick
end-volume

volume cache
  type performance/io-cache
  option cache-size 1GB
  option page-size 128KB
  subvolumes remote
end-volume

volume threads
  type performance/io-threads
#   option thread-count 16                # default 16
  subvolumes cache
end-volume

volume writebehind
  type performance/write-behind
  option flush-behind on    # default is 'off', let's live dangerously
 subvolumes threads
end-volume
-----------------------

bonnie++ say what reality backs:

Version 1.03d       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
gluster-shlevin  4G 66262  90  3437   0  3778   0 35682  49 55111   3 383.6   0
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  1165   2  3695   2  1621   2  1109   1  3787   2  1583   2

Basically sequential block output is extremely slow, and tiobench
helped it to narrow to block sizes below 64k. Below 64k performance is
around 2-3MB/s, above is the normal 60-65MB/s.

Dropping writebehind gave a performance "boost"... 2-3MB/s went up to
10-15MB/s, while, of course, putc performance went down to around
30MB/s.

However I cannot seem to be able to raise block performance below 64k
(especially around 4k) higher than 2-3MB/s (or 9-10MB/s without WB);
it basically doesn't change if I try to remove other translators.

CPU and network load seems to be low on both sides.

Local fuse test gives 70+MB/s for any IO.

Ideas? (Maybe more fuse tweaks? kernel variables?)

Thanks,
Peter