[Gluster-users] Performance optimization tips Gluster 3.3? (small files / directory listings)

[Brian Candler]

On Thu, Jun 07, 2012 at 10:10:03AM +0000, Fernando Frediani (Qube) wrote:
>    Sorry this reply won’t be of any help to your problem, but I am too
>    curious to understand how it can be even slower if monting using
>    Gluster client which I would expect always be quicker than NFS or
>    anything else.

(1) Try it with "ls -aR" or "find ." instead of "ls -alR"

(2) Try it on a gluster non-replicated volume (for fair comparison with
    direct NFS access)

With a replicated volume, many accesses involve sending queries to *both*
servers to check they are in sync - even read accesses.  This in turn can
cause disk seeks on both machines, so the latency you'll get is the larger
of the two.  If you are doing lots of accesses sequentially then the
latencies will all add up.

A stat() is one of those accesses which touches both machines, and "ls -l"
forces a stat() of each file found.

In fact, a quick test suggests ls -l does stat, lstat, getxattr and
lgetxattr:

$ ls -laR . >/dev/null 2>ert; cut -f1 -d'(' ert | sort | uniq -c
     13 access
      1 arch_prctl
      5 brk
    395 close
      4 connect
      1 execve
      1 exit_group
      2 fcntl
    391 fstat
      3 futex
    702 getdents
      1 getrlimit
   1719 getxattr
      3 ioctl
   1721 lgetxattr
      9 lseek
   1721 lstat
     58 mmap
     24 mprotect
     12 munmap
    424 open
     19 read
      2 readlink
      2 rt_sigaction
      1 rt_sigprocmask
      1 set_robust_list
      1 set_tid_address
      4 socket
   1719 stat
      1 statfs
     29 write

Looking at the detail in the strace output, I see these are actually

    lstat(<target-file>, ...)
    lgetxattr(<target-file>, "security.selinux", ...)
    getxattr(<target-file>, "system.posix_acl_access", ...)
    stat("/etc/localtime", ...)

Compare without -l:

$ strace ls -aR . >/dev/null 2>ert; cut -f1 -d'(' ert | sort | uniq -c
      9 access
      1 arch_prctl
      4 brk
    377 close
      1 execve
      1 exit_group
      1 fcntl
    376 fstat
      3 futex
    702 getdents
      1 getrlimit
      3 ioctl
     39 mmap
     16 mprotect
      4 munmap
    388 open
     11 read
      2 rt_sigaction
      1 rt_sigprocmask
      1 set_robust_list
      1 set_tid_address
      1 stat
      1 statfs
      9 write

Regards,

Brian.