[Gluster-users] gfs vs nfs: abysmal performance difference

Tue May 26 13:43:12 UTC 2009

Paolo,
  Do you have a core dump on your system (in /)? Can you get us a backtrace from it?

Thanks,
Avati

----- Original Message -----
From: "Paolo Pisati" <paolo.pisati at tomato.it>
To: "Paolo Pisati" <paolo.pisati at contactlab.com>
Cc: "Anand Avati" <avati at gluster.com>, "Raghavendra G" <raghavendra at gluster.com>, gluster-users at gluster.org
Sent: Tuesday, May 26, 2009 6:11:53 PM GMT +05:30 Chennai, Kolkata, Mumbai, New Delhi
Subject: Re: [Gluster-users] gfs vs nfs: abysmal performance difference

Paolo Pisati wrote:
> Anand Avati wrote:
>   
>> Can you post the client and server logs of this session?
>>     
> i'm doing a parallel build test now (make -j16), later i'll try bonnie
> again.
>
> where does gluster put its logs?
>   
never mind, i found the user guide:

client:

 > sudo bonnie -d /mnt/gfs -m GFS
File '/mnt/gfs/Bonnie.94754', size: 104857600
Writing with putc()...done
Rewriting...done

Bonnie: drastic I/O error (/mnt/gfs/Bonnie.94754): Input/output error

/var/log/messages:
May 26 13:33:25 faustoweb sudo:     piso : TTY=ttyp1 ; PWD=/usr/src ; 
USER=root ; COMMAND=/usr/local/gfs/sbin/glusterfs -f 
/usr/local/gfs/etc/glusterfs/glusterfs.vol -l /var/log/gfs.log -L 
WARNING /mnt/gfs/
May 26 13:33:53 faustoweb sudo:     piso : TTY=ttyp1 ; PWD=/usr/src ; 
USER=root ; COMMAND=/usr/local/bin/bonnie -d /mnt/gfs -m GFS
May 26 13:37:30 faustoweb kernel: pid 94748 (glusterfsd), uid 0: exited 
on signal 11 (core dumped)

/var/log/gfs.log:
================================================================================
Version      : glusterfs 2.0.0 built on May 11 2009 09:41:53
TLA Revision : 7b2e459db65edd302aa12476bc73b3b7a17b1410
Starting Time: 2009-05-26 13:33:25
Command line : /usr/local/gfs/sbin/glusterfs -f 
/usr/local/gfs/etc/glusterfs/glusterfs.vol -l /var/log/gfs.log -L 
WARNING /mnt/gfs/
PID          : 94747
System name  : FreeBSD
Nodename     : faustoweb.tomato.lan
Kernel Release : 7.2-RELEASE
Hardware Identifier: amd64

Given volfile:
+------------------------------------------------------------------------------+
  1: volume client
  2:   type protocol/client
  3:   option transport-type tcp
  4:   option remote-host 10.0.0.1         # IP address of the remote brick
  5:   option remote-subvolume brick        # name of the remote volume
  6: end-volume
  7:
  8: ### Add readahead feature
  9: volume readahead
 10:   type performance/read-ahead
 11:   option page-size 1MB     # unit in bytes
 12:   option page-count 2       # cache per file  = (page-count x 
page-size)
 13:   subvolumes client
 14: end-volume
 15:
 16: ### Add IO-Cache feature
 17: volume iocache
 18:   type performance/io-cache
 19:   option page-size 256KB
 20:   option page-count 2
 21:   subvolumes readahead
 22: end-volume
 23:
 24: ### Add writeback feature
 25: volume writeback
 26:   type performance/write-behind
 27:   option aggregate-size 1MB
 28:   option window-size 2MB
 29:   option flush-behind off
 30:   subvolumes iocache  
 31: end-volume

+------------------------------------------------------------------------------+
2009-05-26 13:33:25 W [xlator.c:555:validate_xlator_volume_options] 
writeback: option 'window-size' is deprecated, preferred is 
'cache-size', continuing with correction
2009-05-26 13:33:25 W [glusterfsd.c:455:_log_if_option_is_invalid] 
writeback: option 'aggregate-size' is not recognized
2009-05-26 13:33:25 W [glusterfsd.c:455:_log_if_option_is_invalid] 
iocache: option 'page-count' is not recognized
2009-05-26 13:33:25 W [glusterfsd.c:455:_log_if_option_is_invalid] 
readahead: option 'page-size' is not recognized
pending frames:
frame : type(1) op(CREATE)

patchset: 7b2e459db65edd302aa12476bc73b3b7a17b1410
signal received: 11
configuration details:db.h 1
dlfcn 1
libpthread 1
spinlock 1
extattr.h 1
st_atimespec.tv_nsec 1
package-string: glusterfs 2.0.0

server:

/var/log/messages:
May 26 12:32:19 gnode sudo:     piso : TTY=ttyp0 ; PWD=/usr/home/piso ; 
USER=root ; COMMAND=/usr/local/gfs/sbin/glusterfsd -f 
/usr/local/gfs/etc/glusterfs/glusterfsd.vol -l /var/log/gfsd.log -L WARNING

/var/log/gfsd.log:
================================================================================
Version      : glusterfs 2.0.0 built on May 11 2009 14:38:00
TLA Revision : 7b2e459db65edd302aa12476bc73b3b7a17b1410
Starting Time: 2009-05-26 12:32:19
Command line : /usr/local/gfs/sbin/glusterfsd -f 
/usr/local/gfs/etc/glusterfs/glusterfsd.vol -l /var/log/gfsd.log -L WARNING
PID          : 2552
System name  : FreeBSD
Nodename     : gnode.tomato.lan
Kernel Release : 7.2-RELEASE
Hardware Identifier: i386

Given volfile:
+------------------------------------------------------------------------------+
  1: volume posix
  2:   type storage/posix                   # POSIX FS translator
  3:   option directory /usr/src                # Export this directory
  4: end-volume
  5:
  6: volume locks
  7:   type features/locks
  8:   option mandatory-locks on
  9:   subvolumes posix
 10: end-volume
 11:
 12: volume brick
 13:   type performance/io-threads
 14:   option thread-count 8
 15:   subvolumes locks
 16: end-volume
 17:
 18: volume server
 19:   type protocol/server
 20:   option transport-type tcp
 21:   subvolumes brick
 22:   option auth.addr.brick.allow * # Allow access to "brick" volume
 23: end-volume

+------------------------------------------------------------------------------+
2009-05-26 12:32:19 W [xlator.c:287:_volume_option_value_validate] 
brick: '8' in 'option thread-count 8' is out of range [16 - 256]

-- 

Paolo Pisati
Reparto Sistemi
---------
ContactLab
Via Natale Battaglia, 10
20127 Milano (Italy)
Tel. +39 02 28 31 181
Fax +39 02 70 03 02 69.
http://www.contactlab.com