[Gluster-users] Trouble with AFR

Steve steeeeeveee at gmx.net
Mon Mar 30 16:49:09 UTC 2009


> Shortest sequence of steps to reproduce?
> 
Two systems in my case:
1: hermes
2: tisiphone


On both systems I do:
# /etc/init.d/glusterfs-client.vu-hosting stop
# /etc/init.d/glusterfs-server.vu-hosting stop
# /etc/init.d/fuse stop
# umount /mnt/glusterfs/vu-hosting/
# mkfs.xfs -f -l version=2 -i attr=2 -L vu-hosting /dev/LVM_RAID_VolGroup01/vu-hosting
# mount /mnt/glusterfs/vu-hosting/
# setfattr -x trusted.glusterfs.version /mnt/glusterfs/vu-hosting
# setfattr -x trusted.glusterfs.createtime /mnt/glusterfs/vu-hosting
# /etc/init.d/glusterfs-server.vu-hosting start
# /etc/init.d/glusterfs-client.vu-hosting start


Then on one of the systems I do:
# mkdir -pv /var/www/vu-hosting/localhost/{admin/ssl,htdocs,log,temp}
# cp -v /etc/ssl/nginx/nginx.* /var/www/vu-hosting/localhost/admin/ssl
# md5sum --binary /var/www/vu-hosting/localhost/admin/ssl/*


The result is:


hermes:
----
hermes ~ # /etc/init.d/glusterfs-client.vu-hosting stop ; /etc/init.d/glusterfs-server.vu-hosting stop ; /etc/init.d/fuse stop ; umount /mnt/glusterfs/vu-hosting/ ; mkfs.xfs -f -l version=2 -i attr=2 -L vu-hosting /dev/LVM_RAID_VolGroup01/vu-hosting ; mount /mnt/glusterfs/vu-hosting/ ; setfattr -x trusted.glusterfs.version /mnt/glusterfs/vu-hosting ; setfattr -x trusted.glusterfs.createtime /mnt/glusterfs/vu-hosting ; /etc/init.d/glusterfs-server.vu-hosting start ; /etc/init.d/glusterfs-client.vu-hosting start
* Stopping glusterfs-client.vu-hosting...                           [ ok ]
* Stopping glusterfs-server.vu-hosting...                           [ ok ]
* Stopping fuse...                                                  [ ok ]
meta-data=/dev/LVM_RAID_VolGroup01/vu-hosting isize=256    agcount=4, agsize=327680 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=1310720, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                   extsz=4096   blocks=0, rtextents=0
setfattr: /mnt/glusterfs/vu-hosting: No such attribute
setfattr: /mnt/glusterfs/vu-hosting: No such attribute
* Starting glusterfs-server.vu-hosting...                           [ ok ]
* Starting fuse...                                                  [ ok ]
* Starting glusterfs-client.vu-hosting...                           [ ok ]
hermes ~ #
----


tisiphone:
----
tisiphone ~ # /etc/init.d/glusterfs-client.vu-hosting stop ; /etc/init.d/glusterfs-server.vu-hosting stop ; /etc/init.d/fuse stop ; umount /mnt/glusterfs/vu-hosting/ ; mkfs.xfs -f -l version=2 -i attr=2 -L vu-hosting /dev/LVM_RAID_VolGroup01/vu-hosting ; mount /mnt/glusterfs/vu-hosting/ ; setfattr -x trusted.glusterfs.version /mnt/glusterfs/vu-hosting ; setfattr -x trusted.glusterfs.createtime /mnt/glusterfs/vu-hosting ; /etc/init.d/glusterfs-server.vu-hosting start ; /etc/init.d/glusterfs-client.vu-hosting start
* Stopping glusterfs-client.vu-hosting...                           [ ok ]
* Stopping glusterfs-server.vu-hosting...                           [ ok ]
* Stopping fuse...                                                  [ ok ]
meta-data=/dev/LVM_RAID_VolGroup01/vu-hosting isize=256    agcount=4, agsize=327680 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=1310720, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=0
realtime =none                   extsz=4096   blocks=0, rtextents=0
setfattr: /mnt/glusterfs/vu-hosting: No such attribute
setfattr: /mnt/glusterfs/vu-hosting: No such attribute
* Starting glusterfs-server.vu-hosting...                           [ ok ]
* Starting fuse...                                                  [ ok ]
* Starting glusterfs-client.vu-hosting...                           [ ok ]
tisiphone ~ #
----


Doing the copy and check job on hermes:
----
hermes ~ # mkdir -pv /var/www/vu-hosting/localhost/{admin/ssl,htdocs,log,temp} ; cp -v /etc/ssl/nginx/nginx.* /var/www/vu-hosting/localhost/admin/ssl ; echo "Source:" ; md5sum --binary /etc/ssl/nginx/nginx.* ; echo "GlusterFS:" ; md5sum --binary /var/www/vu-hosting/localhost/admin/ssl/*
mkdir: created directory `/var/www/vu-hosting/localhost'
mkdir: created directory `/var/www/vu-hosting/localhost/admin'
mkdir: created directory `/var/www/vu-hosting/localhost/admin/ssl'
mkdir: created directory `/var/www/vu-hosting/localhost/htdocs'
mkdir: created directory `/var/www/vu-hosting/localhost/log'
mkdir: created directory `/var/www/vu-hosting/localhost/temp'
`/etc/ssl/nginx/nginx.crt' -> `/var/www/vu-hosting/localhost/admin/ssl/nginx.crt'
`/etc/ssl/nginx/nginx.csr' -> `/var/www/vu-hosting/localhost/admin/ssl/nginx.csr'
`/etc/ssl/nginx/nginx.key' -> `/var/www/vu-hosting/localhost/admin/ssl/nginx.key'
`/etc/ssl/nginx/nginx.pem' -> `/var/www/vu-hosting/localhost/admin/ssl/nginx.pem'
Source:
c80162aa9220f8942fd11137f03243cb */etc/ssl/nginx/nginx.crt
330a48882caf1960180183d7e3d237c8 */etc/ssl/nginx/nginx.csr
20169d74a3493b2941c99213ea24705b */etc/ssl/nginx/nginx.key
c92bad4bdfde4df52db7afe91977247d */etc/ssl/nginx/nginx.pem
GlusterFS:
c80162aa9220f8942fd11137f03243cb */var/www/vu-hosting/localhost/admin/ssl/nginx.crt
330a48882caf1960180183d7e3d237c8 */var/www/vu-hosting/localhost/admin/ssl/nginx.csr
20169d74a3493b2941c99213ea24705b */var/www/vu-hosting/localhost/admin/ssl/nginx.key
c92bad4bdfde4df52db7afe91977247d */var/www/vu-hosting/localhost/admin/ssl/nginx.pem
hermes ~ #
----
==> All fine on hermes.


Doing the check job on tisiphone:
----
tisiphone ~ # md5sum --binary /var/www/vu-hosting/localhost/admin/ssl/*
c80162aa9220f8942fd11137f03243cb */var/www/vu-hosting/localhost/admin/ssl/nginx.crt
330a48882caf1960180183d7e3d237c8 */var/www/vu-hosting/localhost/admin/ssl/nginx.csr
20169d74a3493b2941c99213ea24705b */var/www/vu-hosting/localhost/admin/ssl/nginx.key
d41d8cd98f00b204e9800998ecf8427e */var/www/vu-hosting/localhost/admin/ssl/nginx.pem
tisiphone ~ #
----
==> The pem file is wrong!


Redoing the check on hermes:
----
hermes ~ # md5sum --binary /var/www/vu-hosting/localhost/admin/ssl/*
c80162aa9220f8942fd11137f03243cb */var/www/vu-hosting/localhost/admin/ssl/nginx.crt
330a48882caf1960180183d7e3d237c8 */var/www/vu-hosting/localhost/admin/ssl/nginx.csr
20169d74a3493b2941c99213ea24705b */var/www/vu-hosting/localhost/admin/ssl/nginx.key
c92bad4bdfde4df52db7afe91977247d */var/www/vu-hosting/localhost/admin/ssl/nginx.pem
hermes ~ #
----


Listing on hermes:
----
hermes ~ # ls -lah /var/www/vu-hosting/localhost/admin/ssl/
total 16K
drwxr-xr-x 2 root root   70 Mar 30 18:27 .
drwxr-xr-x 3 root root   16 Mar 30 18:27 ..
-r--r--r-- 1 root root 1017 Mar 30 18:27 nginx.crt
-r--r--r-- 1 root root  737 Mar 30 18:27 nginx.csr
-r-------- 1 root root  887 Mar 30 18:27 nginx.key
-r-------- 1 root root 1.9K Mar 30 18:27 nginx.pem
hermes ~ #
----


Listing on tisiphone:
----
tisiphone ~ # ls -lah /var/www/vu-hosting/localhost/admin/ssl/
total 16K
drwxr-xr-x 2 root root   70 Mar 30 18:27 .
drwxr-xr-x 3 root root   16 Mar 30 18:27 ..
-r--r--r-- 1 root root 1017 Mar 30 18:27 nginx.crt
-r--r--r-- 1 root root  737 Mar 30 18:27 nginx.csr
-r-------- 1 root root  887 Mar 30 18:27 nginx.key
-r-------- 1 root root 1.9K Mar 30 18:27 nginx.pem
tisiphone ~ #
----
==> WHAT? This is new! I never had the whole file there in my tests. Now the file is there and not having 0 bytes size. Let me check MD5 sum.


Recheck on tisiphone:
----
tisiphone ~ # md5sum --binary /var/www/vu-hosting/localhost/admin/ssl/*
c80162aa9220f8942fd11137f03243cb */var/www/vu-hosting/localhost/admin/ssl/nginx.crt
330a48882caf1960180183d7e3d237c8 */var/www/vu-hosting/localhost/admin/ssl/nginx.csr
20169d74a3493b2941c99213ea24705b */var/www/vu-hosting/localhost/admin/ssl/nginx.key
d41d8cd98f00b204e9800998ecf8427e */var/www/vu-hosting/localhost/admin/ssl/nginx.pem
tisiphone ~ #
----
==> Okay. At least this is consistent. The pem file is not the same on both nodes. The size is this time not 0 bytes but the content is not the same. At least if I can trust md5sum.


It's always the pem file which get's destroyed. I don't know why? Another case where I loose data is when I name the file "default_vhost.include". The file it self is normal ASCII but with a reference to the pem file. The effect is the same. I loose the data. The file is 0 bytes long after doing the copy to GlusterFS.


The log in NORMAL mode:


hermes (server):
----
================================================================================
Version      : glusterfs 2.0.0git built on Mar 30 2009 05:38:42
TLA Revision : git://git.sv.gnu.org/gluster.git
Starting Time: 2009-03-30 18:25:31
Command line : /usr/sbin/glusterfsd -N -f /etc/glusterfs/glusterfs-server.vu-hosting.vol -l /var/log/glusterfs/glusterfs-server.vu-hosting.log -L NORMAL
PID          : 6265
System name  : Linux
Nodename     : hermes
Kernel Release : 2.6.29
Hardware Identifier: i686

Given volfile:
+------------------------------------------------------------------------------+
  1: #######################################################
  2: ###  GlusterFS Server Volume Specification          ###
  3: #######################################################
  4:
  5: volume gfs-ds-posix
  6:   type storage/posix
  7:   option directory /mnt/glusterfs/vu-hosting
  8:   # option o-direct off
  9:   # option export-statfs-size yes
 10:   # option mandate-attribute yes
 11:   # option span-devices 1
 12: end-volume
 13:
 14: volume gfs-ds-locks
 15:   type features/locks
 16:   option mandatory-locks off
 17:   subvolumes gfs-ds-posix
 18: end-volume
 19:
 20: volume gfs-ds-io-threads
 21:   type performance/io-threads
 22:   option thread-count 8
 23:   subvolumes gfs-ds-locks
 24: end-volume
 25:
 26: volume gfs-ds-server
 27:   type protocol/server
 28:   option transport-type tcp
 29:   option transport.socket.listen-port 6997
 30:   option auth.addr.gfs-ds-locks.allow 192.168.0.*,127.0.0.1
 31:   option auth.addr.gfs-ds-io-threads.allow 192.168.0.*,127.0.0.1
 32:   subvolumes gfs-ds-io-threads
 33: end-volume

+------------------------------------------------------------------------------+
2009-03-30 18:25:31 N [glusterfsd.c:1134:main] glusterfs: Successfully started
2009-03-30 18:25:31 N [server-protocol.c:7513:mop_setvolume] gfs-ds-server: accepted client from 192.168.0.70:1021
2009-03-30 18:25:31 N [server-protocol.c:7513:mop_setvolume] gfs-ds-server: accepted client from 192.168.0.70:1020
2009-03-30 18:25:33 N [server-protocol.c:7513:mop_setvolume] gfs-ds-server: accepted client from 192.168.0.75:1021
2009-03-30 18:25:33 N [server-protocol.c:7513:mop_setvolume] gfs-ds-server: accepted client from 192.168.0.75:1020
2009-03-30 18:25:37 N [server-protocol.c:8268:notify] gfs-ds-server: 192.168.0.75:1021 disconnected
2009-03-30 18:25:37 N [server-protocol.c:8268:notify] gfs-ds-server: 192.168.0.75:1020 disconnected
2009-03-30 18:25:37 N [server-helpers.c:530:server_connection_destroy] gfs-ds-server: destroyed connection of tisiphone-5818-2009/03/30-18:24:46:612808-gfs-ds-vu-hosting-hermes
2009-03-30 18:25:38 N [server-protocol.c:7513:mop_setvolume] gfs-ds-server: accepted client from 192.168.0.75:1019
2009-03-30 18:25:38 N [server-protocol.c:7513:mop_setvolume] gfs-ds-server: accepted client from 192.168.0.75:1018
2009-03-30 18:29:00 N [server-protocol.c:8268:notify] gfs-ds-server: 192.168.0.75:1019 disconnected
2009-03-30 18:29:00 N [server-protocol.c:8268:notify] gfs-ds-server: 192.168.0.75:1018 disconnected
2009-03-30 18:29:00 N [server-helpers.c:530:server_connection_destroy] gfs-ds-server: destroyed connection of tisiphone-5934-2009/03/30-18:25:41:340960-gfs-ds-vu-hosting-hermes
2009-03-30 18:29:00 N [server-protocol.c:7513:mop_setvolume] gfs-ds-server: accepted client from 192.168.0.75:1023
2009-03-30 18:29:00 N [server-protocol.c:7513:mop_setvolume] gfs-ds-server: accepted client from 192.168.0.75:1022
----


hermes (client):
----
================================================================================
Version      : glusterfs 2.0.0git built on Mar 30 2009 05:38:42
TLA Revision : git://git.sv.gnu.org/gluster.git
Starting Time: 2009-03-30 18:25:31
Command line : /usr/sbin/glusterfsd -N -f /etc/glusterfs/glusterfs-client.vu-hosting.vol -l /var/log/glusterfs/glusterfs-client.vu-hosting.log -L NORMAL /var/www/vu-hosting
PID          : 6300
System name  : Linux
Nodename     : hermes
Kernel Release : 2.6.29
Hardware Identifier: i686

Given volfile:
+------------------------------------------------------------------------------+
  1: #######################################################
  2: ###  GlusterFS Client Volume Specification          ###
  3: #######################################################
  4:
  5: #volume gfs-ds-fuse
  6: #  type mount/fuse
  7: #  option direct-io-mode on
  8: #  option macfuse-local off
  9: #  option mount-point /var/www/vu-hosting
 10: #  option attribute-timeout 0.0
 11: #  option entry-timeout 0.0
 12: #end-volume
 13:
 14: volume gfs-ds-vu-hosting-hermes
 15:   type protocol/client
 16:   option transport-type tcp
 17:   # option username
 18:   # option password
 19:   option remote-host 192.168.0.70
 20:   option remote-port 6997
 21:   option remote-subvolume gfs-ds-io-threads
 22:   option transport-timeout 10
 23:   option ping-timeout 5
 24: end-volume
 25:
 26: volume gfs-ds-vu-hosting-tisiphone
 27:   type protocol/client
 28:   option transport-type tcp
 29:   # option username
 30:   # option password
 31:   option remote-host 192.168.0.75
 32:   option remote-port 6997
 33:   option remote-subvolume gfs-ds-io-threads
 34:   option transport-timeout 10
 35:   option ping-timeout 5
 36: end-volume
 37:
 38: volume gfs-ds-replicate
 39:   type cluster/replicate
 40:   option data-self-heal on
 41:   option metadata-self-heal on
 42:   option entry-self-heal on
 43:   # option read-subvolume
 44:   # option favorite-child
 45:   option data-change-log on
 46:   option metadata-change-log on
 47:   option entry-change-log on
 48:   option data-lock-server-count 1
 49:   option metadata-lock-server-count 1
 50:   option entry-lock-server-count 1
 51:   subvolumes gfs-ds-vu-hosting-hermes gfs-ds-vu-hosting-tisiphone
 52: end-volume
 53:
 54: volume gfs-ds-io-threads
 55:   type performance/io-threads
 56:   option thread-count 8
 57:   subvolumes gfs-ds-replicate
 58: end-volume
 59:
 60: volume gfs-ds-write-behind
 61:   type performance/write-behind
 62:   option block-size 1MB
 63:   option cache-size 4MB
 64:   option flush-behind on
 65:   # opiton disable-for-first-nbytes 1
 66:   # option enable-O_SYNC false
 67:   subvolumes gfs-ds-io-threads
 68: end-volume
 69:
 70: #volume gfs-ds-read-ahead
 71: #  type performance/read-ahead
 72: #  option page-size 65KB
 73: #  option page-count 16
 74: #  option force-atime-update off
 75: #  subvolumes gfs-ds-write-behind
 76: #end-volume
 77:
 78: volume gfs-ds-io-cache
 79:   type performance/io-cache
 80:   option page-size 4KB
 81:   option cache-size 64MB
 82:   option priority *:0
 83:   option cache-timeout 1
 84:   # subvolumes gfs-ds-read-ahead
 85:   subvolumes gfs-ds-write-behind
 86: end-volume

+------------------------------------------------------------------------------+
2009-03-30 18:25:31 E [xlator.c:201:_volume_option_value_validate] gfs-ds-io-cache: '4096' in 'option page-size 4KB' is out of range [16384 - 4194304]
2009-03-30 18:25:31 N [glusterfsd.c:1134:main] glusterfs: Successfully started
2009-03-30 18:25:31 N [client-protocol.c:6169:client_setvolume_cbk] gfs-ds-vu-hosting-tisiphone: connection and handshake succeeded
2009-03-30 18:25:31 N [client-protocol.c:6169:client_setvolume_cbk] gfs-ds-vu-hosting-tisiphone: connection and handshake succeeded
2009-03-30 18:25:31 N [client-protocol.c:6169:client_setvolume_cbk] gfs-ds-vu-hosting-hermes: connection and handshake succeeded
2009-03-30 18:25:31 N [client-protocol.c:6169:client_setvolume_cbk] gfs-ds-vu-hosting-hermes: connection and handshake succeeded
2009-03-30 18:25:37 N [client-protocol.c:6853:notify] gfs-ds-vu-hosting-tisiphone: disconnected
2009-03-30 18:25:42 N [client-protocol.c:6169:client_setvolume_cbk] gfs-ds-vu-hosting-tisiphone: connection and handshake succeeded
2009-03-30 18:25:42 N [client-protocol.c:6169:client_setvolume_cbk] gfs-ds-vu-hosting-tisiphone: connection and handshake succeeded
2009-03-30 18:27:52 E [client-protocol.c:537:client_ping_timer_expired] gfs-ds-vu-hosting-tisiphone: ping timer expired! bailing transport
2009-03-30 18:27:52 E [saved-frames.c:169:saved_frames_unwind] gfs-ds-vu-hosting-tisiphone: forced unwinding frame type(1) op(WRITE)
2009-03-30 18:27:52 E [saved-frames.c:169:saved_frames_unwind] gfs-ds-vu-hosting-tisiphone: forced unwinding frame type(2) op((null))
2009-03-30 18:27:52 E [client-protocol.c:635:client_ping_cbk] gfs-ds-vu-hosting-tisiphone: timer must have expired
2009-03-30 18:27:52 N [client-protocol.c:6853:notify] gfs-ds-vu-hosting-tisiphone: disconnected
2009-03-30 18:27:52 N [client-protocol.c:6169:client_setvolume_cbk] gfs-ds-vu-hosting-tisiphone: connection and handshake succeeded
2009-03-30 18:27:52 N [client-protocol.c:6169:client_setvolume_cbk] gfs-ds-vu-hosting-tisiphone: connection and handshake succeeded
----


tisiphone (server):
----
================================================================================
Version      : glusterfs 2.0.0git built on Mar 30 2009 10:06:55
TLA Revision : git://git.sv.gnu.org/gluster.git
Starting Time: 2009-03-30 18:25:41
Command line : /usr/sbin/glusterfsd -N -f /etc/glusterfs/glusterfs-server.vu-hosting.vol -l /var/log/glusterfs/glusterfs-server.vu-hosting.log -L NORMAL
PID          : 5899
System name  : Linux
Nodename     : tisiphone
Kernel Release : 2.6.29
Hardware Identifier: i686

Given volfile:
+------------------------------------------------------------------------------+
  1: #######################################################
  2: ###  GlusterFS Server Volume Specification          ###
  3: #######################################################
  4:
  5: volume gfs-ds-posix
  6:   type storage/posix
  7:   option directory /mnt/glusterfs/vu-hosting
  8:   # option o-direct off
  9:   # option export-statfs-size yes
 10:   # option mandate-attribute yes
 11:   # option span-devices 1
 12: end-volume
 13:
 14: volume gfs-ds-locks
 15:   type features/locks
 16:   option mandatory-locks off
 17:   subvolumes gfs-ds-posix
 18: end-volume
 19:
 20: volume gfs-ds-io-threads
 21:   type performance/io-threads
 22:   option thread-count 8
 23:   subvolumes gfs-ds-locks
 24: end-volume
 25:
 26: volume gfs-ds-server
 27:   type protocol/server
 28:   option transport-type tcp
 29:   option transport.socket.listen-port 6997
 30:   option auth.addr.gfs-ds-locks.allow 192.168.0.*,127.0.0.1
 31:   option auth.addr.gfs-ds-io-threads.allow 192.168.0.*,127.0.0.1
 32:   subvolumes gfs-ds-io-threads
 33: end-volume

+------------------------------------------------------------------------------+
2009-03-30 18:25:41 N [glusterfsd.c:1134:main] glusterfs: Successfully started
2009-03-30 18:25:41 N [server-protocol.c:7513:mop_setvolume] gfs-ds-server: accepted client from 192.168.0.75:1013
2009-03-30 18:25:41 N [server-protocol.c:7513:mop_setvolume] gfs-ds-server: accepted client from 192.168.0.75:1012
2009-03-30 18:25:46 N [server-protocol.c:7513:mop_setvolume] gfs-ds-server: accepted client from 192.168.0.70:1017
2009-03-30 18:25:46 N [server-protocol.c:7513:mop_setvolume] gfs-ds-server: accepted client from 192.168.0.70:1016
2009-03-30 18:27:56 N [server-protocol.c:8268:notify] gfs-ds-server: 192.168.0.70:1016 disconnected
2009-03-30 18:27:56 N [server-protocol.c:7513:mop_setvolume] gfs-ds-server: accepted client from 192.168.0.70:1023
----


tisiphone (client):
----
================================================================================
Version      : glusterfs 2.0.0git built on Mar 30 2009 10:06:55
TLA Revision : git://git.sv.gnu.org/gluster.git
Starting Time: 2009-03-30 18:25:41
Command line : /usr/sbin/glusterfsd -N -f /etc/glusterfs/glusterfs-client.vu-hosting.vol -l /var/log/glusterfs/glusterfs-client.vu-hosting.log -L NORMAL /var/www/vu-hosting
PID          : 5934
System name  : Linux
Nodename     : tisiphone
Kernel Release : 2.6.29
Hardware Identifier: i686

Given volfile:
+------------------------------------------------------------------------------+
  1: #######################################################
  2: ###  GlusterFS Client Volume Specification          ###
  3: #######################################################
  4:
  5: #volume gfs-ds-fuse
  6: #  type mount/fuse
  7: #  option direct-io-mode on
  8: #  option macfuse-local off
  9: #  option mount-point /var/www/vu-hosting
 10: #  option attribute-timeout 0.0
 11: #  option entry-timeout 0.0
 12: #end-volume
 13:
 14: volume gfs-ds-vu-hosting-hermes
 15:   type protocol/client
 16:   option transport-type tcp
 17:   # option username
 18:   # option password
 19:   option remote-host 192.168.0.70
 20:   option remote-port 6997
 21:   option remote-subvolume gfs-ds-io-threads
 22:   option transport-timeout 10
 23:   option ping-timeout 5
 24: end-volume
 25:
 26: volume gfs-ds-vu-hosting-tisiphone
 27:   type protocol/client
 28:   option transport-type tcp
 29:   # option username
 30:   # option password
 31:   option remote-host 192.168.0.75
 32:   option remote-port 6997
 33:   option remote-subvolume gfs-ds-io-threads
 34:   option transport-timeout 10
 35:   option ping-timeout 5
 36: end-volume
 37:
 38: volume gfs-ds-replicate
 39:   type cluster/replicate
 40:   option data-self-heal on
 41:   option metadata-self-heal on
 42:   option entry-self-heal on
 43:   # option read-subvolume
 44:   # option favorite-child
 45:   option data-change-log on
 46:   option metadata-change-log on
 47:   option entry-change-log on
 48:   option data-lock-server-count 1
 49:   option metadata-lock-server-count 1
 50:   option entry-lock-server-count 1
 51:   subvolumes gfs-ds-vu-hosting-hermes gfs-ds-vu-hosting-tisiphone
 52: end-volume
 53:
 54: volume gfs-ds-io-threads
 55:   type performance/io-threads
 56:   option thread-count 8
 57:   subvolumes gfs-ds-replicate
 58: end-volume
 59:
 60: volume gfs-ds-write-behind
 61:   type performance/write-behind
 62:   option block-size 1MB
 63:   option cache-size 4MB
 64:   option flush-behind on
 65:   # opiton disable-for-first-nbytes 1
 66:   # option enable-O_SYNC false
 67:   subvolumes gfs-ds-io-threads
 68: end-volume
 69:
 70: #volume gfs-ds-read-ahead
 71: #  type performance/read-ahead
 72: #  option page-size 65KB
 73: #  option page-count 16
 74: #  option force-atime-update off
 75: #  subvolumes gfs-ds-write-behind
 76: #end-volume
 77:
 78: volume gfs-ds-io-cache
 79:   type performance/io-cache
 80:   option page-size 4KB
 81:   option cache-size 64MB
 82:   option priority *:0
 83:   option cache-timeout 1
 84:   # subvolumes gfs-ds-read-ahead
 85:   subvolumes gfs-ds-write-behind
 86: end-volume

+------------------------------------------------------------------------------+
2009-03-30 18:25:41 E [xlator.c:201:_volume_option_value_validate] gfs-ds-io-cache: '4096' in 'option page-size 4KB' is out of range [16384 - 4194304]
2009-03-30 18:25:41 N [glusterfsd.c:1134:main] glusterfs: Successfully started
2009-03-30 18:25:41 N [client-protocol.c:6169:client_setvolume_cbk] gfs-ds-vu-hosting-hermes: connection and handshake succeeded
2009-03-30 18:25:41 N [client-protocol.c:6169:client_setvolume_cbk] gfs-ds-vu-hosting-hermes: connection and handshake succeeded
2009-03-30 18:25:41 N [client-protocol.c:6169:client_setvolume_cbk] gfs-ds-vu-hosting-tisiphone: connection and handshake succeeded
2009-03-30 18:25:41 N [client-protocol.c:6169:client_setvolume_cbk] gfs-ds-vu-hosting-tisiphone: connection and handshake succeeded
2009-03-30 18:29:03 E [client-protocol.c:537:client_ping_timer_expired] gfs-ds-vu-hosting-hermes: ping timer expired! bailing transport
2009-03-30 18:29:03 E [saved-frames.c:169:saved_frames_unwind] gfs-ds-vu-hosting-hermes: forced unwinding frame type(1) op(READ)
2009-03-30 18:29:03 E [saved-frames.c:169:saved_frames_unwind] gfs-ds-vu-hosting-hermes: forced unwinding frame type(2) op((null))
2009-03-30 18:29:03 E [client-protocol.c:635:client_ping_cbk] gfs-ds-vu-hosting-hermes: timer must have expired
2009-03-30 18:29:03 N [client-protocol.c:6853:notify] gfs-ds-vu-hosting-hermes: disconnected
2009-03-30 18:29:03 N [client-protocol.c:6169:client_setvolume_cbk] gfs-ds-vu-hosting-hermes: connection and handshake succeeded
2009-03-30 18:29:03 N [client-protocol.c:6169:client_setvolume_cbk] gfs-ds-vu-hosting-hermes: connection and handshake succeeded
----


I see the error with the 4KB being to low. I will fix that, but I don't think that this is the problem. Or could it be?

I don't know why the client has a ping timeout? The system is connected with a brand new HP ProCurve 24 Port Gigabit Switch.

The network cards use 7200 MTU.


> Vikas
>
Steve


> -- 
> Engineer - Z Research
> http://gluster.com/

-- 
Neu: GMX FreeDSL Komplettanschluss mit DSL 6.000 Flatrate + Telefonanschluss für nur 17,95 Euro/mtl.!* http://dsl.gmx.de/?ac=OM.AD.PD003K11308T4569a




More information about the Gluster-users mailing list