[Gluster-devel] Suffering disconnections of glusterfs-clients on glusterfs-server

Ioannis Aslanidis iaslanidis at flumotion.com
Wed May 27 10:35:42 UTC 2009


Hello,

This is yet another problem we are experiencing with glusterfs 2.0.1
(both server and client).

Every few hours the client gets disconnected in a way that processes
trying to access the glusterfs mount volume die in a non-interruptible
input-output state. We are forced to run `umount -l` on the partition
and remount it with `mount` all over again.

The logs in the side of the server are the following:


[2009-05-27 01:18:25] C
[server-protocol.c:6553:_volfile_update_checksum] foo-server: the volume
file got modified between earlier access and now, this may lead to
inconsistency between clien
ts, advised to remount client
[2009-05-27 01:18:25] N [server-protocol.c:7796:notify] foo-server:
192.168.168.210:1020 disconnected
[2009-05-27 01:18:25] N [server-protocol.c:7796:notify] foo-server:
192.168.168.210:1021 disconnected
[2009-05-27 01:18:26] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.210:1021
[2009-05-27 01:18:26] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.210:1020
[2009-05-27 01:52:50] N [server-protocol.c:7796:notify] foo-server:
192.168.168.212:1020 disconnected
[2009-05-27 01:52:50] N [server-protocol.c:7796:notify] foo-server:
192.168.168.212:1021 disconnected
[2009-05-27 01:52:50] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.212:1020
[2009-05-27 01:52:50] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.212:1021
[2009-05-27 04:11:36] N [server-protocol.c:7796:notify] foo-server:
192.168.168.210:1018 disconnected
[2009-05-27 04:11:36] N [server-protocol.c:7796:notify] foo-server:
192.168.168.210:1019 disconnected
[2009-05-27 04:11:37] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.210:1019
[2009-05-27 04:11:37] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.210:1018
[2009-05-27 04:42:49] N [server-protocol.c:7796:notify] foo-server:
192.168.168.212:1018 disconnected
[2009-05-27 04:42:49] N [server-protocol.c:7796:notify] foo-server:
192.168.168.212:1019 disconnected
[2009-05-27 04:42:50] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.212:1018
[2009-05-27 04:42:50] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.212:1019
[2009-05-27 07:25:31] N [server-protocol.c:7796:notify] foo-server:
192.168.168.210:1016 disconnected
[2009-05-27 07:25:31] N [server-protocol.c:7796:notify] foo-server:
192.168.168.210:1017 disconnected
[2009-05-27 07:25:31] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.210:1016
[2009-05-27 07:25:31] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.210:1017
[2009-05-27 07:39:25] N [server-protocol.c:7796:notify] foo-server:
192.168.168.210:1017 disconnected
[2009-05-27 07:39:25] N [server-protocol.c:7796:notify] foo-server:
192.168.168.210:1016 disconnected
[2009-05-27 07:39:25] N [server-helpers.c:723:server_connection_destroy]
foo-server: destroyed connection of
xxx010.bar.foo.net-28685-2009/05/27-07:25:30:922187-foo
[2009-05-27 07:39:26] N [server-protocol.c:7796:notify] foo-server:
192.168.168.210:1015 disconnected
[2009-05-27 07:39:26] N [server-protocol.c:7796:notify] foo-server:
192.168.168.210:1017 disconnected
[2009-05-27 07:39:26] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.210:1017
[2009-05-27 07:39:26] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.210:1015
[2009-05-27 08:00:24] N [server-protocol.c:7796:notify] foo-server:
192.168.168.212:1016 disconnected
[2009-05-27 08:00:24] N [server-protocol.c:7796:notify] foo-server:
192.168.168.212:1017 disconnected
[2009-05-27 08:00:24] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.212:1016
[2009-05-27 08:00:24] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.212:1017
[2009-05-27 08:00:29] N [server-protocol.c:7796:notify] foo-server:
192.168.168.210:1015 disconnected
[2009-05-27 08:00:29] N [server-protocol.c:7796:notify] foo-server:
192.168.168.210:1017 disconnected
[2009-05-27 08:00:29] N [server-helpers.c:723:server_connection_destroy]
foo-server: destroyed connection of
xxx010.bar.foo.net-30040-2009/05/27-07:39:26:412074-foo
[2009-05-27 08:00:30] N [server-protocol.c:7796:notify] foo-server:
192.168.168.210:1014 disconnected
[2009-05-27 08:00:30] N [server-protocol.c:7796:notify] foo-server:
192.168.168.210:1016 disconnected
[2009-05-27 08:00:30] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.210:1014
[2009-05-27 08:00:30] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.210:1016
[2009-05-27 11:37:59] N [server-protocol.c:7796:notify] foo-server:
192.168.168.210:1015 disconnected
[2009-05-27 11:37:59] N [server-protocol.c:7796:notify] foo-server:
192.168.168.210:1017 disconnected
[2009-05-27 11:38:00] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.210:1017
[2009-05-27 11:38:00] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.210:1015
[2009-05-27 11:38:12] N [server-protocol.c:7796:notify] foo-server:
192.168.168.212:1015 disconnected
[2009-05-27 11:38:12] N [server-protocol.c:7796:notify] foo-server:
192.168.168.212:1014 disconnected
[2009-05-27 11:38:12] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.212:1015
[2009-05-27 11:38:12] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.212:1014


In the client:

================================================================================
Version      : glusterfs 2.0.1 built on May 26 2009 05:23:52
TLA Revision : 5c1d9108c1529a1155963cb1911f8870a674ab5b
Starting Time: 2009-05-26 12:09:27
Command line : /usr/sbin/glusterfs2 --log-level=NORMAL
--volfile-server=file01.priv --volfile-server-port=6997 /mnt/file01
PID          : 18548
System name  : Linux
Nodename     : xxx012.bar.foo.net
Kernel Release : 2.6.18-92.1.10.el5PAE
Hardware Identifier: i686

Given volfile:
+------------------------------------------------------------------------------+
  1: volume foo
  2:   type protocol/client
  3:   option transport-type tcp
  4:   option remote-host file01.priv # can be IP or hostname
  5:   option remote-port 6997
  6:   option remote-subvolume foo-remote
  7: end-volume
  8:
  9: volume foo-readahead
 10:   type performance/read-ahead
 11:   option page-count 4
 12:   subvolumes foo
 13: end-volume
 14:
 15: volume foo-writebehind
 16:   type performance/write-behind
 17:   option cache-size 512KB
 18:   subvolumes foo-readahead
 19: end-volume
 20:
 21: volume foo-iothreads
 22:   type performance/io-threads
 23:   option autoscaling on
 24:   option min-threads 16
 25:   option max-threads 256
 26:   subvolumes foo-writebehind
 27: end-volume
 28:
 29: volume foo-iocache
 30:   type performance/io-cache
 31:   option cache-size 64MB
 32:   option page-size 128KB
 33:   option cache-timeout 2
 34:   subvolumes foo-iothreads
 35: end-volume
 36:

+------------------------------------------------------------------------------+
[2009-05-26 12:09:27] N [glusterfsd.c:1152:main] glusterfs: Successfully
started
[2009-05-26 12:09:27] N [client-protocol.c:5557:client_setvolume_cbk]
foo: Connected to 192.168.168.231:6997, attached to remote volume
'foo-remote'.
[2009-05-26 12:09:27] N [client-protocol.c:5557:client_setvolume_cbk]
foo: Connected to 192.168.168.231:6997, attached to remote volume
'foo-remote'.
[2009-05-26 12:09:38] W [fuse-bridge.c:1365:fuse_create_cbk]
glusterfs-fuse: 21: /cust/a => -1 (Read-only file system)
[2009-05-26 12:16:54] W [glusterfsd.c:827:cleanup_and_exit] glusterfs:
shutting down
[2009-05-26 12:16:54] N [fuse-bridge.c:2843:fini] fuse: Unmounting
'/mnt/file01'.
[2009-05-26 12:16:54] W [glusterfsd.c:827:cleanup_and_exit] glusterfs:
shutting down
[2009-05-26 12:20:08] W [socket.c:1319:socket_init] trans: disabling
non-blocking IO
[2009-05-26 12:20:08] W [socket.c:1319:socket_init] trans: disabling
non-blocking IO
[2009-05-26 12:20:08] W [client-protocol.c:6012:init] trans: WARNING:
Failed to set 'ulimit -n 1M': Permission denied
================================================================================
Version      : glusterfs 2.0.1 built on May 26 2009 05:23:52
TLA Revision : 5c1d9108c1529a1155963cb1911f8870a674ab5b
Starting Time: 2009-05-26 12:20:14
Command line : /usr/sbin/glusterfs2 --log-level=NORMAL
--volfile-server=file01.priv --volfile-server-port=6997 /mnt/file01
PID          : 3074
System name  : Linux
Nodename     : xxx012.bar.foo.net
Kernel Release : 2.6.18-92.1.10.el5PAE
Hardware Identifier: i686

Given volfile:
+------------------------------------------------------------------------------+
  1: volume foo
  2:   type protocol/client
  3:   option transport-type tcp
  4:   option remote-host file01.priv # can be IP or hostname
  5:   option remote-port 6997
  6:   option remote-subvolume foo-remote
  7: end-volume
  8:
  9: volume foo-readahead
 10:   type performance/read-ahead
 11:   option page-count 4
 12:   subvolumes foo
 13: end-volume
 14:
 15: volume foo-writebehind
 16:   type performance/write-behind
 17:   option cache-size 512KB
 18:   subvolumes foo-readahead
 19: end-volume
 20:
 21: volume foo-iothreads
 22:   type performance/io-threads
 23:   option autoscaling on
 24:   option min-threads 16
 25:   option max-threads 256
 26:   subvolumes foo-writebehind
 27: end-volume
 28:
 29: volume foo-iocache
 30:   type performance/io-cache
 31:   option cache-size 64MB
 32:   option page-size 128KB
 33:   option cache-timeout 2
 34:   subvolumes foo-iothreads
 35: end-volume
 36:

+------------------------------------------------------------------------------+
[2009-05-26 12:20:15] W [client-protocol.c:6012:init] foo: WARNING:
Failed to set 'ulimit -n 1M': Permission denied
[2009-05-26 12:20:15] N [glusterfsd.c:1152:main] glusterfs: Successfully
started
[2009-05-26 12:20:15] N [client-protocol.c:5557:client_setvolume_cbk]
foo: Connected to 192.168.168.231:6997, attached to remote volume
'foo-remote'.
[2009-05-26 12:20:15] N [client-protocol.c:5557:client_setvolume_cbk]
foo: Connected to 192.168.168.231:6997, attached to remote volume
'foo-remote'.
[2009-05-27 01:52:50] W [socket.c:1319:socket_init] trans: disabling
non-blocking IO
[2009-05-27 01:52:50] W [socket.c:1319:socket_init] trans: disabling
non-blocking IO
================================================================================
Version      : glusterfs 2.0.1 built on May 26 2009 05:23:52
TLA Revision : 5c1d9108c1529a1155963cb1911f8870a674ab5b
Starting Time: 2009-05-27 01:52:50
Command line : /usr/sbin/glusterfs2 --log-level=NORMAL
--volfile-server=file01.priv --volfile-server-port=6997 /mnt/file01
PID          : 30582
System name  : Linux
Nodename     : xxx012.bar.foo.net
Kernel Release : 2.6.18-92.1.10.el5PAE
Hardware Identifier: i686

Given volfile:
+------------------------------------------------------------------------------+
  1: volume foo
  2:   type protocol/client
  3:   option transport-type tcp
  4:   option remote-host file01.priv # can be IP or hostname
  5:   option remote-port 6997
  6:   option remote-subvolume foo-remote
  7: end-volume
  8:
  9: volume foo-readahead
 10:   type performance/read-ahead
 11:   option page-count 4
 12:   subvolumes foo
 13: end-volume
 14:
 15: volume foo-writebehind
 16:   type performance/write-behind
 17:   option cache-size 512KB
 18:   subvolumes foo-readahead
 19: end-volume
 20:
 21: volume foo-iothreads
 22:   type performance/io-threads
 23:   option autoscaling on
 24:   option min-threads 16
 25:   option max-threads 256
 26:   subvolumes foo-writebehind
 27: end-volume
 28:
 29: volume foo-iocache
 30:   type performance/io-cache
 31:   option cache-size 64MB
 32:   option page-size 128KB
 33:   option cache-timeout 2
 34:   subvolumes foo-iothreads
 35: end-volume

+------------------------------------------------------------------------------+
[2009-05-27 01:52:50] N [glusterfsd.c:1152:main] glusterfs: Successfully
started
[2009-05-27 01:52:50] N [client-protocol.c:5557:client_setvolume_cbk]
foo: Connected to 192.168.168.231:6997, attached to remote volume
'foo-remote'.
[2009-05-27 01:52:50] N [client-protocol.c:5557:client_setvolume_cbk]
foo: Connected to 192.168.168.231:6997, attached to remote volume
'foo-remote'.
[2009-05-27 04:42:49] W [socket.c:1319:socket_init] trans: disabling
non-blocking IO
[2009-05-27 04:42:49] W [socket.c:1319:socket_init] trans: disabling
non-blocking IO
================================================================================
Version      : glusterfs 2.0.1 built on May 26 2009 05:23:52
TLA Revision : 5c1d9108c1529a1155963cb1911f8870a674ab5b
Starting Time: 2009-05-27 04:42:49
Command line : /usr/sbin/glusterfs2 --log-level=NORMAL
--volfile-server=file01.priv --volfile-server-port=6997 /mnt/file01
PID          : 13601
System name  : Linux
Nodename     : xxx012.bar.foo.net
Kernel Release : 2.6.18-92.1.10.el5PAE
Hardware Identifier: i686

Given volfile:
+------------------------------------------------------------------------------+
  1: volume foo
  2:   type protocol/client
  3:   option transport-type tcp
  4:   option remote-host file01.priv # can be IP or hostname
  5:   option remote-port 6997
  6:   option remote-subvolume foo-remote
  7: end-volume
  8:
  9: volume foo-readahead
 10:   type performance/read-ahead
 11:   option page-count 4
 12:   subvolumes foo
 13: end-volume
 14:
 15: volume foo-writebehind
 16:   type performance/write-behind
 17:   option cache-size 512KB
 18:   subvolumes foo-readahead
 19: end-volume
 20:
 21: volume foo-iothreads
 22:   type performance/io-threads
 23:   option autoscaling on
 24:   option min-threads 16
 25:   option max-threads 256
 26:   subvolumes foo-writebehind
 27: end-volume
 28:
 29: volume foo-iocache
 30:   type performance/io-cache
 31:   option cache-size 64MB
 32:   option page-size 128KB
 33:   option cache-timeout 2
 34:   subvolumes foo-iothreads
 35: end-volume

+------------------------------------------------------------------------------+
[2009-05-27 04:42:50] N [glusterfsd.c:1152:main] glusterfs: Successfully
started
[2009-05-27 04:42:50] N [client-protocol.c:5557:client_setvolume_cbk]
foo: Connected to 192.168.168.231:6997, attached to remote volume
'foo-remote'.
[2009-05-27 04:42:50] N [client-protocol.c:5557:client_setvolume_cbk]
foo: Connected to 192.168.168.231:6997, attached to remote volume
'foo-remote'.
[2009-05-27 08:00:24] W [socket.c:1319:socket_init] trans: disabling
non-blocking IO
[2009-05-27 08:00:24] W [socket.c:1319:socket_init] trans: disabling
non-blocking IO
================================================================================
Version      : glusterfs 2.0.1 built on May 26 2009 05:23:52
TLA Revision : 5c1d9108c1529a1155963cb1911f8870a674ab5b
Starting Time: 2009-05-27 08:00:24
Command line : /usr/sbin/glusterfs2 --log-level=NORMAL
--volfile-server=file01.priv --volfile-server-port=6997 /mnt/file01
PID          : 30261
System name  : Linux
Nodename     : xxx012.bar.foo.net
Kernel Release : 2.6.18-92.1.10.el5PAE
Hardware Identifier: i686

Given volfile:
+------------------------------------------------------------------------------+
  1: volume foo
  2:   type protocol/client
  3:   option transport-type tcp
  4:   option remote-host file01.priv # can be IP or hostname
  5:   option remote-port 6997
  6:   option remote-subvolume foo-remote
  7: end-volume
  8:
  9: volume foo-readahead
 10:   type performance/read-ahead
 11:   option page-count 4
 12:   subvolumes foo
 13: end-volume
 14:
 15: volume foo-writebehind
 16:   type performance/write-behind
 17:   option cache-size 512KB
 18:   subvolumes foo-readahead
 19: end-volume
 20:
 21: volume foo-iothreads
 22:   type performance/io-threads
 23:   option autoscaling on
 24:   option min-threads 16
 25:   option max-threads 256
 26:   subvolumes foo-writebehind
 27: end-volume
 28:
 29: volume foo-iocache
 30:   type performance/io-cache
 31:   option cache-size 64MB
 32:   option page-size 128KB
 33:   option cache-timeout 2
 34:   subvolumes foo-iothreads
 35: end-volume

+------------------------------------------------------------------------------+
[2009-05-27 08:00:24] N [glusterfsd.c:1152:main] glusterfs: Successfully
started
[2009-05-27 08:00:24] N [client-protocol.c:5557:client_setvolume_cbk]
foo: Connected to 192.168.168.231:6997, attached to remote volume
'foo-remote'.
[2009-05-27 08:00:24] N [client-protocol.c:5557:client_setvolume_cbk]
foo: Connected to 192.168.168.231:6997, attached to remote volume
'foo-remote'.
[2009-05-27 11:38:12] W [socket.c:1319:socket_init] trans: disabling
non-blocking IO
[2009-05-27 11:38:12] W [socket.c:1319:socket_init] trans: disabling
non-blocking IO
================================================================================
Version      : glusterfs 2.0.1 built on May 26 2009 05:23:52
TLA Revision : 5c1d9108c1529a1155963cb1911f8870a674ab5b
Starting Time: 2009-05-27 11:38:12
Command line : /usr/sbin/glusterfs2 --log-level=NORMAL
--volfile-server=file01.priv --volfile-server-port=6997 /mnt/file01
PID          : 16293
System name  : Linux
Nodename     : xxx012.bar.foo.net
Kernel Release : 2.6.18-92.1.10.el5PAE
Hardware Identifier: i686

Given volfile:
+------------------------------------------------------------------------------+
  1: volume foo
  2:   type protocol/client
  3:   option transport-type tcp
  4:   option remote-host file01.priv # can be IP or hostname
  5:   option remote-port 6997
  6:   option remote-subvolume foo-remote
  7: end-volume
  8:
  9: volume foo-readahead
 10:   type performance/read-ahead
 11:   option page-count 4
 12:   subvolumes foo
 13: end-volume
 14:
 15: volume foo-writebehind
 16:   type performance/write-behind
 17:   option cache-size 512KB
 18:   subvolumes foo-readahead
 19: end-volume
 20:
 21: volume foo-iothreads
 22:   type performance/io-threads
 23:   option autoscaling on
 24:   option min-threads 16
 25:   option max-threads 256
 26:   subvolumes foo-writebehind
 27: end-volume
 28:
 29: volume foo-iocache
 30:   type performance/io-cache
 31:   option cache-size 64MB
 32:   option page-size 128KB
 33:   option cache-timeout 2
 34:   subvolumes foo-iothreads
 35: end-volume

+------------------------------------------------------------------------------+
[2009-05-27 11:38:12] N [glusterfsd.c:1152:main] glusterfs: Successfully
started
[2009-05-27 11:38:12] N [client-protocol.c:5557:client_setvolume_cbk]
foo: Connected to 192.168.168.231:6997, attached to remote volume
'foo-remote'.
[2009-05-27 11:38:12] N [client-protocol.c:5557:client_setvolume_cbk]
foo: Connected to 192.168.168.231:6997, attached to remote volume
'foo-remote'.


Thanks,

Ioannis
-------------- next part --------------
A non-text attachment was scrubbed...
Name: iaslanidis.vcf
Type: text/x-vcard
Size: 498 bytes
Desc: not available
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20090527/4d66743b/attachment-0003.vcf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 261 bytes
Desc: OpenPGP digital signature
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20090527/4d66743b/attachment-0003.sig>


More information about the Gluster-devel mailing list