[Gluster-devel] Suffering disconnections of glusterfs-clients on glusterfs-server
Ioannis Aslanidis
iaslanidis at flumotion.com
Wed May 27 10:35:42 UTC 2009
Hello,
This is yet another problem we are experiencing with glusterfs 2.0.1
(both server and client).
Every few hours the client gets disconnected in a way that processes
trying to access the glusterfs mount volume die in a non-interruptible
input-output state. We are forced to run `umount -l` on the partition
and remount it with `mount` all over again.
The logs in the side of the server are the following:
[2009-05-27 01:18:25] C
[server-protocol.c:6553:_volfile_update_checksum] foo-server: the volume
file got modified between earlier access and now, this may lead to
inconsistency between clien
ts, advised to remount client
[2009-05-27 01:18:25] N [server-protocol.c:7796:notify] foo-server:
192.168.168.210:1020 disconnected
[2009-05-27 01:18:25] N [server-protocol.c:7796:notify] foo-server:
192.168.168.210:1021 disconnected
[2009-05-27 01:18:26] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.210:1021
[2009-05-27 01:18:26] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.210:1020
[2009-05-27 01:52:50] N [server-protocol.c:7796:notify] foo-server:
192.168.168.212:1020 disconnected
[2009-05-27 01:52:50] N [server-protocol.c:7796:notify] foo-server:
192.168.168.212:1021 disconnected
[2009-05-27 01:52:50] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.212:1020
[2009-05-27 01:52:50] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.212:1021
[2009-05-27 04:11:36] N [server-protocol.c:7796:notify] foo-server:
192.168.168.210:1018 disconnected
[2009-05-27 04:11:36] N [server-protocol.c:7796:notify] foo-server:
192.168.168.210:1019 disconnected
[2009-05-27 04:11:37] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.210:1019
[2009-05-27 04:11:37] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.210:1018
[2009-05-27 04:42:49] N [server-protocol.c:7796:notify] foo-server:
192.168.168.212:1018 disconnected
[2009-05-27 04:42:49] N [server-protocol.c:7796:notify] foo-server:
192.168.168.212:1019 disconnected
[2009-05-27 04:42:50] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.212:1018
[2009-05-27 04:42:50] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.212:1019
[2009-05-27 07:25:31] N [server-protocol.c:7796:notify] foo-server:
192.168.168.210:1016 disconnected
[2009-05-27 07:25:31] N [server-protocol.c:7796:notify] foo-server:
192.168.168.210:1017 disconnected
[2009-05-27 07:25:31] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.210:1016
[2009-05-27 07:25:31] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.210:1017
[2009-05-27 07:39:25] N [server-protocol.c:7796:notify] foo-server:
192.168.168.210:1017 disconnected
[2009-05-27 07:39:25] N [server-protocol.c:7796:notify] foo-server:
192.168.168.210:1016 disconnected
[2009-05-27 07:39:25] N [server-helpers.c:723:server_connection_destroy]
foo-server: destroyed connection of
xxx010.bar.foo.net-28685-2009/05/27-07:25:30:922187-foo
[2009-05-27 07:39:26] N [server-protocol.c:7796:notify] foo-server:
192.168.168.210:1015 disconnected
[2009-05-27 07:39:26] N [server-protocol.c:7796:notify] foo-server:
192.168.168.210:1017 disconnected
[2009-05-27 07:39:26] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.210:1017
[2009-05-27 07:39:26] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.210:1015
[2009-05-27 08:00:24] N [server-protocol.c:7796:notify] foo-server:
192.168.168.212:1016 disconnected
[2009-05-27 08:00:24] N [server-protocol.c:7796:notify] foo-server:
192.168.168.212:1017 disconnected
[2009-05-27 08:00:24] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.212:1016
[2009-05-27 08:00:24] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.212:1017
[2009-05-27 08:00:29] N [server-protocol.c:7796:notify] foo-server:
192.168.168.210:1015 disconnected
[2009-05-27 08:00:29] N [server-protocol.c:7796:notify] foo-server:
192.168.168.210:1017 disconnected
[2009-05-27 08:00:29] N [server-helpers.c:723:server_connection_destroy]
foo-server: destroyed connection of
xxx010.bar.foo.net-30040-2009/05/27-07:39:26:412074-foo
[2009-05-27 08:00:30] N [server-protocol.c:7796:notify] foo-server:
192.168.168.210:1014 disconnected
[2009-05-27 08:00:30] N [server-protocol.c:7796:notify] foo-server:
192.168.168.210:1016 disconnected
[2009-05-27 08:00:30] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.210:1014
[2009-05-27 08:00:30] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.210:1016
[2009-05-27 11:37:59] N [server-protocol.c:7796:notify] foo-server:
192.168.168.210:1015 disconnected
[2009-05-27 11:37:59] N [server-protocol.c:7796:notify] foo-server:
192.168.168.210:1017 disconnected
[2009-05-27 11:38:00] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.210:1017
[2009-05-27 11:38:00] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.210:1015
[2009-05-27 11:38:12] N [server-protocol.c:7796:notify] foo-server:
192.168.168.212:1015 disconnected
[2009-05-27 11:38:12] N [server-protocol.c:7796:notify] foo-server:
192.168.168.212:1014 disconnected
[2009-05-27 11:38:12] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.212:1015
[2009-05-27 11:38:12] N [server-protocol.c:7035:mop_setvolume]
foo-server: accepted client from 192.168.168.212:1014
In the client:
================================================================================
Version : glusterfs 2.0.1 built on May 26 2009 05:23:52
TLA Revision : 5c1d9108c1529a1155963cb1911f8870a674ab5b
Starting Time: 2009-05-26 12:09:27
Command line : /usr/sbin/glusterfs2 --log-level=NORMAL
--volfile-server=file01.priv --volfile-server-port=6997 /mnt/file01
PID : 18548
System name : Linux
Nodename : xxx012.bar.foo.net
Kernel Release : 2.6.18-92.1.10.el5PAE
Hardware Identifier: i686
Given volfile:
+------------------------------------------------------------------------------+
1: volume foo
2: type protocol/client
3: option transport-type tcp
4: option remote-host file01.priv # can be IP or hostname
5: option remote-port 6997
6: option remote-subvolume foo-remote
7: end-volume
8:
9: volume foo-readahead
10: type performance/read-ahead
11: option page-count 4
12: subvolumes foo
13: end-volume
14:
15: volume foo-writebehind
16: type performance/write-behind
17: option cache-size 512KB
18: subvolumes foo-readahead
19: end-volume
20:
21: volume foo-iothreads
22: type performance/io-threads
23: option autoscaling on
24: option min-threads 16
25: option max-threads 256
26: subvolumes foo-writebehind
27: end-volume
28:
29: volume foo-iocache
30: type performance/io-cache
31: option cache-size 64MB
32: option page-size 128KB
33: option cache-timeout 2
34: subvolumes foo-iothreads
35: end-volume
36:
+------------------------------------------------------------------------------+
[2009-05-26 12:09:27] N [glusterfsd.c:1152:main] glusterfs: Successfully
started
[2009-05-26 12:09:27] N [client-protocol.c:5557:client_setvolume_cbk]
foo: Connected to 192.168.168.231:6997, attached to remote volume
'foo-remote'.
[2009-05-26 12:09:27] N [client-protocol.c:5557:client_setvolume_cbk]
foo: Connected to 192.168.168.231:6997, attached to remote volume
'foo-remote'.
[2009-05-26 12:09:38] W [fuse-bridge.c:1365:fuse_create_cbk]
glusterfs-fuse: 21: /cust/a => -1 (Read-only file system)
[2009-05-26 12:16:54] W [glusterfsd.c:827:cleanup_and_exit] glusterfs:
shutting down
[2009-05-26 12:16:54] N [fuse-bridge.c:2843:fini] fuse: Unmounting
'/mnt/file01'.
[2009-05-26 12:16:54] W [glusterfsd.c:827:cleanup_and_exit] glusterfs:
shutting down
[2009-05-26 12:20:08] W [socket.c:1319:socket_init] trans: disabling
non-blocking IO
[2009-05-26 12:20:08] W [socket.c:1319:socket_init] trans: disabling
non-blocking IO
[2009-05-26 12:20:08] W [client-protocol.c:6012:init] trans: WARNING:
Failed to set 'ulimit -n 1M': Permission denied
================================================================================
Version : glusterfs 2.0.1 built on May 26 2009 05:23:52
TLA Revision : 5c1d9108c1529a1155963cb1911f8870a674ab5b
Starting Time: 2009-05-26 12:20:14
Command line : /usr/sbin/glusterfs2 --log-level=NORMAL
--volfile-server=file01.priv --volfile-server-port=6997 /mnt/file01
PID : 3074
System name : Linux
Nodename : xxx012.bar.foo.net
Kernel Release : 2.6.18-92.1.10.el5PAE
Hardware Identifier: i686
Given volfile:
+------------------------------------------------------------------------------+
1: volume foo
2: type protocol/client
3: option transport-type tcp
4: option remote-host file01.priv # can be IP or hostname
5: option remote-port 6997
6: option remote-subvolume foo-remote
7: end-volume
8:
9: volume foo-readahead
10: type performance/read-ahead
11: option page-count 4
12: subvolumes foo
13: end-volume
14:
15: volume foo-writebehind
16: type performance/write-behind
17: option cache-size 512KB
18: subvolumes foo-readahead
19: end-volume
20:
21: volume foo-iothreads
22: type performance/io-threads
23: option autoscaling on
24: option min-threads 16
25: option max-threads 256
26: subvolumes foo-writebehind
27: end-volume
28:
29: volume foo-iocache
30: type performance/io-cache
31: option cache-size 64MB
32: option page-size 128KB
33: option cache-timeout 2
34: subvolumes foo-iothreads
35: end-volume
36:
+------------------------------------------------------------------------------+
[2009-05-26 12:20:15] W [client-protocol.c:6012:init] foo: WARNING:
Failed to set 'ulimit -n 1M': Permission denied
[2009-05-26 12:20:15] N [glusterfsd.c:1152:main] glusterfs: Successfully
started
[2009-05-26 12:20:15] N [client-protocol.c:5557:client_setvolume_cbk]
foo: Connected to 192.168.168.231:6997, attached to remote volume
'foo-remote'.
[2009-05-26 12:20:15] N [client-protocol.c:5557:client_setvolume_cbk]
foo: Connected to 192.168.168.231:6997, attached to remote volume
'foo-remote'.
[2009-05-27 01:52:50] W [socket.c:1319:socket_init] trans: disabling
non-blocking IO
[2009-05-27 01:52:50] W [socket.c:1319:socket_init] trans: disabling
non-blocking IO
================================================================================
Version : glusterfs 2.0.1 built on May 26 2009 05:23:52
TLA Revision : 5c1d9108c1529a1155963cb1911f8870a674ab5b
Starting Time: 2009-05-27 01:52:50
Command line : /usr/sbin/glusterfs2 --log-level=NORMAL
--volfile-server=file01.priv --volfile-server-port=6997 /mnt/file01
PID : 30582
System name : Linux
Nodename : xxx012.bar.foo.net
Kernel Release : 2.6.18-92.1.10.el5PAE
Hardware Identifier: i686
Given volfile:
+------------------------------------------------------------------------------+
1: volume foo
2: type protocol/client
3: option transport-type tcp
4: option remote-host file01.priv # can be IP or hostname
5: option remote-port 6997
6: option remote-subvolume foo-remote
7: end-volume
8:
9: volume foo-readahead
10: type performance/read-ahead
11: option page-count 4
12: subvolumes foo
13: end-volume
14:
15: volume foo-writebehind
16: type performance/write-behind
17: option cache-size 512KB
18: subvolumes foo-readahead
19: end-volume
20:
21: volume foo-iothreads
22: type performance/io-threads
23: option autoscaling on
24: option min-threads 16
25: option max-threads 256
26: subvolumes foo-writebehind
27: end-volume
28:
29: volume foo-iocache
30: type performance/io-cache
31: option cache-size 64MB
32: option page-size 128KB
33: option cache-timeout 2
34: subvolumes foo-iothreads
35: end-volume
+------------------------------------------------------------------------------+
[2009-05-27 01:52:50] N [glusterfsd.c:1152:main] glusterfs: Successfully
started
[2009-05-27 01:52:50] N [client-protocol.c:5557:client_setvolume_cbk]
foo: Connected to 192.168.168.231:6997, attached to remote volume
'foo-remote'.
[2009-05-27 01:52:50] N [client-protocol.c:5557:client_setvolume_cbk]
foo: Connected to 192.168.168.231:6997, attached to remote volume
'foo-remote'.
[2009-05-27 04:42:49] W [socket.c:1319:socket_init] trans: disabling
non-blocking IO
[2009-05-27 04:42:49] W [socket.c:1319:socket_init] trans: disabling
non-blocking IO
================================================================================
Version : glusterfs 2.0.1 built on May 26 2009 05:23:52
TLA Revision : 5c1d9108c1529a1155963cb1911f8870a674ab5b
Starting Time: 2009-05-27 04:42:49
Command line : /usr/sbin/glusterfs2 --log-level=NORMAL
--volfile-server=file01.priv --volfile-server-port=6997 /mnt/file01
PID : 13601
System name : Linux
Nodename : xxx012.bar.foo.net
Kernel Release : 2.6.18-92.1.10.el5PAE
Hardware Identifier: i686
Given volfile:
+------------------------------------------------------------------------------+
1: volume foo
2: type protocol/client
3: option transport-type tcp
4: option remote-host file01.priv # can be IP or hostname
5: option remote-port 6997
6: option remote-subvolume foo-remote
7: end-volume
8:
9: volume foo-readahead
10: type performance/read-ahead
11: option page-count 4
12: subvolumes foo
13: end-volume
14:
15: volume foo-writebehind
16: type performance/write-behind
17: option cache-size 512KB
18: subvolumes foo-readahead
19: end-volume
20:
21: volume foo-iothreads
22: type performance/io-threads
23: option autoscaling on
24: option min-threads 16
25: option max-threads 256
26: subvolumes foo-writebehind
27: end-volume
28:
29: volume foo-iocache
30: type performance/io-cache
31: option cache-size 64MB
32: option page-size 128KB
33: option cache-timeout 2
34: subvolumes foo-iothreads
35: end-volume
+------------------------------------------------------------------------------+
[2009-05-27 04:42:50] N [glusterfsd.c:1152:main] glusterfs: Successfully
started
[2009-05-27 04:42:50] N [client-protocol.c:5557:client_setvolume_cbk]
foo: Connected to 192.168.168.231:6997, attached to remote volume
'foo-remote'.
[2009-05-27 04:42:50] N [client-protocol.c:5557:client_setvolume_cbk]
foo: Connected to 192.168.168.231:6997, attached to remote volume
'foo-remote'.
[2009-05-27 08:00:24] W [socket.c:1319:socket_init] trans: disabling
non-blocking IO
[2009-05-27 08:00:24] W [socket.c:1319:socket_init] trans: disabling
non-blocking IO
================================================================================
Version : glusterfs 2.0.1 built on May 26 2009 05:23:52
TLA Revision : 5c1d9108c1529a1155963cb1911f8870a674ab5b
Starting Time: 2009-05-27 08:00:24
Command line : /usr/sbin/glusterfs2 --log-level=NORMAL
--volfile-server=file01.priv --volfile-server-port=6997 /mnt/file01
PID : 30261
System name : Linux
Nodename : xxx012.bar.foo.net
Kernel Release : 2.6.18-92.1.10.el5PAE
Hardware Identifier: i686
Given volfile:
+------------------------------------------------------------------------------+
1: volume foo
2: type protocol/client
3: option transport-type tcp
4: option remote-host file01.priv # can be IP or hostname
5: option remote-port 6997
6: option remote-subvolume foo-remote
7: end-volume
8:
9: volume foo-readahead
10: type performance/read-ahead
11: option page-count 4
12: subvolumes foo
13: end-volume
14:
15: volume foo-writebehind
16: type performance/write-behind
17: option cache-size 512KB
18: subvolumes foo-readahead
19: end-volume
20:
21: volume foo-iothreads
22: type performance/io-threads
23: option autoscaling on
24: option min-threads 16
25: option max-threads 256
26: subvolumes foo-writebehind
27: end-volume
28:
29: volume foo-iocache
30: type performance/io-cache
31: option cache-size 64MB
32: option page-size 128KB
33: option cache-timeout 2
34: subvolumes foo-iothreads
35: end-volume
+------------------------------------------------------------------------------+
[2009-05-27 08:00:24] N [glusterfsd.c:1152:main] glusterfs: Successfully
started
[2009-05-27 08:00:24] N [client-protocol.c:5557:client_setvolume_cbk]
foo: Connected to 192.168.168.231:6997, attached to remote volume
'foo-remote'.
[2009-05-27 08:00:24] N [client-protocol.c:5557:client_setvolume_cbk]
foo: Connected to 192.168.168.231:6997, attached to remote volume
'foo-remote'.
[2009-05-27 11:38:12] W [socket.c:1319:socket_init] trans: disabling
non-blocking IO
[2009-05-27 11:38:12] W [socket.c:1319:socket_init] trans: disabling
non-blocking IO
================================================================================
Version : glusterfs 2.0.1 built on May 26 2009 05:23:52
TLA Revision : 5c1d9108c1529a1155963cb1911f8870a674ab5b
Starting Time: 2009-05-27 11:38:12
Command line : /usr/sbin/glusterfs2 --log-level=NORMAL
--volfile-server=file01.priv --volfile-server-port=6997 /mnt/file01
PID : 16293
System name : Linux
Nodename : xxx012.bar.foo.net
Kernel Release : 2.6.18-92.1.10.el5PAE
Hardware Identifier: i686
Given volfile:
+------------------------------------------------------------------------------+
1: volume foo
2: type protocol/client
3: option transport-type tcp
4: option remote-host file01.priv # can be IP or hostname
5: option remote-port 6997
6: option remote-subvolume foo-remote
7: end-volume
8:
9: volume foo-readahead
10: type performance/read-ahead
11: option page-count 4
12: subvolumes foo
13: end-volume
14:
15: volume foo-writebehind
16: type performance/write-behind
17: option cache-size 512KB
18: subvolumes foo-readahead
19: end-volume
20:
21: volume foo-iothreads
22: type performance/io-threads
23: option autoscaling on
24: option min-threads 16
25: option max-threads 256
26: subvolumes foo-writebehind
27: end-volume
28:
29: volume foo-iocache
30: type performance/io-cache
31: option cache-size 64MB
32: option page-size 128KB
33: option cache-timeout 2
34: subvolumes foo-iothreads
35: end-volume
+------------------------------------------------------------------------------+
[2009-05-27 11:38:12] N [glusterfsd.c:1152:main] glusterfs: Successfully
started
[2009-05-27 11:38:12] N [client-protocol.c:5557:client_setvolume_cbk]
foo: Connected to 192.168.168.231:6997, attached to remote volume
'foo-remote'.
[2009-05-27 11:38:12] N [client-protocol.c:5557:client_setvolume_cbk]
foo: Connected to 192.168.168.231:6997, attached to remote volume
'foo-remote'.
Thanks,
Ioannis
-------------- next part --------------
A non-text attachment was scrubbed...
Name: iaslanidis.vcf
Type: text/x-vcard
Size: 498 bytes
Desc: not available
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20090527/4d66743b/attachment-0003.vcf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 261 bytes
Desc: OpenPGP digital signature
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20090527/4d66743b/attachment-0003.sig>
More information about the Gluster-devel
mailing list