[Gluster-users] Fwd: to mailing list gluster.org
Bekk, Klaus (IKP)
klaus.bekk at kit.edu
Wed Jan 7 14:07:03 UTC 2015
Hi all,
we are running 4 server with Ubuntu 14.04 as glusterfs server with 3.5.2-ubuntu1~trusty1.
When starting quotas on any of the volumes the brick will be not available after some time. So we have to disable quota on this volume and stop and start this volume.
We have the following configuration
root at ikpsrv01:/var/log/glusterfs# gluster volume info
Volume Name: gltestn
Type: Distribute
Volume ID: e50bd30d-dbc8-4279-ac9d-2e557328a643
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: ikpsrv01:/gltest2/gl
Volume Name: glwww1
Type: Replicate
Volume ID: 811e4c2a-4520-4e61-bb84-955afbdb9dff
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: ikpsrv03:/glwwwn/gl
Brick2: ikpsrv01:/glwwwn2/gl
Options Reconfigured:
features.quota: on
Volume Name: corsika
Type: Distribute
Volume ID: d6e07f3e-4b35-4736-9b71-afc3224f0f29
Status: Started
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: ikpsrv03:/glcors3-1/gl
Brick2: ikpsrv03:/glcors3-2/gl
Brick3: ikpsrv02:/glcors2-1/gl
Brick4: ikpsrv02:/glcors2-2/gl
Volume Name: gldata
Type: Distribute
Volume ID: 82485d03-9ef4-4151-a2cc-881072e871a1
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: ikpsrv02:/gldata2-1/gl
Brick2: ikpsrv01:/gldata1-2/gl
Options Reconfigured:
features.quota: on
Volume Name: gltest
Type: Distribute
Volume ID: 9e69afcc-d421-41c9-824b-7d9a31240072
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: ikpsrv01:/gltest3/gl
Brick2: ikpsrv01:/gltest4/gl
Options Reconfigured:
features.quota: off
Volume Name: glusersold
Type: Distribute
Volume ID: 91a79e1a-4ff3-4191-b0ae-73357105774c
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: ikpsrv02:/glusers2-2/gl
Volume Name: glusers
Type: Distribute
Volume ID: f48b1941-9c19-4311-bd9e-497660fbcc80
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: ikpsrv02:/glusers2-1/gl
Options Reconfigured:
diagnostics.client-log-level: CRITICAL
diagnostics.brick-log-level: CRITICAL
features.quota: off
When starting quotas on any of the volumes the brick will be not available after some time. So we have to disable quota on this volume and stop and start this volume.
The command
Gluster volume quota gltestn enable
gives the following output in quotad.log
[2015-01-07 12:01:55.522229] W [glusterfsd.c:1095:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fb455bd9fbd] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7fb455ead182] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xd5) [0x7fb4569a7265]))) 0-: received signum (15), shutting down
[2015-01-07 12:01:56.534931] I [glusterfsd.c:1959:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.5.2 (/usr/sbin/glusterfs -s localhost --volfile-id gluster/quotad -p /var/lib/glusterd/quotad/run/quotad.pid -l /var/log/glusterfs/quotad.log -S /var/run/fa869d5552bf340ec7506747e21b2841.socket --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off)
[2015-01-07 12:01:56.537061] I [socket.c:3561:socket_init] 0-socket.glusterfsd: SSL support is NOT enabled
[2015-01-07 12:01:56.537134] I [socket.c:3576:socket_init] 0-socket.glusterfsd: using system polling thread
[2015-01-07 12:01:56.537355] I [socket.c:3561:socket_init] 0-glusterfs: SSL support is NOT enabled
[2015-01-07 12:01:56.537383] I [socket.c:3576:socket_init] 0-glusterfs: using system polling thread
[2015-01-07 12:01:56.541480] I [graph.c:254:gf_add_cmdline_options] 0-glwww1-replicate-0: adding option 'entry-self-heal' for volume 'glwww1-replicate-0' with value 'off'
[2015-01-07 12:01:56.541516] I [graph.c:254:gf_add_cmdline_options] 0-glwww1-replicate-0: adding option 'metadata-self-heal' for volume 'glwww1-replicate-0' with value 'off'
[2015-01-07 12:01:56.541536] I [graph.c:254:gf_add_cmdline_options] 0-glwww1-replicate-0: adding option 'data-self-heal' for volume 'glwww1-replicate-0' with value 'off'
[2015-01-07 12:01:56.543122] I [socket.c:3561:socket_init] 0-socket.quotad: SSL support is NOT enabled
[2015-01-07 12:01:56.543157] I [socket.c:3576:socket_init] 0-socket.quotad: using system polling thread
[2015-01-07 12:01:56.543404] I [dht-shared.c:311:dht_init_regex] 0-glwww1: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
[2015-01-07 12:01:56.547124] I [socket.c:3561:socket_init] 0-glwww1-client-1: SSL support is NOT enabled
[2015-01-07 12:01:56.547158] I [socket.c:3576:socket_init] 0-glwww1-client-1: using system polling thread
[2015-01-07 12:01:56.547728] I [socket.c:3561:socket_init] 0-glwww1-client-0: SSL support is NOT enabled
[2015-01-07 12:01:56.547754] I [socket.c:3576:socket_init] 0-glwww1-client-0: using system polling thread
[2015-01-07 12:01:56.547820] I [dht-shared.c:311:dht_init_regex] 0-gltestn: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
[2015-01-07 12:01:56.549471] I [socket.c:3561:socket_init] 0-gltestn-client-0: SSL support is NOT enabled
[2015-01-07 12:01:56.549501] I [socket.c:3576:socket_init] 0-gltestn-client-0: using system polling thread
[2015-01-07 12:01:56.549565] I [dht-shared.c:311:dht_init_regex] 0-gldata: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
[2015-01-07 12:01:56.551155] I [socket.c:3561:socket_init] 0-gldata-client-1: SSL support is NOT enabled
[2015-01-07 12:01:56.551184] I [socket.c:3576:socket_init] 0-gldata-client-1: using system polling thread
[2015-01-07 12:01:56.551744] I [socket.c:3561:socket_init] 0-gldata-client-0: SSL support is NOT enabled
[2015-01-07 12:01:56.551770] I [socket.c:3576:socket_init] 0-gldata-client-0: using system polling thread
[2015-01-07 12:01:56.551790] W [graph.c:329:_log_if_unknown_option] 0-quotad: option 'rpc-auth.auth-glusterfs' is not recognized
[2015-01-07 12:01:56.551819] W [graph.c:329:_log_if_unknown_option] 0-quotad: option 'rpc-auth.auth-unix' is not recognized
[2015-01-07 12:01:56.551838] W [graph.c:329:_log_if_unknown_option] 0-quotad: option 'rpc-auth.auth-null' is not recognized
[2015-01-07 12:01:56.551854] W [graph.c:329:_log_if_unknown_option] 0-quotad: option 'glwww1.volume-id' is not recognized
[2015-01-07 12:01:56.551870] W [graph.c:329:_log_if_unknown_option] 0-quotad: option 'gltestn.volume-id' is not recognized
[2015-01-07 12:01:56.551885] W [graph.c:329:_log_if_unknown_option] 0-quotad: option 'gldata.volume-id' is not recognized
[2015-01-07 12:01:56.551922] I [client.c:2294:notify] 0-glwww1-client-0: parent translators are ready, attempting connect on transport
[2015-01-07 12:01:56.552579] I [client.c:2294:notify] 0-glwww1-client-1: parent translators are ready, attempting connect on transport
[2015-01-07 12:01:56.553179] I [client.c:2294:notify] 0-gltestn-client-0: parent translators are ready, attempting connect on transport
[2015-01-07 12:01:56.553757] I [client.c:2294:notify] 0-gldata-client-0: parent translators are ready, attempting connect on transport
[2015-01-07 12:01:56.554372] I [client.c:2294:notify] 0-gldata-client-1: parent translators are ready, attempting connect on transport
Final graph:
+------------------------------------------------------------------------------+
1: volume glwww1-client-0
2: type protocol/client
3: option remote-host ikpsrv03
4: option remote-subvolume /glwwwn/gl
5: option transport-type socket
6: option username 7c5931b4-d41e-40f7-978e-4b5a40f49481
7: option password 6ab1661b-4752-42a4-80cc-ca0bd0167a9f
8: end-volume
9:
10: volume glwww1-client-1
11: type protocol/client
12: option remote-host ikpsrv01
13: option remote-subvolume /glwwwn2/gl
14: option transport-type socket
15: option username 7c5931b4-d41e-40f7-978e-4b5a40f49481
16: option password 6ab1661b-4752-42a4-80cc-ca0bd0167a9f
17: end-volume
18:
19: volume glwww1-replicate-0
20: type cluster/replicate
21: option data-self-heal off
22: option metadata-self-heal off
23: option entry-self-heal off
24: subvolumes glwww1-client-0 glwww1-client-1
25: end-volume
26:
27: volume glwww1
28: type cluster/distribute
29: subvolumes glwww1-replicate-0
30: end-volume
31:
32: volume gltestn-client-0
33: type protocol/client
34: option remote-host ikpsrv01
35: option remote-subvolume /gltest2/gl
36: option transport-type socket
37: option username 76c8bf85-a5c5-445d-8e04-e596b6eb173b
38: option password ceec81fd-5afd-4d6b-a413-00b2688f4c62
39: end-volume
40:
41: volume gltestn
42: type cluster/distribute
43: subvolumes gltestn-client-0
44: end-volume
45:
46: volume gldata-client-0
47: type protocol/client
48: option remote-host ikpsrv02
49: option remote-subvolume /gldata2-1/gl
50: option transport-type socket
51: option username e9dcdbec-1110-4ab2-b667-63625126b246
52: option password 05a9e37d-67a0-476b-88df-93d612a4e08e
53: end-volume
54:
55: volume gldata-client-1
56: type protocol/client
57: option remote-host ikpsrv01
58: option remote-subvolume /gldata1-2/gl
59: option transport-type socket
60: option username e9dcdbec-1110-4ab2-b667-63625126b246
61: option password 05a9e37d-67a0-476b-88df-93d612a4e08e
62: end-volume
63:
64: volume gldata
65: type cluster/distribute
66: subvolumes gldata-client-0 gldata-client-1
67: end-volume
68:
69: volume quotad
70: type features/quotad
71: option rpc-auth.auth-glusterfs on
72: option rpc-auth.auth-unix on
73: option rpc-auth.auth-null on
74: option transport.socket.listen-path /tmp/quotad.socket
75: option transport-type socket
76: option transport.address-family unix
77: option glwww1.volume-id glwww1
78: option gltestn.volume-id gltestn
79: option gldata.volume-id gldata
80: subvolumes glwww1 gltestn gldata
81: end-volume
82:
+------------------------------------------------------------------------------+
[2015-01-07 12:01:56.555618] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-gldata-client-0: changing port to 49160 (from 0)
[2015-01-07 12:01:56.556529] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-glwww1-client-1: changing port to 49165 (from 0)
[2015-01-07 12:01:56.556597] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-gltestn-client-0: changing port to 49166 (from 0)
[2015-01-07 12:01:56.556643] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-gldata-client-1: changing port to 49178 (from 0)
[2015-01-07 12:01:56.558382] W [client-callback.c:31:client_cbk_fetchspec] 0-glwww1-client-0: this function should not be called
[2015-01-07 12:01:56.558484] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2015-01-07 12:01:56.558601] I [client-handshake.c:1677:select_server_supported_programs] 0-gldata-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-01-07 12:01:56.559594] I [client-handshake.c:1677:select_server_supported_programs] 0-gltestn-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-01-07 12:01:56.559686] I [client-handshake.c:1677:select_server_supported_programs] 0-glwww1-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-01-07 12:01:56.559768] I [client-handshake.c:1677:select_server_supported_programs] 0-gldata-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-01-07 12:01:56.559852] I [client-handshake.c:1462:client_setvolume_cbk] 0-gldata-client-0: Connected to 141.52.67.86:49160, attached to remote volume '/gldata2-1/gl'.
[2015-01-07 12:01:56.559873] I [client-handshake.c:1474:client_setvolume_cbk] 0-gldata-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2015-01-07 12:01:56.559996] I [client-handshake.c:1462:client_setvolume_cbk] 0-gltestn-client-0: Connected to 141.52.67.85:49166, attached to remote volume '/gltest2/gl'.
[2015-01-07 12:01:56.560017] I [client-handshake.c:1474:client_setvolume_cbk] 0-gltestn-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2015-01-07 12:01:56.560089] I [client-handshake.c:450:client_set_lk_version_cbk] 0-gldata-client-0: Server lk version = 1
[2015-01-07 12:01:56.560139] I [client-handshake.c:1462:client_setvolume_cbk] 0-glwww1-client-1: Connected to 141.52.67.85:49165, attached to remote volume '/glwwwn2/gl'.
[2015-01-07 12:01:56.560158] I [client-handshake.c:1474:client_setvolume_cbk] 0-glwww1-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2015-01-07 12:01:56.560204] I [afr-common.c:4131:afr_notify] 0-glwww1-replicate-0: Subvolume 'glwww1-client-1' came back up; going online.
[2015-01-07 12:01:56.560262] I [client-handshake.c:1462:client_setvolume_cbk] 0-gldata-client-1: Connected to 141.52.67.85:49178, attached to remote volume '/gldata1-2/gl'.
[2015-01-07 12:01:56.560301] I [client-handshake.c:1474:client_setvolume_cbk] 0-gldata-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2015-01-07 12:01:56.560385] I [client-handshake.c:450:client_set_lk_version_cbk] 0-gltestn-client-0: Server lk version = 1
[2015-01-07 12:01:56.560467] I [client-handshake.c:450:client_set_lk_version_cbk] 0-glwww1-client-1: Server lk version = 1
[2015-01-07 12:01:56.560546] I [client-handshake.c:450:client_set_lk_version_cbk] 0-gldata-client-1: Server lk version = 1
[2015-01-07 12:01:57.255911] I [glusterfsd-mgmt.c:1307:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2015-01-07 12:01:57.557002] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-glwww1-client-0: changing port to 49153 (from 0)
[2015-01-07 12:01:57.557924] I [client-handshake.c:1677:select_server_supported_programs] 0-glwww1-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2015-01-07 12:01:57.558360] I [client-handshake.c:1462:client_setvolume_cbk] 0-glwww1-client-0: Connected to 141.52.67.96:49153, attached to remote volume '/glwwwn/gl'.
[2015-01-07 12:01:57.558405] I [client-handshake.c:1474:client_setvolume_cbk] 0-glwww1-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2015-01-07 12:01:57.558706] I [client-handshake.c:450:client_set_lk_version_cbk] 0-glwww1-client-0: Server lk version = 1
what means
2015-01-07 12:01:56.559594] I [client-handshake.c:1677:select_server_supported_programs] 0-gltestn-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
In the log output.
Our glusterfs clients all have 3.5.2 glusterfs version.
Can someone help?
kind regards
Klaus Bekk
----------------------------------------------------------
Karlsruher Institut of Technologie (KIT)
Institut für Kernphysik (IKP)
Dr. Klaus Bekk
Hermann-von-Helmholtz-Platz 1
Campus Nord Gebäude 425
76344 Eggenstein-Leopoldshafen
Telefon: +49 721 608-23382
Fax: +49 721 608-23321
E-Mail: Klaus.Bekk at kit.edu<mailto:Klaus.Bekk at kit.edu>
http://www.kit.edu/
KIT – University of the State of Baden-Wuerttemberg and
National Research Center of the Helmholtz Association
----------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150107/008d34fd/attachment.html>
More information about the Gluster-users
mailing list