[Gluster-users] Fwd: to mailing list gluster.org
Pranith Kumar Karampuri
pkarampu at redhat.com
Wed Jan 21 19:16:34 UTC 2015
+ quota devs
Pranith
On 01/07/2015 07:37 PM, Bekk, Klaus (IKP) wrote:
>
>> Hi all,
>>
>> we are running 4 server with Ubuntu 14.04 as glusterfs server with
>> 3.5.2-ubuntu1~trusty1.
>>
>
>> When starting quotas on any of the volumes the brick will be not
>> available after some time. So we have to disable quota on this volume
>> and stop and start this volume.
>>
>> We have the following configuration
>>
>> root at ikpsrv01:/var/log/glusterfs# gluster volume info
>>
>> Volume Name: gltestn
>>
>> Type: Distribute
>>
>> Volume ID: e50bd30d-dbc8-4279-ac9d-2e557328a643
>>
>> Status: Started
>>
>> Number of Bricks: 1
>>
>> Transport-type: tcp
>>
>> Bricks:
>>
>> Brick1: ikpsrv01:/gltest2/gl
>>
>> Volume Name: glwww1
>>
>> Type: Replicate
>>
>> Volume ID: 811e4c2a-4520-4e61-bb84-955afbdb9dff
>>
>> Status: Started
>>
>> Number of Bricks: 1 x 2 = 2
>>
>> Transport-type: tcp
>>
>> Bricks:
>>
>> Brick1: ikpsrv03:/glwwwn/gl
>>
>> Brick2: ikpsrv01:/glwwwn2/gl
>>
>> Options Reconfigured:
>>
>> features.quota: on
>>
>> Volume Name: corsika
>>
>> Type: Distribute
>>
>> Volume ID: d6e07f3e-4b35-4736-9b71-afc3224f0f29
>>
>> Status: Started
>>
>> Number of Bricks: 4
>>
>> Transport-type: tcp
>>
>> Bricks:
>>
>> Brick1: ikpsrv03:/glcors3-1/gl
>>
>> Brick2: ikpsrv03:/glcors3-2/gl
>>
>> Brick3: ikpsrv02:/glcors2-1/gl
>>
>> Brick4: ikpsrv02:/glcors2-2/gl
>>
>> Volume Name: gldata
>>
>> Type: Distribute
>>
>> Volume ID: 82485d03-9ef4-4151-a2cc-881072e871a1
>>
>> Status: Started
>>
>> Number of Bricks: 2
>>
>> Transport-type: tcp
>>
>> Bricks:
>>
>> Brick1: ikpsrv02:/gldata2-1/gl
>>
>> Brick2: ikpsrv01:/gldata1-2/gl
>>
>> Options Reconfigured:
>>
>> features.quota: on
>>
>> Volume Name: gltest
>>
>> Type: Distribute
>>
>> Volume ID: 9e69afcc-d421-41c9-824b-7d9a31240072
>>
>> Status: Started
>>
>> Number of Bricks: 2
>>
>> Transport-type: tcp
>>
>> Bricks:
>>
>> Brick1: ikpsrv01:/gltest3/gl
>>
>> Brick2: ikpsrv01:/gltest4/gl
>>
>> Options Reconfigured:
>>
>> features.quota: off
>>
>> Volume Name: glusersold
>>
>> Type: Distribute
>>
>> Volume ID: 91a79e1a-4ff3-4191-b0ae-73357105774c
>>
>> Status: Started
>>
>> Number of Bricks: 1
>>
>> Transport-type: tcp
>>
>> Bricks:
>>
>> Brick1: ikpsrv02:/glusers2-2/gl
>>
>> Volume Name: glusers
>>
>> Type: Distribute
>>
>> Volume ID: f48b1941-9c19-4311-bd9e-497660fbcc80
>>
>> Status: Started
>>
>> Number of Bricks: 1
>>
>> Transport-type: tcp
>>
>> Bricks:
>>
>> Brick1: ikpsrv02:/glusers2-1/gl
>>
>> Options Reconfigured:
>>
>> diagnostics.client-log-level: CRITICAL
>>
>> diagnostics.brick-log-level: CRITICAL
>>
>> features.quota: off
>>
>> When starting quotas on any of the volumes the brick will be not
>> available after some time. So we have to disable quota on this volume
>> and stop and start this volume.
>>
>> The command
>>
>> Gluster volume quota gltestn enable
>>
>> gives the following output in quotad.log
>>
>> [2015-01-07 12:01:55.522229] W [glusterfsd.c:1095:cleanup_and_exit]
>> (-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7fb455bd9fbd]
>> (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7fb455ead182]
>> (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xd5) [0x7fb4569a7265])))
>> 0-: received signum (15), shutting down
>>
>> [2015-01-07 12:01:56.534931] I [glusterfsd.c:1959:main]
>> 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version
>> 3.5.2 (/usr/sbin/glusterfs -s localhost --volfile-id gluster/quotad
>> -p /var/lib/glusterd/quotad/run/quotad.pid -l
>> /var/log/glusterfs/quotad.log -S
>> /var/run/fa869d5552bf340ec7506747e21b2841.socket --xlator-option
>> *replicate*.data-self-heal=off --xlator-option
>> *replicate*.metadata-self-heal=off --xlator-option
>> *replicate*.entry-self-heal=off)
>>
>> [2015-01-07 12:01:56.537061] I [socket.c:3561:socket_init]
>> 0-socket.glusterfsd: SSL support is NOT enabled
>>
>> [2015-01-07 12:01:56.537134] I [socket.c:3576:socket_init]
>> 0-socket.glusterfsd: using system polling thread
>>
>> [2015-01-07 12:01:56.537355] I [socket.c:3561:socket_init]
>> 0-glusterfs: SSL support is NOT enabled
>>
>> [2015-01-07 12:01:56.537383] I [socket.c:3576:socket_init]
>> 0-glusterfs: using system polling thread
>>
>> [2015-01-07 12:01:56.541480] I [graph.c:254:gf_add_cmdline_options]
>> 0-glwww1-replicate-0: adding option 'entry-self-heal' for volume
>> 'glwww1-replicate-0' with value 'off'
>>
>> [2015-01-07 12:01:56.541516] I [graph.c:254:gf_add_cmdline_options]
>> 0-glwww1-replicate-0: adding option 'metadata-self-heal' for volume
>> 'glwww1-replicate-0' with value 'off'
>>
>> [2015-01-07 12:01:56.541536] I [graph.c:254:gf_add_cmdline_options]
>> 0-glwww1-replicate-0: adding option 'data-self-heal' for volume
>> 'glwww1-replicate-0' with value 'off'
>>
>> [2015-01-07 12:01:56.543122] I [socket.c:3561:socket_init]
>> 0-socket.quotad: SSL support is NOT enabled
>>
>> [2015-01-07 12:01:56.543157] I [socket.c:3576:socket_init]
>> 0-socket.quotad: using system polling thread
>>
>> [2015-01-07 12:01:56.543404] I [dht-shared.c:311:dht_init_regex]
>> 0-glwww1: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>>
>> [2015-01-07 12:01:56.547124] I [socket.c:3561:socket_init]
>> 0-glwww1-client-1: SSL support is NOT enabled
>>
>> [2015-01-07 12:01:56.547158] I [socket.c:3576:socket_init]
>> 0-glwww1-client-1: using system polling thread
>>
>> [2015-01-07 12:01:56.547728] I [socket.c:3561:socket_init]
>> 0-glwww1-client-0: SSL support is NOT enabled
>>
>> [2015-01-07 12:01:56.547754] I [socket.c:3576:socket_init]
>> 0-glwww1-client-0: using system polling thread
>>
>> [2015-01-07 12:01:56.547820] I [dht-shared.c:311:dht_init_regex]
>> 0-gltestn: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>>
>> [2015-01-07 12:01:56.549471] I [socket.c:3561:socket_init]
>> 0-gltestn-client-0: SSL support is NOT enabled
>>
>> [2015-01-07 12:01:56.549501] I [socket.c:3576:socket_init]
>> 0-gltestn-client-0: using system polling thread
>>
>> [2015-01-07 12:01:56.549565] I [dht-shared.c:311:dht_init_regex]
>> 0-gldata: using regex rsync-hash-regex = ^\.(.+)\.[^.]+$
>>
>> [2015-01-07 12:01:56.551155] I [socket.c:3561:socket_init]
>> 0-gldata-client-1: SSL support is NOT enabled
>>
>> [2015-01-07 12:01:56.551184] I [socket.c:3576:socket_init]
>> 0-gldata-client-1: using system polling thread
>>
>> [2015-01-07 12:01:56.551744] I [socket.c:3561:socket_init]
>> 0-gldata-client-0: SSL support is NOT enabled
>>
>> [2015-01-07 12:01:56.551770] I [socket.c:3576:socket_init]
>> 0-gldata-client-0: using system polling thread
>>
>> [2015-01-07 12:01:56.551790] W [graph.c:329:_log_if_unknown_option]
>> 0-quotad: option 'rpc-auth.auth-glusterfs' is not recognized
>>
>> [2015-01-07 12:01:56.551819] W [graph.c:329:_log_if_unknown_option]
>> 0-quotad: option 'rpc-auth.auth-unix' is not recognized
>>
>> [2015-01-07 12:01:56.551838] W [graph.c:329:_log_if_unknown_option]
>> 0-quotad: option 'rpc-auth.auth-null' is not recognized
>>
>> [2015-01-07 12:01:56.551854] W [graph.c:329:_log_if_unknown_option]
>> 0-quotad: option 'glwww1.volume-id' is not recognized
>>
>> [2015-01-07 12:01:56.551870] W [graph.c:329:_log_if_unknown_option]
>> 0-quotad: option 'gltestn.volume-id' is not recognized
>>
>> [2015-01-07 12:01:56.551885] W [graph.c:329:_log_if_unknown_option]
>> 0-quotad: option 'gldata.volume-id' is not recognized
>>
>> [2015-01-07 12:01:56.551922] I [client.c:2294:notify]
>> 0-glwww1-client-0: parent translators are ready, attempting connect
>> on transport
>>
>> [2015-01-07 12:01:56.552579] I [client.c:2294:notify]
>> 0-glwww1-client-1: parent translators are ready, attempting connect
>> on transport
>>
>> [2015-01-07 12:01:56.553179] I [client.c:2294:notify]
>> 0-gltestn-client-0: parent translators are ready, attempting connect
>> on transport
>>
>> [2015-01-07 12:01:56.553757] I [client.c:2294:notify]
>> 0-gldata-client-0: parent translators are ready, attempting connect
>> on transport
>>
>> [2015-01-07 12:01:56.554372] I [client.c:2294:notify]
>> 0-gldata-client-1: parent translators are ready, attempting connect
>> on transport
>>
>> Final graph:
>>
>> +------------------------------------------------------------------------------+
>>
>> 1: volume glwww1-client-0
>>
>> 2: type protocol/client
>>
>> 3: option remote-host ikpsrv03
>>
>> 4: option remote-subvolume /glwwwn/gl
>>
>> 5: option transport-type socket
>>
>> 6: option username 7c5931b4-d41e-40f7-978e-4b5a40f49481
>>
>> 7: option password 6ab1661b-4752-42a4-80cc-ca0bd0167a9f
>>
>> 8: end-volume
>>
>> 9:
>>
>> 10: volume glwww1-client-1
>>
>> 11: type protocol/client
>>
>> 12: option remote-host ikpsrv01
>>
>> 13: option remote-subvolume /glwwwn2/gl
>>
>> 14: option transport-type socket
>>
>> 15: option username 7c5931b4-d41e-40f7-978e-4b5a40f49481
>>
>> 16: option password 6ab1661b-4752-42a4-80cc-ca0bd0167a9f
>>
>> 17: end-volume
>>
>> 18:
>>
>> 19: volume glwww1-replicate-0
>>
>> 20: type cluster/replicate
>>
>> 21: option data-self-heal off
>>
>> 22: option metadata-self-heal off
>>
>> 23: option entry-self-heal off
>>
>> 24: subvolumes glwww1-client-0 glwww1-client-1
>>
>> 25: end-volume
>>
>> 26:
>>
>> 27: volume glwww1
>>
>> 28: type cluster/distribute
>>
>> 29: subvolumes glwww1-replicate-0
>>
>> 30: end-volume
>>
>> 31:
>>
>> 32: volume gltestn-client-0
>>
>> 33: type protocol/client
>>
>> 34: option remote-host ikpsrv01
>>
>> 35: option remote-subvolume /gltest2/gl
>>
>> 36: option transport-type socket
>>
>> 37: option username 76c8bf85-a5c5-445d-8e04-e596b6eb173b
>>
>> 38: option password ceec81fd-5afd-4d6b-a413-00b2688f4c62
>>
>> 39: end-volume
>>
>> 40:
>>
>> 41: volume gltestn
>>
>> 42: type cluster/distribute
>>
>> 43: subvolumes gltestn-client-0
>>
>> 44: end-volume
>>
>> 45:
>>
>> 46: volume gldata-client-0
>>
>> 47: type protocol/client
>>
>> 48: option remote-host ikpsrv02
>>
>> 49: option remote-subvolume /gldata2-1/gl
>>
>> 50: option transport-type socket
>>
>> 51: option username e9dcdbec-1110-4ab2-b667-63625126b246
>>
>> 52: option password 05a9e37d-67a0-476b-88df-93d612a4e08e
>>
>> 53: end-volume
>>
>> 54:
>>
>> 55: volume gldata-client-1
>>
>> 56: type protocol/client
>>
>> 57: option remote-host ikpsrv01
>>
>> 58: option remote-subvolume /gldata1-2/gl
>>
>> 59: option transport-type socket
>>
>> 60: option username e9dcdbec-1110-4ab2-b667-63625126b246
>>
>> 61: option password 05a9e37d-67a0-476b-88df-93d612a4e08e
>>
>> 62: end-volume
>>
>> 63:
>>
>> 64: volume gldata
>>
>> 65: type cluster/distribute
>>
>> 66: subvolumes gldata-client-0 gldata-client-1
>>
>> 67: end-volume
>>
>> 68:
>>
>> 69: volume quotad
>>
>> 70: type features/quotad
>>
>> 71: option rpc-auth.auth-glusterfs on
>>
>> 72: option rpc-auth.auth-unix on
>>
>> 73: option rpc-auth.auth-null on
>>
>> 74: option transport.socket.listen-path /tmp/quotad.socket
>>
>> 75: option transport-type socket
>>
>> 76: option transport.address-family unix
>>
>> 77: option glwww1.volume-id glwww1
>>
>> 78: option gltestn.volume-id gltestn
>>
>> 79: option gldata.volume-id gldata
>>
>> 80: subvolumes glwww1 gltestn gldata
>>
>> 81: end-volume
>>
>> 82:
>>
>> +------------------------------------------------------------------------------+
>>
>> [2015-01-07 12:01:56.555618] I [rpc-clnt.c:1729:rpc_clnt_reconfig]
>> 0-gldata-client-0: changing port to 49160 (from 0)
>>
>> [2015-01-07 12:01:56.556529] I [rpc-clnt.c:1729:rpc_clnt_reconfig]
>> 0-glwww1-client-1: changing port to 49165 (from 0)
>>
>> [2015-01-07 12:01:56.556597] I [rpc-clnt.c:1729:rpc_clnt_reconfig]
>> 0-gltestn-client-0: changing port to 49166 (from 0)
>>
>> [2015-01-07 12:01:56.556643] I [rpc-clnt.c:1729:rpc_clnt_reconfig]
>> 0-gldata-client-1: changing port to 49178 (from 0)
>>
>> [2015-01-07 12:01:56.558382] W
>> [client-callback.c:31:client_cbk_fetchspec] 0-glwww1-client-0: this
>> function should not be called
>>
>> [2015-01-07 12:01:56.558484] I [glusterfsd-mgmt.c:56:mgmt_cbk_spec]
>> 0-mgmt: Volume file changed
>>
>> [2015-01-07 12:01:56.558601] I
>> [client-handshake.c:1677:select_server_supported_programs]
>> 0-gldata-client-0: Using Program GlusterFS 3.3, Num (1298437),
>> Version (330)
>>
>> [2015-01-07 12:01:56.559594] I
>> [client-handshake.c:1677:select_server_supported_programs]
>> 0-gltestn-client-0: Using Program GlusterFS 3.3, Num (1298437),
>> Version (330)
>>
>> [2015-01-07 12:01:56.559686] I
>> [client-handshake.c:1677:select_server_supported_programs]
>> 0-glwww1-client-1: Using Program GlusterFS 3.3, Num (1298437),
>> Version (330)
>>
>> [2015-01-07 12:01:56.559768] I
>> [client-handshake.c:1677:select_server_supported_programs]
>> 0-gldata-client-1: Using Program GlusterFS 3.3, Num (1298437),
>> Version (330)
>>
>> [2015-01-07 12:01:56.559852] I
>> [client-handshake.c:1462:client_setvolume_cbk] 0-gldata-client-0:
>> Connected to 141.52.67.86:49160, attached to remote volume
>> '/gldata2-1/gl'.
>>
>> [2015-01-07 12:01:56.559873] I
>> [client-handshake.c:1474:client_setvolume_cbk] 0-gldata-client-0:
>> Server and Client lk-version numbers are not same, reopening the fds
>>
>> [2015-01-07 12:01:56.559996] I
>> [client-handshake.c:1462:client_setvolume_cbk] 0-gltestn-client-0:
>> Connected to 141.52.67.85:49166, attached to remote volume '/gltest2/gl'.
>>
>> [2015-01-07 12:01:56.560017] I
>> [client-handshake.c:1474:client_setvolume_cbk] 0-gltestn-client-0:
>> Server and Client lk-version numbers are not same, reopening the fds
>>
>> [2015-01-07 12:01:56.560089] I
>> [client-handshake.c:450:client_set_lk_version_cbk] 0-gldata-client-0:
>> Server lk version = 1
>>
>> [2015-01-07 12:01:56.560139] I
>> [client-handshake.c:1462:client_setvolume_cbk] 0-glwww1-client-1:
>> Connected to 141.52.67.85:49165, attached to remote volume '/glwwwn2/gl'.
>>
>> [2015-01-07 12:01:56.560158] I
>> [client-handshake.c:1474:client_setvolume_cbk] 0-glwww1-client-1:
>> Server and Client lk-version numbers are not same, reopening the fds
>>
>> [2015-01-07 12:01:56.560204] I [afr-common.c:4131:afr_notify]
>> 0-glwww1-replicate-0: Subvolume 'glwww1-client-1' came back up; going
>> online.
>>
>> [2015-01-07 12:01:56.560262] I
>> [client-handshake.c:1462:client_setvolume_cbk] 0-gldata-client-1:
>> Connected to 141.52.67.85:49178, attached to remote volume
>> '/gldata1-2/gl'.
>>
>> [2015-01-07 12:01:56.560301] I
>> [client-handshake.c:1474:client_setvolume_cbk] 0-gldata-client-1:
>> Server and Client lk-version numbers are not same, reopening the fds
>>
>> [2015-01-07 12:01:56.560385] I
>> [client-handshake.c:450:client_set_lk_version_cbk]
>> 0-gltestn-client-0: Server lk version = 1
>>
>> [2015-01-07 12:01:56.560467] I
>> [client-handshake.c:450:client_set_lk_version_cbk] 0-glwww1-client-1:
>> Server lk version = 1
>>
>> [2015-01-07 12:01:56.560546] I
>> [client-handshake.c:450:client_set_lk_version_cbk] 0-gldata-client-1:
>> Server lk version = 1
>>
>> [2015-01-07 12:01:57.255911] I
>> [glusterfsd-mgmt.c:1307:mgmt_getspec_cbk] 0-glusterfs: No change in
>> volfile, continuing
>>
>> [2015-01-07 12:01:57.557002] I [rpc-clnt.c:1729:rpc_clnt_reconfig]
>> 0-glwww1-client-0: changing port to 49153 (from 0)
>>
>> [2015-01-07 12:01:57.557924] I
>> [client-handshake.c:1677:select_server_supported_programs]
>> 0-glwww1-client-0: Using Program GlusterFS 3.3, Num (1298437),
>> Version (330)
>>
>> [2015-01-07 12:01:57.558360] I
>> [client-handshake.c:1462:client_setvolume_cbk] 0-glwww1-client-0:
>> Connected to 141.52.67.96:49153, attached to remote volume '/glwwwn/gl'.
>>
>> [2015-01-07 12:01:57.558405] I
>> [client-handshake.c:1474:client_setvolume_cbk] 0-glwww1-client-0:
>> Server and Client lk-version numbers are not same, reopening the fds
>>
>> [2015-01-07 12:01:57.558706] I
>> [client-handshake.c:450:client_set_lk_version_cbk] 0-glwww1-client-0:
>> Server lk version = 1
>>
>> what means
>>
>> 2015-01-07 12:01:56.559594] I
>> [client-handshake.c:1677:select_server_supported_programs]
>> 0-gltestn-client-0: Using Program GlusterFS 3.3, Num (1298437),
>> Version (330)
>>
>> In the log output.
>>
>> Our glusterfs clients all have 3.5.2 glusterfs version.
>>
>> Can someone help?
>>
>> kind regards
>>
>> Klaus Bekk
>>
>> ----------------------------------------------------------
>>
>> Karlsruher Institut of Technologie (KIT)
>>
>> Institut für Kernphysik (IKP)
>>
>> Dr. Klaus Bekk
>>
>> Hermann-von-Helmholtz-Platz 1
>>
>> Campus Nord Gebäude 425
>>
>> 76344 Eggenstein-Leopoldshafen
>>
>> Telefon: +49 721 608-23382
>>
>> Fax: +49 721 608-23321
>>
>> E-Mail: Klaus.Bekk at kit.edu <mailto:Klaus.Bekk at kit.edu>
>>
>> http://www.kit.edu/
>>
>> KIT -- University of the State of Baden-Wuerttemberg and
>>
>> National Research Center of the Helmholtz Association
>>
>> ----------------------------------------------------------
>>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150122/b247d983/attachment.html>
More information about the Gluster-users
mailing list