[Gluster-users] Strange Logs
Christian Reiss
email at christian-reiss.de
Sun Feb 16 08:13:02 UTC 2020
Yes,
You are right. Ovirt might be the culprit.
The logs looked like errors or "unable to do things" to me.
Seems this is all okay, huh.
Thanks for replying and clearing that up.
-Chris.
On 14/02/2020 22:17, Artem Russakovskii wrote:
> I've been seeing the same thing happen, and in our case, it's because of
> running a script that checks gluster from time to time
> (https://github.com/jtopjian/scripts/blob/master/gluster/gluster-status.sh
> in our case).
>
> Do you have a job that runs and periodically checks for gluster health?
>
> Sincerely,
> Artem
>
> --
> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
> <http://www.apkmirror.com/>, Illogical Robot LLC
> beerpla.net <http://beerpla.net/> | @ArtemR <http://twitter.com/ArtemR>
>
>
> On Fri, Feb 14, 2020 at 3:10 AM Christian Reiss
> <email at christian-reiss.de <mailto:email at christian-reiss.de>> wrote:
>
> Hey folks,
>
> my logs are constantly (every few secs, continuously) swamped with
>
> [2020-02-14 11:05:20.258542] I [MSGID: 114046]
> [client-handshake.c:1106:client_setvolume_cbk] 0-ssd_storage-client-0:
> Connected to ssd_storage-client-0, attached to remote volume
> '/gluster_bricks/node01.company.com/gluster
> <http://node01.company.com/gluster>'.
> [2020-02-14 11:05:20.258559] I [MSGID: 108005]
> [afr-common.c:5280:__afr_handle_child_up_event]
> 0-ssd_storage-replicate-0: Subvolume 'ssd_storage-client-0' came back
> up; going online.
> [2020-02-14 11:05:20.258920] I [rpc-clnt.c:1963:rpc_clnt_reconfig]
> 0-ssd_storage-client-2: changing port to 49152 (from 0)
> [2020-02-14 11:05:20.259132] I [socket.c:864:__socket_shutdown]
> 0-ssd_storage-client-2: intentional socket shutdown(11)
> [2020-02-14 11:05:20.260010] I [MSGID: 114057]
> [client-handshake.c:1376:select_server_supported_programs]
> 0-ssd_storage-client-1: Using Program GlusterFS 4.x v1, Num (1298437),
> Version (400)
> [2020-02-14 11:05:20.261077] I [MSGID: 114046]
> [client-handshake.c:1106:client_setvolume_cbk] 0-ssd_storage-client-1:
> Connected to ssd_storage-client-1, attached to remote volume
> '/gluster_bricks/node02.company.com/gluster
> <http://node02.company.com/gluster>'.
> [2020-02-14 11:05:20.261089] I [MSGID: 108002]
> [afr-common.c:5647:afr_notify] 0-ssd_storage-replicate-0: Client-quorum
> is met
> [2020-02-14 11:05:20.262005] I [MSGID: 114057]
> [client-handshake.c:1376:select_server_supported_programs]
> 0-ssd_storage-client-2: Using Program GlusterFS 4.x v1, Num (1298437),
> Version (400)
> [2020-02-14 11:05:20.262685] I [MSGID: 114046]
> [client-handshake.c:1106:client_setvolume_cbk] 0-ssd_storage-client-2:
> Connected to ssd_storage-client-2, attached to remote volume
> '/gluster_bricks/node03.company.com/gluster
> <http://node03.company.com/gluster>'.
> [2020-02-14 11:05:20.263909] I [MSGID: 108031]
> [afr-common.c:2580:afr_local_discovery_cbk] 0-ssd_storage-replicate-0:
> selecting local read_child ssd_storage-client-0
> [2020-02-14 11:05:20.264124] I [MSGID: 104041]
> [glfs-resolve.c:954:__glfs_active_subvol] 0-ssd_storage: switched to
> graph 6e6f6465-3031-2e64-632d-6475732e6461 (0)
> [2020-02-14 11:05:22.407851] I [MSGID: 114007]
> [client.c:2478:client_check_remote_host] 0-ssd_storage-snapd-client:
> Remote host is not set. Assuming the volfile server as remote host
> [Invalid argument]
> [2020-02-14 11:05:22.409711] I [MSGID: 104045]
> [glfs-master.c:80:notify]
> 0-gfapi: New graph 6e6f6465-3031-2e64-632d-6475732e6461 (0) coming up
> [2020-02-14 11:05:22.409738] I [MSGID: 114020] [client.c:2436:notify]
> 0-ssd_storage-client-0: parent translators are ready, attempting
> connect
> on transport
> [2020-02-14 11:05:22.412949] I [MSGID: 114020] [client.c:2436:notify]
> 0-ssd_storage-client-1: parent translators are ready, attempting
> connect
> on transport
> [2020-02-14 11:05:22.413130] I [rpc-clnt.c:1963:rpc_clnt_reconfig]
> 0-ssd_storage-client-0: changing port to 49152 (from 0)
> [2020-02-14 11:05:22.413154] I [socket.c:864:__socket_shutdown]
> 0-ssd_storage-client-0: intentional socket shutdown(10)
> [2020-02-14 11:05:22.415534] I [MSGID: 114020] [client.c:2436:notify]
> 0-ssd_storage-client-2: parent translators are ready, attempting
> connect
> on transport
> [2020-02-14 11:05:22.417836] I [MSGID: 114057]
> [client-handshake.c:1376:select_server_supported_programs]
> 0-ssd_storage-client-0: Using Program GlusterFS 4.x v1, Num (1298437),
> Version (400)
> [2020-02-14 11:05:22.418036] I [rpc-clnt.c:1963:rpc_clnt_reconfig]
> 0-ssd_storage-client-1: changing port to 49152 (from 0)
> [2020-02-14 11:05:22.418095] I [socket.c:864:__socket_shutdown]
> 0-ssd_storage-client-1: intentional socket shutdown(12)
> [2020-02-14 11:05:22.420029] I [MSGID: 114020] [client.c:2436:notify]
> 0-ssd_storage-snapd-client: parent translators are ready, attempting
> connect on transport
> [2020-02-14 11:05:22.420533] E [MSGID: 101075]
> [common-utils.c:505:gf_resolve_ip6] 0-resolver: getaddrinfo failed
> (family:2) (Name or service not known)
> [2020-02-14 11:05:22.420545] E
> [name.c:266:af_inet_client_get_remote_sockaddr]
> 0-ssd_storage-snapd-client: DNS resolution failed on host
> /var/run/glusterd.socket
> Final graph:
> +------------------------------------------------------------------------------+
> 1: volume ssd_storage-client-0
> 2: type protocol/client
> 3: option opversion 70000
> 4: option clnt-lk-version 1
> 5: option volfile-checksum 0
> 6: option volfile-key ssd_storage
> 7: option client-version 7.0
> 8: option process-name gfapi.glfsheal
> 9: option process-uuid
> CTX_ID:50cec79e-6028-4e6f-b8ed-dda9db36b2d0-GRAPH_ID:0-PID:24926-HOST:node01.company.com-PC_NAME:ssd_storage-client-0-RECON_NO:-0
> 10: option fops-version 1298437
> 11: option ping-timeout 42
> 12: option remote-host node01.company.com
> <http://node01.company.com>
> 13: option remote-subvolume
> /gluster_bricks/node01.company.com/gluster
> <http://node01.company.com/gluster>
> 14: option transport-type socket
> 15: option transport.address-family inet
> 16: option username 96bcf4d4-932f-4654-86c3-470a081d5021
> 17: option password 069e7ee9-b17d-4228-a612-b0f33588a9ec
> 18: option transport.socket.ssl-enabled off
> 19: option transport.tcp-user-timeout 0
> 20: option transport.socket.keepalive-time 20
> 21: option transport.socket.keepalive-interval 2
> 22: option transport.socket.keepalive-count 9
> 23: option send-gids true
> 24: end-volume
> 25:
> 26: volume ssd_storage-client-1
> 27: type protocol/client
> 28: option ping-timeout 42
> 29: option remote-host node02.company.com
> <http://node02.company.com>
> 30: option remote-subvolume
> /gluster_bricks/node02.company.com/gluster
> <http://node02.company.com/gluster>
> 31: option transport-type socket
> 32: option transport.address-family inet
> 33: option username 96bcf4d4-932f-4654-86c3-470a081d5021
> 34: option password 069e7ee9-b17d-4228-a612-b0f33588a9ec
> 35: option transport.socket.ssl-enabled off
> 36: option transport.tcp-user-timeout 0
> 37: option transport.socket.keepalive-time 20
> 38: option transport.socket.keepalive-interval 2
> 39: option transport.socket.keepalive-count 9
> 40: option send-gids true
> 41: end-volume
> 42:
> 43: volume ssd_storage-client-2
> 44: type protocol/client
> 45: option ping-timeout 42
> 46: option remote-host node03.company.com
> <http://node03.company.com>
> 47: option remote-subvolume
> /gluster_bricks/node03.company.com/gluster
> <http://node03.company.com/gluster>
> 48: option transport-type socket
> 49: option transport.address-family inet
> 50: option username 96bcf4d4-932f-4654-86c3-470a081d5021
> 51: option password 069e7ee9-b17d-4228-a612-b0f33588a9ec
> 52: option transport.socket.ssl-enabled off
> 53: option transport.tcp-user-timeout 0
> 54: option transport.socket.keepalive-time 20
> 55: option transport.socket.keepalive-interval 2
> 56: option transport.socket.keepalive-count 9
> 57: option send-gids true
> 58: end-volume
> 59:
> 60: volume ssd_storage-replicate-0
> 61: type cluster/replicate
> 62: option background-self-heal-count 0
> 63: option afr-pending-xattr
> ssd_storage-client-0,ssd_storage-client-1,ssd_storage-client-2
> 64: option metadata-self-heal on
> 65: option data-self-heal on
> 66: option entry-self-heal on
> 67: option data-self-heal-algorithm full
> 68: option use-compound-fops off
> 69: subvolumes ssd_storage-client-0 ssd_storage-client-1
> ssd_storage-client-2
> 70: end-volume
> 71:
> 72: volume ssd_storage-dht
> 73: type cluster/distribute
> 74: option readdir-optimize on
> 75: option lock-migration off
> 76: option force-migration off
> 77: subvolumes ssd_storage-replicate-0
> 78: end-volume
> 79:
> 80: volume ssd_storage-utime
> 81: type features/utime
> 82: option noatime on
> 83: subvolumes ssd_storage-dht
> 84: end-volume
> 85:
> 86: volume ssd_storage-write-behind
> 87: type performance/write-behind
> 88: subvolumes ssd_storage-utime
> 89: end-volume
> 90:
> 91: volume ssd_storage-read-ahead
> 92: type performance/read-ahead
> 93: subvolumes ssd_storage-write-behind
> 94: end-volume
> 95:
> 96: volume ssd_storage-readdir-ahead
> 97: type performance/readdir-ahead
> 98: option parallel-readdir off
> 99: option rda-request-size 131072
> 100: option rda-cache-limit 10MB
> 101: subvolumes ssd_storage-read-ahead
> 102: end-volume
> 103:
> 104: volume ssd_storage-io-cache
> 105: type performance/io-cache
> 106: subvolumes ssd_storage-readdir-ahead
> 107: end-volume
> 108:
> 109: volume ssd_storage-open-behind
> 110: type performance/open-behind
> 111: subvolumes ssd_storage-io-cache
> 112: end-volume
> 113:
> 114: volume ssd_storage-quick-read
> 115: type performance/quick-read
> 116: subvolumes ssd_storage-open-behind
> 117: end-volume
> 118:
> 119: volume ssd_storage-md-cache
> 120: type performance/md-cache
> 121: subvolumes ssd_storage-quick-read
> 122: end-volume
> 123:
> 124: volume ssd_storage-snapd-client
> 125: type protocol/client
> 126: option remote-host /var/run/glusterd.socket
> 127: option ping-timeout 42
> 128: option remote-subvolume snapd-ssd_storage
> 129: option transport-type socket
> 130: option transport.address-family inet
> 131: option username 96bcf4d4-932f-4654-86c3-470a081d5021
> 132: option password 069e7ee9-b17d-4228-a612-b0f33588a9ec
> 133: option transport.socket.ssl-enabled off
> 134: option transport.tcp-user-timeout 0
> 135: option transport.socket.keepalive-time 20
> 136: option transport.socket.keepalive-interval 2
> 137: option transport.socket.keepalive-count 9
> 138: option send-gids true
> 139: end-volume
> 140:
> 141: volume ssd_storage-snapview-client
> 142: type features/snapview-client
> 143: option snapshot-directory .snaps
> 144: option show-snapshot-directory on
> 145: subvolumes ssd_storage-md-cache ssd_storage-snapd-client
> 146: end-volume
> 147:
> 148: volume ssd_storage
> 149: type debug/io-stats
> 150: option log-level INFO
> 151: option threads 16
> 152: option latency-measurement off
> 153: option count-fop-hits off
> 154: option global-threading off
> 155: subvolumes ssd_storage-snapview-client
> 156: end-volume
> 157:
> 158: volume meta-autoload
> 159: type meta
> 160: subvolumes ssd_storage
> 161: end-volume
> 162:
> +------------------------------------------------------------------------------+
> [2020-02-14 11:05:22.421366] I [MSGID: 114046]
> [client-handshake.c:1106:client_setvolume_cbk] 0-ssd_storage-client-0:
> Connected to ssd_storage-client-0, attached to remote volume
> '/gluster_bricks/node01.company.com/gluster
> <http://node01.company.com/gluster>'.
> [2020-02-14 11:05:22.421379] I [MSGID: 108005]
> [afr-common.c:5280:__afr_handle_child_up_event]
> 0-ssd_storage-replicate-0: Subvolume 'ssd_storage-client-0' came back
> up; going online.
> [2020-02-14 11:05:22.421669] I [rpc-clnt.c:1963:rpc_clnt_reconfig]
> 0-ssd_storage-client-2: changing port to 49152 (from 0)
> [2020-02-14 11:05:22.421686] I [socket.c:864:__socket_shutdown]
> 0-ssd_storage-client-2: intentional socket shutdown(11)
> [2020-02-14 11:05:22.422460] I [MSGID: 114057]
> [client-handshake.c:1376:select_server_supported_programs]
> 0-ssd_storage-client-1: Using Program GlusterFS 4.x v1, Num (1298437),
> Version (400)
> [2020-02-14 11:05:22.423377] I [MSGID: 114046]
> [client-handshake.c:1106:client_setvolume_cbk] 0-ssd_storage-client-1:
> Connected to ssd_storage-client-1, attached to remote volume
> '/gluster_bricks/node02.company.com/gluster
> <http://node02.company.com/gluster>'.
> [2020-02-14 11:05:22.423391] I [MSGID: 108002]
> [afr-common.c:5647:afr_notify] 0-ssd_storage-replicate-0: Client-quorum
> is met
> [2020-02-14 11:05:22.424586] I [MSGID: 114057]
> [client-handshake.c:1376:select_server_supported_programs]
> 0-ssd_storage-client-2: Using Program GlusterFS 4.x v1, Num (1298437),
> Version (400)
> [2020-02-14 11:05:22.425323] I [MSGID: 114046]
> [client-handshake.c:1106:client_setvolume_cbk] 0-ssd_storage-client-2:
> Connected to ssd_storage-client-2, attached to remote volume
> '/gluster_bricks/node03.company.com/gluster
> <http://node03.company.com/gluster>'.
> [2020-02-14 11:05:22.426613] I [MSGID: 108031]
> [afr-common.c:2580:afr_local_discovery_cbk] 0-ssd_storage-replicate-0:
> selecting local read_child ssd_storage-client-0
> [2020-02-14 11:05:22.426758] I [MSGID: 104041]
> [glfs-resolve.c:954:__glfs_active_subvol] 0-ssd_storage: switched to
> graph 6e6f6465-3031-2e64-632d-6475732e6461 (0)
>
>
> Can you guys make any sense out of this? 5 unsynced entries remain.
>
> --
> with kind regards,
> mit freundlichen Gruessen,
>
> Christian Reiss
>
> ________
>
> Community Meeting Calendar:
>
> APAC Schedule -
> Every 2nd and 4th Tuesday at 11:30 AM IST
> Bridge: https://bluejeans.com/441850968
>
> NA/EMEA Schedule -
> Every 1st and 3rd Tuesday at 01:00 PM EDT
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
--
with kind regards,
mit freundlichen Gruessen,
Christian Reiss
More information about the Gluster-users
mailing list