[Gluster-users] Shared Storage is unmount after stopping glusterd

Tue Nov 27 10:14:20 UTC 2018

Hello Rafi,
a few days ago I had the scenario described below. Here are the log
entries. As you can see glusterd was stopped at 15:22:48. The same time
FUSE of shared_storage was stopped. It seems to be nearly the same error
message. You can also see that I restarted glusterd manually but FUSE of
shared_storage was not restarted

*#-> /var/log/glusterfs/glusterd.log[2018-11-21 15:22:37.406032] I [MSGID:
106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list reqThe message "I [MSGID: 106488]
[glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req" repeated 7 times between [2018-11-21 15:21:45.377310]
and [2018-11-21 15:22:39.382300][2018-11-21 15:22:48.623951] W
[glusterfsd.c:1514:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7559)
[0x7fb1e5f11559] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xfd)
[0x561c3626340d] -->/usr/sbin/glusterd(cleanup_and_exit+0x5a)
[0x561c362631fa] ) 0-: received signum (15), shutting down[2018-11-21
15:22:53.491120] I [MSGID: 100030] [glusterfsd.c:2741:main]
0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 4.1.5
(args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level
INFO)[2018-11-21 15:22:53.494326] I [MSGID: 106478] [glusterd.c:1423:init]
0-management: Maximum allowed open file descriptors set to 65536[2018-11-21
15:22:53.494372] I [MSGID: 106479] [glusterd.c:1481:init] 0-management:
Using /var/lib/glusterd as working directory[2018-11-21 15:22:53.494393] I
[MSGID: 106479] [glusterd.c:1486:init] 0-management: Using /var/run/gluster
as pid file working directory[2018-11-21 15:22:53.496564] I
[socket.c:4477:socket_init] 0-socket.management: SSL support on the I/O
path is ENABLED[2018-11-21 15:22:53.496598] I [socket.c:4480:socket_init]
0-socket.management: SSL support for glusterd is ENABLED[2018-11-21
15:22:53.496607] I [socket.c:4497:socket_init] 0-socket.management: using
private polling thread[2018-11-21 15:22:53.496615] I
[socket.c:4507:socket_init] 0-socket.management: using certificate depth
1[2018-11-21 15:22:53.496891] E [socket.c:4554:socket_init]
0-socket.management: failed to open /etc/ssl/dhparam.pem, DH ciphers are
disabled[2018-11-21 15:22:53.499933] W [MSGID: 103071]
[rdma.c:4629:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event
channel creation failed [No such device][2018-11-21 15:22:53.499970] W
[MSGID: 103055] [rdma.c:4938:init] 0-rdma.management: Failed to initialize
IB Device[2018-11-21 15:22:53.499983] W
[rpc-transport.c:351:rpc_transport_load] 0-rpc-transport: 'rdma'
initialization failed[2018-11-21 15:22:53.500148] W
[rpcsvc.c:1781:rpcsvc_create_listener] 0-rpc-service: cannot create
listener, initing the transport failed[2018-11-21 15:22:53.500174] E
[MSGID: 106244] [glusterd.c:1764:init] 0-management: creation of 1
listeners failed, continuing with succeeded transport[2018-11-21
15:22:53.500361] I [socket.c:4480:socket_init] 0-socket.management: SSL
support for glusterd is ENABLED[2018-11-21 15:22:53.500373] I
[socket.c:4507:socket_init] 0-socket.management: using certificate depth
1[2018-11-21 15:22:53.500615] E [socket.c:4554:socket_init]
0-socket.management: failed to open /etc/ssl/dhparam.pem, DH ciphers are
disabled[2018-11-21 15:22:55.353929] I [MSGID: 106513]
[glusterd-store.c:2240:glusterd_restore_op_version] 0-glusterd: retrieved
op-version: 40100[2018-11-21 15:22:55.423433] I [MSGID: 106544]
[glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID:
7e7fd30a-94c9-4c21-9c4b-3817508ef121[2018-11-21 15:22:55.429214] I [MSGID:
106498] [glusterd-handler.c:3614:glusterd_friend_add_from_peerinfo]
0-management: connect returned 0[2018-11-21 15:22:55.429317] I [MSGID:
106498] [glusterd-handler.c:3614:glusterd_friend_add_from_peerinfo]
0-management: connect returned 0[2018-11-21 15:22:55.429379] W [MSGID:
106061] [glusterd-handler.c:3408:glusterd_transport_inet_options_build]
0-glusterd: Failed to get tcp-user-timeout[2018-11-21 15:22:55.429443] I
[rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting
frame-timeout to 600[2018-11-21 15:22:55.429623] I
[socket.c:4477:socket_init] 0-management: SSL support on the I/O path is
ENABLED[2018-11-21 15:22:55.429635] I [socket.c:4480:socket_init]
0-management: SSL support for glusterd is ENABLED[2018-11-21
15:22:55.429643] I [socket.c:4497:socket_init] 0-management: using private
polling thread[2018-11-21 15:22:55.429650] I [socket.c:4507:socket_init]
0-management: using certificate depth 1[2018-11-21 15:22:55.429914] E
[socket.c:4554:socket_init] 0-management: failed to open
/etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:22:55.431676] I
[rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting
frame-timeout to 600[2018-11-21 15:22:55.431846] I
[socket.c:4477:socket_init] 0-management: SSL support on the I/O path is
ENABLED[2018-11-21 15:22:55.431858] I [socket.c:4480:socket_init]
0-management: SSL support for glusterd is ENABLED[2018-11-21
15:22:55.431866] I [socket.c:4497:socket_init] 0-management: using private
polling thread[2018-11-21 15:22:55.431873] I [socket.c:4507:socket_init]
0-management: using certificate depth 1[2018-11-21 15:22:55.432136] E
[socket.c:4554:socket_init] 0-management: failed to open
/etc/ssl/dhparam.pem, DH ciphers are disabledFinal
graph:+------------------------------------------------------------------------------+
1: volume management  2:     type mgmt/glusterd  3:     option
transport.socket.ssl-enabled on  4:     option rpc-auth.auth-glusterfs on
5:     option rpc-auth.auth-unix on  6:     option rpc-auth.auth-null on
7:     option rpc-auth-allow-insecure on  8:     option
transport.listen-backlog 10  9:     option event-threads 1 10:     option
ping-timeout 0 11:     option transport.socket.read-fail-log off 12:
option transport.socket.keepalive-interval 2 13:     option
transport.socket.keepalive-time 10 14:     option transport-type
rdma 15:     option working-directory /var/lib/glusterd 16: end-volume 17:
+------------------------------------------------------------------------------+[2018-11-21
15:22:55.431634] W [MSGID: 106061]
[glusterd-handler.c:3408:glusterd_transport_inet_options_build] 0-glusterd:
Failed to get tcp-user-timeout[2018-11-21 15:22:55.433711] I [MSGID:
101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started
thread with index 1[2018-11-21 15:22:58.975934] I [MSGID: 106142]
[glusterd-pmap.c:297:pmap_registry_bind] 0-pmap: adding brick
/gluster/brick2/glusterbrick on port 49153[2018-11-21 15:22:58.979840] I
[MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind] 0-pmap: adding
brick /gluster/brick1/glusterbrick on port 49152[2018-11-21
15:22:59.462323] I [MSGID: 106493]
[glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC
from uuid: 4d5226a7-bb9c-4432-ae6b-2f2a8446d41b, host: fs-apollo-c1-n1,
port: 0[2018-11-21 15:22:59.463676] I [MSGID: 106492]
[glusterd-handler.c:2726:__glusterd_handle_friend_update] 0-glusterd:
Received friend update from uuid:
4d5226a7-bb9c-4432-ae6b-2f2a8446d41b[2018-11-21 15:22:59.463728] I [MSGID:
106502] [glusterd-handler.c:2771:__glusterd_handle_friend_update]
0-management: Received my uuid as Friend[2018-11-21 15:22:59.464948] I
[rpc-clnt.c:1059:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to
600[2018-11-21 15:22:59.465116] I [socket.c:4480:socket_init] 0-nfs: SSL
support for glusterd is ENABLED[2018-11-21 15:22:59.465130] I
[socket.c:4507:socket_init] 0-nfs: using certificate depth 1[2018-11-21
15:22:59.465387] E [socket.c:4554:socket_init] 0-nfs: failed to open
/etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:22:59.466154] I
[MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management:
nfs already stopped[2018-11-21 15:22:59.466195] I [MSGID: 106568]
[glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: nfs service is
stopped[2018-11-21 15:22:59.467179] I
[rpc-clnt.c:1059:rpc_clnt_connection_init] 0-glustershd: setting
frame-timeout to 600[2018-11-21 15:22:59.467309] I
[socket.c:4480:socket_init] 0-glustershd: SSL support for glusterd is
ENABLED[2018-11-21 15:22:59.467321] I [socket.c:4507:socket_init]
0-glustershd: using certificate depth 1[2018-11-21 15:22:59.467553] E
[socket.c:4554:socket_init] 0-glustershd: failed to open
/etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:22:59.473619] I
[MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop] 0-management:
Stopping glustershd daemon running in pid: 4143[2018-11-21 15:22:59.478144]
E [socket.c:2792:socket_poller] 0-socket.management: socket_poller
127.0.0.1:49149 <http://127.0.0.1:49149> failed (Success)[2018-11-21
15:23:00.473882] I [MSGID: 106568]
[glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: glustershd
service is stopped[2018-11-21 15:23:00.474011] I [MSGID: 106567]
[glusterd-svc-mgmt.c:203:glusterd_svc_start] 0-management: Starting
glustershd service[2018-11-21 15:23:01.478838] I
[rpc-clnt.c:1059:rpc_clnt_connection_init] 0-quotad: setting frame-timeout
to 600[2018-11-21 15:23:01.479039] I [socket.c:4480:socket_init] 0-quotad:
SSL support for glusterd is ENABLED[2018-11-21 15:23:01.479054] I
[socket.c:4507:socket_init] 0-quotad: using certificate depth 1[2018-11-21
15:23:01.479336] E [socket.c:4554:socket_init] 0-quotad: failed to open
/etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:01.480664] I
[MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management:
quotad already stopped[2018-11-21 15:23:01.480711] I [MSGID: 106568]
[glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: quotad service is
stopped[2018-11-21 15:23:01.480777] I
[rpc-clnt.c:1059:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to
600[2018-11-21 15:23:01.480902] I [socket.c:4480:socket_init] 0-bitd: SSL
support for glusterd is ENABLED[2018-11-21 15:23:01.480912] I
[socket.c:4507:socket_init] 0-bitd: using certificate depth 1[2018-11-21
15:23:01.481145] E [socket.c:4554:socket_init] 0-bitd: failed to open
/etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:01.486112] I
[MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop] 0-management:
Stopping bitd daemon running in pid: 4343[2018-11-21 15:23:02.486437] I
[MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management:
bitd service is stopped[2018-11-21 15:23:02.486531] I [MSGID: 106567]
[glusterd-svc-mgmt.c:203:glusterd_svc_start] 0-management: Starting bitd
service[2018-11-21 15:23:03.492955] I
[rpc-clnt.c:1059:rpc_clnt_connection_init] 0-scrub: setting frame-timeout
to 600[2018-11-21 15:23:03.493131] I [socket.c:4480:socket_init] 0-scrub:
SSL support for glusterd is ENABLED[2018-11-21 15:23:03.493147] I
[socket.c:4507:socket_init] 0-scrub: using certificate depth 1[2018-11-21
15:23:03.493416] E [socket.c:4554:socket_init] 0-scrub: failed to open
/etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:03.498042] I
[MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop] 0-management:
Stopping scrub daemon running in pid: 4354[2018-11-21 15:23:04.498341] I
[MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management:
scrub service is stopped[2018-11-21 15:23:04.498426] I [MSGID: 106567]
[glusterd-svc-mgmt.c:203:glusterd_svc_start] 0-management: Starting scrub
service[2018-11-21 15:23:05.503972] I
[glusterd-utils.c:5994:glusterd_brick_start] 0-management: discovered
already-running brick /gluster/brick1/glusterbrick[2018-11-21
15:23:05.504033] I [MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind]
0-pmap: adding brick /gluster/brick1/glusterbrick on port 49152[2018-11-21
15:23:05.504103] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management:
setting frame-timeout to 600[2018-11-21 15:23:05.504229] I
[socket.c:4480:socket_init] 0-management: SSL support for glusterd is
ENABLED[2018-11-21 15:23:05.504241] I [socket.c:4507:socket_init]
0-management: using certificate depth 1[2018-11-21 15:23:05.504505] E
[socket.c:4554:socket_init] 0-management: failed to open
/etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:05.510177] I
[glusterd-utils.c:5994:glusterd_brick_start] 0-management: discovered
already-running brick /gluster/brick2/glusterbrick[2018-11-21
15:23:05.510210] I [MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind]
0-pmap: adding brick /gluster/brick2/glusterbrick on port 49153[2018-11-21
15:23:05.510259] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management:
setting frame-timeout to 600[2018-11-21 15:23:05.510368] I
[socket.c:4480:socket_init] 0-management: SSL support for glusterd is
ENABLED[2018-11-21 15:23:05.510380] I [socket.c:4507:socket_init]
0-management: using certificate depth 1[2018-11-21 15:23:05.510635] E
[socket.c:4554:socket_init] 0-management: failed to open
/etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:05.516357] I
[glusterd-utils.c:5994:glusterd_brick_start] 0-management: discovered
already-running brick /var/lib/glusterd/ss_brick[2018-11-21
15:23:05.516387] I [MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind]
0-pmap: adding brick /var/lib/glusterd/ss_brick on port 49154[2018-11-21
15:23:05.516433] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management:
setting frame-timeout to 600[2018-11-21 15:23:05.516538] I
[socket.c:4480:socket_init] 0-management: SSL support for glusterd is
ENABLED[2018-11-21 15:23:05.516550] I [socket.c:4507:socket_init]
0-management: using certificate depth 1[2018-11-21 15:23:05.516795] E
[socket.c:4554:socket_init] 0-management: failed to open
/etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:05.522223] I
[rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout
to 600[2018-11-21 15:23:05.522301] I [socket.c:4480:socket_init] 0-snapd:
SSL support for glusterd is ENABLED[2018-11-21 15:23:05.522308] I
[socket.c:4507:socket_init] 0-snapd: using certificate depth 1[2018-11-21
15:23:05.522709] E [socket.c:4554:socket_init] 0-snapd: failed to open
/etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:05.523245] I
[rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout
to 600[2018-11-21 15:23:05.523364] I [socket.c:4480:socket_init] 0-snapd:
SSL support for glusterd is ENABLED[2018-11-21 15:23:05.523379] I
[socket.c:4507:socket_init] 0-snapd: using certificate depth 1[2018-11-21
15:23:05.523544] E [socket.c:4554:socket_init] 0-snapd: failed to open
/etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:05.523993] I
[rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout
to 600[2018-11-21 15:23:05.524564] I [socket.c:4480:socket_init] 0-snapd:
SSL support for glusterd is ENABLED[2018-11-21 15:23:05.524585] I
[socket.c:4507:socket_init] 0-snapd: using certificate depth 1[2018-11-21
15:23:05.524804] E [socket.c:4554:socket_init] 0-snapd: failed to open
/etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:05.525379] I
[rpc-clnt.c:1059:rpc_clnt_connection_init] 0-gfproxyd: setting
frame-timeout to 600[2018-11-21 15:23:05.525451] I
[socket.c:4480:socket_init] 0-gfproxyd: SSL support for glusterd is
ENABLED[2018-11-21 15:23:05.525458] I [socket.c:4507:socket_init]
0-gfproxyd: using certificate depth 1[2018-11-21 15:23:05.525649] E
[socket.c:4554:socket_init] 0-gfproxyd: failed to open
/etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:05.526277] I
[rpc-clnt.c:1059:rpc_clnt_connection_init] 0-gfproxyd: setting
frame-timeout to 600[2018-11-21 15:23:05.526348] I
[socket.c:4480:socket_init] 0-gfproxyd: SSL support for glusterd is
ENABLED[2018-11-21 15:23:05.526355] I [socket.c:4507:socket_init]
0-gfproxyd: using certificate depth 1[2018-11-21 15:23:05.526507] E
[socket.c:4554:socket_init] 0-gfproxyd: failed to open
/etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:05.527193] I
[rpc-clnt.c:1059:rpc_clnt_connection_init] 0-gfproxyd: setting
frame-timeout to 600[2018-11-21 15:23:05.527265] I
[socket.c:4480:socket_init] 0-gfproxyd: SSL support for glusterd is
ENABLED[2018-11-21 15:23:05.527274] I [socket.c:4507:socket_init]
0-gfproxyd: using certificate depth 1[2018-11-21 15:23:05.527437] E
[socket.c:4554:socket_init] 0-gfproxyd: failed to open
/etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:05.529501] I
[MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk]
0-glusterd: Received ACC from uuid: 94d352c3-4fac-4926-bbde-e90af52bcad7,
host: fs-apollo-c1-n3, port: 0[2018-11-21 15:23:05.530466] I [MSGID:
106492] [glusterd-handler.c:2726:__glusterd_handle_friend_update]
0-glusterd: Received friend update from uuid:
94d352c3-4fac-4926-bbde-e90af52bcad7[2018-11-21 15:23:05.531325] I [MSGID:
106502] [glusterd-handler.c:2771:__glusterd_handle_friend_update]
0-management: Received my uuid as Friend[2018-11-21 15:23:05.531934] I
[MSGID: 106493] [glusterd-rpc-ops.c:702:__glusterd_friend_update_cbk]
0-management: Received ACC from uuid:
4d5226a7-bb9c-4432-ae6b-2f2a8446d41b[2018-11-21 15:23:05.532036] I [MSGID:
106163] [glusterd-handshake.c:1356:__glusterd_mgmt_hndsk_versions_ack]
0-management: using the op-version 40100[2018-11-21 15:23:05.532812] I
[MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind] 0-pmap: adding
brick /var/lib/glusterd/ss_brick on port 49154[2018-11-21 15:23:05.533478]
I [MSGID: 106163]
[glusterd-handshake.c:1356:__glusterd_mgmt_hndsk_versions_ack]
0-management: using the op-version 40100[2018-11-21 15:23:05.534519] I
[MSGID: 106493] [glusterd-rpc-ops.c:702:__glusterd_friend_update_cbk]
0-management: Received ACC from uuid:
94d352c3-4fac-4926-bbde-e90af52bcad7[2018-11-21 15:23:05.535549] I [MSGID:
106490] [glusterd-handler.c:2548:__glusterd_handle_incoming_friend_req]
0-glusterd: Received probe from uuid:
4d5226a7-bb9c-4432-ae6b-2f2a8446d41b[2018-11-21 15:23:05.538671] I [MSGID:
106493] [glusterd-handler.c:3811:glusterd_xfer_friend_add_resp] 0-glusterd:
Responded to fs-apollo-c1-n1 (0), ret: 0, op_ret: 0[2018-11-21
15:23:05.541086] I [MSGID: 106490]
[glusterd-handler.c:2548:__glusterd_handle_incoming_friend_req] 0-glusterd:
Received probe from uuid: 94d352c3-4fac-4926-bbde-e90af52bcad7[2018-11-21
15:23:05.543763] I [MSGID: 106493]
[glusterd-handler.c:3811:glusterd_xfer_friend_add_resp] 0-glusterd:
Responded to fs-apollo-c1-n3 (0), ret: 0, op_ret: 0[2018-11-21
15:23:05.546026] I [MSGID: 106493]
[glusterd-rpc-ops.c:702:__glusterd_friend_update_cbk] 0-management:
Received ACC from uuid: 4d5226a7-bb9c-4432-ae6b-2f2a8446d41b[2018-11-21
15:23:05.547097] I [MSGID: 106492]
[glusterd-handler.c:2726:__glusterd_handle_friend_update] 0-glusterd:
Received friend update from uuid:
4d5226a7-bb9c-4432-ae6b-2f2a8446d41b[2018-11-21 15:23:05.547154] I [MSGID:
106502] [glusterd-handler.c:2771:__glusterd_handle_friend_update]
0-management: Received my uuid as Friend[2018-11-21 15:23:05.547765] I
[MSGID: 106492] [glusterd-handler.c:2726:__glusterd_handle_friend_update]
0-glusterd: Received friend update from uuid:
94d352c3-4fac-4926-bbde-e90af52bcad7[2018-11-21 15:23:05.548774] I [MSGID:
106502] [glusterd-handler.c:2771:__glusterd_handle_friend_update]
0-management: Received my uuid as Friend[2018-11-21 15:23:05.548958] I
[MSGID: 106493] [glusterd-rpc-ops.c:702:__glusterd_friend_update_cbk]
0-management: Received ACC from uuid:
94d352c3-4fac-4926-bbde-e90af52bcad7[2018-11-21 15:23:31.411509] I [MSGID:
106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list req[2018-11-21 15:23:33.372957] I [MSGID:
106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume]
0-management: Received get vol reqThe message "I [MSGID: 106488]
[glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req" repeated 3 times between [2018-11-21 15:23:33.372957]
and [2018-11-21 15:23:33.376516][2018-11-21 15:24:25.407528] I [MSGID:
106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list req[2018-11-21 15:24:27.370775] I [MSGID:
106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume]
0-management: Received get vol reqThe message "I [MSGID: 106488]
[glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req" repeated 3 times between [2018-11-21 15:24:27.370775]
and [2018-11-21 15:24:27.373796][2018-11-21 15:25:19.407885] I [MSGID:
106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list req[2018-11-21 15:25:21.371482] I [MSGID:
106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume]
0-management: Received get vol req[2018-11-21 15:26:13.415390] I [MSGID:
106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list reqThe message "I [MSGID: 106488]
[glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req" repeated 7 times between [2018-11-21 15:25:21.371482]
and [2018-11-21 15:26:15.378777][2018-11-21 15:27:07.403685] I [MSGID:
106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list req[2018-11-21 15:27:09.371804] I [MSGID:
106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume]
0-management: Received get vol req[2018-11-21 15:28:02.728185] E
[socket.c:2792:socket_poller] 0-socket.management: socket_poller
10.10.12.64:49133 <http://10.10.12.64:49133> failed (Input/output
error)[2018-11-21 15:28:01.415055] I [MSGID: 106487]
[glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list reqThe message "I [MSGID: 106488]
[glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req" repeated 7 times between [2018-11-21 15:27:09.371804]
and [2018-11-21 15:28:03.370301][2018-11-21 15:28:55.414118] I [MSGID:
106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list req[2018-11-21 15:28:57.375508] I [MSGID:
106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume]
0-management: Received get vol reqThe message "I [MSGID: 106487]
[glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req" repeated 2 times between [2018-11-21
15:28:55.414118] and [2018-11-21 15:30:43.419479]The message "I [MSGID:
106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume]
0-management: Received get vol req" repeated 11 times between [2018-11-21
15:28:57.375508] and [2018-11-21 15:30:45.383263][2018-11-21
15:31:37.415788] I [MSGID: 106487]
[glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req[2018-11-21 15:31:39.375304] I [MSGID: 106488]
[glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req[2018-11-21 15:32:31.415510] I [MSGID: 106487]
[glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list reqThe message "I [MSGID: 106488]
[glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req" repeated 7 times between [2018-11-21 15:31:39.375304]
and [2018-11-21 15:32:33.375024][2018-11-21 15:33:25.407192] I [MSGID:
106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list req[2018-11-21 15:33:27.379564] I [MSGID:
106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume]
0-management: Received get vol req[2018-11-21 15:34:20.746832] E
[socket.c:2792:socket_poller] 0-socket.management: socket_poller
10.10.12.64:49133 <http://10.10.12.64:49133> failed (Input/output
error)[2018-11-21 15:34:19.411130] I [MSGID: 106487]
[glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list reqThe message "I [MSGID: 106488]
[glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req" repeated 7 times between [2018-11-21 15:33:27.379564]
and [2018-11-21 15:34:21.368809][2018-11-21 15:35:13.415187] I [MSGID:
106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list req[2018-11-21 15:35:15.371647] I [MSGID:
106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume]
0-management: Received get vol req[2018-11-21 15:36:00.958833] E [MSGID:
106115] [glusterd-mgmt.c:124:gd_mgmt_v3_collate_errors] 0-management:
Locking failed on fs-apollo-c1-n3. Please check log file for
details.[2018-11-21 15:36:00.960197] W
[glusterd-locks.c:622:glusterd_mgmt_v3_lock]
(-->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0x3fc43)
[0x7f206876cc43]
-->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0x30c73)
[0x7f206875dc73]
-->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0xdb5d3)
[0x7f20688085d3] ) 0-management: Lock for archive1 held by
7e7fd30a-94c9-4c21-9c4b-3817508ef121[2018-11-21 15:36:00.960234] E [MSGID:
106118] [glusterd-op-sm.c:4173:glusterd_op_ac_lock] 0-management: Unable to
acquire lock for archive1[2018-11-21 15:36:00.960311] E [MSGID: 106376]
[glusterd-op-sm.c:8305:glusterd_op_sm] 0-management: handler returned:
-1[2018-11-21 15:36:00.961494] E [MSGID: 106115]
[glusterd-mgmt.c:124:gd_mgmt_v3_collate_errors] 0-management: Locking
failed on fs-apollo-c1-n1. Please check log file for details.[2018-11-21
15:36:00.961603] E [MSGID: 106150]
[glusterd-syncop.c:1957:gd_sync_task_begin] 0-management: Locking Peers
Failed.[2018-11-21 15:36:00.962117] W
[glusterd-locks.c:856:glusterd_mgmt_v3_unlock]
(-->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0x3fc43)
[0x7f206876cc43]
-->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0x3099f)
[0x7f206875d99f]
-->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0xdba74)
[0x7f2068808a74] ) 0-management: Lock owner mismatch. Lock for vol archive1
held by 7e7fd30a-94c9-4c21-9c4b-3817508ef121[2018-11-21 15:36:00.962144] E
[MSGID: 106117] [glusterd-op-sm.c:4236:glusterd_op_ac_unlock] 0-management:
Unable to release lock for archive1[2018-11-21 15:36:00.962222] E [MSGID:
106376] [glusterd-op-sm.c:8305:glusterd_op_sm] 0-management: handler
returned: 1[2018-11-21 15:36:00.962406] E [MSGID: 106115]
[glusterd-mgmt.c:124:gd_mgmt_v3_collate_errors] 0-management: Unlocking
failed on fs-apollo-c1-n1. Please check log file for details.[2018-11-21
15:36:00.962973] E [MSGID: 106115]
[glusterd-mgmt.c:124:gd_mgmt_v3_collate_errors] 0-management: Unlocking
failed on fs-apollo-c1-n3. Please check log file for details.[2018-11-21
15:36:00.963033] E [MSGID: 106151]
[glusterd-syncop.c:1640:gd_unlock_op_phase] 0-management: Failed to unlock
on some peer(s)[2018-11-21 15:36:00.991734] W
[rpc-clnt.c:1753:rpc_clnt_submit] 0-scrub: error returned while attempting
to connect to host:(null), port:0[2018-11-21 15:36:01.186043] W
[rpc-clnt.c:1753:rpc_clnt_submit] 0-scrub: error returned while attempting
to connect to host:(null), port:0[2018-11-21 15:36:07.416984] I [MSGID:
106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list reqThe message "I [MSGID: 106488]
[glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req" repeated 11 times between [2018-11-21
15:35:15.371647] and [2018-11-21 15:36:09.383236][2018-11-21
15:37:01.410979] I [MSGID: 106487]
[glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req[2018-11-21 15:37:03.383408] I [MSGID: 106488]
[glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management:
Received get vol reqThe message "I [MSGID: 106487]
[glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req" repeated 2 times between [2018-11-21
15:37:01.410979] and [2018-11-21 15:38:49.410710]The message "I [MSGID:
106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume]
0-management: Received get vol req" repeated 11 times between [2018-11-21
15:37:03.383408] and [2018-11-21 15:38:51.378524][2018-11-21
15:39:43.421483] I [MSGID: 106487]
[glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req[2018-11-21 15:39:44.786383] E
[socket.c:2792:socket_poller] 0-socket.management: socket_poller
10.10.12.64:49133 <http://10.10.12.64:49133> failed (Input/output
error)[2018-11-21 15:39:45.373093] I [MSGID: 106488]
[glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req[2018-11-21 15:40:37.412592] I [MSGID: 106487]
[glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list reqThe message "I [MSGID: 106488]
[glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req" repeated 7 times between [2018-11-21 15:39:45.373093]
and [2018-11-21 15:40:39.378904][2018-11-21 15:41:31.415866] I [MSGID:
106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list req[2018-11-21 15:41:33.374993] I [MSGID:
106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume]
0-management: Received get vol req[2018-11-21 15:42:25.415585] I [MSGID:
106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list reqThe message "I [MSGID: 106488]
[glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req" repeated 7 times between [2018-11-21 15:41:33.374993]
and [2018-11-21 15:42:27.379013][2018-11-21 15:43:19.411237] I [MSGID:
106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list req[2018-11-21 15:43:21.376718] I [MSGID:
106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume]
0-management: Received get vol req[2018-11-21 15:44:13.425206] I [MSGID:
106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list reqThe message "I [MSGID: 106488]
[glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req" repeated 7 times between [2018-11-21 15:43:21.376718]
and [2018-11-21 15:44:15.375796][2018-11-21 15:45:07.424344] I [MSGID:
106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list req[2018-11-21 15:45:09.382299] I [MSGID:
106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume]
0-management: Received get vol req[2018-11-21 15:46:02.769673] E
[socket.c:2792:socket_poller] 0-socket.management: socket_poller
10.10.12.64:49133 <http://10.10.12.64:49133> failed (Input/output
error)[2018-11-21 15:46:01.422742] I [MSGID: 106487]
[glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list reqThe message "I [MSGID: 106488]
[glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req" repeated 7 times between [2018-11-21 15:45:09.382299]
and [2018-11-21 15:46:03.370463][2018-11-21 15:46:55.426992] I [MSGID:
106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list req[2018-11-21 15:46:57.379488] I [MSGID:
106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume]
0-management: Received get vol req#->
/var/log/glusterfs/run-gluster-shared_storage.log[2018-11-21
15:22:48.623969] W [glusterfsd.c:1514:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7559) [0x7f9e25569559]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xfd) [0x561fd19c840d]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x5a) [0x561fd19c81fa] ) 0-:
received signum (15), shutting down*

*[2018-11-21 15:22:48.624042] I [fuse-bridge.c:5981:fini] 0-fuse:
Unmounting '/run/gluster/shared_storage'.[2018-11-21 15:22:48.624185] E
[fuse-bridge.c:212:check_and_dump_fuse_W] 0-glusterfs-fuse: writing to fuse
device failed: Bad file descriptor[2018-11-21 15:22:48.624246] I
[fuse-bridge.c:5199:fuse_thread_proc] 0-fuse: initating unmount of
/run/gluster/shared_storage[2018-11-21 15:22:48.624679] I
[fuse-bridge.c:5986:fini] 0-fuse: Closing fuse connection to
'/run/gluster/shared_storage'.#->
/var/log/glusterfs/bricks/var-lib-glusterd-ss_brick.log[2018-11-21
15:22:48.626860] E [socket.c:2792:socket_poller]
0-tcp.gluster_shared_storage-server: socket_poller 10.10.12.62:49151
<http://10.10.12.62:49151> failed (Success)[2018-11-21 15:22:48.627048] I
[MSGID: 115036] [server.c:483:server_rpc_notify]
0-gluster_shared_storage-server: disconnecting connection from
CTX_ID:797e6f66-b69a-4045-bc43-c38d26a2c413-GRAPH_ID:0-PID:4447-HOST:fs-apollo-c1-n2-PC_NAME:gluster_shared_storage-client-0-RECON_NO:-0[2018-11-21
15:22:48.627214] I [MSGID: 101055] [client_t.c:444:gf_client_unref]
0-gluster_shared_storage-server: Shutting down connection
CTX_ID:797e6f66-b69a-4045-bc43-c38d26a2c413-GRAPH_ID:0-PID:4447-HOST:fs-apollo-c1-n2-PC_NAME:gluster_shared_storage-client-0-RECON_NO:-0[2018-11-21
15:22:48.632303] W [socket.c:599:__socket_rwv] 0-glusterfs: readv on
10.10.12.62:24007 <http://10.10.12.62:24007> failed (Success)[2018-11-21
15:22:48.632531] I [glusterfsd-mgmt.c:2348:mgmt_rpc_notify]
0-glusterfsd-mgmt: disconnected from remote-host:
fs-apollo-c1-n2[2018-11-21 15:22:48.632558] I
[glusterfsd-mgmt.c:2369:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all
volfile servers[2018-11-21 15:22:59.478134] I [MSGID: 115036]
[server.c:483:server_rpc_notify] 0-gluster_shared_storage-server:
disconnecting connection from
CTX_ID:87d36408-3140-457a-b844-bf3ee87f5595-GRAPH_ID:0-PID:4143-HOST:fs-apollo-c1-n2-PC_NAME:gluster_shared_storage-client-0-RECON_NO:-0[2018-11-21
15:22:59.478287] I [MSGID: 101055] [client_t.c:444:gf_client_unref]
0-gluster_shared_storage-server: Shutting down connection
CTX_ID:87d36408-3140-457a-b844-bf3ee87f5595-GRAPH_ID:0-PID:4143-HOST:fs-apollo-c1-n2-PC_NAME:gluster_shared_storage-client-0-RECON_NO:-0[2018-11-21
15:23:05.529596] I [glusterfsd-mgmt.c:1953:mgmt_getspec_cbk] 0-glusterfs:
No change in volfile,continuing[2018-11-21 15:23:05.625165] I
[addr.c:55:compare_addr_and_update] 0-/var/lib/glusterd/ss_brick: allowed =
"*", received addr = "10.10.12.62"[2018-11-21 15:23:05.625203] I
[login.c:45:gf_auth] 0-auth/login: connecting user name:
fs-apollo-c1-n2[2018-11-21 15:23:05.625238] I [login.c:111:gf_auth]
0-auth/login: allowed user names: *[2018-11-21 15:23:05.625254] I [MSGID:
115029] [server-handshake.c:763:server_setvolume]
0-gluster_shared_storage-server: accepted client from
CTX_ID:1c203d59-02da-46ec-b836-371ebb47e3dc-GRAPH_ID:0-PID:20295-HOST:fs-apollo-c1-n2-PC_NAME:gluster_shared_storage-client-0-RECON_NO:-0
(version: 4.1.5)[2018-11-21 15:36:01.009586] I
[glusterfsd-mgmt.c:53:mgmt_cbk_spec] 0-mgmt: Volume file changed[2018-11-21
15:36:01.017077] I [glusterfsd-mgmt.c:1953:mgmt_getspec_cbk] 0-glusterfs:
No change in volfile,continuing[2018-11-21 15:36:01.206445] I
[glusterfsd-mgmt.c:53:mgmt_cbk_spec] 0-mgmt: Volume file changed[2018-11-21
15:36:01.212266] I [glusterfsd-mgmt.c:1953:mgmt_getspec_cbk] 0-glusterfs:
No change in volfile,continuing*

Am Fr., 9. Nov. 2018 um 11:53 Uhr schrieb David Spisla <spisla80 at gmail.com>:

> Sure, I will do that the next time shared_storage is unmount after
> stopping glusterd
>
> Am Fr., 9. Nov. 2018 um 10:23 Uhr schrieb Rafi Kavungal Chundattu Parambil
> <rkavunga at redhat.com>:
>
>> Hi David Spisla,
>>
>> gluster_shared_storage works just like yet another volume(There are some
>> special handling related to authentication). So bringing down a glusterd
>> node shouldn't have lead to unmount of the volume.
>>
>> Can you please share the mount log, brick logs at time when you stopped
>> glusterd. Also make sure your shared storage bricks up and running.
>>
>> Regards
>> Rafi KC
>>
>> ----- Original Message -----
>> From: "David Spisla" <spisla80 at gmail.com>
>> To: gluster-users at gluster.org
>> Sent: Tuesday, November 6, 2018 3:25:18 PM
>> Subject: [Gluster-users] Shared Storage is unmount after stopping glusterd
>>
>> Dear Gluster Community,
>> I have a v4.1.5 4-node-cluster with SLES15 machines. The last time I
>> observe that after stopping glusterd with
>> $ sudo systemctl stop glusterd
>>
>> sometimes also the FUSE mount of the shared_storage on the same node is
>> unmount but shared_storage ist still in the volume list. I mean really
>> "sometimes" because it is not reproducible in a deterministic way and I am
>> not sure if there is really a dependency between stopping glusterd and
>> shared_storage.
>>
>> What is the expected behaviour if stopping glusterd? In my opinion all
>> volumes included shared_storage should still have FUSE mounts.
>>
>> Regards
>> David Spisla
>>
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20181127/e97d51ef/attachment.html>