<div dir="ltr"><div dir="ltr"><div>Hello Rafi,</div><div>a few days ago I had the scenario described below. Here are the log entries. As you can see glusterd was stopped at 15:22:48. The same time FUSE of shared_storage was stopped. It seems to be nearly the same error message. You can also see that I restarted glusterd manually but FUSE of shared_storage was not restarted<br></div><div><br></div><div><i>#-> /var/log/glusterfs/glusterd.log<br><br>[2018-11-21 15:22:37.406032] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br>The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:21:45.377310] and [2018-11-21 15:22:39.382300]<br>[2018-11-21 15:22:48.623951] W [glusterfsd.c:1514:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7559) [0x7fb1e5f11559] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xfd) [0x561c3626340d] -->/usr/sbin/glusterd(cleanup_and_exit+0x5a) [0x561c362631fa] ) 0-: received signum (15), shutting down<br>[2018-11-21 15:22:53.491120] I [MSGID: 100030] [glusterfsd.c:2741:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 4.1.5 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)<br>[2018-11-21 15:22:53.494326] I [MSGID: 106478] [glusterd.c:1423:init] 0-management: Maximum allowed open file descriptors set to 65536<br>[2018-11-21 15:22:53.494372] I [MSGID: 106479] [glusterd.c:1481:init] 0-management: Using /var/lib/glusterd as working directory<br>[2018-11-21 15:22:53.494393] I [MSGID: 106479] [glusterd.c:1486:init] 0-management: Using /var/run/gluster as pid file working directory<br>[2018-11-21 15:22:53.496564] I [socket.c:4477:socket_init] 0-socket.management: SSL support on the I/O path is ENABLED<br>[2018-11-21 15:22:53.496598] I [socket.c:4480:socket_init] 0-socket.management: SSL support for glusterd is ENABLED<br>[2018-11-21 15:22:53.496607] I [socket.c:4497:socket_init] 0-socket.management: using private polling thread<br>[2018-11-21 15:22:53.496615] I [socket.c:4507:socket_init] 0-socket.management: using certificate depth 1<br>[2018-11-21 15:22:53.496891] E [socket.c:4554:socket_init] 0-socket.management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled<br>[2018-11-21 15:22:53.499933] W [MSGID: 103071] [rdma.c:4629:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device]<br>[2018-11-21 15:22:53.499970] W [MSGID: 103055] [rdma.c:4938:init] 0-rdma.management: Failed to initialize IB Device<br>[2018-11-21 15:22:53.499983] W [rpc-transport.c:351:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed<br>[2018-11-21 15:22:53.500148] W [rpcsvc.c:1781:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed<br>[2018-11-21 15:22:53.500174] E [MSGID: 106244] [glusterd.c:1764:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport<br>[2018-11-21 15:22:53.500361] I [socket.c:4480:socket_init] 0-socket.management: SSL support for glusterd is ENABLED<br>[2018-11-21 15:22:53.500373] I [socket.c:4507:socket_init] 0-socket.management: using certificate depth 1<br>[2018-11-21 15:22:53.500615] E [socket.c:4554:socket_init] 0-socket.management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled<br>[2018-11-21 15:22:55.353929] I [MSGID: 106513] [glusterd-store.c:2240:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 40100<br>[2018-11-21 15:22:55.423433] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 7e7fd30a-94c9-4c21-9c4b-3817508ef121<br>[2018-11-21 15:22:55.429214] I [MSGID: 106498] [glusterd-handler.c:3614:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0<br>[2018-11-21 15:22:55.429317] I [MSGID: 106498] [glusterd-handler.c:3614:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0<br>[2018-11-21 15:22:55.429379] W [MSGID: 106061] [glusterd-handler.c:3408:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout<br>[2018-11-21 15:22:55.429443] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<br>[2018-11-21 15:22:55.429623] I [socket.c:4477:socket_init] 0-management: SSL support on the I/O path is ENABLED<br>[2018-11-21 15:22:55.429635] I [socket.c:4480:socket_init] 0-management: SSL support for glusterd is ENABLED<br>[2018-11-21 15:22:55.429643] I [socket.c:4497:socket_init] 0-management: using private polling thread<br>[2018-11-21 15:22:55.429650] I [socket.c:4507:socket_init] 0-management: using certificate depth 1<br>[2018-11-21 15:22:55.429914] E [socket.c:4554:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled<br>[2018-11-21 15:22:55.431676] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<br>[2018-11-21 15:22:55.431846] I [socket.c:4477:socket_init] 0-management: SSL support on the I/O path is ENABLED<br>[2018-11-21 15:22:55.431858] I [socket.c:4480:socket_init] 0-management: SSL support for glusterd is ENABLED<br>[2018-11-21 15:22:55.431866] I [socket.c:4497:socket_init] 0-management: using private polling thread<br>[2018-11-21 15:22:55.431873] I [socket.c:4507:socket_init] 0-management: using certificate depth 1<br>[2018-11-21 15:22:55.432136] E [socket.c:4554:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled<br>Final graph:<br>+------------------------------------------------------------------------------+<br> 1: volume management<br> 2: type mgmt/glusterd<br> 3: option transport.socket.ssl-enabled on<br> 4: option rpc-auth.auth-glusterfs on<br> 5: option rpc-auth.auth-unix on<br> 6: option rpc-auth.auth-null on<br> 7: option rpc-auth-allow-insecure on<br> 8: option transport.listen-backlog 10<br> 9: option event-threads 1<br> 10: option ping-timeout 0<br> 11: option transport.socket.read-fail-log off<br> 12: option transport.socket.keepalive-interval 2<br> 13: option transport.socket.keepalive-time 10<br> 14: option transport-type rdma<br> 15: option working-directory /var/lib/glusterd<br> 16: end-volume<br> 17: <br>+------------------------------------------------------------------------------+<br>[2018-11-21 15:22:55.431634] W [MSGID: 106061] [glusterd-handler.c:3408:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout<br>[2018-11-21 15:22:55.433711] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1<br>[2018-11-21 15:22:58.975934] I [MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind] 0-pmap: adding brick /gluster/brick2/glusterbrick on port 49153<br>[2018-11-21 15:22:58.979840] I [MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind] 0-pmap: adding brick /gluster/brick1/glusterbrick on port 49152<br>[2018-11-21 15:22:59.462323] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 4d5226a7-bb9c-4432-ae6b-2f2a8446d41b, host: fs-apollo-c1-n1, port: 0<br>[2018-11-21 15:22:59.463676] I [MSGID: 106492] [glusterd-handler.c:2726:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 4d5226a7-bb9c-4432-ae6b-2f2a8446d41b<br>[2018-11-21 15:22:59.463728] I [MSGID: 106502] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend<br>[2018-11-21 15:22:59.464948] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600<br>[2018-11-21 15:22:59.465116] I [socket.c:4480:socket_init] 0-nfs: SSL support for glusterd is ENABLED<br>[2018-11-21 15:22:59.465130] I [socket.c:4507:socket_init] 0-nfs: using certificate depth 1<br>[2018-11-21 15:22:59.465387] E [socket.c:4554:socket_init] 0-nfs: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled<br>[2018-11-21 15:22:59.466154] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped<br>[2018-11-21 15:22:59.466195] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: nfs service is stopped<br>[2018-11-21 15:22:59.467179] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600<br>[2018-11-21 15:22:59.467309] I [socket.c:4480:socket_init] 0-glustershd: SSL support for glusterd is ENABLED<br>[2018-11-21 15:22:59.467321] I [socket.c:4507:socket_init] 0-glustershd: using certificate depth 1<br>[2018-11-21 15:22:59.467553] E [socket.c:4554:socket_init] 0-glustershd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled<br>[2018-11-21 15:22:59.473619] I [MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop] 0-management: Stopping glustershd daemon running in pid: 4143<br>[2018-11-21 15:22:59.478144] E [socket.c:2792:socket_poller] 0-socket.management: socket_poller <a href="http://127.0.0.1:49149">127.0.0.1:49149</a> failed (Success)<br>[2018-11-21 15:23:00.473882] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: glustershd service is stopped<br>[2018-11-21 15:23:00.474011] I [MSGID: 106567] [glusterd-svc-mgmt.c:203:glusterd_svc_start] 0-management: Starting glustershd service<br>[2018-11-21 15:23:01.478838] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600<br>[2018-11-21 15:23:01.479039] I [socket.c:4480:socket_init] 0-quotad: SSL support for glusterd is ENABLED<br>[2018-11-21 15:23:01.479054] I [socket.c:4507:socket_init] 0-quotad: using certificate depth 1<br>[2018-11-21 15:23:01.479336] E [socket.c:4554:socket_init] 0-quotad: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled<br>[2018-11-21 15:23:01.480664] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped<br>[2018-11-21 15:23:01.480711] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: quotad service is stopped<br>[2018-11-21 15:23:01.480777] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600<br>[2018-11-21 15:23:01.480902] I [socket.c:4480:socket_init] 0-bitd: SSL support for glusterd is ENABLED<br>[2018-11-21 15:23:01.480912] I [socket.c:4507:socket_init] 0-bitd: using certificate depth 1<br>[2018-11-21 15:23:01.481145] E [socket.c:4554:socket_init] 0-bitd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled<br>[2018-11-21 15:23:01.486112] I [MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop] 0-management: Stopping bitd daemon running in pid: 4343<br>[2018-11-21 15:23:02.486437] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: bitd service is stopped<br>[2018-11-21 15:23:02.486531] I [MSGID: 106567] [glusterd-svc-mgmt.c:203:glusterd_svc_start] 0-management: Starting bitd service<br>[2018-11-21 15:23:03.492955] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600<br>[2018-11-21 15:23:03.493131] I [socket.c:4480:socket_init] 0-scrub: SSL support for glusterd is ENABLED<br>[2018-11-21 15:23:03.493147] I [socket.c:4507:socket_init] 0-scrub: using certificate depth 1<br>[2018-11-21 15:23:03.493416] E [socket.c:4554:socket_init] 0-scrub: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled<br>[2018-11-21 15:23:03.498042] I [MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop] 0-management: Stopping scrub daemon running in pid: 4354<br>[2018-11-21 15:23:04.498341] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: scrub service is stopped<br>[2018-11-21 15:23:04.498426] I [MSGID: 106567] [glusterd-svc-mgmt.c:203:glusterd_svc_start] 0-management: Starting scrub service<br>[2018-11-21 15:23:05.503972] I [glusterd-utils.c:5994:glusterd_brick_start] 0-management: discovered already-running brick /gluster/brick1/glusterbrick<br>[2018-11-21 15:23:05.504033] I [MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind] 0-pmap: adding brick /gluster/brick1/glusterbrick on port 49152<br>[2018-11-21 15:23:05.504103] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<br>[2018-11-21 15:23:05.504229] I [socket.c:4480:socket_init] 0-management: SSL support for glusterd is ENABLED<br>[2018-11-21 15:23:05.504241] I [socket.c:4507:socket_init] 0-management: using certificate depth 1<br>[2018-11-21 15:23:05.504505] E [socket.c:4554:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled<br>[2018-11-21 15:23:05.510177] I [glusterd-utils.c:5994:glusterd_brick_start] 0-management: discovered already-running brick /gluster/brick2/glusterbrick<br>[2018-11-21 15:23:05.510210] I [MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind] 0-pmap: adding brick /gluster/brick2/glusterbrick on port 49153<br>[2018-11-21 15:23:05.510259] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<br>[2018-11-21 15:23:05.510368] I [socket.c:4480:socket_init] 0-management: SSL support for glusterd is ENABLED<br>[2018-11-21 15:23:05.510380] I [socket.c:4507:socket_init] 0-management: using certificate depth 1<br>[2018-11-21 15:23:05.510635] E [socket.c:4554:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled<br>[2018-11-21 15:23:05.516357] I [glusterd-utils.c:5994:glusterd_brick_start] 0-management: discovered already-running brick /var/lib/glusterd/ss_brick<br>[2018-11-21 15:23:05.516387] I [MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind] 0-pmap: adding brick /var/lib/glusterd/ss_brick on port 49154<br>[2018-11-21 15:23:05.516433] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<br>[2018-11-21 15:23:05.516538] I [socket.c:4480:socket_init] 0-management: SSL support for glusterd is ENABLED<br>[2018-11-21 15:23:05.516550] I [socket.c:4507:socket_init] 0-management: using certificate depth 1<br>[2018-11-21 15:23:05.516795] E [socket.c:4554:socket_init] 0-management: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled<br>[2018-11-21 15:23:05.522223] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600<br>[2018-11-21 15:23:05.522301] I [socket.c:4480:socket_init] 0-snapd: SSL support for glusterd is ENABLED<br>[2018-11-21 15:23:05.522308] I [socket.c:4507:socket_init] 0-snapd: using certificate depth 1<br>[2018-11-21 15:23:05.522709] E [socket.c:4554:socket_init] 0-snapd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled<br>[2018-11-21 15:23:05.523245] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600<br>[2018-11-21 15:23:05.523364] I [socket.c:4480:socket_init] 0-snapd: SSL support for glusterd is ENABLED<br>[2018-11-21 15:23:05.523379] I [socket.c:4507:socket_init] 0-snapd: using certificate depth 1<br>[2018-11-21 15:23:05.523544] E [socket.c:4554:socket_init] 0-snapd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled<br>[2018-11-21 15:23:05.523993] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600<br>[2018-11-21 15:23:05.524564] I [socket.c:4480:socket_init] 0-snapd: SSL support for glusterd is ENABLED<br>[2018-11-21 15:23:05.524585] I [socket.c:4507:socket_init] 0-snapd: using certificate depth 1<br>[2018-11-21 15:23:05.524804] E [socket.c:4554:socket_init] 0-snapd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled<br>[2018-11-21 15:23:05.525379] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-gfproxyd: setting frame-timeout to 600<br>[2018-11-21 15:23:05.525451] I [socket.c:4480:socket_init] 0-gfproxyd: SSL support for glusterd is ENABLED<br>[2018-11-21 15:23:05.525458] I [socket.c:4507:socket_init] 0-gfproxyd: using certificate depth 1<br>[2018-11-21 15:23:05.525649] E [socket.c:4554:socket_init] 0-gfproxyd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled<br>[2018-11-21 15:23:05.526277] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-gfproxyd: setting frame-timeout to 600<br>[2018-11-21 15:23:05.526348] I [socket.c:4480:socket_init] 0-gfproxyd: SSL support for glusterd is ENABLED<br>[2018-11-21 15:23:05.526355] I [socket.c:4507:socket_init] 0-gfproxyd: using certificate depth 1<br>[2018-11-21 15:23:05.526507] E [socket.c:4554:socket_init] 0-gfproxyd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled<br>[2018-11-21 15:23:05.527193] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-gfproxyd: setting frame-timeout to 600<br>[2018-11-21 15:23:05.527265] I [socket.c:4480:socket_init] 0-gfproxyd: SSL support for glusterd is ENABLED<br>[2018-11-21 15:23:05.527274] I [socket.c:4507:socket_init] 0-gfproxyd: using certificate depth 1<br>[2018-11-21 15:23:05.527437] E [socket.c:4554:socket_init] 0-gfproxyd: failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled<br>[2018-11-21 15:23:05.529501] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 94d352c3-4fac-4926-bbde-e90af52bcad7, host: fs-apollo-c1-n3, port: 0<br>[2018-11-21 15:23:05.530466] I [MSGID: 106492] [glusterd-handler.c:2726:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 94d352c3-4fac-4926-bbde-e90af52bcad7<br>[2018-11-21 15:23:05.531325] I [MSGID: 106502] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend<br>[2018-11-21 15:23:05.531934] I [MSGID: 106493] [glusterd-rpc-ops.c:702:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 4d5226a7-bb9c-4432-ae6b-2f2a8446d41b<br>[2018-11-21 15:23:05.532036] I [MSGID: 106163] [glusterd-handshake.c:1356:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 40100<br>[2018-11-21 15:23:05.532812] I [MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind] 0-pmap: adding brick /var/lib/glusterd/ss_brick on port 49154<br>[2018-11-21 15:23:05.533478] I [MSGID: 106163] [glusterd-handshake.c:1356:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 40100<br>[2018-11-21 15:23:05.534519] I [MSGID: 106493] [glusterd-rpc-ops.c:702:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 94d352c3-4fac-4926-bbde-e90af52bcad7<br>[2018-11-21 15:23:05.535549] I [MSGID: 106490] [glusterd-handler.c:2548:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 4d5226a7-bb9c-4432-ae6b-2f2a8446d41b<br>[2018-11-21 15:23:05.538671] I [MSGID: 106493] [glusterd-handler.c:3811:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to fs-apollo-c1-n1 (0), ret: 0, op_ret: 0<br>[2018-11-21 15:23:05.541086] I [MSGID: 106490] [glusterd-handler.c:2548:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 94d352c3-4fac-4926-bbde-e90af52bcad7<br>[2018-11-21 15:23:05.543763] I [MSGID: 106493] [glusterd-handler.c:3811:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to fs-apollo-c1-n3 (0), ret: 0, op_ret: 0<br>[2018-11-21 15:23:05.546026] I [MSGID: 106493] [glusterd-rpc-ops.c:702:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 4d5226a7-bb9c-4432-ae6b-2f2a8446d41b<br>[2018-11-21 15:23:05.547097] I [MSGID: 106492] [glusterd-handler.c:2726:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 4d5226a7-bb9c-4432-ae6b-2f2a8446d41b<br>[2018-11-21 15:23:05.547154] I [MSGID: 106502] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend<br>[2018-11-21 15:23:05.547765] I [MSGID: 106492] [glusterd-handler.c:2726:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 94d352c3-4fac-4926-bbde-e90af52bcad7<br>[2018-11-21 15:23:05.548774] I [MSGID: 106502] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend<br>[2018-11-21 15:23:05.548958] I [MSGID: 106493] [glusterd-rpc-ops.c:702:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 94d352c3-4fac-4926-bbde-e90af52bcad7<br>[2018-11-21 15:23:31.411509] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br>[2018-11-21 15:23:33.372957] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req<br>The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 3 times between [2018-11-21 15:23:33.372957] and [2018-11-21 15:23:33.376516]<br>[2018-11-21 15:24:25.407528] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br>[2018-11-21 15:24:27.370775] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req<br>The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 3 times between [2018-11-21 15:24:27.370775] and [2018-11-21 15:24:27.373796]<br>[2018-11-21 15:25:19.407885] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br>[2018-11-21 15:25:21.371482] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req<br>[2018-11-21 15:26:13.415390] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br>The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:25:21.371482] and [2018-11-21 15:26:15.378777]<br>[2018-11-21 15:27:07.403685] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br>[2018-11-21 15:27:09.371804] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req<br>[2018-11-21 15:28:02.728185] E [socket.c:2792:socket_poller] 0-socket.management: socket_poller <a href="http://10.10.12.64:49133">10.10.12.64:49133</a> failed (Input/output error)<br>[2018-11-21 15:28:01.415055] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br>The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:27:09.371804] and [2018-11-21 15:28:03.370301]<br>[2018-11-21 15:28:55.414118] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br>[2018-11-21 15:28:57.375508] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req<br>The message "I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated 2 times between [2018-11-21 15:28:55.414118] and [2018-11-21 15:30:43.419479]<br>The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 11 times between [2018-11-21 15:28:57.375508] and [2018-11-21 15:30:45.383263]<br>[2018-11-21 15:31:37.415788] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br>[2018-11-21 15:31:39.375304] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req<br>[2018-11-21 15:32:31.415510] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br>The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:31:39.375304] and [2018-11-21 15:32:33.375024]<br>[2018-11-21 15:33:25.407192] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br>[2018-11-21 15:33:27.379564] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req<br>[2018-11-21 15:34:20.746832] E [socket.c:2792:socket_poller] 0-socket.management: socket_poller <a href="http://10.10.12.64:49133">10.10.12.64:49133</a> failed (Input/output error)<br>[2018-11-21 15:34:19.411130] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br>The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:33:27.379564] and [2018-11-21 15:34:21.368809]<br>[2018-11-21 15:35:13.415187] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br>[2018-11-21 15:35:15.371647] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req<br>[2018-11-21 15:36:00.958833] E [MSGID: 106115] [glusterd-mgmt.c:124:gd_mgmt_v3_collate_errors] 0-management: Locking failed on fs-apollo-c1-n3. Please check log file for details.<br>[2018-11-21 15:36:00.960197] W [glusterd-locks.c:622:glusterd_mgmt_v3_lock] (-->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0x3fc43) [0x7f206876cc43] -->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0x30c73) [0x7f206875dc73] -->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0xdb5d3) [0x7f20688085d3] ) 0-management: Lock for archive1 held by 7e7fd30a-94c9-4c21-9c4b-3817508ef121<br>[2018-11-21 15:36:00.960234] E [MSGID: 106118] [glusterd-op-sm.c:4173:glusterd_op_ac_lock] 0-management: Unable to acquire lock for archive1<br>[2018-11-21 15:36:00.960311] E [MSGID: 106376] [glusterd-op-sm.c:8305:glusterd_op_sm] 0-management: handler returned: -1<br>[2018-11-21 15:36:00.961494] E [MSGID: 106115] [glusterd-mgmt.c:124:gd_mgmt_v3_collate_errors] 0-management: Locking failed on fs-apollo-c1-n1. Please check log file for details.<br>[2018-11-21 15:36:00.961603] E [MSGID: 106150] [glusterd-syncop.c:1957:gd_sync_task_begin] 0-management: Locking Peers Failed.<br>[2018-11-21 15:36:00.962117] W [glusterd-locks.c:856:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0x3fc43) [0x7f206876cc43] -->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0x3099f) [0x7f206875d99f] -->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0xdba74) [0x7f2068808a74] ) 0-management: Lock owner mismatch. Lock for vol archive1 held by 7e7fd30a-94c9-4c21-9c4b-3817508ef121<br>[2018-11-21 15:36:00.962144] E [MSGID: 106117] [glusterd-op-sm.c:4236:glusterd_op_ac_unlock] 0-management: Unable to release lock for archive1<br>[2018-11-21 15:36:00.962222] E [MSGID: 106376] [glusterd-op-sm.c:8305:glusterd_op_sm] 0-management: handler returned: 1<br>[2018-11-21 15:36:00.962406] E [MSGID: 106115] [glusterd-mgmt.c:124:gd_mgmt_v3_collate_errors] 0-management: Unlocking failed on fs-apollo-c1-n1. Please check log file for details.<br>[2018-11-21 15:36:00.962973] E [MSGID: 106115] [glusterd-mgmt.c:124:gd_mgmt_v3_collate_errors] 0-management: Unlocking failed on fs-apollo-c1-n3. Please check log file for details.<br>[2018-11-21 15:36:00.963033] E [MSGID: 106151] [glusterd-syncop.c:1640:gd_unlock_op_phase] 0-management: Failed to unlock on some peer(s)<br>[2018-11-21 15:36:00.991734] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-scrub: error returned while attempting to connect to host:(null), port:0<br>[2018-11-21 15:36:01.186043] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-scrub: error returned while attempting to connect to host:(null), port:0<br>[2018-11-21 15:36:07.416984] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br>The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 11 times between [2018-11-21 15:35:15.371647] and [2018-11-21 15:36:09.383236]<br>[2018-11-21 15:37:01.410979] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br>[2018-11-21 15:37:03.383408] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req<br>The message "I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req" repeated 2 times between [2018-11-21 15:37:01.410979] and [2018-11-21 15:38:49.410710]<br>The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 11 times between [2018-11-21 15:37:03.383408] and [2018-11-21 15:38:51.378524]<br>[2018-11-21 15:39:43.421483] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br>[2018-11-21 15:39:44.786383] E [socket.c:2792:socket_poller] 0-socket.management: socket_poller <a href="http://10.10.12.64:49133">10.10.12.64:49133</a> failed (Input/output error)<br>[2018-11-21 15:39:45.373093] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req<br>[2018-11-21 15:40:37.412592] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br>The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:39:45.373093] and [2018-11-21 15:40:39.378904]<br>[2018-11-21 15:41:31.415866] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br>[2018-11-21 15:41:33.374993] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req<br>[2018-11-21 15:42:25.415585] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br>The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:41:33.374993] and [2018-11-21 15:42:27.379013]<br>[2018-11-21 15:43:19.411237] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br>[2018-11-21 15:43:21.376718] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req<br>[2018-11-21 15:44:13.425206] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br>The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:43:21.376718] and [2018-11-21 15:44:15.375796]<br>[2018-11-21 15:45:07.424344] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br>[2018-11-21 15:45:09.382299] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req<br>[2018-11-21 15:46:02.769673] E [socket.c:2792:socket_poller] 0-socket.management: socket_poller <a href="http://10.10.12.64:49133">10.10.12.64:49133</a> failed (Input/output error)<br>[2018-11-21 15:46:01.422742] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br>The message "I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 7 times between [2018-11-21 15:45:09.382299] and [2018-11-21 15:46:03.370463]<br>[2018-11-21 15:46:55.426992] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<br>[2018-11-21 15:46:57.379488] I [MSGID: 106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management: Received get vol req<br><br><br><br><br>#-> /var/log/glusterfs/run-gluster-shared_storage.log<br><br>[2018-11-21 15:22:48.623969] W [glusterfsd.c:1514:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7559) [0x7f9e25569559] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xfd) [0x561fd19c840d] -->/usr/sbin/glusterfs(cleanup_and_exit+0x5a) [0x561fd19c81fa] ) 0-: received signum (15), shutting down</i></div><div><i>[2018-11-21 15:22:48.624042] I [fuse-bridge.c:5981:fini] 0-fuse: Unmounting '/run/gluster/shared_storage'.<br>[2018-11-21 15:22:48.624185] E [fuse-bridge.c:212:check_and_dump_fuse_W] 0-glusterfs-fuse: writing to fuse device failed: Bad file descriptor<br>[2018-11-21 15:22:48.624246] I [fuse-bridge.c:5199:fuse_thread_proc] 0-fuse: initating unmount of /run/gluster/shared_storage<br>[2018-11-21 15:22:48.624679] I [fuse-bridge.c:5986:fini] 0-fuse: Closing fuse connection to '/run/gluster/shared_storage'.<br><br><br><br>#-> /var/log/glusterfs/bricks/var-lib-glusterd-ss_brick.log<br>[2018-11-21 15:22:48.626860] E [socket.c:2792:socket_poller] 0-tcp.gluster_shared_storage-server: socket_poller <a href="http://10.10.12.62:49151">10.10.12.62:49151</a> failed (Success)<br>[2018-11-21 15:22:48.627048] I [MSGID: 115036] [server.c:483:server_rpc_notify] 0-gluster_shared_storage-server: disconnecting connection from CTX_ID:797e6f66-b69a-4045-bc43-c38d26a2c413-GRAPH_ID:0-PID:4447-HOST:fs-apollo-c1-n2-PC_NAME:gluster_shared_storage-client-0-RECON_NO:-0<br>[2018-11-21 15:22:48.627214] I [MSGID: 101055] [client_t.c:444:gf_client_unref] 0-gluster_shared_storage-server: Shutting down connection CTX_ID:797e6f66-b69a-4045-bc43-c38d26a2c413-GRAPH_ID:0-PID:4447-HOST:fs-apollo-c1-n2-PC_NAME:gluster_shared_storage-client-0-RECON_NO:-0<br>[2018-11-21 15:22:48.632303] W [socket.c:599:__socket_rwv] 0-glusterfs: readv on <a href="http://10.10.12.62:24007">10.10.12.62:24007</a> failed (Success)<br>[2018-11-21 15:22:48.632531] I [glusterfsd-mgmt.c:2348:mgmt_rpc_notify] 0-glusterfsd-mgmt: disconnected from remote-host: fs-apollo-c1-n2<br>[2018-11-21 15:22:48.632558] I [glusterfsd-mgmt.c:2369:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers<br>[2018-11-21 15:22:59.478134] I [MSGID: 115036] [server.c:483:server_rpc_notify] 0-gluster_shared_storage-server: disconnecting connection from CTX_ID:87d36408-3140-457a-b844-bf3ee87f5595-GRAPH_ID:0-PID:4143-HOST:fs-apollo-c1-n2-PC_NAME:gluster_shared_storage-client-0-RECON_NO:-0<br>[2018-11-21 15:22:59.478287] I [MSGID: 101055] [client_t.c:444:gf_client_unref] 0-gluster_shared_storage-server: Shutting down connection CTX_ID:87d36408-3140-457a-b844-bf3ee87f5595-GRAPH_ID:0-PID:4143-HOST:fs-apollo-c1-n2-PC_NAME:gluster_shared_storage-client-0-RECON_NO:-0<br>[2018-11-21 15:23:05.529596] I [glusterfsd-mgmt.c:1953:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing<br>[2018-11-21 15:23:05.625165] I [addr.c:55:compare_addr_and_update] 0-/var/lib/glusterd/ss_brick: allowed = "*", received addr = "10.10.12.62"<br>[2018-11-21 15:23:05.625203] I [login.c:45:gf_auth] 0-auth/login: connecting user name: fs-apollo-c1-n2<br>[2018-11-21 15:23:05.625238] I [login.c:111:gf_auth] 0-auth/login: allowed user names: *<br>[2018-11-21 15:23:05.625254] I [MSGID: 115029] [server-handshake.c:763:server_setvolume] 0-gluster_shared_storage-server: accepted client from CTX_ID:1c203d59-02da-46ec-b836-371ebb47e3dc-GRAPH_ID:0-PID:20295-HOST:fs-apollo-c1-n2-PC_NAME:gluster_shared_storage-client-0-RECON_NO:-0 (version: 4.1.5)<br>[2018-11-21 15:36:01.009586] I [glusterfsd-mgmt.c:53:mgmt_cbk_spec] 0-mgmt: Volume file changed<br>[2018-11-21 15:36:01.017077] I [glusterfsd-mgmt.c:1953:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing<br>[2018-11-21 15:36:01.206445] I [glusterfsd-mgmt.c:53:mgmt_cbk_spec] 0-mgmt: Volume file changed<br>[2018-11-21 15:36:01.212266] I [glusterfsd-mgmt.c:1953:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing<br><br></i><br></div><div><br><div class="gmail_quote"><div dir="ltr">Am Fr., 9. Nov. 2018 um 11:53 Uhr schrieb David Spisla <<a href="mailto:spisla80@gmail.com">spisla80@gmail.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Sure, I will do that the next time shared_storage is unmount after stopping glusterd<br></div><br><div class="gmail_quote"><div dir="ltr">Am Fr., 9. Nov. 2018 um 10:23 Uhr schrieb Rafi Kavungal Chundattu Parambil <<a href="mailto:rkavunga@redhat.com" target="_blank">rkavunga@redhat.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi David Spisla,<br>
<br>
gluster_shared_storage works just like yet another volume(There are some special handling related to authentication). So bringing down a glusterd node shouldn't have lead to unmount of the volume.<br>
<br>
Can you please share the mount log, brick logs at time when you stopped glusterd. Also make sure your shared storage bricks up and running.<br>
<br>
Regards<br>
Rafi KC<br>
<br>
----- Original Message -----<br>
From: "David Spisla" <<a href="mailto:spisla80@gmail.com" target="_blank">spisla80@gmail.com</a>><br>
To: <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
Sent: Tuesday, November 6, 2018 3:25:18 PM<br>
Subject: [Gluster-users] Shared Storage is unmount after stopping glusterd<br>
<br>
Dear Gluster Community, <br>
I have a v4.1.5 4-node-cluster with SLES15 machines. The last time I observe that after stopping glusterd with <br>
$ sudo systemctl stop glusterd <br>
<br>
sometimes also the FUSE mount of the shared_storage on the same node is unmount but shared_storage ist still in the volume list. I mean really "sometimes" because it is not reproducible in a deterministic way and I am not sure if there is really a dependency between stopping glusterd and shared_storage. <br>
<br>
What is the expected behaviour if stopping glusterd? In my opinion all volumes included shared_storage should still have FUSE mounts. <br>
<br>
Regards <br>
David Spisla <br>
<br>
<br>
<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div>
</blockquote></div></div></div></div>