[Bugs] [Bug 1372356] New: glusterd experiencing repeated connect/ disconnect messages when shd is down

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1372356

            Bug ID: 1372356
           Summary: glusterd experiencing repeated connect/disconnect
                    messages when shd is down
           Product: GlusterFS
           Version: mainline
         Component: protocol
          Assignee: bugs at gluster.org
          Reporter: ravishankar at redhat.com
                CC: bugs at gluster.org



Description of problem:
Found this while testing the afr eventing framework patch:

On a 1x2 replicate volume, single node setup, when I kill glustershd, I see
glusterd sending events toggling repeatedly between connected and disconnected.

2016-09-01 16:35:20  SVC_DISCONNECTED e3876d55-89ab-406b-9a49-e32899f70e5d
svc_name=glustershd
2016-09-01 16:35:21  SVC_CONNECTED e3876d55-89ab-406b-9a49-e32899f70e5d
svc_name=glustershd
2016-09-01 16:35:21  SVC_DISCONNECTED e3876d55-89ab-406b-9a49-e32899f70e5d
svc_name=glustershd
2016-09-01 16:35:24  SVC_CONNECTED e3876d55-89ab-406b-9a49-e32899f70e5d
svc_name=glustershd
2016-09-01 16:35:24  SVC_DISCONNECTED e3876d55-89ab-406b-9a49-e32899f70e5d
svc_name=glustershd
2016-09-01 16:35:27  SVC_CONNECTED e3876d55-89ab-406b-9a49-e32899f70e5d
svc_name=glustershd
2016-09-01 16:35:27  SVC_DISCONNECTED e3876d55-89ab-406b-9a49-e32899f70e5d
svc_name=glustershd


How reproducible:
Always.

Steps to Reproduce:
1. Start glusterd in debug mode
2. Create a 1x2 replica volume and start it.
3. Kill the self-heal daemon
4. From the glusterd logs:


2016-09-01 13:37:59.915866] D [socket.c:3058:socket_connect] 0-glustershd:
connection attempt on /var/run/gluster/628f74c872e682fe7c5003e222cab86a.socket
failed, (Connection refused)
[2016-09-01 13:37:59.916053] D [MSGID: 0]
[glusterd-svc-mgmt.c:318:glusterd_svc_common_rpc_notify] 0-management:
glustershd has connected with glusterd.
[2016-09-01 13:37:59.916268] D [socket.c:2384:socket_event_handler]
0-transport: disconnecting now
[2016-09-01 13:37:59.917535] D
[rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
/usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x1e9)[0x7f9e23ab4191] (-->
/usr/local/lib/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0xc5)[0x7f9e2387fe02]
(-->
/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xcb)[0x7f9e23879ba9]
(--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x1aa)[0x7f9e2387a689] (-->
/usr/local/lib/libgfrpc.so.0(rpc_transport_notify+0x10f)[0x7f9e23876b70] )))))
0-: /var/run/gluster/628f74c872e682fe7c5003e222cab86a.socket: ping timer event
already removed
[2016-09-01 13:37:59.917649] I [MSGID: 106006]
[glusterd-svc-mgmt.c:327:glusterd_svc_common_rpc_notify] 0-management:
glustershd has disconnected from glusterd.
[2016-09-01 13:37:59.917802] D [MSGID: 0]
[event-epoll.c:587:event_dispatch_epoll_handler] 0-epoll: generation bumped on
idx=2 from gen=13 to slot->gen=14, fd=6, slot->fd=6








[2016-09-01 13:38:02.916508] D [socket.c:3058:socket_connect] 0-glustershd:
connection attempt on /var/run/gluster/628f74c872e682fe7c5003e222cab86a.socket
failed, (Connection refused)
[2016-09-01 13:38:02.916567] D [MSGID: 0]
[glusterd-svc-mgmt.c:318:glusterd_svc_common_rpc_notify] 0-management:
glustershd has connected with glusterd.
[2016-09-01 13:38:02.916667] D [socket.c:2384:socket_event_handler]
0-transport: disconnecting now
[2016-09-01 13:38:02.916811] D
[rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
/usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x1e9)[0x7f9e23ab4191] (-->
/usr/local/lib/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0xc5)[0x7f9e2387fe02]
(-->
/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xcb)[0x7f9e23879ba9]
(--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x1aa)[0x7f9e2387a689] (-->
/usr/local/lib/libgfrpc.so.0(rpc_transport_notify+0x10f)[0x7f9e23876b70] )))))
0-: /var/run/gluster/628f74c872e682fe7c5003e222cab86a.socket: ping timer event
already removed
[2016-09-01 13:38:02.916830] I [MSGID: 106006]
[glusterd-svc-mgmt.c:327:glusterd_svc_common_rpc_notify] 0-management:
glustershd has disconnected from glusterd.
[2016-09-01 13:38:02.916888] D [MSGID: 0]
[event-epoll.c:587:event_dispatch_epoll_handler] 0-epoll: generation bumped on
idx=2 from gen=16 to slot->gen=17, fd=6, slot->fd=6













[2016-09-01 13:38:05.917660] D [socket.c:3058:socket_connect] 0-glustershd:
connection attempt on /var/run/gluster/628f74c872e682fe7c5003e222cab86a.socket
failed, (Connection refused)
[2016-09-01 13:38:05.917720] D [MSGID: 0]
[glusterd-svc-mgmt.c:318:glusterd_svc_common_rpc_notify] 0-management:
glustershd has connected with glusterd.
[2016-09-01 13:38:05.917940] D [socket.c:2384:socket_event_handler]
0-transport: disconnecting now
[2016-09-01 13:38:05.918274] D
[rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (-->
/usr/local/lib/libglusterfs.so.0(_gf_log_callingfn+0x1e9)[0x7f9e23ab4191] (-->
/usr/local/lib/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0xc5)[0x7f9e2387fe02]
(-->
/usr/local/lib/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xcb)[0x7f9e23879ba9]
(--> /usr/local/lib/libgfrpc.so.0(rpc_clnt_notify+0x1aa)[0x7f9e2387a689] (-->
/usr/local/lib/libgfrpc.so.0(rpc_transport_notify+0x10f)[0x7f9e23876b70] )))))
0-: /var/run/gluster/628f74c872e682fe7c5003e222cab86a.socket: ping timer event
already removed
[2016-09-01 13:38:05.918299] I [MSGID: 106006]
[glusterd-svc-mgmt.c:327:glusterd_svc_common_rpc_notify] 0-management:
glustershd has disconnected from glusterd.
[2016-09-01 13:38:05.918467] D [MSGID: 0]
[event-epoll.c:587:event_dispatch_epoll_handler] 0-epoll: generation bumped on
idx=2 from gen=19 to slot->gen=20, fd=6, slot->fd=6

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.