[Gluster-users] Worrying

[Hiren Joshi]

Hello all,
 
I have a 2 servers each exporting 6 bricks. The client mirrors the 2
servers and AFRs the 6 mirrors it creates.
 
Running bonnie, the servers kept dropping (according to gluster logs
they stopped responding to pings in 10 seconds) so I set the ping
timeout to 30 second, now although bonnie runs I still see dropouts in
the log.
 
The worrying thing is that one of the servers is localhost! What's
happening here? I'm frustratingly close to putting this system on our
live network.
 
The log:
[2009-09-03 02:10:01] E
[client-protocol.c:437:client_ping_timer_expired] glust1a_1: Server
127.0.0.1:6996 has not responded in the last 30 seconds, disconnecting.
[2009-09-03 01:10:01] E
[client-protocol.c:437:client_ping_timer_expired] glust1b_1: Server
192.168.4.51:6996 has not responded in the last 30 seconds,
disconnecting.
[2009-09-03 01:10:01] E [saved-frames.c:165:saved_frames_unwind]
glust1a_1: forced unwinding frame type(2) op(PING)
[2009-09-03 01:10:01] E
[client-protocol.c:437:client_ping_timer_expired] glust1a_2: Server
127.0.0.1:6996 has not responded in the last 30 seconds, disconnecting.
[2009-09-03 01:10:01] E
[client-protocol.c:437:client_ping_timer_expired] glust1b_2: Server
192.168.4.51:6996 has not responded in the last 30 seconds,
disconnecting.
[2009-09-03 01:10:01] E
[client-protocol.c:437:client_ping_timer_expired] glust1a_3: Server
127.0.0.1:6996 has not responded in the last 30 seconds, disconnecting.
[2009-09-03 01:10:01] E
[client-protocol.c:437:client_ping_timer_expired] glust1b_3: Server
192.168.4.51:6996 has not responded in the last 30 seconds,
disconnecting.
[2009-09-03 01:10:01] E
[client-protocol.c:437:client_ping_timer_expired] glust1a_5: Server
127.0.0.1:6996 has not responded in the last 30 seconds, disconnecting.
[2009-09-03 01:10:01] E
[client-protocol.c:437:client_ping_timer_expired] glust1b_5: Server
192.168.4.51:6996 has not responded in the last 30 seconds,
disconnecting.
[2009-09-03 01:10:01] E
[client-protocol.c:437:client_ping_timer_expired] glust1a_6: Server
127.0.0.1:6996 has not responded in the last 30 seconds, disconnecting.
[2009-09-03 01:10:01] E
[client-protocol.c:437:client_ping_timer_expired] glust1b_6: Server
192.168.4.51:6996 has not responded in the last 30 seconds,
disconnecting.
[2009-09-03 01:10:01] N [client-protocol.c:6246:notify] glust1a_1:
disconnected
[2009-09-03 01:10:01] E [saved-frames.c:165:saved_frames_unwind]
glust1b_6: forced unwinding frame type(2) op(PING)
[2009-09-03 01:10:01] N [client-protocol.c:6246:notify] glust1b_6:
disconnected
[2009-09-03 01:10:01] E [saved-frames.c:165:saved_frames_unwind]
glust1a_6: forced unwinding frame type(2) op(PING)
[2009-09-03 01:10:01] N [client-protocol.c:6246:notify] glust1a_6:
disconnected
[2009-09-03 01:10:01] E [afr.c:2228:notify] mirror1_6: All subvolumes
are down. Going offline until atleast one of them comes back up.
[2009-09-03 01:10:01] E [saved-frames.c:165:saved_frames_unwind]
glust1b_5: forced unwinding frame type(2) op(PING)
[2009-09-03 01:10:01] N [client-protocol.c:6246:notify] glust1b_5:
disconnected
[2009-09-03 01:10:01] E [saved-frames.c:165:saved_frames_unwind]
glust1a_5: forced unwinding frame type(2) op(PING)
[2009-09-03 01:10:01] N [client-protocol.c:6246:notify] glust1a_5:
disconnected
[2009-09-03 01:10:01] E [afr.c:2228:notify] mirror1_5: All subvolumes
are down. Going offline until atleast one of them comes back up.
[2009-09-03 01:10:01] E [saved-frames.c:165:saved_frames_unwind]
glust1b_3: forced unwinding frame type(2) op(PING)
[2009-09-03 01:10:01] N [client-protocol.c:6246:notify] glust1b_3:
disconnected
[2009-09-03 01:10:01] E [saved-frames.c:165:saved_frames_unwind]
glust1a_3: forced unwinding frame type(2) op(PING)
[2009-09-03 01:10:01] N [client-protocol.c:6246:notify] glust1a_3:
disconnected
[2009-09-03 01:10:01] E [afr.c:2228:notify] mirror1_3: All subvolumes
are down. Going offline until atleast one of them comes back up.
[2009-09-03 01:10:01] E [saved-frames.c:165:saved_frames_unwind]
glust1b_2: forced unwinding frame type(2) op(PING)
[2009-09-03 01:10:01] N [client-protocol.c:6246:notify] glust1b_2:
disconnected
[2009-09-03 01:10:01] E [saved-frames.c:165:saved_frames_unwind]
glust1a_2: forced unwinding frame type(2) op(PING)
[2009-09-03 01:10:01] N [client-protocol.c:6246:notify] glust1a_2:
disconnected
[2009-09-03 01:10:01] E [afr.c:2228:notify] mirror1_2: All subvolumes
are down. Going offline until atleast one of them comes back up.
[2009-09-03 01:10:01] E [saved-frames.c:165:saved_frames_unwind]
glust1b_1: forced unwinding frame type(2) op(PING)
[2009-09-03 01:10:02] N [client-protocol.c:6246:notify] glust1b_1:
disconnected
[2009-09-03 01:10:02] E [afr.c:2228:notify] mirror1_1: All subvolumes
are down. Going offline until atleast one of them comes back up.
[2009-09-03 01:10:31] E
[client-protocol.c:437:client_ping_timer_expired] glust1a_4: Server
127.0.0.1:6996 has not responded in the last 30 seconds, disconnecting.
[2009-09-03 01:10:31] E
[client-protocol.c:437:client_ping_timer_expired] glust1b_4: Server
192.168.4.51:6996 has not responded in the last 30 seconds,
disconnecting.
[2009-09-03 01:10:31] E [saved-frames.c:165:saved_frames_unwind]
glust1b_4: forced unwinding frame type(1) op(LOOKUP)
[2009-09-03 01:10:31] E [saved-frames.c:165:saved_frames_unwind]
glust1b_4: forced unwinding frame type(2) op(PING)
[2009-09-03 01:10:31] E [saved-frames.c:165:saved_frames_unwind]
glust1b_4: forced unwinding frame type(2) op(PING)
[2009-09-03 01:10:31] E [saved-frames.c:165:saved_frames_unwind]
glust1b_4: forced unwinding frame type(3) op(RELEASE)
[2009-09-03 01:10:31] E [saved-frames.c:165:saved_frames_unwind]
glust1b_4: forced unwinding frame type(3) op(FORGET)
[2009-09-03 01:10:31] N [client-protocol.c:6246:notify] glust1b_4:
disconnected
[2009-09-03 01:10:31] E [saved-frames.c:165:saved_frames_unwind]
glust1a_4: forced unwinding frame type(1) op(LOOKUP)
[2009-09-03 01:10:31] E [saved-frames.c:165:saved_frames_unwind]
glust1a_4: forced unwinding frame type(2) op(PING)
[2009-09-03 01:10:31] E [saved-frames.c:165:saved_frames_unwind]
glust1a_4: forced unwinding frame type(2) op(PING)
[2009-09-03 01:10:31] E [saved-frames.c:165:saved_frames_unwind]
glust1a_4: forced unwinding frame type(3) op(RELEASE)
[2009-09-03 01:10:31] E [saved-frames.c:165:saved_frames_unwind]
glust1a_4: forced unwinding frame type(3) op(FORGET)
[2009-09-03 01:10:31] N [client-protocol.c:6246:notify] glust1a_4:
disconnected
[2009-09-03 01:10:31] E [afr.c:2228:notify] mirror1_4: All subvolumes
are down. Going offline until atleast one of them comes back up.
[2009-09-03 01:10:31] N [client-protocol.c:5559:client_setvolume_cbk]
glust1b_6: Connected to 192.168.4.51:6996, attached to remote volume
'brick6'.
[2009-09-03 01:10:31] N [afr.c:2204:notify] mirror1_6: Subvolume
'glust1b_6' came back up; going online.
[2009-09-03 01:10:31] N [client-protocol.c:5559:client_setvolume_cbk]
glust1b_6: Connected to 192.168.4.51:6996, attached to remote volume
'brick6'.
[2009-09-03 01:10:31] N [afr.c:2204:notify] mirror1_6: Subvolume
'glust1b_6' came back up; going online.
[2009-09-03 01:10:31] N [client-protocol.c:5559:client_setvolume_cbk]
glust1b_5: Connected to 192.168.4.51:6996, attached to remote volume
'brick5'.
[2009-09-03 01:10:31] N [afr.c:2204:notify] mirror1_5: Subvolume
'glust1b_5' came back up; going online.
[2009-09-03 01:10:31] N [client-protocol.c:5559:client_setvolume_cbk]
glust1b_5: Connected to 192.168.4.51:6996, attached to remote volume
'brick5'.
[2009-09-03 01:10:31] N [afr.c:2204:notify] mirror1_5: Subvolume
'glust1b_5' came back up; going online.
[2009-09-03 01:10:31] N [client-protocol.c:5559:client_setvolume_cbk]
glust1b_3: Connected to 192.168.4.51:6996, attached to remote volume
'brick3'.
[2009-09-03 01:10:31] N [afr.c:2204:notify] mirror1_3: Subvolume
'glust1b_3' came back up; going online.
[2009-09-03 01:10:31] N [client-protocol.c:5559:client_setvolume_cbk]
glust1b_3: Connected to 192.168.4.51:6996, attached to remote volume
'brick3'.
[2009-09-03 01:10:31] N [afr.c:2204:notify] mirror1_3: Subvolume
'glust1b_3' came back up; going online.
[2009-09-03 01:10:31] N [client-protocol.c:5559:client_setvolume_cbk]
glust1b_2: Connected to 192.168.4.51:6996, attached to remote volume
'brick2'.
[2009-09-03 01:10:31] N [afr.c:2204:notify] mirror1_2: Subvolume
'glust1b_2' came back up; going online.
[2009-09-03 01:10:31] N [client-protocol.c:5559:client_setvolume_cbk]
glust1b_2: Connected to 192.168.4.51:6996, attached to remote volume
'brick2'.
[2009-09-03 01:10:31] N [afr.c:2204:notify] mirror1_2: Subvolume
'glust1b_2' came back up; going online.
[2009-09-03 01:10:31] N [client-protocol.c:5559:client_setvolume_cbk]
glust1b_1: Connected to 192.168.4.51:6996, attached to remote volume
'brick1'.
[2009-09-03 01:10:31] N [afr.c:2204:notify] mirror1_1: Subvolume
'glust1b_1' came back up; going online.
[2009-09-03 01:10:31] N [client-protocol.c:5559:client_setvolume_cbk]
glust1b_1: Connected to 192.168.4.51:6996, attached to remote volume
'brick1'.
[2009-09-03 01:10:31] N [afr.c:2204:notify] mirror1_1: Subvolume
'glust1b_1' came back up; going online.
[2009-09-03 01:10:31] N [client-protocol.c:5559:client_setvolume_cbk]
glust1b_4: Connected to 192.168.4.51:6996, attached to remote volume
'brick4'.
[2009-09-03 01:10:31] N [afr.c:2204:notify] mirror1_4: Subvolume
'glust1b_4' came back up; going online.
[2009-09-03 01:10:32] N [client-protocol.c:5559:client_setvolume_cbk]
glust1b_4: Connected to 192.168.4.51:6996, attached to remote volume
'brick4'.
[2009-09-03 01:10:32] N [afr.c:2204:notify] mirror1_4: Subvolume
'glust1b_4' came back up; going online.
[2009-09-03 01:10:55] N [client-protocol.c:5559:client_setvolume_cbk]
glust1a_1: Connected to 127.0.0.1:6996, attached to remote volume
'brick1'.
[2009-09-03 01:10:55] N [client-protocol.c:5559:client_setvolume_cbk]
glust1a_1: Connected to 127.0.0.1:6996, attached to remote volume
'brick1'.
[2009-09-03 01:10:55] N [client-protocol.c:5559:client_setvolume_cbk]
glust1a_6: Connected to 127.0.0.1:6996, attached to remote volume
'brick6'.
[2009-09-03 01:10:55] N [client-protocol.c:5559:client_setvolume_cbk]
glust1a_6: Connected to 127.0.0.1:6996, attached to remote volume
'brick6'.
[2009-09-03 01:10:55] N [client-protocol.c:5559:client_setvolume_cbk]
glust1a_5: Connected to 127.0.0.1:6996, attached to remote volume
'brick5'.
[2009-09-03 01:10:55] N [client-protocol.c:5559:client_setvolume_cbk]
glust1a_5: Connected to 127.0.0.1:6996, attached to remote volume
'brick5'.
[2009-09-03 01:10:55] N [client-protocol.c:5559:client_setvolume_cbk]
glust1a_3: Connected to 127.0.0.1:6996, attached to remote volume
'brick3'.
[2009-09-03 01:10:55] N [client-protocol.c:5559:client_setvolume_cbk]
glust1a_3: Connected to 127.0.0.1:6996, attached to remote volume
'brick3'.
[2009-09-03 01:10:55] N [client-protocol.c:5559:client_setvolume_cbk]
glust1a_2: Connected to 127.0.0.1:6996, attached to remote volume
'brick2'.
[2009-09-03 01:10:55] N [client-protocol.c:5559:client_setvolume_cbk]
glust1a_2: Connected to 127.0.0.1:6996, attached to remote volume
'brick2'.
[2009-09-03 01:10:55] N [client-protocol.c:5559:client_setvolume_cbk]
glust1a_4: Connected to 127.0.0.1:6996, attached to remote volume
'brick4'.
[2009-09-03 01:10:57] N [client-protocol.c:5559:client_setvolume_cbk]
glust1a_4: Connected to 127.0.0.1:6996, attached to remote volume
'brick4'.
[2009-09-03 01:13:44] E
[client-protocol.c:437:client_ping_timer_expired] glust1a_4: Server
127.0.0.1:6996 has not responded in the last 30 seconds, disconnecting.
[2009-09-03 01:13:44] N [client-protocol.c:6246:notify] glust1a_4:
disconnected
[2009-09-03 01:13:44] N [client-protocol.c:5559:client_setvolume_cbk]
glust1a_4: Connected to 127.0.0.1:6996, attached to remote volume
'brick4'.
[2009-09-03 01:13:44] N [client-protocol.c:5559:client_setvolume_cbk]
glust1a_4: Connected to 127.0.0.1:6996, attached to remote volume
'brick4'.
[2009-09-03 01:13:57] E
[client-protocol.c:437:client_ping_timer_expired] glust1b_4: Server
192.168.4.51:6996 has not responded in the last 30 seconds,
disconnecting.
[2009-09-03 01:13:57] N [client-protocol.c:6246:notify] glust1b_4:
disconnected
[2009-09-03 01:13:57] N [client-protocol.c:5559:client_setvolume_cbk]
glust1b_4: Connected to 192.168.4.51:6996, attached to remote volume
'brick4'.
[2009-09-03 01:13:57] N [client-protocol.c:5559:client_setvolume_cbk]
glust1b_4: Connected to 192.168.4.51:6996, attached to remote volume
'brick4'.