[Bugs] [Bug 1677160] Gluster 5 client can't access Gluster 3.12 servers

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1677160

Darrell <budic at onholyground.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |budic at onholyground.com



--- Comment #12 from Darrell <budic at onholyground.com> ---
I encountered this with 5.3 and 5.5 clients connecting to gluster 3.12.15
servers. Might be multiple problems.

At first, I encountered https://bugzilla.redhat.com/show_bug.cgi?id=1651246
with 5.3 clients, and 5.5 resolved that problem. I've hit a new one though, so
adding my details.

Initially, a new 5.5 mount to a 3.12.15 cluster of 3 servers succeeds and
everything works well. If you reboot one of the servers, however, all clients
no longer connect to it and the other servers are forced to heal everything to
the 3rd server. Restarting the clients (new mounts) will cause them to
reconnect until you restart a server again. Affects both fuse and glfapi
clients.

Server brick example from rebooted server (lots of these repeating):
[2019-03-25 17:45:37.588519] I [socket.c:3679:socket_submit_reply]
0-socket.mana
gement: not connected (priv->connected = -1)
[2019-03-25 17:45:37.588571] E [rpcsvc.c:1364:rpcsvc_submit_generic]
0-rpc-servi
ce: failed to submit message (XID: 0x542ab, Program: GF-DUMP, ProgVers: 1,
Proc:
 2) to rpc-transport (socket.management)
[2019-03-25 17:48:25.944496] I [socket.c:3679:socket_submit_reply]
0-socket.mana
gement: not connected (priv->connected = -1)
[2019-03-25 17:48:25.944547] E [rpcsvc.c:1364:rpcsvc_submit_generic]
0-rpc-servi
ce: failed to submit message (XID: 0x38036, Program: GF-DUMP, ProgVers: 1,
Proc:
 2) to rpc-transport (socket.management)
[2019-03-25 17:50:34.306141] I [socket.c:3679:socket_submit_reply]
0-socket.mana
gement: not connected (priv->connected = -1)
[2019-03-25 17:50:34.306206] E [rpcsvc.c:1364:rpcsvc_submit_generic]
0-rpc-servi
ce: failed to submit message (XID: 0x1e050e, Program: GF-DUMP, ProgVers: 1,
Proc
: 2) to rpc-transport (socket.management)
[2019-03-25 17:51:58.082944] I [socket.c:3679:socket_submit_reply]
0-socket.mana
gement: not connected (priv->connected = -1)
[2019-03-25 17:51:58.082999] E [rpcsvc.c:1364:rpcsvc_submit_generic]
0-rpc-servi
ce: failed to submit message (XID: 0x1ec5, Program: GF-DUMP, ProgVers: 1, Proc: 
2) to rpc-transport (socket.management)

Client brick example (also lots repeating):
[2019-03-26 14:55:50.582757] W [rpc-clnt-ping.c:215:rpc_clnt_ping_cbk]
0-gv1-client-1: socket disconnected
[2019-03-26 14:55:54.582490] I [rpc-clnt.c:2042:rpc_clnt_reconfig]
0-gv1-client-1: changing port to 50155 (from 0)
[2019-03-26 14:55:54.585627] E [rpc-clnt.c:346:saved_frames_unwind] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a5164efbb] (-->
/lib64/libgfrpc.so.0(+0xce11)[0x7f4a51417e11] (-->
/lib64/libgfrpc.so.0(+0xcf2e)[0x7f4a51417f2e] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x91)[0x7f4a51419531] (-->
/lib64/libgfrpc.so.0(+0xf0d8)[0x7f4a5141a0d8] ))))) 0-gv1-client-1: forced
unwinding frame type(GF-DUMP) op(NULL(2)) called at 2019-03-26 14:55:54.585283
(xid=0x3ef42)
[2019-03-26 14:55:54.585644] W [rpc-clnt-ping.c:215:rpc_clnt_ping_cbk]
0-gv1-client-1: socket disconnected
[2019-03-26 14:55:58.585636] I [rpc-clnt.c:2042:rpc_clnt_reconfig]
0-gv1-client-1: changing port to 50155 (from 0)
[2019-03-26 14:55:58.588760] E [rpc-clnt.c:346:saved_frames_unwind] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a5164efbb] (-->
/lib64/libgfrpc.so.0(+0xce11)[0x7f4a51417e11] (-->
/lib64/libgfrpc.so.0(+0xcf2e)[0x7f4a51417f2e] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x91)[0x7f4a51419531] (-->
/lib64/libgfrpc.so.0(+0xf0d8)[0x7f4a5141a0d8] ))))) 0-gv1-client-1: forced
unwinding frame type(GF-DUMP) op(NULL(2)) called at 2019-03-26 14:55:58.588478
(xid=0x3ef47)
[2019-03-26 14:55:58.588779] W [rpc-clnt-ping.c:215:rpc_clnt_ping_cbk]
0-gv1-client-1: socket disconnected
[2019-03-26 14:56:02.589009] I [rpc-clnt.c:2042:rpc_clnt_reconfig]
0-gv1-client-1: changing port to 50155 (from 0)
[2019-03-26 14:56:02.592150] E [rpc-clnt.c:346:saved_frames_unwind] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a5164efbb] (-->
/lib64/libgfrpc.so.0(+0xce11)[0x7f4a51417e11] (-->
/lib64/libgfrpc.so.0(+0xcf2e)[0x7f4a51417f2e] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x91)[0x7f4a51419531] (-->
/lib64/libgfrpc.so.0(+0xf0d8)[0x7f4a5141a0d8] ))))) 0-gv1-client-1: forced
unwinding frame type(GF-DUMP) op(NULL(2)) called at 2019-03-26 14:56:02.591818
(xid=0x3ef4c)
[2019-03-26 14:56:02.592166] W [rpc-clnt-ping.c:215:rpc_clnt_ping_cbk]
0-gv1-client-1: socket disconnected
[2019-03-26 14:56:06.592208] I [rpc-clnt.c:2042:rpc_clnt_reconfig]
0-gv1-client-1: changing port to 50155 (from 0)
[2019-03-26 14:56:06.595306] E [rpc-clnt.c:346:saved_frames_unwind] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a5164efbb] (-->
/lib64/libgfrpc.so.0(+0xce11)[0x7f4a51417e11] (-->
/lib64/libgfrpc.so.0(+0xcf2e)[0x7f4a51417f2e] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x91)[0x7f4a51419531] (-->
/lib64/libgfrpc.so.0(+0xf0d8)[0x7f4a5141a0d8] ))))) 0-gv1-client-1: forced
unwinding frame type(GF-DUMP) op(NULL(2)) called at 2019-03-26 14:56:06.594965
(xid=0x3ef51)
[2019-03-26 14:56:06.595343] W [rpc-clnt-ping.c:215:rpc_clnt_ping_cbk]
0-gv1-client-1: socket disconnected
[2019-03-26 14:56:10.594781] I [rpc-clnt.c:2042:rpc_clnt_reconfig]
0-gv1-client-1: changing port to 50155 (from 0)
[2019-03-26 14:56:10.597780] E [rpc-clnt.c:346:saved_frames_unwind] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a5164efbb] (-->
/lib64/libgfrpc.so.0(+0xce11)[0x7f4a51417e11] (-->
/lib64/libgfrpc.so.0(+0xcf2e)[0x7f4a51417f2e] (-->
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x91)[0x7f4a51419531] (-->
/lib64/libgfrpc.so.0(+0xf0d8)[0x7f4a5141a0d8] ))))) 0-gv1-client-1: forced
unwinding frame type(GF-DUMP) op(NULL(2)) called at 2019-03-26 14:56:10.597488
(xid=0x3ef56)
[2019-03-26 14:56:10.597796] W [rpc-clnt-ping.c:215:rpc_clnt_ping_cbk]
0-gv1-client-1: socket disconnected
[2019-03-26 14:56:14.597866] I [rpc-clnt.c:2042:rpc_clnt_reconfig]
0-gv1-client-1: changing port to 50155 (from 0)

Bricks didn't crash, just the clients wouldn't talk to them.

Upgrading the currently affected server to 5.5 and rebooting it caused the
clients to reconnect to normally.

-- 
You are receiving this mail because:
You are on the CC list for the bug.