[2018-07-11 19:16:14.918389] W [socket.c:593:__socket_rwv] 0-gv1-client-2: readv on 10.4.16.19:49153 failed (No data available)
[2018-07-11 19:16:14.918495] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 0-gv1-client-2: disconnected from gv1-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2018-07-11 19:16:14.920183] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fd882449efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fd88220ee6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fd88220ef8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fd882210710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fd882211200] ))))) 0-gv1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2018-07-11 19:15:47.548069 (xid=0x1be2e4)
[2018-07-11 19:16:14.920207] E [MSGID: 114031] [client-rpc-fops.c:1557:client3_3_finodelk_cbk] 0-gv1-client-2: remote operation failed [Transport endpoint is not connected]
[2018-07-11 19:16:14.920386] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fd882449efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fd88220ee6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fd88220ef8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fd882210710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fd882211200] ))))) 0-gv1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2018-07-11 19:15:52.555712 (xid=0x1be2e5)
[2018-07-11 19:16:14.920559] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fd882449efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fd88220ee6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fd88220ef8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fd882210710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fd882211200] ))))) 0-gv1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2018-07-11 19:15:52.555852 (xid=0x1be2e6)
[2018-07-11 19:16:14.920732] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fd882449efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fd88220ee6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fd88220ef8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fd882210710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fd882211200] ))))) 0-gv1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2018-07-11 19:15:52.556010 (xid=0x1be2e7)
[2018-07-11 19:16:14.920919] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fd882449efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fd88220ee6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fd88220ef8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fd882210710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fd882211200] ))))) 0-gv1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2018-07-11 19:15:53.419502 (xid=0x1be2e9)
[2018-07-11 19:16:14.921094] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fd882449efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fd88220ee6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fd88220ef8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fd882210710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fd882211200] ))))) 0-gv1-client-2: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at 2018-07-11 19:15:53.341989 (xid=0x1be2e8)
[2018-07-11 19:16:14.921114] W [rpc-clnt-ping.c:223:rpc_clnt_ping_cbk] 0-gv1-client-2: socket disconnected
[2018-07-11 19:16:14.921274] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fd882449efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fd88220ee6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fd88220ef8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fd882210710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fd882211200] ))))) 0-gv1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FSTAT(25)) called at 2018-07-11 19:15:54.292390 (xid=0x1be2ea)
[2018-07-11 19:16:14.921292] W [MSGID: 114031] [client-rpc-fops.c:1459:client3_3_fstat_cbk] 0-gv1-client-2: remote operation failed [Transport endpoint is not connected]
[2018-07-11 19:16:14.921328] W [MSGID: 114061] [client-common.c:704:client_pre_fstat] 0-gv1-client-2:  (037fd1d1-71e7-4553-92ba-9a84c02bb63c) remote_fd is -1. EBADFD [File descriptor in bad state]
[2018-07-11 19:16:16.231994] W [socket.c:593:__socket_rwv] 0-gv1-client-4: readv on 10.4.16.12:49152 failed (No data available)
[2018-07-11 19:16:16.232035] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 0-gv1-client-4: disconnected from gv1-client-4. Client process will keep trying to connect to glusterd until brick's port is available
[2018-07-11 19:16:16.232056] W [MSGID: 108001] [afr-common.c:5319:afr_notify] 0-gv1-replicate-0: Client-quorum is not met
[2018-07-11 19:16:16.232602] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fd882449efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fd88220ee6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fd88220ef8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fd882210710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fd882211200] ))))) 0-gv1-client-4: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2018-07-11 19:15:47.548098 (xid=0x2f323e)
[2018-07-11 19:16:16.232621] E [MSGID: 114031] [client-rpc-fops.c:1557:client3_3_finodelk_cbk] 0-gv1-client-4: remote operation failed [Transport endpoint is not connected]
[2018-07-11 19:16:16.232792] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fd882449efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fd88220ee6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fd88220ef8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fd882210710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fd882211200] ))))) 0-gv1-client-4: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2018-07-11 19:15:52.555732 (xid=0x2f323f)
[2018-07-11 19:16:16.233004] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fd882449efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fd88220ee6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fd88220ef8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fd882210710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fd882211200] ))))) 0-gv1-client-4: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2018-07-11 19:15:52.555884 (xid=0x2f3240)
[2018-07-11 19:16:16.233179] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fd882449efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fd88220ee6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fd88220ef8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fd882210710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fd882211200] ))))) 0-gv1-client-4: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2018-07-11 19:15:52.556033 (xid=0x2f3241)
[2018-07-11 19:16:16.233350] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fd882449efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fd88220ee6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fd88220ef8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fd882210710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fd882211200] ))))) 0-gv1-client-4: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2018-07-11 19:15:53.419536 (xid=0x2f3242)
[2018-07-11 19:16:16.233521] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fd882449efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fd88220ee6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fd88220ef8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fd882210710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fd882211200] ))))) 0-gv1-client-4: forced unwinding frame type(GlusterFS 3.3) op(FSTAT(25)) called at 2018-07-11 19:16:04.200401 (xid=0x2f3243)
The message "E [MSGID: 114031] [client-rpc-fops.c:1557:client3_3_finodelk_cbk] 0-gv1-client-4: remote operation failed [Transport endpoint is not connected]" repeated 4 times between [2018-07-11 19:16:16.232621] and [2018-07-11 19:16:16.233368]
[2018-07-11 19:16:16.233540] W [MSGID: 114031] [client-rpc-fops.c:1459:client3_3_fstat_cbk] 0-gv1-client-4: remote operation failed [Transport endpoint is not connected]
[2018-07-11 19:16:16.233578] W [MSGID: 114061] [client-common.c:704:client_pre_fstat] 0-gv1-client-2:  (de4eb8bc-f5fa-4da8-b907-1562cbd38dfe) remote_fd is -1. EBADFD [File descriptor in bad state]
[2018-07-11 19:16:16.233609] W [MSGID: 114061] [client-common.c:704:client_pre_fstat] 0-gv1-client-4:  (de4eb8bc-f5fa-4da8-b907-1562cbd38dfe) remote_fd is -1. EBADFD [File descriptor in bad state]
[2018-07-11 19:16:16.233779] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fd882449efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fd88220ee6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fd88220ef8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fd882210710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fd882211200] ))))) 0-gv1-client-4: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at 2018-07-11 19:16:14.345044 (xid=0x2f3244)
[2018-07-11 19:16:16.233797] W [rpc-clnt-ping.c:223:rpc_clnt_ping_cbk] 0-gv1-client-4: socket disconnected
[2018-07-11 19:16:16.233968] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fd882449efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fd88220ee6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fd88220ef8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fd882210710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fd882211200] ))))) 0-gv1-client-4: forced unwinding frame type(GlusterFS 3.3) op(FSTAT(25)) called at 2018-07-11 19:16:14.921385 (xid=0x2f3245)
[2018-07-11 19:16:16.233992] W [MSGID: 114031] [client-rpc-fops.c:1459:client3_3_fstat_cbk] 0-gv1-client-4: remote operation failed [Transport endpoint is not connected]
[2018-07-11 19:16:16.362403] W [socket.c:593:__socket_rwv] 0-gv1-client-3: readv on 10.4.16.11:49153 failed (No data available)
[2018-07-11 19:16:16.362449] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 0-gv1-client-3: disconnected from gv1-client-3. Client process will keep trying to connect to glusterd until brick's port is available
[2018-07-11 19:16:16.362472] E [MSGID: 108006] [afr-common.c:5092:__afr_handle_child_down_event] 0-gv1-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
[2018-07-11 19:16:16.362729] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fd882449efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fd88220ee6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fd88220ef8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fd882210710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fd882211200] ))))) 0-gv1-client-3: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2018-07-11 19:15:47.548081 (xid=0x1e3320)
[2018-07-11 19:16:16.362749] E [MSGID: 114031] [client-rpc-fops.c:1557:client3_3_finodelk_cbk] 0-gv1-client-3: remote operation failed [Transport endpoint is not connected]
[2018-07-11 19:16:16.362775] E [MSGID: 114031] [client-rpc-fops.c:1557:client3_3_finodelk_cbk] 0-gv1-client-3: remote operation failed [Transport endpoint is not connected]
[2018-07-11 19:16:16.362778] E [MSGID: 114031] [client-rpc-fops.c:1557:client3_3_finodelk_cbk] 0-gv1-client-4: remote operation failed [Transport endpoint is not connected]
[2018-07-11 19:16:16.362800] W [MSGID: 108019] [afr-lk-common.c:1102:is_blocking_locks_count_sufficient] 0-gv1-replicate-0: Unable to obtain blocking inode lock on even one child for gfid:59555375-1db1-428a-a7f2-08dcd79f0800.
[2018-07-11 19:16:16.362811] I [MSGID: 108019] [afr-transaction.c:1820:afr_post_blocking_inodelk_cbk] 0-gv1-replicate-0: Blocking inodelks failed.
[2018-07-11 19:16:16.363048] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fd882449efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fd88220ee6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fd88220ef8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fd882210710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fd882211200] ))))) 0-gv1-client-3: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2018-07-11 19:15:52.555722 (xid=0x1e3322)
[2018-07-11 19:16:16.363068] E [MSGID: 114031] [client-rpc-fops.c:1557:client3_3_finodelk_cbk] 0-gv1-client-3: remote operation failed [Transport endpoint is not connected]
[2018-07-11 19:16:16.363090] E [MSGID: 114031] [client-rpc-fops.c:1557:client3_3_finodelk_cbk] 0-gv1-client-3: remote operation failed [Transport endpoint is not connected]
[2018-07-11 19:16:16.363092] E [MSGID: 114031] [client-rpc-fops.c:1557:client3_3_finodelk_cbk] 0-gv1-client-4: remote operation failed [Transport endpoint is not connected]
[2018-07-11 19:16:16.363112] W [MSGID: 108019] [afr-lk-common.c:1102:is_blocking_locks_count_sufficient] 0-gv1-replicate-0: Unable to obtain blocking inode lock on even one child for gfid:59555375-1db1-428a-a7f2-08dcd79f0800.
[2018-07-11 19:16:16.363122] I [MSGID: 108019] [afr-transaction.c:1820:afr_post_blocking_inodelk_cbk] 0-gv1-replicate-0: Blocking inodelks failed.
[2018-07-11 19:16:16.363317] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fd882449efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fd88220ee6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fd88220ef8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fd882210710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fd882211200] ))))) 0-gv1-client-3: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2018-07-11 19:15:52.555873 (xid=0x1e3323)
[2018-07-11 19:16:16.363340] E [MSGID: 114031] [client-rpc-fops.c:1557:client3_3_finodelk_cbk] 0-gv1-client-3: remote operation failed [Transport endpoint is not connected]
[2018-07-11 19:16:16.363361] E [MSGID: 114031] [client-rpc-fops.c:1557:client3_3_finodelk_cbk] 0-gv1-client-3: remote operation failed [Transport endpoint is not connected]
[2018-07-11 19:16:16.363363] E [MSGID: 114031] [client-rpc-fops.c:1557:client3_3_finodelk_cbk] 0-gv1-client-4: remote operation failed [Transport endpoint is not connected]
[2018-07-11 19:16:16.363383] W [MSGID: 108019] [afr-lk-common.c:1102:is_blocking_locks_count_sufficient] 0-gv1-replicate-0: Unable to obtain blocking inode lock on even one child for gfid:59555375-1db1-428a-a7f2-08dcd79f0800.
[2018-07-11 19:16:16.363393] I [MSGID: 108019] [afr-transaction.c:1820:afr_post_blocking_inodelk_cbk] 0-gv1-replicate-0: Blocking inodelks failed.
[2018-07-11 19:16:16.363586] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fd882449efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fd88220ee6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fd88220ef8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fd882210710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fd882211200] ))))) 0-gv1-client-3: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2018-07-11 19:15:52.556024 (xid=0x1e3324)
[2018-07-11 19:16:16.363604] E [MSGID: 114031] [client-rpc-fops.c:1557:client3_3_finodelk_cbk] 0-gv1-client-3: remote operation failed [Transport endpoint is not connected]
[2018-07-11 19:16:16.363624] E [MSGID: 114031] [client-rpc-fops.c:1557:client3_3_finodelk_cbk] 0-gv1-client-3: remote operation failed [Transport endpoint is not connected]
[2018-07-11 19:16:16.363626] E [MSGID: 114031] [client-rpc-fops.c:1557:client3_3_finodelk_cbk] 0-gv1-client-4: remote operation failed [Transport endpoint is not connected]
[2018-07-11 19:16:16.363647] W [MSGID: 108019] [afr-lk-common.c:1102:is_blocking_locks_count_sufficient] 0-gv1-replicate-0: Unable to obtain blocking inode lock on even one child for gfid:59555375-1db1-428a-a7f2-08dcd79f0800.
[2018-07-11 19:16:16.363657] I [MSGID: 108019] [afr-transaction.c:1820:afr_post_blocking_inodelk_cbk] 0-gv1-replicate-0: Blocking inodelks failed.
[2018-07-11 19:16:16.363848] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fd882449efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fd88220ee6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fd88220ef8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fd882210710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fd882211200] ))))) 0-gv1-client-3: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2018-07-11 19:15:53.419521 (xid=0x1e3325)
[2018-07-11 19:16:16.363877] E [MSGID: 114031] [client-rpc-fops.c:1557:client3_3_finodelk_cbk] 0-gv1-client-3: remote operation failed [Transport endpoint is not connected]
[2018-07-11 19:16:16.363903] E [MSGID: 114031] [client-rpc-fops.c:1557:client3_3_finodelk_cbk] 0-gv1-client-3: remote operation failed [Transport endpoint is not connected]
[2018-07-11 19:16:16.363905] E [MSGID: 114031] [client-rpc-fops.c:1557:client3_3_finodelk_cbk] 0-gv1-client-4: remote operation failed [Transport endpoint is not connected]
[2018-07-11 19:16:16.363927] W [MSGID: 108019] [afr-lk-common.c:1102:is_blocking_locks_count_sufficient] 0-gv1-replicate-0: Unable to obtain blocking inode lock on even one child for gfid:59555375-1db1-428a-a7f2-08dcd79f0800.
[2018-07-11 19:16:16.363937] I [MSGID: 108019] [afr-transaction.c:1820:afr_post_blocking_inodelk_cbk] 0-gv1-replicate-0: Blocking inodelks failed.
[2018-07-11 19:16:16.363978] W [fuse-bridge.c:1381:fuse_err_cbk] 0-glusterfs-fuse: 3673856: FSYNC() ERR => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:16.364185] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fd882449efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fd88220ee6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fd88220ef8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fd882210710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fd882211200] ))))) 0-gv1-client-3: forced unwinding frame type(GlusterFS 3.3) op(STAT(1)) called at 2018-07-11 19:15:45.968944 (xid=0x1e331e)
[2018-07-11 19:16:16.364203] W [MSGID: 114031] [client-rpc-fops.c:493:client3_3_stat_cbk] 0-gv1-client-3: remote operation failed [Transport endpoint is not connected]
[2018-07-11 19:16:16.364239] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-gv1-client-2: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]
[2018-07-11 19:16:16.364258] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-gv1-client-3: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]
[2018-07-11 19:16:16.364273] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-gv1-client-4: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]
[2018-07-11 19:16:16.364324] I [MSGID: 108006] [afr-common.c:5372:afr_local_init] 0-gv1-replicate-0: no subvolumes up
[2018-07-11 19:16:16.364345] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673843: LOOKUP() / => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:16.364527] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fd882449efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fd88220ee6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fd88220ef8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fd882210710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fd882211200] ))))) 0-gv1-client-3: forced unwinding frame type(GlusterFS 3.3) op(STAT(1)) called at 2018-07-11 19:15:46.656801 (xid=0x1e331f)
[2018-07-11 19:16:16.364546] W [MSGID: 114031] [client-rpc-fops.c:493:client3_3_stat_cbk] 0-gv1-client-3: remote operation failed [Transport endpoint is not connected]
[2018-07-11 19:16:16.364575] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-gv1-client-2: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]
[2018-07-11 19:16:16.364593] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-gv1-client-3: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]
[2018-07-11 19:16:16.364608] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-gv1-client-4: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]
[2018-07-11 19:16:16.364642] I [MSGID: 108006] [afr-common.c:5372:afr_local_init] 0-gv1-replicate-0: no subvolumes up
[2018-07-11 19:16:16.364660] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673844: LOOKUP() / => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:16.364841] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fd882449efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fd88220ee6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fd88220ef8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fd882210710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fd882211200] ))))) 0-gv1-client-3: forced unwinding frame type(GlusterFS 3.3) op(STAT(1)) called at 2018-07-11 19:15:48.587690 (xid=0x1e3321)
[2018-07-11 19:16:16.364866] W [MSGID: 114031] [client-rpc-fops.c:493:client3_3_stat_cbk] 0-gv1-client-3: remote operation failed [Transport endpoint is not connected]
[2018-07-11 19:16:16.364911] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-gv1-client-2: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]
[2018-07-11 19:16:16.364931] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-gv1-client-3: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]
[2018-07-11 19:16:16.364948] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-gv1-client-4: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]
[2018-07-11 19:16:16.364986] I [MSGID: 108006] [afr-common.c:5372:afr_local_init] 0-gv1-replicate-0: no subvolumes up
[2018-07-11 19:16:16.365004] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673847: LOOKUP() / => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:16.365187] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fd882449efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fd88220ee6e
] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fd88220ef8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fd882210710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fd882211200] ))))) 0-gv1-client-3: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at 2018-07-11 19:16:03.343398 (xid=0x1e3326)
[2018-07-11 19:16:16.365205] W [rpc-clnt-ping.c:223:rpc_clnt_ping_cbk] 0-gv1-client-3: socket disconnected
[2018-07-11 19:16:16.365363] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fd882449efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fd88220ee6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fd88220ef8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fd882210710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fd882211200] ))))) 0-gv1-client-3: forced unwinding frame type(GlusterFS 3.3) op(FSTAT(25)) called at 2018-07-11 19:16:09.304742 (xid=0x1e3327)
[2018-07-11 19:16:16.365381] W [MSGID: 114031] [client-rpc-fops.c:1459:client3_3_fstat_cbk] 0-gv1-client-3: remote operation failed [Transport endpoint is not connected]
[2018-07-11 19:16:16.365409] W [MSGID: 114061] [client-common.c:704:client_pre_fstat] 0-gv1-client-2:  (59555375-1db1-428a-a7f2-08dcd79f0800) remote_fd is -1. EBADFD [File descriptor in bad state]
[2018-07-11 19:16:16.365423] W [MSGID: 114061] [client-common.c:704:client_pre_fstat] 0-gv1-client-3:  (59555375-1db1-428a-a7f2-08dcd79f0800) remote_fd is -1. EBADFD [File descriptor in bad state]
[2018-07-11 19:16:16.365436] W [MSGID: 114061] [client-common.c:704:client_pre_fstat] 0-gv1-client-4:  (59555375-1db1-428a-a7f2-08dcd79f0800) remote_fd is -1. EBADFD [File descriptor in bad state]
[2018-07-11 19:16:16.365455] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673859: FSTAT() /9f87196f-17eb-4b9a-9f7f-adb377255b50/images/45e8d993-aa03-4905-b56c-5ead8e3a59a6/3e91c130-6aa0-44d7-8e8c-3128badb7607 => -1 (File descriptor in bad state)
[2018-07-11 19:16:16.365638] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fd882449efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fd88220ee6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fd88220ef8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fd882210710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fd882211200] ))))) 0-gv1-client-3: forced unwinding frame type(GlusterFS 3.3) op(FSTAT(25)) called at 2018-07-11 19:16:14.921371 (xid=0x1e3328)
[2018-07-11 19:16:16.365656] W [MSGID: 114031] [client-rpc-fops.c:1459:client3_3_fstat_cbk] 0-gv1-client-3: remote operation failed [Transport endpoint is not connected]
[2018-07-11 19:16:16.365677] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673857: FSTAT() /9f87196f-17eb-4b9a-9f7f-adb377255b50/images/38b23e13-da8b-431f-92fa-f019ce4c1cce/e03029a3-e8c6-4e08-8d69-14d8b079f5f6 => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:16.365963] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fd882449efb] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7fd88220ee6e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fd88220ef8e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x90)[0x7fd882210710] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7fd882211200] ))))) 0-gv1-client-3: forced unwinding frame type(GlusterFS 3.3) op(FSTAT(25)) called at 2018-07-11 19:16:16.233605 (xid=0x1e3329)
[2018-07-11 19:16:16.366021] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673858: FSTAT() /9f87196f-17eb-4b9a-9f7f-adb377255b50/dom_md/ids => -1 (File descriptor in bad state)
[2018-07-11 19:16:16.365999] W [MSGID: 114031] [client-rpc-fops.c:1459:client3_3_fstat_cbk] 0-gv1-client-3: remote operation failed [Transport endpoint is not connected]
[2018-07-11 19:16:16.369427] I [MSGID: 108006] [afr-common.c:5372:afr_local_init] 0-gv1-replicate-0: no subvolumes up
[2018-07-11 19:16:16.369483] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673860: FSTAT() /9f87196f-17eb-4b9a-9f7f-adb377255b50/images/38b23e13-da8b-431f-92fa-f019ce4c1cce/e03029a3-e8c6-4e08-8d69-14d8b079f5f6 => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:16.369573] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673861: FSTAT() /9f87196f-17eb-4b9a-9f7f-adb377255b50/images/38b23e13-da8b-431f-92fa-f019ce4c1cce/e03029a3-e8c6-4e08-8d69-14d8b079f5f6 => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:16.372119] W [fuse-bridge.c:1381:fuse_err_cbk] 0-glusterfs-fuse: 3673862: FSYNC() ERR => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:16.372807] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673863: FSTAT() /9f87196f-17eb-4b9a-9f7f-adb377255b50/images/45e8d993-aa03-4905-b56c-5ead8e3a59a6/3e91c130-6aa0-44d7-8e8c-3128badb7607 => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:16.372906] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673864: FSTAT() /9f87196f-17eb-4b9a-9f7f-adb377255b50/images/45e8d993-aa03-4905-b56c-5ead8e3a59a6/3e91c130-6aa0-44d7-8e8c-3128badb7607 => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:16.377819] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673865: FSTAT() /9f87196f-17eb-4b9a-9f7f-adb377255b50/images/45e8d993-aa03-4905-b56c-5ead8e3a59a6/3e91c130-6aa0-44d7-8e8c-3128badb7607 => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:16.378597] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673866: FSTAT() /9f87196f-17eb-4b9a-9f7f-adb377255b50/images/45e8d993-aa03-4905-b56c-5ead8e3a59a6/3e91c130-6aa0-44d7-8e8c-3128badb7607 => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:16.378657] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673867: FSTAT() /9f87196f-17eb-4b9a-9f7f-adb377255b50/images/45e8d993-aa03-4905-b56c-5ead8e3a59a6/3e91c130-6aa0-44d7-8e8c-3128badb7607 => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:16.387724] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673868: FSTAT() /9f87196f-17eb-4b9a-9f7f-adb377255b50/images/45e8d993-aa03-4905-b56c-5ead8e3a59a6/3e91c130-6aa0-44d7-8e8c-3128badb7607 => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:16.866371] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673869: FSTAT() /9f87196f-17eb-4b9a-9f7f-adb377255b50/dom_md/ids => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:17.366758] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673870: FSTAT() /9f87196f-17eb-4b9a-9f7f-adb377255b50/dom_md/ids => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:17.867175] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673871: FSTAT() /9f87196f-17eb-4b9a-9f7f-adb377255b50/dom_md/ids => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:18.367548] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673872: FSTAT() /9f87196f-17eb-4b9a-9f7f-adb377255b50/dom_md/ids => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:18.378199] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673873: LOOKUP() / => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:18.586617] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673874: LOOKUP() / => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:18.867968] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673875: FSTAT() /9f87196f-17eb-4b9a-9f7f-adb377255b50/dom_md/ids => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:19.368384] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673876: FSTAT() /9f87196f-17eb-4b9a-9f7f-adb377255b50/dom_md/ids => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:19.387163] I [glusterfsd-mgmt.c:1888:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing
[2018-07-11 19:16:19.868734] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673877: FSTAT() /9f87196f-17eb-4b9a-9f7f-adb377255b50/dom_md/ids => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:20.369113] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673878: FSTAT() /9f87196f-17eb-4b9a-9f7f-adb377255b50/dom_md/ids => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:20.390977] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673879: LOOKUP() / => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:20.621602] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673880: LOOKUP() / => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:20.869520] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673881: FSTAT() /9f87196f-17eb-4b9a-9f7f-adb377255b50/dom_md/ids => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:20.975257] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673882: LOOKUP() / => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:21.369905] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673883: FSTAT() /9f87196f-17eb-4b9a-9f7f-adb377255b50/dom_md/ids => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:21.870310] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673884: FSTAT() /9f87196f-17eb-4b9a-9f7f-adb377255b50/dom_md/ids => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:22.081291] W [fuse-bridge.c:1381:fuse_err_cbk] 0-glusterfs-fuse: 3673887: FSYNC() ERR => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:22.084700] W [fuse-bridge.c:1381:fuse_err_cbk] 0-glusterfs-fuse: 3673888: FSYNC() ERR => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:22.094940] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673889: FSTAT() /9f87196f-17eb-4b9a-9f7f-adb377255b50/images/38b23e13-da8b-431f-92fa-f019ce4c1cce/e03029a3-e8c6-4e08-8d69-14d8b079f5f6 => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:22.370778] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673890: FSTAT() /9f87196f-17eb-4b9a-9f7f-adb377255b50/dom_md/ids => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:22.423209] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673891: LOOKUP() / => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:22.574824] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673892: LOOKUP() / => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:22.871227] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673893: FSTAT() /9f87196f-17eb-4b9a-9f7f-adb377255b50/dom_md/ids => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:23.371628] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673894: FSTAT() /9f87196f-17eb-4b9a-9f7f-adb377255b50/dom_md/ids => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:23.872026] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673895: FSTAT() /9f87196f-17eb-4b9a-9f7f-adb377255b50/dom_md/ids => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:24.372444] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673896: FSTAT() /9f87196f-17eb-4b9a-9f7f-adb377255b50/dom_md/ids => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:24.436119] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673897: LOOKUP() / => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:24.872850] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673898: FSTAT() /9f87196f-17eb-4b9a-9f7f-adb377255b50/dom_md/ids => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:25.373254] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673899: FSTAT() /9f87196f-17eb-4b9a-9f7f-adb377255b50/dom_md/ids => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:25.873651] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673900: FSTAT() /9f87196f-17eb-4b9a-9f7f-adb377255b50/dom_md/ids => -1 (Transport endpoint is not connected)
The message "I [MSGID: 108006] [afr-common.c:5372:afr_local_init] 0-gv1-replicate-0: no subvolumes up" repeated 47 times between [2018-07-11 19:16:16.369427] and [2018-07-11 19:16:25.873639]
[2018-07-11 19:16:26.039513] E [MSGID: 114058] [client-handshake.c:1565:client_query_portmap_cbk] 0-gv1-client-2: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2018-07-11 19:16:26.039611] I [MSGID: 114018] [client.c:2285:client_rpc_notify] 0-gv1-client-2: disconnected from gv1-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2018-07-11 19:16:26.039616] E [MSGID: 108006] [afr-common.c:5092:__afr_handle_child_down_event] 0-gv1-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.
[2018-07-11 19:16:26.361070] I [rpc-clnt.c:1986:rpc_clnt_reconfig] 0-gv1-client-4: changing port to 49154 (from 0)
[2018-07-11 19:16:26.365057] I [MSGID: 114057] [client-handshake.c:1478:select_server_supported_programs] 0-gv1-client-4: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2018-07-11 19:16:26.365831] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] 0-gv1-client-4: Connected to gv1-client-4, attached to remote volume '/glusterfs/data1/gv1'.
[2018-07-11 19:16:26.365940] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] 0-gv1-client-4: Server and Client lk-version numbers are not same, reopening the fds
[2018-07-11 19:16:26.365971] I [MSGID: 114042] [client-handshake.c:1047:client_post_handshake] 0-gv1-client-4: 5 fds open - Delaying child_up until they are re-opened
[2018-07-11 19:16:26.366730] I [MSGID: 108006] [afr-common.c:5372:afr_local_init] 0-gv1-replicate-0: no subvolumes up
[2018-07-11 19:16:26.366813] W [fuse-bridge.c:871:fuse_attr_cbk] 0-glusterfs-fuse: 3673901: LOOKUP() / => -1 (Transport endpoint is not connected)
[2018-07-11 19:16:26.366803] I [MSGID: 108006] [afr-common.c:5372:afr_local_init] 0-gv1-replicate-0: no subvolumes up
[2018-07-11 19:16:26.366887] I [MSGID: 114041] [client-handshake.c:678:client_child_up_reopen_done] 0-gv1-client-4: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2018-07-11 19:16:26.366952] I [MSGID: 108005] [afr-common.c:5015:__afr_handle_child_up_event] 0-gv1-replicate-0: Subvolume 'gv1-client-4' came back up; going online.
[2018-07-11 19:16:26.367114] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-gv1-client-4: Server lk version = 1
[2018-07-11 19:16:26.378705] W [fuse-bridge.c:2402:fuse_writev_cbk] 0-glusterfs-fuse: 3673912: WRITE => -1 gfid=de4eb8bc-f5fa-4da8-b907-1562cbd38dfe fd=0x7fd870001790 (Read-only file system)
[2018-07-11 19:16:26.882546] W [fuse-bridge.c:2402:fuse_writev_cbk] 0-glusterfs-fuse: 3673939: WRITE => -1 gfid=de4eb8bc-f5fa-4da8-b907-1562cbd38dfe fd=0x7fd870001790 (Read-only file system)
[2018-07-11 19:16:27.364149] I [rpc-clnt.c:1986:rpc_clnt_reconfig] 0-gv1-client-3: changing port to 49152 (from 0)
[2018-07-11 19:16:27.367908] I [MSGID: 114057] [client-handshake.c:1478:select_server_supported_programs] 0-gv1-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2018-07-11 19:16:27.368659] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] 0-gv1-client-3: Connected to gv1-client-3, attached to remote volume '/glusterfs/data1/gv1'.
[2018-07-11 19:16:27.368698] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] 0-gv1-client-3: Server and Client lk-version numbers are not same, reopening the fds
[2018-07-11 19:16:27.368716] I [MSGID: 114042] [client-handshake.c:1047:client_post_handshake] 0-gv1-client-3: 5 fds open - Delaying child_up until they are re-opened
[2018-07-11 19:16:27.369663] I [MSGID: 114041] [client-handshake.c:678:client_child_up_reopen_done] 0-gv1-client-3: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2018-07-11 19:16:27.369729] I [MSGID: 108002] [afr-common.c:5312:afr_notify] 0-gv1-replicate-0: Client-quorum is met
[2018-07-11 19:16:27.369853] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-gv1-client-3: Server lk version = 1
[2018-07-11 19:16:27.369853] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-gv1-client-3: Server lk version = 1
[2018-07-11 19:16:31.575635] I [rpc-clnt.c:1986:rpc_clnt_reconfig] 0-gv1-client-2: changing port to 49152 (from 0)
[2018-07-11 19:16:31.580371] I [MSGID: 114057] [client-handshake.c:1478:select_server_supported_programs] 0-gv1-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2018-07-11 19:16:31.581181] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] 0-gv1-client-2: Connected to gv1-client-2, attached to remote volume '/glusterfs/data1/gv1'.
[2018-07-11 19:16:31.581213] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] 0-gv1-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2018-07-11 19:16:31.581230] I [MSGID: 114042] [client-handshake.c:1047:client_post_handshake] 0-gv1-client-2: 5 fds open - Delaying child_up until they are re-opened
[2018-07-11 19:16:31.582124] I [MSGID: 114041] [client-handshake.c:678:client_child_up_reopen_done] 0-gv1-client-2: last fd open'd/lock-self-heal'd - notifying CHILD-UP
[2018-07-11 19:16:31.582476] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-gv1-client-2: Server lk version = 1
[2018-07-11 19:16:36.388675] W [fuse-bridge.c:1381:fuse_err_cbk] 0-glusterfs-fuse: 3674028: FSYNC() ERR => -1 (Success)
[2018-07-11 19:16:36.414829] W [fuse-bridge.c:1381:fuse_err_cbk] 0-glusterfs-fuse: 3674029: FSYNC() ERR => -1 (Success)
The message "E [MSGID: 114031] [client-rpc-fops.c:1557:client3_3_finodelk_cbk] 0-gv1-client-2: remote operation failed [Transport endpoint is not connected]" repeated 9 times between [2018-07-11 19:16:14.920207] and [2018-07-11 19:16:16.363900]