[Bugs] [Bug 1740413] Gluster volume bricks crashes when running a security scan on glusterfs ports
bugzilla at redhat.com
bugzilla at redhat.com
Fri Aug 16 15:44:42 UTC 2019
https://bugzilla.redhat.com/show_bug.cgi?id=1740413
--- Comment #3 from Marvin <motillito1 at hotmail.com> ---
Hello, sorry for the delay on providing this log. I still don't have more
information about the scripts used on the security scan but I will provide it
as soon as I get it from the security team. Thank you for your help.
Brick log with TRACE log level enabled (Part 1):
[2019-08-04 10:24:04.372080] T [MSGID: 0] [posix.c:4388:pl_statfs]
0-stack-trace: stack-address: 0x7fb188000aa8, winding from shared-locks to
shared-access-control
[2019-08-04 10:24:04.372096] T [MSGID: 0] [defaults.c:3158:default_statfs]
0-stack-trace: stack-address: 0x7fb188000aa8, winding from
shared-access-control to shared-bitrot-stub
[2019-08-04 10:24:04.372110] T [MSGID: 0] [defaults.c:3158:default_statfs]
0-stack-trace: stack-address: 0x7fb188000aa8, winding from shared-bitrot-stub
to shared-changelog
[2019-08-04 10:24:04.372136] T [MSGID: 0] [defaults.c:3158:default_statfs]
0-stack-trace: stack-address: 0x7fb188000aa8, winding from shared-changelog to
shared-trash
[2019-08-04 10:24:04.372150] T [MSGID: 0] [defaults.c:3158:default_statfs]
0-stack-trace: stack-address: 0x7fb188000aa8, winding from shared-trash to
shared-posix
[2019-08-04 10:24:04.372211] T [MSGID: 0]
[posix-inode-fd-ops.c:2287:posix_statfs] 0-stack-trace: stack-address:
0x7fb188000aa8, shared-posix returned 0
[2019-08-04 10:24:04.372228] T [MSGID: 0] [posix.c:4379:pl_statfs_cbk]
0-stack-trace: stack-address: 0x7fb188000aa8, shared-locks returned 0
[2019-08-04 10:24:04.372242] T [MSGID: 0] [upcall.c:1211:up_statfs_cbk]
0-stack-trace: stack-address: 0x7fb188000aa8, shared-upcall returned 0
[2019-08-04 10:24:04.372255] T [MSGID: 0] [defaults.c:1642:default_statfs_cbk]
0-stack-trace: stack-address: 0x7fb188000aa8, shared-io-threads returned 0
[2019-08-04 10:24:04.372282] T [rpcsvc.c:1533:rpcsvc_submit_generic]
0-rpc-service: Tx message: 108
[2019-08-04 10:24:04.372298] T [rpcsvc.c:1069:rpcsvc_record_build_header]
0-rpc-service: Reply fraglen 132, payload: 108, rpc hdr: 24
[2019-08-04 10:24:04.372348] T [rpcsvc.c:1585:rpcsvc_submit_generic]
0-rpc-service: submitted reply for rpc-message (XID: 0x1d89a, Program:
GlusterFS 4.x v1, ProgVers: 400, Proc: 14) to rpc-transport (tcp.shared-server)
[2019-08-04 10:24:04.372436] D [client_t.c:433:gf_client_unref]
(-->/usr/lib64/glusterfs/6.1/xlator/protocol/server.so(+0x5afc3)
[0x7fb1c2a7efc3] -->/usr/lib64/glusterfs/6.1/xlator/protocol/server.so(+0xadeb)
[0x7fb1c2a2edeb] -->/lib64/libglusterfs.so.0(gf_client_unref+0x7b)
[0x7fb1d7b0fd2b] ) 0-client_t:
CTX_ID:ca89ac1a-1d4d-4aea-b264-fce7b1378aa4-GRAPH_ID:2-PID:22025-HOST:gfs-migration701-PC_NAME:shared-client-1-RECON_NO:-2:
ref-count 1
[2019-08-04 10:24:14.266296] D [logging.c:2006:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to
flush least recently used log message to disk
[2019-08-04 10:24:04.372271] T [MSGID: 0] [io-stats.c:2354:io_stats_statfs_cbk]
0-stack-trace: stack-address: 0x7fb188000aa8, shared-io-stats returned 0
[2019-08-04 10:24:14.266290] T [MSGID: 0]
[posix-helpers.c:1469:posix_janitor_task] 0-shared-posix: janitor cleaning out
/opt/data/shared/.glusterfs/landfill
[2019-08-04 10:24:14.266439] D [MSGID: 0]
[posix-metadata.c:118:posix_fetch_mdata_xattr] 0-shared-posix: No such
attribute:trusted.glusterfs.mdata for file /opt/data/shared/.glusterfs/landfill
gfid: null
[2019-08-04 10:24:14.372639] T [socket.c:2884:socket_event_handler]
0-tcp.shared-server: server (sock:11) in:1, out:0, err:0
[2019-08-04 10:24:14.372670] T [socket.c:2910:socket_event_handler]
0-tcp.shared-server: Server socket (11) is already connected
[2019-08-04 10:24:14.372676] T [socket.c:520:__socket_ssl_readv]
0-tcp.shared-server: ***** reading over non-SSL
[2019-08-04 10:24:14.372685] T [socket.c:520:__socket_ssl_readv]
0-tcp.shared-server: ***** reading over non-SSL
[2019-08-04 10:24:14.372699] T [rpcsvc.c:744:rpcsvc_handle_rpc_call] 0-rpcsvc:
Client port: 49090
[2019-08-04 10:24:14.372706] T [rpcsvc-auth.c:445:rpcsvc_auth_request_init]
0-rpc-service: Auth handler: AUTH_GLUSTERFS-v3
[2019-08-04 10:24:14.372713] T [rpcsvc.c:549:rpcsvc_request_create]
0-rpc-service: received rpc-message (XID: 0x120987, Ver: 2, Program: 1298437,
ProgVers: 400, Proc: 14) from rpc-transport (tcp.shared-server)
[2019-08-04 10:24:14.372724] T
[auth-glusterfs.c:363:auth_glusterfs_v3_authenticate] 0-rpc-service: Auth Info:
pid: 8258, uid: 694, gid: 692, owner: 0000000000000000, flags: 0
[2019-08-04 10:24:14.372729] T [rpcsvc.c:375:rpcsvc_program_actor]
0-rpc-service: Actor found: GlusterFS 4.x v1 - STATFS for 10.7.1.209:49090
[2019-08-04 10:24:14.372744] T [socket.c:2928:socket_event_handler]
0-tcp.shared-server: (sock:11) socket_event_poll_in returned 0
[2019-08-04 10:24:14.372770] T [rpcsvc.c:375:rpcsvc_program_actor]
0-rpc-service: Actor found: GlusterFS 4.x v1 - STATFS for 10.7.1.209:49090
[2019-08-04 10:24:14.372848] D [client_t.c:324:gf_client_ref]
(-->/usr/lib64/glusterfs/6.1/xlator/protocol/server.so(+0x34dd5)
[0x7fb1c2a58dd5]
-->/usr/lib64/glusterfs/6.1/xlator/protocol/server.so(+0x115ed)
[0x7fb1c2a355ed] -->/lib64/libglusterfs.so.0(gf_client_ref+0x6e)
[0x7fb1d7b0fbde] ) 0-client_t:
CTX_ID:ca89ac1a-1d4d-4aea-b264-fce7b1378aa4-GRAPH_ID:2-PID:22025-HOST:gfs-migration701-PC_NAME:shared-client-1-RECON_NO:-2:
ref-count 2
[2019-08-04 10:24:14.372866] T [MSGID: 0]
[server-rpc-fops_v2.c:2802:server4_statfs_resume] 0-stack-trace: stack-address:
0x7fb188000aa8, winding from shared-server to /opt/data/shared
[2019-08-04 10:24:14.372879] T [MSGID: 0] [defaults.c:3158:default_statfs]
0-stack-trace: stack-address: 0x7fb188000aa8, winding from /opt/data/shared to
shared-io-stats
[2019-08-04 10:24:14.372887] T [MSGID: 0] [io-stats.c:2906:io_stats_statfs]
0-stack-trace: stack-address: 0x7fb188000aa8, winding from shared-io-stats to
shared-quota
[2019-08-04 10:24:14.372895] T [MSGID: 0] [quota.c:4557:quota_statfs]
0-stack-trace: stack-address: 0x7fb188000aa8, winding from shared-quota to
shared-index
[2019-08-04 10:24:14.372902] T [MSGID: 0] [defaults.c:3158:default_statfs]
0-stack-trace: stack-address: 0x7fb188000aa8, winding from shared-index to
shared-barrier
[2019-08-04 10:24:14.372908] T [MSGID: 0] [defaults.c:3158:default_statfs]
0-stack-trace: stack-address: 0x7fb188000aa8, winding from shared-barrier to
shared-marker
[2019-08-04 10:24:14.372914] T [MSGID: 0] [defaults.c:3158:default_statfs]
0-stack-trace: stack-address: 0x7fb188000aa8, winding from shared-marker to
shared-selinux
[2019-08-04 10:24:14.372921] T [MSGID: 0] [defaults.c:3158:default_statfs]
0-stack-trace: stack-address: 0x7fb188000aa8, winding from shared-selinux to
shared-io-threads
[2019-08-04 10:24:14.372931] D [MSGID: 0] [io-threads.c:376:iot_schedule]
0-shared-io-threads: STATFS scheduled as fast priority fop
[2019-08-04 10:24:14.372970] T [MSGID: 0]
[defaults.c:2325:default_statfs_resume] 0-stack-trace: stack-address:
0x7fb188000aa8, winding from shared-io-threads to shared-upcall
[2019-08-04 10:24:14.372991] T [MSGID: 0] [upcall.c:1232:up_statfs]
0-stack-trace: stack-address: 0x7fb188000aa8, winding from shared-upcall to
shared-leases
[2019-08-04 10:24:14.373001] T [MSGID: 0] [defaults.c:3158:default_statfs]
0-stack-trace: stack-address: 0x7fb188000aa8, winding from shared-leases to
shared-read-only
[2019-08-04 10:24:14.373013] T [MSGID: 0] [defaults.c:3158:default_statfs]
0-stack-trace: stack-address: 0x7fb188000aa8, winding from shared-read-only to
shared-worm
[2019-08-04 10:24:14.373020] T [MSGID: 0] [defaults.c:3158:default_statfs]
0-stack-trace: stack-address: 0x7fb188000aa8, winding from shared-worm to
shared-locks
[2019-08-04 10:24:14.373028] T [MSGID: 0] [posix.c:4388:pl_statfs]
0-stack-trace: stack-address: 0x7fb188000aa8, winding from shared-locks to
shared-access-control
[2019-08-04 10:24:14.373035] T [MSGID: 0] [defaults.c:3158:default_statfs]
0-stack-trace: stack-address: 0x7fb188000aa8, winding from
shared-access-control to shared-bitrot-stub
[2019-08-04 10:24:14.373041] T [MSGID: 0] [defaults.c:3158:default_statfs]
0-stack-trace: stack-address: 0x7fb188000aa8, winding from shared-bitrot-stub
to shared-changelog
[2019-08-04 10:24:14.373047] T [MSGID: 0] [defaults.c:3158:default_statfs]
0-stack-trace: stack-address: 0x7fb188000aa8, winding from shared-changelog to
shared-trash
[2019-08-04 10:24:14.373054] T [MSGID: 0] [defaults.c:3158:default_statfs]
0-stack-trace: stack-address: 0x7fb188000aa8, winding from shared-trash to
shared-posix
[2019-08-04 10:24:14.373089] T [MSGID: 0]
[posix-inode-fd-ops.c:2287:posix_statfs] 0-stack-trace: stack-address:
0x7fb188000aa8, shared-posix returned 0
[2019-08-04 10:24:14.373097] T [MSGID: 0] [posix.c:4379:pl_statfs_cbk]
0-stack-trace: stack-address: 0x7fb188000aa8, shared-locks returned 0
[2019-08-04 10:24:14.373104] T [MSGID: 0] [upcall.c:1211:up_statfs_cbk]
0-stack-trace: stack-address: 0x7fb188000aa8, shared-upcall returned 0
[2019-08-04 10:24:14.373134] D [logging.c:2006:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to
flush least recently used log message to disk
[2019-08-04 10:24:14.373129] T [MSGID: 0] [defaults.c:1642:default_statfs_cbk]
0-stack-trace: stack-address: 0x7fb188000aa8, shared-io-threads returned 0
[2019-08-04 10:24:14.373132] T [MSGID: 0] [io-stats.c:2354:io_stats_statfs_cbk]
0-stack-trace: stack-address: 0x7fb188000aa8, shared-io-stats returned 0
[2019-08-04 10:24:14.373161] T [rpcsvc.c:1533:rpcsvc_submit_generic]
0-rpc-service: Tx message: 108
[2019-08-04 10:24:14.373168] T [rpcsvc.c:1069:rpcsvc_record_build_header]
0-rpc-service: Reply fraglen 132, payload: 108, rpc hdr: 24
[2019-08-04 10:24:14.373192] T [rpcsvc.c:1585:rpcsvc_submit_generic]
0-rpc-service: submitted reply for rpc-message (XID: 0x1d89b, Program:
GlusterFS 4.x v1, ProgVers: 400, Proc: 14) to rpc-transport (tcp.shared-server)
[2019-08-04 10:24:14.373231] D [client_t.c:433:gf_client_unref]
(-->/usr/lib64/glusterfs/6.1/xlator/protocol/server.so(+0x5afc3)
[0x7fb1c2a7efc3] -->/usr/lib64/glusterfs/6.1/xlator/protocol/server.so(+0xadeb)
[0x7fb1c2a2edeb] -->/lib64/libglusterfs.so.0(gf_client_unref+0x7b)
[0x7fb1d7b0fd2b] ) 0-client_t:
CTX_ID:ca89ac1a-1d4d-4aea-b264-fce7b1378aa4-GRAPH_ID:2-PID:22025-HOST:gfs-migration701-PC_NAME:shared-client-1-RECON_NO:-2:
ref-count 1
[2019-08-04 10:24:18.703786] T [socket.c:2884:socket_event_handler]
0-tcp.shared-server: server (sock:5) in:1, out:0, err:0
[2019-08-04 10:24:18.703846] T [socket.c:2891:socket_event_handler]
0-tcp.shared-server: server (sock:5) socket is not connected, completing
connection
[2019-08-04 10:24:18.703876] T [socket.c:2898:socket_event_handler]
0-tcp.shared-server: (sock:5) socket_complete_connection() returned 1
[2019-08-04 10:24:18.703887] T [socket.c:2902:socket_event_handler]
0-tcp.shared-server: (sock:5) returning to wait on socket
[2019-08-04 10:24:18.703894] T [socket.c:2884:socket_event_handler]
0-tcp.shared-server: server (sock:5) in:1, out:0, err:0
[2019-08-04 10:24:18.703910] T [socket.c:2910:socket_event_handler]
0-tcp.shared-server: Server socket (5) is already connected
[2019-08-04 10:24:18.703916] T [socket.c:520:__socket_ssl_readv]
0-tcp.shared-server: ***** reading over non-SSL
[2019-08-04 10:24:18.703934] T [socket.c:2928:socket_event_handler]
0-tcp.shared-server: (sock:5) socket_event_poll_in returned -12
[2019-08-04 10:24:18.703949] T [socket.c:231:socket_dump_info]
0-tcp.shared-server: $$$ server: disconnecting from (af:2,sock:5) 10.7.3.217
non-SSL (errno:0:Success)
[2019-08-04 10:24:18.703954] D [socket.c:2946:socket_event_handler]
0-transport: EPOLLERR - disconnecting (sock:5) (non-SSL)
[2019-08-04 10:24:18.703954] T [socket.c:2884:socket_event_handler]
0-tcp.shared-server: server (sock:5) in:1, out:0, err:0
[2019-08-04 10:24:18.703968] T [socket.c:4583:fini] 0-tcp.shared-server:
transport 0x7fb1b80053f0 destroyed
[2019-08-04 10:24:18.703967] T [socket.c:2910:socket_event_handler]
0-tcp.shared-server: Server socket (-1) is already connected
[2019-08-04 10:24:18.704080] E [socket.c:2317:__socket_proto_state_machine]
(-->/lib64/libpthread.so.0(+0x7dd5) [0x7fb1d68eddd5]
-->/lib64/libglusterfs.so.0(+0x8c286) [0x7fb1d7b13286]
-->/usr/lib64/glusterfs/6.1/rpc-transport/socket.so(+0xc972) [0x7fb1cbe74972] )
0-socket: invalid argument: this->private [Invalid argument]
[2019-08-04 10:24:18.704093] T [socket.c:2928:socket_event_handler]
0-tcp.shared-server: (sock:-1) socket_event_poll_in returned -1
[2019-08-04 10:24:18.704102] T [socket.c:231:socket_dump_info]
0-tcp.shared-server: $$$ server: disconnecting from (af:2,sock:-1) 10.7.3.217
non-SSL (errno:-1:Unknown error -1)
[2019-08-04 10:24:18.704107] D [socket.c:2946:socket_event_handler]
0-transport: EPOLLERR - disconnecting (sock:-1) (non-SSL)
[2019-08-04 10:24:18.704132] E [socket.c:1303:socket_event_poll_err]
(-->/lib64/libglusterfs.so.0(+0x8c286) [0x7fb1d7b13286]
-->/usr/lib64/glusterfs/6.1/rpc-transport/socket.so(+0xa48a) [0x7fb1cbe7248a]
-->/usr/lib64/glusterfs/6.1/rpc-transport/socket.so(+0x81fc) [0x7fb1cbe701fc] )
0-socket: invalid argument: this->private [Invalid argument]
[2019-08-04 10:24:18.704759] T [socket.c:961:__socket_nodelay] 0-shared-server:
NODELAY enabled for socket 5
[2019-08-04 10:24:18.704776] T [socket.c:1050:__socket_keepalive]
0-shared-server: Keep-alive enabled for socket: 5, (idle: 20, interval: 2,
max-probes: 9, timeout: 42)
[2019-08-04 10:24:18.704803] T [socket.c:3086:socket_server_event_handler]
0-tcp.shared-server: XXX server:10.7.4.110:49156, client:10.7.3.217:60044
[2019-08-04 10:24:18.704822] T [socket.c:2884:socket_event_handler]
0-tcp.shared-server: server (sock:5) in:1, out:0, err:0
[2019-08-04 10:24:18.704827] T [socket.c:2891:socket_event_handler]
0-tcp.shared-server: server (sock:5) socket is not connected, completing
connection
[2019-08-04 10:24:18.704834] T [socket.c:2898:socket_event_handler]
0-tcp.shared-server: (sock:5) socket_complete_connection() returned 1
[2019-08-04 10:24:18.704838] T [socket.c:2902:socket_event_handler]
0-tcp.shared-server: (sock:5) returning to wait on socket
[2019-08-04 10:24:18.704840] T [socket.c:2884:socket_event_handler]
0-tcp.shared-server: server (sock:5) in:1, out:0, err:0
[2019-08-04 10:24:18.704847] T [socket.c:2910:socket_event_handler]
0-tcp.shared-server: Server socket (5) is already connected
[2019-08-04 10:24:18.704851] T [socket.c:520:__socket_ssl_readv]
0-tcp.shared-server: ***** reading over non-SSL
[2019-08-04 10:24:18.704858] T [socket.c:520:__socket_ssl_readv]
0-tcp.shared-server: ***** reading over non-SSL
[2019-08-04 10:24:18.704863] T [socket.c:2349:__socket_proto_state_machine]
0-tcp.shared-server: partial fragment header read
[2019-08-04 10:24:18.704870] T [socket.c:2928:socket_event_handler]
0-tcp.shared-server: (sock:5) socket_event_poll_in returned 0
[2019-08-04 10:24:24.371032] T [socket.c:2884:socket_event_handler]
0-tcp.shared-server: server (sock:11) in:1, out:0, err:0
[2019-08-04 10:24:24.371067] T [socket.c:2910:socket_event_handler]
0-tcp.shared-server: Server socket (11) is already connected
[2019-08-04 10:24:24.371074] T [socket.c:520:__socket_ssl_readv]
0-tcp.shared-server: ***** reading over non-SSL
[2019-08-04 10:24:24.371084] T [socket.c:520:__socket_ssl_readv]
0-tcp.shared-server: ***** reading over non-SSL
[2019-08-04 10:24:24.371100] T [rpcsvc.c:744:rpcsvc_handle_rpc_call] 0-rpcsvc:
Client port: 49090
[2019-08-04 10:24:24.371109] T [rpcsvc-auth.c:445:rpcsvc_auth_request_init]
0-rpc-service: Auth handler: AUTH_GLUSTERFS-v3
[2019-08-04 10:24:24.371117] T [rpcsvc.c:549:rpcsvc_request_create]
0-rpc-service: received rpc-message (XID: 0x120988, Ver: 2, Program: 1298437,
ProgVers: 400, Proc: 14) from rpc-transport (tcp.shared-server)
[2019-08-04 10:24:24.371128] T
[auth-glusterfs.c:363:auth_glusterfs_v3_authenticate] 0-rpc-service: Auth Info:
pid: 8323, uid: 694, gid: 692, owner: 0000000000000000, flags: 0
[2019-08-04 10:24:24.371135] T [rpcsvc.c:375:rpcsvc_program_actor]
0-rpc-service: Actor found: GlusterFS 4.x v1 - STATFS for 10.7.1.209:49090
[2019-08-04 10:24:24.371146] T [socket.c:2928:socket_event_handler]
0-tcp.shared-server: (sock:11) socket_event_poll_in returned 0
[2019-08-04 10:24:24.371159] T [rpcsvc.c:375:rpcsvc_program_actor]
0-rpc-service: Actor found: GlusterFS 4.x v1 - STATFS for 10.7.1.209:49090
[2019-08-04 10:24:24.371229] D [client_t.c:324:gf_client_ref]
(-->/usr/lib64/glusterfs/6.1/xlator/protocol/server.so(+0x34dd5)
[0x7fb1c2a58dd5]
-->/usr/lib64/glusterfs/6.1/xlator/protocol/server.so(+0x115ed)
[0x7fb1c2a355ed] -->/lib64/libglusterfs.so.0(gf_client_ref+0x6e)
[0x7fb1d7b0fbde] ) 0-client_t:
CTX_ID:ca89ac1a-1d4d-4aea-b264-fce7b1378aa4-GRAPH_ID:2-PID:22025-HOST:gfs-migration701-PC_NAME:shared-client-1-RECON_NO:-2:
ref-count 2
[2019-08-04 10:24:24.371253] D [logging.c:2006:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to
flush least recently used log message to disk
The message "T [MSGID: 0] [server-rpc-fops_v2.c:2802:server4_statfs_resume]
0-stack-trace: stack-address: 0x7fb18c000aa8, winding from shared-server to
/opt/data/shared" repeated 2 times between [2019-08-04 10:23:34.373258] and
[2019-08-04 10:24:24.371247]
[2019-08-04 10:24:24.371252] T [MSGID: 0] [defaults.c:3158:default_statfs]
0-stack-trace: stack-address: 0x7fb18c000aa8, winding from /opt/data/shared to
shared-io-stats
[2019-08-04 10:24:24.371291] T [MSGID: 0] [io-stats.c:2906:io_stats_statfs]
0-stack-trace: stack-address: 0x7fb18c000aa8, winding from shared-io-stats to
shared-quota
[2019-08-04 10:24:24.371300] T [MSGID: 0] [quota.c:4557:quota_statfs]
0-stack-trace: stack-address: 0x7fb18c000aa8, winding from shared-quota to
shared-index
[2019-08-04 10:24:24.371309] T [MSGID: 0] [defaults.c:3158:default_statfs]
0-stack-trace: stack-address: 0x7fb18c000aa8, winding from shared-index to
shared-barrier
[2019-08-04 10:24:24.371316] T [MSGID: 0] [defaults.c:3158:default_statfs]
0-stack-trace: stack-address: 0x7fb18c000aa8, winding from shared-barrier to
shared-marker
[2019-08-04 10:24:24.371323] T [MSGID: 0] [defaults.c:3158:default_statfs]
0-stack-trace: stack-address: 0x7fb18c000aa8, winding from shared-marker to
shared-selinux
[2019-08-04 10:24:24.371331] T [MSGID: 0] [defaults.c:3158:default_statfs]
0-stack-trace: stack-address: 0x7fb18c000aa8, winding from shared-selinux to
shared-io-threads
[2019-08-04 10:24:24.371342] D [MSGID: 0] [io-threads.c:376:iot_schedule]
0-shared-io-threads: STATFS scheduled as fast priority fop
[2019-08-04 10:24:24.371419] T [MSGID: 0]
[defaults.c:2325:default_statfs_resume] 0-stack-trace: stack-address:
0x7fb18c000aa8, winding from shared-io-threads to shared-upcall
[2019-08-04 10:24:24.371442] T [MSGID: 0] [upcall.c:1232:up_statfs]
0-stack-trace: stack-address: 0x7fb18c000aa8, winding from shared-upcall to
shared-leases
[2019-08-04 10:24:24.371451] T [MSGID: 0] [defaults.c:3158:default_statfs]
0-stack-trace: stack-address: 0x7fb18c000aa8, winding from shared-leases to
shared-read-only
[2019-08-04 10:24:24.371458] T [MSGID: 0] [defaults.c:3158:default_statfs]
0-stack-trace: stack-address: 0x7fb18c000aa8, winding from shared-read-only to
shared-worm
[2019-08-04 10:24:24.371465] T [MSGID: 0] [defaults.c:3158:default_statfs]
0-stack-trace: stack-address: 0x7fb18c000aa8, winding from shared-worm to
shared-locks
[2019-08-04 10:24:24.371473] T [MSGID: 0] [posix.c:4388:pl_statfs]
0-stack-trace: stack-address: 0x7fb18c000aa8, winding from shared-locks to
shared-access-control
[2019-08-04 10:24:24.371480] T [MSGID: 0] [defaults.c:3158:default_statfs]
0-stack-trace: stack-address: 0x7fb18c000aa8, winding from
shared-access-control to shared-bitrot-stub
[2019-08-04 10:24:24.371486] T [MSGID: 0] [defaults.c:3158:default_statfs]
0-stack-trace: stack-address: 0x7fb18c000aa8, winding from shared-bitrot-stub
to shared-changelog
[2019-08-04 10:24:24.371492] T [MSGID: 0] [defaults.c:3158:default_statfs]
0-stack-trace: stack-address: 0x7fb18c000aa8, winding from shared-changelog to
shared-trash
[2019-08-04 10:24:24.371499] T [MSGID: 0] [defaults.c:3158:default_statfs]
0-stack-trace: stack-address: 0x7fb18c000aa8, winding from shared-trash to
shared-posix
[2019-08-04 10:24:24.371531] T [MSGID: 0]
[posix-inode-fd-ops.c:2287:posix_statfs] 0-stack-trace: stack-address:
0x7fb18c000aa8, shared-posix returned 0
[2019-08-04 10:24:24.371539] T [MSGID: 0] [posix.c:4379:pl_statfs_cbk]
0-stack-trace: stack-address: 0x7fb18c000aa8, shared-locks returned 0
[2019-08-04 10:24:24.371545] T [MSGID: 0] [upcall.c:1211:up_statfs_cbk]
0-stack-trace: stack-address: 0x7fb18c000aa8, shared-upcall returned 0
[2019-08-04 10:24:24.371561] T [rpcsvc.c:1533:rpcsvc_submit_generic]
0-rpc-service: Tx message: 108
[2019-08-04 10:24:24.371568] T [rpcsvc.c:1069:rpcsvc_record_build_header]
0-rpc-service: Reply fraglen 132, payload: 108, rpc hdr: 24
[2019-08-04 10:24:24.371593] T [rpcsvc.c:1585:rpcsvc_submit_generic]
0-rpc-service: submitted reply for rpc-message (XID: 0x1d89c, Program:
GlusterFS 4.x v1, ProgVers: 400, Proc: 14) to rpc-transport (tcp.shared-server)
[2019-08-04 10:24:24.371647] D [client_t.c:433:gf_client_unref]
(-->/usr/lib64/glusterfs/6.1/xlator/protocol/server.so(+0x5afc3)
[0x7fb1c2a7efc3] -->/usr/lib64/glusterfs/6.1/xlator/protocol/server.so(+0xadeb)
[0x7fb1c2a2edeb] -->/lib64/libglusterfs.so.0(gf_client_unref+0x7b)
[0x7fb1d7b0fd2b] ) 0-client_t:
CTX_ID:ca89ac1a-1d4d-4aea-b264-fce7b1378aa4-GRAPH_ID:2-PID:22025-HOST:gfs-migration701-PC_NAME:shared-client-1-RECON_NO:-2:
ref-count 1
[2019-08-04 10:24:25.278101] D [logging.c:2006:_gf_msg_internal]
0-logging-infra: Buffer overflow of a buffer whose size limit is 5. About to
flush least recently used log message to disk
The message "T [MSGID: 0] [io-stats.c:2354:io_stats_statfs_cbk] 0-stack-trace:
stack-address: 0x7fb18c000aa8, shared-io-stats returned 0" repeated 2 times
between [2019-08-04 10:23:34.373484] and [2019-08-04 10:24:24.371555]
[2019-08-04 10:24:25.278096] T [MSGID: 0]
[posix-helpers.c:1469:posix_janitor_task] 0-shared-posix: janitor cleaning out
/opt/data/shared/.glusterfs/landfill
[2019-08-04 10:24:25.278248] D [MSGID: 0]
[posix-metadata.c:118:posix_fetch_mdata_xattr] 0-shared-posix: No such
attribute:trusted.glusterfs.mdata for file /opt/data/shared/.glusterfs/landfill
gfid: null
[2019-08-04 10:24:26.709885] T [socket.c:2884:socket_event_handler]
0-tcp.shared-server: server (sock:5) in:1, out:0, err:0
[2019-08-04 10:24:26.709931] T [socket.c:2910:socket_event_handler]
0-tcp.shared-server: Server socket (5) is already connected
[2019-08-04 10:24:26.709938] T [socket.c:520:__socket_ssl_readv]
0-tcp.shared-server: ***** reading over non-SSL
[2019-08-04 10:24:26.709957] D [socket.c:692:__socket_rwv] 0-tcp.shared-server:
EOF on socket 5 (errno:22:Invalid argument); returning ENODATA
[2019-08-04 10:24:26.709963] T [socket.c:2928:socket_event_handler]
0-tcp.shared-server: (sock:5) socket_event_poll_in returned -1
[2019-08-04 10:24:26.709972] T [socket.c:231:socket_dump_info]
0-tcp.shared-server: $$$ server: disconnecting from (af:2,sock:5) 10.7.3.217
non-SSL (errno:0:Success)
[2019-08-04 10:24:26.709977] D [socket.c:2946:socket_event_handler]
0-transport: EPOLLERR - disconnecting (sock:5) (non-SSL)
[2019-08-04 10:24:26.709992] T [socket.c:4583:fini] 0-tcp.shared-server:
transport 0x7fb1bc08d9b0 destroyed
[2019-08-04 10:24:26.711716] T [socket.c:961:__socket_nodelay] 0-shared-server:
NODELAY enabled for socket 5
[2019-08-04 10:24:26.711739] T [socket.c:1050:__socket_keepalive]
0-shared-server: Keep-alive enabled for socket: 5, (idle: 20, interval: 2,
max-probes: 9, timeout: 42)
[2019-08-04 10:24:26.711753] T [socket.c:3086:socket_server_event_handler]
0-tcp.shared-server: XXX server:10.7.4.110:49156, client:10.7.3.217:34494
[2019-08-04 10:24:26.711772] T [socket.c:2884:socket_event_handler]
0-tcp.shared-server: server (sock:5) in:1, out:0, err:0
[2019-08-04 10:24:26.711786] T [socket.c:2891:socket_event_handler]
0-tcp.shared-server: server (sock:5) socket is not connected, completing
connection
[2019-08-04 10:24:26.711796] T [socket.c:2898:socket_event_handler]
0-tcp.shared-server: (sock:5) socket_complete_connection() returned 1
[2019-08-04 10:24:26.711800] T [socket.c:2902:socket_event_handler]
0-tcp.shared-server: (sock:5) returning to wait on socket
[2019-08-04 10:24:26.711810] T [socket.c:2884:socket_event_handler]
0-tcp.shared-server: server (sock:5) in:1, out:0, err:0
[2019-08-04 10:24:26.711824] T [socket.c:2910:socket_event_handler]
0-tcp.shared-server: Server socket (5) is already connected
[2019-08-04 10:24:26.711830] T [socket.c:520:__socket_ssl_readv]
0-tcp.shared-server: ***** reading over non-SSL
[2019-08-04 10:24:26.712216] D [MSGID: 0] [iobuf.c:568:iobuf_get2] 0-iobuf:
request for iobuf of size 369295620 is serviced using standard calloc()
(0x7fb1b8004f20) as it exceeds the maximum available buffer size
[2019-08-04 10:24:26.712243] T [socket.c:520:__socket_ssl_readv]
0-tcp.shared-server: ***** reading over non-SSL
[2019-08-04 10:24:26.712252] E [socket.c:2252:__socket_read_frag] 0-rpc: wrong
MSG-TYPE (-788331683) received from 10.7.3.217:34494
[2019-08-04 10:24:26.712257] T [socket.c:2928:socket_event_handler]
0-tcp.shared-server: (sock:5) socket_event_poll_in returned -1
[2019-08-04 10:24:26.712267] T [socket.c:231:socket_dump_info]
0-tcp.shared-server: $$$ server: disconnecting from (af:2,sock:5) 10.7.3.217
non-SSL (errno:0:Success)
[2019-08-04 10:24:26.712281] D [socket.c:2946:socket_event_handler]
0-transport: EPOLLERR - disconnecting (sock:5) (non-SSL)
[2019-08-04 10:24:26.712287] D [MSGID: 0] [iobuf.c:683:__iobuf_put] 0-iobuf:
freeing the iobuf (0x7fb1b8004f20) allocated with standard calloc()
[2019-08-04 10:24:26.712319] T [socket.c:4583:fini] 0-tcp.shared-server:
transport 0x7fb1bc08d9b0 destroyed
[2019-08-04 10:24:26.712835] T [socket.c:961:__socket_nodelay] 0-shared-server:
NODELAY enabled for socket 5
[2019-08-04 10:24:26.712858] T [socket.c:1050:__socket_keepalive]
0-shared-server: Keep-alive enabled for socket: 5, (idle: 20, interval: 2,
max-probes: 9, timeout: 42)
[2019-08-04 10:24:26.712871] T [socket.c:3086:socket_server_event_handler]
0-tcp.shared-server: XXX server:10.7.4.110:49156, client:10.7.3.217:34496
[2019-08-04 10:24:26.712890] T [socket.c:2884:socket_event_handler]
0-tcp.shared-server: server (sock:5) in:1, out:0, err:0
[2019-08-04 10:24:26.712895] T [socket.c:2891:socket_event_handler]
0-tcp.shared-server: server (sock:5) socket is not connected, completing
connection
[2019-08-04 10:24:26.712904] T [socket.c:2898:socket_event_handler]
0-tcp.shared-server: (sock:5) socket_complete_connection() returned 1
[2019-08-04 10:24:26.712908] T [socket.c:2902:socket_event_handler]
0-tcp.shared-server: (sock:5) returning to wait on socket
[2019-08-04 10:24:26.712915] T [socket.c:2884:socket_event_handler]
0-tcp.shared-server: server (sock:5) in:1, out:0, err:0
[2019-08-04 10:24:26.712928] T [socket.c:2910:socket_event_handler]
0-tcp.shared-server: Server socket (5) is already connected
[2019-08-04 10:24:26.712934] T [socket.c:520:__socket_ssl_readv]
0-tcp.shared-server: ***** reading over non-SSL
[2019-08-04 10:24:26.713088] D [MSGID: 0] [iobuf.c:568:iobuf_get2] 0-iobuf:
request for iobuf of size 369295622 is serviced using standard calloc()
(0x7fb1bc002160) as it exceeds the maximum available buffer size
[2019-08-04 10:24:26.713101] T [socket.c:520:__socket_ssl_readv]
0-tcp.shared-server: ***** reading over non-SSL
[2019-08-04 10:24:26.713109] E [socket.c:2252:__socket_read_frag] 0-rpc: wrong
MSG-TYPE (134415197) received from 10.7.3.217:34496
[2019-08-04 10:24:26.713114] T [socket.c:2928:socket_event_handler]
0-tcp.shared-server: (sock:5) socket_event_poll_in returned -1
[2019-08-04 10:24:26.713123] T [socket.c:231:socket_dump_info]
0-tcp.shared-server: $$$ server: disconnecting from (af:2,sock:5) 10.7.3.217
non-SSL (errno:0:Success)
[2019-08-04 10:24:26.713128] D [socket.c:2946:socket_event_handler]
0-transport: EPOLLERR - disconnecting (sock:5) (non-SSL)
[2019-08-04 10:24:26.713133] D [MSGID: 0] [iobuf.c:683:__iobuf_put] 0-iobuf:
freeing the iobuf (0x7fb1bc002160) allocated with standard calloc()
[2019-08-04 10:24:26.713157] T [socket.c:4583:fini] 0-tcp.shared-server:
transport 0x7fb1b80053f0 destroyed
[2019-08-04 10:24:26.713633] T [socket.c:961:__socket_nodelay] 0-shared-server:
NODELAY enabled for socket 5
[2019-08-04 10:24:26.713649] T [socket.c:1050:__socket_keepalive]
0-shared-server: Keep-alive enabled for socket: 5, (idle: 20, interval: 2,
max-probes: 9, timeout: 42)
[2019-08-04 10:24:26.713674] T [socket.c:3086:socket_server_event_handler]
0-tcp.shared-server: XXX server:10.7.4.110:49156, client:10.7.3.217:34498
[2019-08-04 10:24:26.713689] T [socket.c:2884:socket_event_handler]
0-tcp.shared-server: server (sock:5) in:1, out:0, err:0
[2019-08-04 10:24:26.713694] T [socket.c:2891:socket_event_handler]
0-tcp.shared-server: server (sock:5) socket is not connected, completing
connection
[2019-08-04 10:24:26.713702] T [socket.c:2898:socket_event_handler]
0-tcp.shared-server: (sock:5) socket_complete_connection() returned 1
[2019-08-04 10:24:26.713706] T [socket.c:2902:socket_event_handler]
0-tcp.shared-server: (sock:5) returning to wait on socket
[2019-08-04 10:24:26.713711] T [socket.c:2884:socket_event_handler]
0-tcp.shared-server: server (sock:5) in:1, out:0, err:0
[2019-08-04 10:24:26.713721] T [socket.c:2910:socket_event_handler]
0-tcp.shared-server: Server socket (5) is already connected
[2019-08-04 10:24:26.713730] T [socket.c:520:__socket_ssl_readv]
0-tcp.shared-server: ***** reading over non-SSL
[2019-08-04 10:24:26.713878] D [MSGID: 0] [iobuf.c:568:iobuf_get2] 0-iobuf:
request for iobuf of size 369295622 is serviced using standard calloc()
(0x7fb1b8015f70) as it exceeds the maximum available buffer size
[2019-08-04 10:24:26.713890] T [socket.c:520:__socket_ssl_readv]
0-tcp.shared-server: ***** reading over non-SSL
[2019-08-04 10:24:26.713898] E [socket.c:2252:__socket_read_frag] 0-rpc: wrong
MSG-TYPE (134415197) received from 10.7.3.217:34498
[2019-08-04 10:24:26.713902] T [socket.c:2928:socket_event_handler]
0-tcp.shared-server: (sock:5) socket_event_poll_in returned -1
[2019-08-04 10:24:26.713912] T [socket.c:231:socket_dump_info]
0-tcp.shared-server: $$$ server: disconnecting from (af:2,sock:5) 10.7.3.217
non-SSL (errno:0:Success)
[2019-08-04 10:24:26.713916] D [socket.c:2946:socket_event_handler]
0-transport: EPOLLERR - disconnecting (sock:5) (non-SSL)
[2019-08-04 10:24:26.713921] D [MSGID: 0] [iobuf.c:683:__iobuf_put] 0-iobuf:
freeing the iobuf (0x7fb1b8015f70) allocated with standard calloc()
[2019-08-04 10:24:26.713947] T [socket.c:4583:fini] 0-tcp.shared-server:
transport 0x7fb1bc08d9b0 destroyed
[2019-08-04 10:24:26.714466] T [socket.c:961:__socket_nodelay] 0-shared-server:
NODELAY enabled for socket 5
[2019-08-04 10:24:26.714479] T [socket.c:1050:__socket_keepalive]
0-shared-server: Keep-alive enabled for socket: 5, (idle: 20, interval: 2,
max-probes: 9, timeout: 42)
[2019-08-04 10:24:26.714489] T [socket.c:3086:socket_server_event_handler]
0-tcp.shared-server: XXX server:10.7.4.110:49156, client:10.7.3.217:34500
[2019-08-04 10:24:26.714504] T [socket.c:2884:socket_event_handler]
0-tcp.shared-server: server (sock:5) in:1, out:0, err:0
[2019-08-04 10:24:26.714509] T [socket.c:2891:socket_event_handler]
0-tcp.shared-server: server (sock:5) socket is not connected, completing
connection
[2019-08-04 10:24:26.714517] T [socket.c:2898:socket_event_handler]
0-tcp.shared-server: (sock:5) socket_complete_connection() returned 1
[2019-08-04 10:24:26.714522] T [socket.c:2902:socket_event_handler]
0-tcp.shared-server: (sock:5) returning to wait on socket
[2019-08-04 10:24:26.714526] T [socket.c:2884:socket_event_handler]
0-tcp.shared-server: server (sock:5) in:1, out:0, err:0
[2019-08-04 10:24:26.714535] T [socket.c:2910:socket_event_handler]
0-tcp.shared-server: Server socket (5) is already connected
[2019-08-04 10:24:26.714539] T [socket.c:520:__socket_ssl_readv]
0-tcp.shared-server: ***** reading over non-SSL
[2019-08-04 10:24:26.714694] D [MSGID: 0] [iobuf.c:568:iobuf_get2] 0-iobuf:
request for iobuf of size 369295622 is serviced using standard calloc()
(0x7fb1bc002160) as it exceeds the maximum available buffer size
[2019-08-04 10:24:26.714709] T [socket.c:520:__socket_ssl_readv]
0-tcp.shared-server: ***** reading over non-SSL
[2019-08-04 10:24:26.714716] E [socket.c:2252:__socket_read_frag] 0-rpc: wrong
MSG-TYPE (134415197) received from 10.7.3.217:34500
[2019-08-04 10:24:26.714720] T [socket.c:2928:socket_event_handler]
0-tcp.shared-server: (sock:5) socket_event_poll_in returned -1
[2019-08-04 10:24:26.714729] T [socket.c:231:socket_dump_info]
0-tcp.shared-server: $$$ server: disconnecting from (af:2,sock:5) 10.7.3.217
non-SSL (errno:0:Success)
[2019-08-04 10:24:26.714733] D [socket.c:2946:socket_event_handler]
0-transport: EPOLLERR - disconnecting (sock:5) (non-SSL)
[2019-08-04 10:24:26.714738] D [MSGID: 0] [iobuf.c:683:__iobuf_put] 0-iobuf:
freeing the iobuf (0x7fb1bc002160) allocated with standard calloc()
[2019-08-04 10:24:26.714760] T [socket.c:4583:fini] 0-tcp.shared-server:
transport 0x7fb1b80053f0 destroyed
[2019-08-04 10:24:26.715392] T [socket.c:961:__socket_nodelay] 0-shared-server:
NODELAY enabled for socket 5
[2019-08-04 10:24:26.715404] T [socket.c:1050:__socket_keepalive]
0-shared-server: Keep-alive enabled for socket: 5, (idle: 20, interval: 2,
max-probes: 9, timeout: 42)
[2019-08-04 10:24:26.715412] T [socket.c:3086:socket_server_event_handler]
0-tcp.shared-server: XXX server:10.7.4.110:49156, client:10.7.3.217:34502
[2019-08-04 10:24:26.715431] T [socket.c:2884:socket_event_handler]
0-tcp.shared-server: server (sock:5) in:1, out:0, err:0
[2019-08-04 10:24:26.715436] T [socket.c:2891:socket_event_handler]
0-tcp.shared-server: server (sock:5) socket is not connected, completing
connection
[2019-08-04 10:24:26.715444] T [socket.c:2898:socket_event_handler]
0-tcp.shared-server: (sock:5) socket_complete_connection() returned 1
[2019-08-04 10:24:26.715448] T [socket.c:2902:socket_event_handler]
0-tcp.shared-server: (sock:5) returning to wait on socket
[2019-08-04 10:24:26.715453] T [socket.c:2884:socket_event_handler]
0-tcp.shared-server: server (sock:5) in:1, out:0, err:0
[2019-08-04 10:24:26.715464] T [socket.c:2910:socket_event_handler]
0-tcp.shared-server: Server socket (5) is already connected
[2019-08-04 10:24:26.715468] T [socket.c:520:__socket_ssl_readv]
0-tcp.shared-server: ***** reading over non-SSL
[2019-08-04 10:24:26.715624] D [MSGID: 0] [iobuf.c:568:iobuf_get2] 0-iobuf:
request for iobuf of size 369295622 is serviced using standard calloc()
(0x7fb1b8015f70) as it exceeds the maximum available buffer size
[2019-08-04 10:24:26.715639] T [socket.c:520:__socket_ssl_readv]
0-tcp.shared-server: ***** reading over non-SSL
[2019-08-04 10:24:26.715647] E [socket.c:2252:__socket_read_frag] 0-rpc: wrong
MSG-TYPE (134415197) received from 10.7.3.217:34502
[2019-08-04 10:24:26.715652] T [socket.c:2928:socket_event_handler]
0-tcp.shared-server: (sock:5) socket_event_poll_in returned -1
[2019-08-04 10:24:26.715660] T [socket.c:231:socket_dump_info]
0-tcp.shared-server: $$$ server: disconnecting from (af:2,sock:5) 10.7.3.217
non-SSL (errno:0:Success)
[2019-08-04 10:24:26.715665] D [socket.c:2946:socket_event_handler]
0-transport: EPOLLERR - disconnecting (sock:5) (non-SSL)
[2019-08-04 10:24:26.715670] D [MSGID: 0] [iobuf.c:683:__iobuf_put] 0-iobuf:
freeing the iobuf (0x7fb1b8015f70) allocated with standard calloc()
[2019-08-04 10:24:26.715694] T [socket.c:4583:fini] 0-tcp.shared-server:
transport 0x7fb1bc08d9b0 destroyed
[2019-08-04 10:24:26.716172] T [socket.c:961:__socket_nodelay] 0-shared-server:
NODELAY enabled for socket 5
[2019-08-04 10:24:26.716186] T [socket.c:1050:__socket_keepalive]
0-shared-server: Keep-alive enabled for socket: 5, (idle: 20, interval: 2,
max-probes: 9, timeout: 42)
[2019-08-04 10:24:26.716195] T [socket.c:3086:socket_server_event_handler]
0-tcp.shared-server: XXX server:10.7.4.110:49156, client:10.7.3.217:34504
[2019-08-04 10:24:26.716216] T [socket.c:2884:socket_event_handler]
0-tcp.shared-server: server (sock:5) in:1, out:0, err:0
[2019-08-04 10:24:26.716221] T [socket.c:2891:socket_event_handler]
0-tcp.shared-server: server (sock:5) socket is not connected, completing
connection
[2019-08-04 10:24:26.716228] T [socket.c:2898:socket_event_handler]
0-tcp.shared-server: (sock:5) socket_complete_connection() returned 1
[2019-08-04 10:24:26.716232] T [socket.c:2902:socket_event_handler]
0-tcp.shared-server: (sock:5) returning to wait on socket
[2019-08-04 10:24:26.716234] T [socket.c:2884:socket_event_handler]
0-tcp.shared-server: server (sock:5) in:1, out:0, err:0
[2019-08-04 10:24:26.716244] T [socket.c:2910:socket_event_handler]
0-tcp.shared-server: Server socket (5) is already connected
[2019-08-04 10:24:26.716248] T [socket.c:520:__socket_ssl_readv]
0-tcp.shared-server: ***** reading over non-SSL
[2019-08-04 10:24:26.716397] D [MSGID: 0] [iobuf.c:568:iobuf_get2] 0-iobuf:
request for iobuf of size 369296132 is serviced using standard calloc()
(0x7fb1bc002160) as it exceeds the maximum available buffer size
[2019-08-04 10:24:26.716407] T [socket.c:520:__socket_ssl_readv]
0-tcp.shared-server: ***** reading over non-SSL
[2019-08-04 10:24:26.716415] E [socket.c:2252:__socket_read_frag] 0-rpc: wrong
MSG-TYPE (-1543306403) received from 10.7.3.217:34504
[2019-08-04 10:24:26.716419] T [socket.c:2928:socket_event_handler]
0-tcp.shared-server: (sock:5) socket_event_poll_in returned -1
[2019-08-04 10:24:26.716428] T [socket.c:231:socket_dump_info]
0-tcp.shared-server: $$$ server: disconnecting from (af:2,sock:5) 10.7.3.217
non-SSL (errno:0:Success)
[2019-08-04 10:24:26.716437] D [socket.c:2946:socket_event_handler]
0-transport: EPOLLERR - disconnecting (sock:5) (non-SSL)
[2019-08-04 10:24:26.716442] D [MSGID: 0] [iobuf.c:683:__iobuf_put] 0-iobuf:
freeing the iobuf (0x7fb1bc002160) allocated with standard calloc()
[2019-08-04 10:24:26.716464] T [socket.c:4583:fini] 0-tcp.shared-server:
transport 0x7fb1b80053f0 destroyed
[2019-08-04 10:24:26.717115] T [socket.c:961:__socket_nodelay] 0-shared-server:
NODELAY enabled for socket 5
[2019-08-04 10:24:26.717130] T [socket.c:1050:__socket_keepalive]
0-shared-server: Keep-alive enabled for socket: 5, (idle: 20, interval: 2,
max-probes: 9, timeout: 42)
[2019-08-04 10:24:26.717140] T [socket.c:3086:socket_server_event_handler]
0-tcp.shared-server: XXX server:10.7.4.110:49156, client:10.7.3.217:34506
[2019-08-04 10:24:26.717308] T [socket.c:2884:socket_event_handler]
0-tcp.shared-server: server (sock:5) in:1, out:0, err:0
[2019-08-04 10:24:26.717314] T [socket.c:2891:socket_event_handler]
0-tcp.shared-server: server (sock:5) socket is not connected, completing
connection
[2019-08-04 10:24:26.717323] T [socket.c:2898:socket_event_handler]
0-tcp.shared-server: (sock:5) socket_complete_connection() returned 1
[2019-08-04 10:24:26.717327] T [socket.c:2902:socket_event_handler]
0-tcp.shared-server: (sock:5) returning to wait on socket
[2019-08-04 10:24:26.717332] T [socket.c:2884:socket_event_handler]
0-tcp.shared-server: server (sock:5) in:1, out:0, err:0
[2019-08-04 10:24:26.717336] T [socket.c:2910:socket_event_handler]
0-tcp.shared-server: Server socket (5) is already connected
[2019-08-04 10:24:26.717340] T [socket.c:520:__socket_ssl_readv]
0-tcp.shared-server: ***** reading over non-SSL
[2019-08-04 10:24:26.717480] D [MSGID: 0] [iobuf.c:568:iobuf_get2] 0-iobuf:
request for iobuf of size 369295620 is serviced using standard calloc()
(0x7fb1bc002160) as it exceeds the maximum available buffer size
[2019-08-04 10:24:26.717490] T [socket.c:520:__socket_ssl_readv]
0-tcp.shared-server: ***** reading over non-SSL
[2019-08-04 10:24:26.717497] E [socket.c:2252:__socket_read_frag] 0-rpc: wrong
MSG-TYPE (-100466339) received from 10.7.3.217:34506
[2019-08-04 10:24:26.717501] T [socket.c:2928:socket_event_handler]
0-tcp.shared-server: (sock:5) socket_event_poll_in returned -1
[2019-08-04 10:24:26.717509] T [socket.c:231:socket_dump_info]
0-tcp.shared-server: $$$ server: disconnecting from (af:2,sock:5) 10.7.3.217
non-SSL (errno:0:Success)
[2019-08-04 10:24:26.717514] D [socket.c:2946:socket_event_handler]
0-transport: EPOLLERR - disconnecting (sock:5) (non-SSL)
[2019-08-04 10:24:26.717518] D [MSGID: 0] [iobuf.c:683:__iobuf_put] 0-iobuf:
freeing the iobuf (0x7fb1bc002160) allocated with standard calloc()
[2019-08-04 10:24:26.717540] T [socket.c:4583:fini] 0-tcp.shared-server:
transport 0x7fb1bc08d9b0 destroyed
[2019-08-04 10:24:26.718269] T [socket.c:961:__socket_nodelay] 0-shared-server:
NODELAY enabled for socket 5
[2019-08-04 10:24:26.718286] T [socket.c:1050:__socket_keepalive]
0-shared-server: Keep-alive enabled for socket: 5, (idle: 20, interval: 2,
max-probes: 9, timeout: 42)
[2019-08-04 10:24:26.718299] T [socket.c:3086:socket_server_event_handler]
0-tcp.shared-server: XXX server:10.7.4.110:49156, client:10.7.3.217:34508
[2019-08-04 10:24:26.718319] T [socket.c:2884:socket_event_handler]
0-tcp.shared-server: server (sock:5) in:1, out:0, err:0
[2019-08-04 10:24:26.718327] T [socket.c:2891:socket_event_handler]
0-tcp.shared-server: server (sock:5) socket is not connected, completing
connection
[2019-08-04 10:24:26.718339] T [socket.c:2898:socket_event_handler]
0-tcp.shared-server: (sock:5) socket_complete_connection() returned 1
[2019-08-04 10:24:26.718346] T [socket.c:2902:socket_event_handler]
0-tcp.shared-server: (sock:5) returning to wait on socket
[2019-08-04 10:24:26.718348] T [socket.c:2884:socket_event_handler]
0-tcp.shared-server: server (sock:5) in:1, out:0, err:0
[2019-08-04 10:24:26.718360] T [socket.c:2910:socket_event_handler]
0-tcp.shared-server: Server socket (5) is already connected
[2019-08-04 10:24:26.718369] T [socket.c:520:__socket_ssl_readv]
0-tcp.shared-server: ***** reading over non-SSL
[2019-08-04 10:24:26.718516] D [MSGID: 0] [iobuf.c:568:iobuf_get2] 0-iobuf:
request for iobuf of size 369295620 is serviced using standard calloc()
(0x7fb1b8015f70) as it exceeds the maximum available buffer size
[2019-08-04 10:24:26.718527] T [socket.c:520:__socket_ssl_readv]
0-tcp.shared-server: ***** reading over non-SSL
[2019-08-04 10:24:26.718534] E [socket.c:2252:__socket_read_frag] 0-rpc: wrong
MSG-TYPE (-100466339) received from 10.7.3.217:34508
[2019-08-04 10:24:26.718539] T [socket.c:2928:socket_event_handler]
0-tcp.shared-server: (sock:5) socket_event_poll_in returned -1
[2019-08-04 10:24:26.718548] T [socket.c:231:socket_dump_info]
0-tcp.shared-server: $$$ server: disconnecting from (af:2,sock:5) 10.7.3.217
non-SSL (errno:0:Success)
[2019-08-04 10:24:26.718552] D [socket.c:2946:socket_event_handler]
0-transport: EPOLLERR - disconnecting (sock:5) (non-SSL)
[2019-08-04 10:24:26.718557] D [MSGID: 0] [iobuf.c:683:__iobuf_put] 0-iobuf:
freeing the iobuf (0x7fb1b8015f70) allocated with standard calloc()
[2019-08-04 10:24:26.718582] T [socket.c:4583:fini] 0-tcp.shared-server:
transport 0x7fb1bc08d9b0 destroyed
[2019-08-04 10:24:26.719198] T [socket.c:961:__socket_nodelay] 0-shared-server:
NODELAY enabled for socket 5
[2019-08-04 10:24:26.719215] T [socket.c:1050:__socket_keepalive]
0-shared-server: Keep-alive enabled for socket: 5, (idle: 20, interval: 2,
max-probes: 9, timeout: 42)
[2019-08-04 10:24:26.719225] T [socket.c:3086:socket_server_event_handler]
0-tcp.shared-server: XXX server:10.7.4.110:49156, client:10.7.3.217:34510
[2019-08-04 10:24:26.719241] T [socket.c:2884:socket_event_handler]
0-tcp.shared-server: server (sock:5) in:1, out:0, err:0
[2019-08-04 10:24:26.719246] T [socket.c:2891:socket_event_handler]
0-tcp.shared-server: server (sock:5) socket is not connected, completing
connection
[2019-08-04 10:24:26.719254] T [socket.c:2898:socket_event_handler]
0-tcp.shared-server: (sock:5) socket_complete_connection() returned 1
[2019-08-04 10:24:26.719258] T [socket.c:2902:socket_event_handler]
0-tcp.shared-server: (sock:5) returning to wait on socket
[2019-08-04 10:24:26.719266] T [socket.c:2884:socket_event_handler]
0-tcp.shared-server: server (sock:5) in:1, out:0, err:0
[2019-08-04 10:24:26.719279] T [socket.c:2910:socket_event_handler]
0-tcp.shared-server: Server socket (5) is already connected
[2019-08-04 10:24:26.719286] T [socket.c:520:__socket_ssl_readv]
0-tcp.shared-server: ***** reading over non-SSL
[2019-08-04 10:24:26.719446] D [MSGID: 0] [iobuf.c:568:iobuf_get2] 0-iobuf:
request for iobuf of size 369295620 is serviced using standard calloc()
(0x7fb1bc002160) as it exceeds the maximum available buffer size
[2019-08-04 10:24:26.719458] T [socket.c:520:__socket_ssl_readv]
0-tcp.shared-server: ***** reading over non-SSL
[2019-08-04 10:24:26.719466] E [socket.c:2252:__socket_read_frag] 0-rpc: wrong
MSG-TYPE (-1107099299) received from 10.7.3.217:34510
[2019-08-04 10:24:26.719470] T [socket.c:2928:socket_event_handler]
0-tcp.shared-server: (sock:5) socket_event_poll_in returned -1
[2019-08-04 10:24:26.719479] T [socket.c:231:socket_dump_info]
0-tcp.shared-server: $$$ server: disconnecting from (af:2,sock:5) 10.7.3.217
non-SSL (errno:0:Success)
[2019-08-04 10:24:26.719483] D [socket.c:2946:socket_event_handler]
0-transport: EPOLLERR - disconnecting (sock:5) (non-SSL)
[2019-08-04 10:24:26.719488] D [MSGID: 0] [iobuf.c:683:__iobuf_put] 0-iobuf:
freeing the iobuf (0x7fb1bc002160) allocated with standard calloc()
[2019-08-04 10:24:26.719511] T [socket.c:4583:fini] 0-tcp.shared-server:
transport 0x7fb1b80053f0 destroyed
[2019-08-04 10:24:26.719975] T [socket.c:961:__socket_nodelay] 0-shared-server:
NODELAY enabled for socket 5
[2019-08-04 10:24:26.719990] T [socket.c:1050:__socket_keepalive]
0-shared-server: Keep-alive enabled for socket: 5, (idle: 20, interval: 2,
max-probes: 9, timeout: 42)
[2019-08-04 10:24:26.720004] T [socket.c:3086:socket_server_event_handler]
0-tcp.shared-server: XXX server:10.7.4.110:49156, client:10.7.3.217:34512
[2019-08-04 10:24:26.720019] T [socket.c:2884:socket_event_handler]
0-tcp.shared-server: server (sock:5) in:1, out:0, err:0
[2019-08-04 10:24:26.720024] T [socket.c:2891:socket_event_handler]
0-tcp.shared-server: server (sock:5) socket is not connected, completing
connection
[2019-08-04 10:24:26.720031] T [socket.c:2898:socket_event_handler]
0-tcp.shared-server: (sock:5) socket_complete_connection() returned 1
[2019-08-04 10:24:26.720035] T [socket.c:2902:socket_event_handler]
0-tcp.shared-server: (sock:5) returning to wait on socket
[2019-08-04 10:24:26.720040] T [socket.c:2884:socket_event_handler]
0-tcp.shared-server: server (sock:5) in:1, out:0, err:0
[2019-08-04 10:24:26.720050] T [socket.c:2910:socket_event_handler]
0-tcp.shared-server: Server socket (5) is already connected
[2019-08-04 10:24:26.720054] T [socket.c:520:__socket_ssl_readv]
0-tcp.shared-server: ***** reading over non-SSL
[2019-08-04 10:24:26.720198] D [MSGID: 0] [iobuf.c:568:iobuf_get2] 0-iobuf:
request for iobuf of size 369295620 is serviced using standard calloc()
(0x7fb1b8015f70) as it exceeds the maximum available buffer size
[2019-08-04 10:24:26.720209] T [socket.c:520:__socket_ssl_readv]
0-tcp.shared-server: ***** reading over non-SSL
[2019-08-04 10:24:26.720216] E [socket.c:2252:__socket_read_frag] 0-rpc: wrong
MSG-TYPE (-2080177827) received from 10.7.3.217:34512
[2019-08-04 10:24:26.720221] T [socket.c:2928:socket_event_handler]
0-tcp.shared-server: (sock:5) socket_event_poll_in returned -1
[2019-08-04 10:24:26.720229] T [socket.c:231:socket_dump_info]
0-tcp.shared-server: $$$ server: disconnecting from (af:2,sock:5) 10.7.3.217
non-SSL (errno:0:Success)
[2019-08-04 10:24:26.720234] D [socket.c:2946:socket_event_handler]
0-transport: EPOLLERR - disconnecting (sock:5) (non-SSL)
[2019-08-04 10:24:26.720239] D [MSGID: 0] [iobuf.c:683:__iobuf_put] 0-iobuf:
freeing the iobuf (0x7fb1b8015f70) allocated with standard calloc()
[2019-08-04 10:24:26.720262] T [socket.c:4583:fini] 0-tcp.shared-server:
transport 0x7fb1bc08d9b0 destroyed
[2019-08-04 10:24:26.720613] T [socket.c:961:__socket_nodelay] 0-shared-server:
NODELAY enabled for socket 5
[2019-08-04 10:24:26.720637] T [socket.c:1050:__socket_keepalive]
0-shared-server: Keep-alive enabled for socket: 5, (idle: 20, interval: 2,
max-probes: 9, timeout: 42)
[2019-08-04 10:24:26.720647] T [socket.c:3086:socket_server_event_handler]
0-tcp.shared-server: XXX server:10.7.4.110:49156, client:10.7.3.217:34514
[2019-08-04 10:24:26.720661] T [socket.c:2884:socket_event_handler]
0-tcp.shared-server: server (sock:5) in:1, out:0, err:0
[2019-08-04 10:24:26.720666] T [socket.c:2891:socket_event_handler]
0-tcp.shared-server: server (sock:5) socket is not connected, completing
connection
[2019-08-04 10:24:26.720674] T [socket.c:2898:socket_event_handler]
0-tcp.shared-server: (sock:5) socket_complete_connection() returned 1
[2019-08-04 10:24:26.720678] T [socket.c:2902:socket_event_handler]
0-tcp.shared-server: (sock:5) returning to wait on socket
[2019-08-04 10:24:26.720681] T [socket.c:2884:socket_event_handler]
0-tcp.shared-server: server (sock:5) in:1, out:0, err:0
[2019-08-04 10:24:26.720693] T [socket.c:2910:socket_event_handler]
0-tcp.shared-server: Server socket (5) is already connected
[2019-08-04 10:24:26.720697] T [socket.c:520:__socket_ssl_readv]
0-tcp.shared-server: ***** reading over non-SSL
[2019-08-04 10:24:26.720848] D [MSGID: 0] [iobuf.c:568:iobuf_get2] 0-iobuf:
request for iobuf of size 369295364 is serviced using standard calloc()
(0x7fb1bc002160) as it exceeds the maximum available buffer size
[2019-08-04 10:24:26.720859] T [socket.c:520:__socket_ssl_readv]
0-tcp.shared-server: ***** reading over non-SSL
[2019-08-04 10:24:26.720866] E [socket.c:2252:__socket_read_frag] 0-rpc: wrong
MSG-TYPE (1812136029) received from 10.7.3.217:34514
[2019-08-04 10:24:26.720871] T [socket.c:2928:socket_event_handler]
0-tcp.shared-server: (sock:5) socket_event_poll_in returned -1
[2019-08-04 10:24:26.720883] T [socket.c:231:socket_dump_info]
0-tcp.shared-server: $$$ server: disconnecting from (af:2,sock:5) 10.7.3.217
non-SSL (errno:0:Success)
[2019-08-04 10:24:26.720888] D [socket.c:2946:socket_event_handler]
0-transport: EPOLLERR - disconnecting (sock:5) (non-SSL)
[2019-08-04 10:24:26.720892] D [MSGID: 0] [iobuf.c:683:__iobuf_put] 0-iobuf:
freeing the iobuf (0x7fb1bc002160) allocated with standard calloc()
[2019-08-04 10:24:26.720914] T [socket.c:4583:fini] 0-tcp.shared-server:
transport 0x7fb1b80053f0 destroyed
[2019-08-04 10:24:26.721297] T [socket.c:961:__socket_nodelay] 0-shared-server:
NODELAY enabled for socket 5
[2019-08-04 10:24:26.721308] T [socket.c:1050:__socket_keepalive]
0-shared-server: Keep-alive enabled for socket: 5, (idle: 20, interval: 2,
max-probes: 9, timeout: 42)
[2019-08-04 10:24:26.721317] T [socket.c:3086:socket_server_event_handler]
0-tcp.shared-server: XXX server:10.7.4.110:49156, client:10.7.3.217:34516
[2019-08-04 10:24:26.721344] T [socket.c:2884:socket_event_handler]
0-tcp.shared-server: server (sock:5) in:1, out:0, err:0
[2019-08-04 10:24:26.721349] T [socket.c:2891:socket_event_handler]
0-tcp.shared-server: server (sock:5) socket is not connected, completing
connection
[2019-08-04 10:24:26.721358] T [socket.c:2898:socket_event_handler]
0-tcp.shared-server: (sock:5) socket_complete_connection() returned 1
[2019-08-04 10:24:26.721362] T [socket.c:2902:socket_event_handler]
0-tcp.shared-server: (sock:5) returning to wait on socket
[2019-08-04 10:24:26.721371] T [socket.c:2884:socket_event_handler]
0-tcp.shared-server: server (sock:5) in:1, out:0, err:0
[2019-08-04 10:24:26.721383] T [socket.c:2910:socket_event_handler]
0-tcp.shared-server: Server socket (5) is already connected
[2019-08-04 10:24:26.721388] T [socket.c:520:__socket_ssl_readv]
0-tcp.shared-server: ***** reading over non-SSL
[2019-08-04 10:24:26.721533] D [MSGID: 0] [iobuf.c:568:iobuf_get2] 0-iobuf:
request for iobuf of size 369295364 is serviced using standard calloc()
(0x7fb1b8015f70) as it exceeds the maximum available buffer size
[2019-08-04 10:24:26.721544] T [socket.c:520:__socket_ssl_readv]
0-tcp.shared-server: ***** reading over non-SSL
[2019-08-04 10:24:26.721551] E [socket.c:2252:__socket_read_frag] 0-rpc: wrong
MSG-TYPE (1241710685) received from 10.7.3.217:34516
[2019-08-04 10:24:26.721555] T [socket.c:2928:socket_event_handler]
0-tcp.shared-server: (sock:5) socket_event_poll_in returned -1
[2019-08-04 10:24:26.721564] T [socket.c:231:socket_dump_info]
0-tcp.shared-server: $$$ server: disconnecting from (af:2,sock:5) 10.7.3.217
non-SSL (errno:0:Success)
[2019-08-04 10:24:26.721569] D [socket.c:2946:socket_event_handler]
0-transport: EPOLLERR - disconnecting (sock:5) (non-SSL)
[2019-08-04 10:24:26.721574] D [MSGID: 0] [iobuf.c:683:__iobuf_put] 0-iobuf:
freeing the iobuf (0x7fb1b8015f70) allocated with standard calloc()
--
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.
More information about the Bugs
mailing list