[Gluster-users] crashing a lot

Joe Julian joe at julianfamily.org
Sat Feb 15 05:32:42 UTC 2020


I don't have the system configured to dump core, but I'll change that and try.

On February 14, 2020 6:39:23 PM PST, Mohit Agrawal <moagrawa at redhat.com> wrote:
>I don't think it is similar to Xavi fixed in
>https://review.gluster.org/#/c/glusterfs/+/24099/.
>Is it possible to share the output "thread apply all bt full" after
>attaching the core with gdb?
>
>Regards,
>Mohit Agrawal
>
>On Sat, Feb 15, 2020 at 7:25 AM Amar Tumballi <amar at kadalu.io> wrote:
>
>> Is this crash seen already ?  Does
>> https://review.gluster.org/#/c/glusterfs/+/24099/ fix this?
>>
>> On Sat, Feb 15, 2020 at 4:32 AM Joe Julian <joe at julianfamily.org>
>wrote:
>>
>>> These crashes have been happening almost daily. Any thoughts on how
>to
>>> stabilize this?
>>>
>>>
>>> [2020-02-14 19:02:13.932178] I [MSGID: 100030]
>[glusterfsd.c:2865:main]
>>> 0-/usr/bin/glusterfs: Started running /usr/bin/glusterfs version 7.0
>>> (args: /usr/bin/glusterfs --process-name fuse
>--volfile-server=gluster
>>> --volfile-id=kube /tmp/hostpath_pv)
>>> [2020-02-14 19:02:13.939208] I [glusterfsd.c:2593:daemonize]
>>> 0-glusterfs: Pid of current running process is 1828083
>>> [2020-02-14 19:02:13.970829] I [MSGID: 101190]
>>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started
>thread
>>> with index 0
>>> [2020-02-14 19:02:13.970870] I [MSGID: 101190]
>>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started
>thread
>>> with index 1
>>> [2020-02-14 19:02:14.026097] W [MSGID: 108003]
>>> [afr.c:98:fix_quorum_options] 0-kube-replicate-0: quorum-type none
>>> overriding quorum-count 1
>>> [2020-02-14 19:02:14.027753] I [MSGID: 101190]
>>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started
>thread
>>> with index 2
>>> [2020-02-14 19:02:14.027871] I [MSGID: 101190]
>>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started
>thread
>>> with index 4
>>> [2020-02-14 19:02:14.027939] I [MSGID: 101190]
>>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started
>thread
>>> with index 5
>>> [2020-02-14 19:02:14.027984] I [MSGID: 101190]
>>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started
>thread
>>> with index 3
>>> [2020-02-14 19:02:14.028010] I [MSGID: 101190]
>>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started
>thread
>>> with index 6
>>> [2020-02-14 19:02:14.028410] I [MSGID: 101190]
>>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started
>thread
>>> with index 7
>>> [2020-02-14 19:02:14.028826] I [MSGID: 114020]
>[client.c:2434:notify]
>>> 0-kube-client-0: parent translators are ready, attempting connect on
>>> transport
>>> [2020-02-14 19:02:14.031361] I [MSGID: 114020]
>[client.c:2434:notify]
>>> 0-kube-client-1: parent translators are ready, attempting connect on
>>> transport
>>> [2020-02-14 19:02:14.031840] I [rpc-clnt.c:1962:rpc_clnt_reconfig]
>>> 0-kube-client-0: changing port to 49159 (from 0)
>>> [2020-02-14 19:02:14.031880] I [socket.c:863:__socket_shutdown]
>>> 0-kube-client-0: intentional socket shutdown(12)
>>> Final graph:
>>>
>>>
>+------------------------------------------------------------------------------+
>>>    1: volume kube-client-0
>>>    2:     type protocol/client
>>>    3:     option ping-timeout 42
>>>    4:     option remote-host strabo
>>>    5:     option remote-subvolume /data/gluster/kube
>>>    6:     option transport-type socket
>>>    7:     option transport.address-family inet
>>>    8:     option username 51b95740-bff7-4744-bbfc-dbae17670997
>>>    9:     option password 021da395-9404-43ac-b358-2f5dfc5acc71
>>>   10:     option transport.socket.ssl-enabled off
>>>   11:     option filter-O_DIRECT on
>>>   12:     option event-threads 8
>>>   13:     option transport.tcp-user-timeout 0
>>>   14:     option transport.socket.keepalive-time 20
>>>   15:     option transport.socket.keepalive-interval 2
>>>   16:     option transport.socket.keepalive-count 9
>>>   17:     option send-gids true
>>>   18: end-volume
>>>   19:
>>>   20: volume kube-client-1
>>>   21:     type protocol/client
>>>   22:     option ping-timeout 42
>>>   23:     option remote-host nightshade
>>>   24:     option remote-subvolume /data/gluster/kube
>>>   25:     option transport-type socket
>>>   26:     option transport.address-family inet
>>>   27:     option username 51b95740-bff7-4744-bbfc-dbae17670997
>>>   28:     option password 021da395-9404-43ac-b358-2f5dfc5acc71
>>>   29:     option transport.socket.ssl-enabled off
>>>   30:     option filter-O_DIRECT on
>>>   31:     option event-threads 8
>>>   32:     option transport.tcp-user-timeout 0
>>>   33:     option transport.socket.keepalive-time 20
>>>   34:     option transport.socket.keepalive-interval 2
>>>   35:     option transport.socket.keepalive-count 9
>>>   36:     option send-gids true
>>>   37: end-volume
>>>   38:
>>>   39: volume kube-replicate-0
>>>   40:     type cluster/replicate
>>>   41:     option afr-pending-xattr kube-client-0,kube-client-1
>>>   42:     option quorum-count 1
>>>   43:     option use-compound-fops off
>>>   44:     subvolumes kube-client-0 kube-client-1
>>>   45: end-volume
>>>   46:
>>>   47: volume kube-dht
>>>   48:     type cluster/distribute
>>>   49:     option readdir-optimize on
>>>   50:     option lock-migration off
>>>   51:     option force-migration off
>>>   52:     subvolumes kube-replicate-0
>>>   53: end-volume
>>>   54:
>>>   55: volume kube-io-cache
>>>   56:     type performance/io-cache
>>>   57:     option cache-timeout 30
>>>   58:     subvolumes kube-dht
>>>   59: end-volume
>>>   60:
>>>   61: volume kube-open-behind
>>>   62:     type performance/open-behind
>>>   63:     subvolumes kube-io-cache
>>>   64: end-volume
>>>   65:
>>>   66: volume kube-md-cache
>>>   67:     type performance/md-cache
>>>   68:     option cache-invalidation on
>>>   69:     subvolumes kube-open-behind
>>>   70: end-volume
>>>   71:
>>>   72: volume kube-io-threads
>>>   73:     type performance/io-threads
>>>   74:     subvolumes kube-md-cache
>>>   75: end-volume
>>>   76:
>>>   77: volume kube
>>>   78:     type debug/io-stats
>>>   79:     option log-level INFO
>>>   80:     option threads 16
>>>   81:     option latency-measurement off
>>>   82:     option count-fop-hits off
>>>   83:     option global-threading off
>>>   84:     subvolumes kube-io-threads
>>>   85: end-volume
>>>   86:
>>>   87: volume meta-autoload
>>>   88:     type meta
>>>   89:     subvolumes kube
>>>   90: end-volume
>>>   91:
>>>
>>>
>+------------------------------------------------------------------------------+
>>> [2020-02-14 19:02:14.052685] I [rpc-clnt.c:1962:rpc_clnt_reconfig]
>>> 0-kube-client-1: changing port to 49159 (from 0)
>>> [2020-02-14 19:02:14.052792] I [socket.c:863:__socket_shutdown]
>>> 0-kube-client-1: intentional socket shutdown(13)
>>> [2020-02-14 19:02:14.056240] I [MSGID: 114057]
>>> [client-handshake.c:1373:select_server_supported_programs]
>>> 0-kube-client-1: Using Program GlusterFS 4.x v1, Num (1298437),
>Version
>>> (400)
>>> [2020-02-14 19:02:14.076058] I [MSGID: 114046]
>>> [client-handshake.c:1104:client_setvolume_cbk] 0-kube-client-1:
>>> Connected to kube-client-1, attached to remote volume
>>> '/data/gluster/kube'.
>>> [2020-02-14 19:02:14.076113] I [MSGID: 108005]
>>> [afr-common.c:5277:__afr_handle_child_up_event] 0-kube-replicate-0:
>>> Subvolume 'kube-client-1' came back up; going online.
>>> [2020-02-14 19:02:24.031553] I [fuse-bridge.c:5162:fuse_init]
>>> 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24
>>> kernel 7.31
>>> [2020-02-14 19:02:24.031605] I [fuse-bridge.c:5777:fuse_graph_sync]
>>> 0-fuse: switched to graph 0
>>> [2020-02-14 19:04:27.205776] I [rpc-clnt.c:1962:rpc_clnt_reconfig]
>>> 0-kube-client-0: changing port to 49159 (from 0)
>>> [2020-02-14 19:06:40.325688] I [rpc-clnt.c:1962:rpc_clnt_reconfig]
>>> 0-kube-client-0: changing port to 49159 (from 0)
>>> [2020-02-14 19:08:53.444994] I [rpc-clnt.c:1962:rpc_clnt_reconfig]
>>> 0-kube-client-0: changing port to 49159 (from 0)
>>> [2020-02-14 19:11:06.565527] I [rpc-clnt.c:1962:rpc_clnt_reconfig]
>>> 0-kube-client-0: changing port to 49159 (from 0)
>>> pending frames:
>>> frame : type(0) op(0)
>>> frame : type(0) op(0)
>>> frame : type(1) op(LOOKUP)
>>> frame : type(1) op(FLUSH)
>>> frame : type(1) op(LOOKUP)
>>> frame : type(1) op(LOOKUP)
>>> frame : type(1) op(LOOKUP)
>>> frame : type(1) op(LOOKUP)
>>> frame : type(1) op(FLUSH)
>>> frame : type(1) op(LOOKUP)
>>> frame : type(1) op(LOOKUP)
>>> frame : type(1) op(LOOKUP)
>>> frame : type(1) op(LOOKUP)
>>> frame : type(1) op(LOOKUP)
>>> frame : type(1) op(LOOKUP)
>>> frame : type(1) op(LOOKUP)
>>> frame : type(1) op(LOOKUP)
>>> frame : type(1) op(LOOKUP)
>>> frame : type(1) op(LOOKUP)
>>> patchset: git://git.gluster.org/glusterfs.git
>>> signal received: 11
>>> time of crash:
>>> 2020-02-14 19:13:00
>>> configuration details:
>>> argp 1
>>> backtrace 1
>>> dlfcn 1
>>> libpthread 1
>>> llistxattr 1
>>> setfsid 1
>>> spinlock 1
>>> epoll.h 1
>>> xattr.h 1
>>> st_atim.tv_nsec 1
>>> package-string: glusterfs 7.0
>>> /usr/lib/libglusterfs.so.0(+0x25e12)[0x7fc026b55e12]
>>> /usr/lib/libglusterfs.so.0(gf_print_trace+0x354)[0x7fc026b60b74]
>>> /usr/lib/libc.so.6(+0x3bfb0)[0x7fc0268e3fb0]
>>> /usr/lib/libglusterfs.so.0(mem_get+0xf0)[0x7fc026b81850]
>>> /usr/lib/libglusterfs.so.0(+0x1a4cd)[0x7fc026b4a4cd]
>>> /usr/lib/libglusterfs.so.0(dict_setn+0x1f1)[0x7fc026b4a941]
>>> /usr/lib/libglusterfs.so.0(dict_set_dynptr+0x32)[0x7fc026b4e342]
>>>
>/usr/lib/glusterfs/7.0/xlator/protocol/client.so(+0x3b318)[0x7fc0210cb318]
>>>
>/usr/lib/glusterfs/7.0/xlator/protocol/client.so(+0x56d03)[0x7fc0210e6d03]
>>> /usr/lib/libgfrpc.so.0(+0xe957)[0x7fc026afe957]
>>> /usr/lib/libgfrpc.so.0(+0xecc9)[0x7fc026afecc9]
>>> /usr/lib/libgfrpc.so.0(rpc_transport_notify+0x26)[0x7fc026afb816]
>>>
>/usr/lib/glusterfs/7.0/rpc-transport/socket.so(+0x4629)[0x7fc022294629]
>>>
>/usr/lib/glusterfs/7.0/rpc-transport/socket.so(+0xc45c)[0x7fc02229c45c]
>>> /usr/lib/libglusterfs.so.0(+0x8cedc)[0x7fc026bbcedc]
>>> /usr/lib/libpthread.so.0(+0x94cf)[0x7fc026a794cf]
>>> /usr/lib/libc.so.6(clone+0x43)[0x7fc0269a72d3]
>>> ---------
>>>
>>> ________
>>>
>>> Community Meeting Calendar:
>>>
>>> APAC Schedule -
>>> Every 2nd and 4th Tuesday at 11:30 AM IST
>>> Bridge: https://bluejeans.com/441850968
>>>
>>> NA/EMEA Schedule -
>>> Every 1st and 3rd Tuesday at 01:00 PM EDT
>>> Bridge: https://bluejeans.com/441850968
>>>
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>
>>
>> --
>> --
>> https://kadalu.io
>> Container Storage made easy!
>>
>>

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200214/6f24a3e9/attachment.html>


More information about the Gluster-users mailing list