<div dir="ltr">Is this crash seen already ? Does <a href="https://review.gluster.org/#/c/glusterfs/+/24099/">https://review.gluster.org/#/c/glusterfs/+/24099/</a> fix this?</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Feb 15, 2020 at 4:32 AM Joe Julian <<a href="mailto:joe@julianfamily.org">joe@julianfamily.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">These crashes have been happening almost daily. Any thoughts on how to <br>
stabilize this?<br>
<br>
<br>
[2020-02-14 19:02:13.932178] I [MSGID: 100030] [glusterfsd.c:2865:main] <br>
0-/usr/bin/glusterfs: Started running /usr/bin/glusterfs version 7.0 <br>
(args: /usr/bin/glusterfs --process-name fuse --volfile-server=gluster <br>
--volfile-id=kube /tmp/hostpath_pv)<br>
[2020-02-14 19:02:13.939208] I [glusterfsd.c:2593:daemonize] <br>
0-glusterfs: Pid of current running process is 1828083<br>
[2020-02-14 19:02:13.970829] I [MSGID: 101190] <br>
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread <br>
with index 0<br>
[2020-02-14 19:02:13.970870] I [MSGID: 101190] <br>
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread <br>
with index 1<br>
[2020-02-14 19:02:14.026097] W [MSGID: 108003] <br>
[afr.c:98:fix_quorum_options] 0-kube-replicate-0: quorum-type none <br>
overriding quorum-count 1<br>
[2020-02-14 19:02:14.027753] I [MSGID: 101190] <br>
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread <br>
with index 2<br>
[2020-02-14 19:02:14.027871] I [MSGID: 101190] <br>
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread <br>
with index 4<br>
[2020-02-14 19:02:14.027939] I [MSGID: 101190] <br>
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread <br>
with index 5<br>
[2020-02-14 19:02:14.027984] I [MSGID: 101190] <br>
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread <br>
with index 3<br>
[2020-02-14 19:02:14.028010] I [MSGID: 101190] <br>
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread <br>
with index 6<br>
[2020-02-14 19:02:14.028410] I [MSGID: 101190] <br>
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Started thread <br>
with index 7<br>
[2020-02-14 19:02:14.028826] I [MSGID: 114020] [client.c:2434:notify] <br>
0-kube-client-0: parent translators are ready, attempting connect on <br>
transport<br>
[2020-02-14 19:02:14.031361] I [MSGID: 114020] [client.c:2434:notify] <br>
0-kube-client-1: parent translators are ready, attempting connect on <br>
transport<br>
[2020-02-14 19:02:14.031840] I [rpc-clnt.c:1962:rpc_clnt_reconfig] <br>
0-kube-client-0: changing port to 49159 (from 0)<br>
[2020-02-14 19:02:14.031880] I [socket.c:863:__socket_shutdown] <br>
0-kube-client-0: intentional socket shutdown(12)<br>
Final graph:<br>
+------------------------------------------------------------------------------+<br>
1: volume kube-client-0<br>
2: type protocol/client<br>
3: option ping-timeout 42<br>
4: option remote-host strabo<br>
5: option remote-subvolume /data/gluster/kube<br>
6: option transport-type socket<br>
7: option transport.address-family inet<br>
8: option username 51b95740-bff7-4744-bbfc-dbae17670997<br>
9: option password 021da395-9404-43ac-b358-2f5dfc5acc71<br>
10: option transport.socket.ssl-enabled off<br>
11: option filter-O_DIRECT on<br>
12: option event-threads 8<br>
13: option transport.tcp-user-timeout 0<br>
14: option transport.socket.keepalive-time 20<br>
15: option transport.socket.keepalive-interval 2<br>
16: option transport.socket.keepalive-count 9<br>
17: option send-gids true<br>
18: end-volume<br>
19:<br>
20: volume kube-client-1<br>
21: type protocol/client<br>
22: option ping-timeout 42<br>
23: option remote-host nightshade<br>
24: option remote-subvolume /data/gluster/kube<br>
25: option transport-type socket<br>
26: option transport.address-family inet<br>
27: option username 51b95740-bff7-4744-bbfc-dbae17670997<br>
28: option password 021da395-9404-43ac-b358-2f5dfc5acc71<br>
29: option transport.socket.ssl-enabled off<br>
30: option filter-O_DIRECT on<br>
31: option event-threads 8<br>
32: option transport.tcp-user-timeout 0<br>
33: option transport.socket.keepalive-time 20<br>
34: option transport.socket.keepalive-interval 2<br>
35: option transport.socket.keepalive-count 9<br>
36: option send-gids true<br>
37: end-volume<br>
38:<br>
39: volume kube-replicate-0<br>
40: type cluster/replicate<br>
41: option afr-pending-xattr kube-client-0,kube-client-1<br>
42: option quorum-count 1<br>
43: option use-compound-fops off<br>
44: subvolumes kube-client-0 kube-client-1<br>
45: end-volume<br>
46:<br>
47: volume kube-dht<br>
48: type cluster/distribute<br>
49: option readdir-optimize on<br>
50: option lock-migration off<br>
51: option force-migration off<br>
52: subvolumes kube-replicate-0<br>
53: end-volume<br>
54:<br>
55: volume kube-io-cache<br>
56: type performance/io-cache<br>
57: option cache-timeout 30<br>
58: subvolumes kube-dht<br>
59: end-volume<br>
60:<br>
61: volume kube-open-behind<br>
62: type performance/open-behind<br>
63: subvolumes kube-io-cache<br>
64: end-volume<br>
65:<br>
66: volume kube-md-cache<br>
67: type performance/md-cache<br>
68: option cache-invalidation on<br>
69: subvolumes kube-open-behind<br>
70: end-volume<br>
71:<br>
72: volume kube-io-threads<br>
73: type performance/io-threads<br>
74: subvolumes kube-md-cache<br>
75: end-volume<br>
76:<br>
77: volume kube<br>
78: type debug/io-stats<br>
79: option log-level INFO<br>
80: option threads 16<br>
81: option latency-measurement off<br>
82: option count-fop-hits off<br>
83: option global-threading off<br>
84: subvolumes kube-io-threads<br>
85: end-volume<br>
86:<br>
87: volume meta-autoload<br>
88: type meta<br>
89: subvolumes kube<br>
90: end-volume<br>
91:<br>
+------------------------------------------------------------------------------+<br>
[2020-02-14 19:02:14.052685] I [rpc-clnt.c:1962:rpc_clnt_reconfig] <br>
0-kube-client-1: changing port to 49159 (from 0)<br>
[2020-02-14 19:02:14.052792] I [socket.c:863:__socket_shutdown] <br>
0-kube-client-1: intentional socket shutdown(13)<br>
[2020-02-14 19:02:14.056240] I [MSGID: 114057] <br>
[client-handshake.c:1373:select_server_supported_programs] <br>
0-kube-client-1: Using Program GlusterFS 4.x v1, Num (1298437), Version <br>
(400)<br>
[2020-02-14 19:02:14.076058] I [MSGID: 114046] <br>
[client-handshake.c:1104:client_setvolume_cbk] 0-kube-client-1: <br>
Connected to kube-client-1, attached to remote volume '/data/gluster/kube'.<br>
[2020-02-14 19:02:14.076113] I [MSGID: 108005] <br>
[afr-common.c:5277:__afr_handle_child_up_event] 0-kube-replicate-0: <br>
Subvolume 'kube-client-1' came back up; going online.<br>
[2020-02-14 19:02:24.031553] I [fuse-bridge.c:5162:fuse_init] <br>
0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 <br>
kernel 7.31<br>
[2020-02-14 19:02:24.031605] I [fuse-bridge.c:5777:fuse_graph_sync] <br>
0-fuse: switched to graph 0<br>
[2020-02-14 19:04:27.205776] I [rpc-clnt.c:1962:rpc_clnt_reconfig] <br>
0-kube-client-0: changing port to 49159 (from 0)<br>
[2020-02-14 19:06:40.325688] I [rpc-clnt.c:1962:rpc_clnt_reconfig] <br>
0-kube-client-0: changing port to 49159 (from 0)<br>
[2020-02-14 19:08:53.444994] I [rpc-clnt.c:1962:rpc_clnt_reconfig] <br>
0-kube-client-0: changing port to 49159 (from 0)<br>
[2020-02-14 19:11:06.565527] I [rpc-clnt.c:1962:rpc_clnt_reconfig] <br>
0-kube-client-0: changing port to 49159 (from 0)<br>
pending frames:<br>
frame : type(0) op(0)<br>
frame : type(0) op(0)<br>
frame : type(1) op(LOOKUP)<br>
frame : type(1) op(FLUSH)<br>
frame : type(1) op(LOOKUP)<br>
frame : type(1) op(LOOKUP)<br>
frame : type(1) op(LOOKUP)<br>
frame : type(1) op(LOOKUP)<br>
frame : type(1) op(FLUSH)<br>
frame : type(1) op(LOOKUP)<br>
frame : type(1) op(LOOKUP)<br>
frame : type(1) op(LOOKUP)<br>
frame : type(1) op(LOOKUP)<br>
frame : type(1) op(LOOKUP)<br>
frame : type(1) op(LOOKUP)<br>
frame : type(1) op(LOOKUP)<br>
frame : type(1) op(LOOKUP)<br>
frame : type(1) op(LOOKUP)<br>
frame : type(1) op(LOOKUP)<br>
patchset: git://<a href="http://git.gluster.org/glusterfs.git" rel="noreferrer" target="_blank">git.gluster.org/glusterfs.git</a><br>
signal received: 11<br>
time of crash:<br>
2020-02-14 19:13:00<br>
configuration details:<br>
argp 1<br>
backtrace 1<br>
dlfcn 1<br>
libpthread 1<br>
llistxattr 1<br>
setfsid 1<br>
spinlock 1<br>
epoll.h 1<br>
xattr.h 1<br>
st_atim.tv_nsec 1<br>
package-string: glusterfs 7.0<br>
/usr/lib/libglusterfs.so.0(+0x25e12)[0x7fc026b55e12]<br>
/usr/lib/libglusterfs.so.0(gf_print_trace+0x354)[0x7fc026b60b74]<br>
/usr/lib/libc.so.6(+0x3bfb0)[0x7fc0268e3fb0]<br>
/usr/lib/libglusterfs.so.0(mem_get+0xf0)[0x7fc026b81850]<br>
/usr/lib/libglusterfs.so.0(+0x1a4cd)[0x7fc026b4a4cd]<br>
/usr/lib/libglusterfs.so.0(dict_setn+0x1f1)[0x7fc026b4a941]<br>
/usr/lib/libglusterfs.so.0(dict_set_dynptr+0x32)[0x7fc026b4e342]<br>
/usr/lib/glusterfs/7.0/xlator/protocol/client.so(+0x3b318)[0x7fc0210cb318]<br>
/usr/lib/glusterfs/7.0/xlator/protocol/client.so(+0x56d03)[0x7fc0210e6d03]<br>
/usr/lib/libgfrpc.so.0(+0xe957)[0x7fc026afe957]<br>
/usr/lib/libgfrpc.so.0(+0xecc9)[0x7fc026afecc9]<br>
/usr/lib/libgfrpc.so.0(rpc_transport_notify+0x26)[0x7fc026afb816]<br>
/usr/lib/glusterfs/7.0/rpc-transport/socket.so(+0x4629)[0x7fc022294629]<br>
/usr/lib/glusterfs/7.0/rpc-transport/socket.so(+0xc45c)[0x7fc02229c45c]<br>
/usr/lib/libglusterfs.so.0(+0x8cedc)[0x7fc026bbcedc]<br>
/usr/lib/libpthread.so.0(+0x94cf)[0x7fc026a794cf]<br>
/usr/lib/libc.so.6(clone+0x43)[0x7fc0269a72d3]<br>
---------<br>
<br>
________<br>
<br>
Community Meeting Calendar:<br>
<br>
APAC Schedule -<br>
Every 2nd and 4th Tuesday at 11:30 AM IST<br>
Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
<br>
NA/EMEA Schedule -<br>
Every 1st and 3rd Tuesday at 01:00 PM EDT<br>
Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr">--<div><a href="https://kadalu.io" target="_blank">https://kadalu.io</a></div><div>Container Storage made easy!</div><div><br></div></div></div>