[Gluster-users] chrome / chromium crash on gluster

Thu Jan 31 13:54:06 UTC 2019

On 1/31/19 2:21 AM, Amar Tumballi Suryanarayan wrote:
> Interesting, I run F29 for all development, and didn't see anything like 
> this.
> 
> Please share 'gluster volume info'. And also logs from mount process.

[root at gluster1 ~]# gluster volume info

Volume Name: volume1
Type: Replicate
Volume ID: 91ef5aed-94be-44ff-a19d-c41682808159
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: gluster1:/gluster/brick1/data
Brick2: gluster2:/gluster/brick2/data
Options Reconfigured:
nfs.disable: on
server.allow-insecure: on
cluster.favorite-child-policy: mtime

And a client mount log is below - although the log is full megabytes of:

The message "E [MSGID: 101191] 
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to 
dispatch handler" repeated 20178 times between [2019-01-31 
13:44:14.962950] and [2019-01-31 13:46:00.013310]

and

[2019-01-31 13:46:07.470163] W [dict.c:761:dict_ref] 
(-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7c45) 
[0x7fb0e0b49c45] 
-->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaba1) 
[0x7fb0e0b5cba1] -->/lib64/libglusterfs.so.0(dict_ref+0x60) 
[0x7fb0f2457c40] ) 0-dict: dict is NULL [Invalid argument]

so I've just shown the start of the log. I guess that's related to 
https://bugzilla.redhat.com/show_bug.cgi?id=1651246.

- Mike

Mount log:

[2019-01-31 13:44:00.775353] I [MSGID: 100030] [glusterfsd.c:2715:main] 
0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 5.3 
(args: /usr/sbin/glusterfs --process-name fuse --volfile-server=gluster1 
--volfile-server=gluster2 --volfile-id=/volume1 /fileserver2)
[2019-01-31 13:44:00.817140] I [MSGID: 101190] 
[event-epoll.c:622:event_dispatch_epoll_worker] 0-epoll: Started thread 
with index 1
[2019-01-31 13:44:00.926491] I [MSGID: 101190] 
[event-epoll.c:622:event_dispatch_epoll_worker] 0-epoll: Started thread 
with index 2
[2019-01-31 13:44:00.928102] I [MSGID: 114020] [client.c:2354:notify] 
0-volume1-client-0: parent translators are ready, attempting connect on 
transport
[2019-01-31 13:44:00.931063] I [MSGID: 114020] [client.c:2354:notify] 
0-volume1-client-1: parent translators are ready, attempting connect on 
transport
[2019-01-31 13:44:00.932144] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 
0-volume1-client-0: changing port to 49152 (from 0)
Final graph:
+------------------------------------------------------------------------------+
   1: volume volume1-client-0
   2:     type protocol/client
   3:     option ping-timeout 42
   4:     option remote-host gluster1
   5:     option remote-subvolume /gluster/brick1/data
   6:     option transport-type socket
   7:     option transport.tcp-user-timeout 0
   8:     option transport.socket.keepalive-time 20
   9:     option transport.socket.keepalive-interval 2
  10:     option transport.socket.keepalive-count 9
  11:     option send-gids true
  12: end-volume
  13:
  14: volume volume1-client-1
  15:     type protocol/client
  16:     option ping-timeout 42
  17:     option remote-host gluster2
  18:     option remote-subvolume /gluster/brick2/data
  19:     option transport-type socket
  20:     option transport.tcp-user-timeout 0
  21:     option transport.socket.keepalive-time 20
  22:     option transport.socket.keepalive-interval 2
  23:     option transport.socket.keepalive-count 9
  24:     option send-gids true
  25: end-volume
  26:
  27: volume volume1-replicate-0
  28:     type cluster/replicate
  29:     option afr-pending-xattr volume1-client-0,volume1-client-1
  30:     option favorite-child-policy mtime
  31:     option use-compound-fops off
  32:     subvolumes volume1-client-0 volume1-client-1
  33: end-volume
  34:
  35: volume volume1-dht
  36:     type cluster/distribute
[2019-01-31 13:44:00.932495] E [MSGID: 101191] 
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to 
dispatch handler
  37:     option lock-migration off
  38:     option force-migration off
  39:     subvolumes volume1-replicate-0
  40: end-volume
  41:
  42: volume volume1-write-behind
  43:     type performance/write-behind
  44:     subvolumes volume1-dht
  45: end-volume
  46:
  47: volume volume1-read-ahead
  48:     type performance/read-ahead
  49:     subvolumes volume1-write-behind
  50: end-volume
  51:
  52: volume volume1-readdir-ahead
  53:     type performance/readdir-ahead
  54:     option parallel-readdir off
  55:     option rda-request-size 131072
  56:     option rda-cache-limit 10MB
  57:     subvolumes volume1-read-ahead
  58: end-volume
  59:
  60: volume volume1-io-cache
  61:     type performance/io-cache
  62:     subvolumes volume1-readdir-ahead
  63: end-volume
  64:
  65: volume volume1-quick-read
  66:     type performance/quick-read
  67:     subvolumes volume1-io-cache
  68: end-volume
  69:
  70: volume volume1-open-behind
  71:     type performance/open-behind
  72:     subvolumes volume1-quick-read
  73: end-volume
  74:
  75: volume volume1-md-cache
  76:     type performance/md-cache
  77:     subvolumes volume1-open-behind
  78: end-volume
  79:
  80: volume volume1
  81:     type debug/io-stats
  82:     option log-level INFO
  83:     option latency-measurement off
  84:     option count-fop-hits off
  85:     subvolumes volume1-md-cache
  86: end-volume
  87:
  88: volume meta-autoload
  89:     type meta
  90:     subvolumes volume1
  91: end-volume
  92:
+------------------------------------------------------------------------------+
[2019-01-31 13:44:00.933375] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 
0-volume1-client-1: changing port to 49152 (from 0)
[2019-01-31 13:44:00.933549] E [MSGID: 101191] 
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to 
dispatch handler
[2019-01-31 13:44:00.934170] I [MSGID: 114046] 
[client-handshake.c:1107:client_setvolume_cbk] 0-volume1-client-0: 
Connected to volume1-client-0, attached to remote volume 
'/gluster/brick1/data'.
[2019-01-31 13:44:00.934210] I [MSGID: 108005] 
[afr-common.c:5237:__afr_handle_child_up_event] 0-volume1-replicate-0: 
Subvolume 'volume1-client-0' came back up; going online.
[2019-01-31 13:44:00.935291] I [MSGID: 114046] 
[client-handshake.c:1107:client_setvolume_cbk] 0-volume1-client-1: 
Connected to volume1-client-1, attached to remote volume 
'/gluster/brick2/data'.
[2019-01-31 13:44:00.937661] I [fuse-bridge.c:4267:fuse_init] 
0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 
kernel 7.28
[2019-01-31 13:44:00.937691] I [fuse-bridge.c:4878:fuse_graph_sync] 
0-fuse: switched to graph 0
[2019-01-31 13:44:14.852144] W [dict.c:761:dict_ref] 
(-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7c45) 
[0x7fb0e0b49c45] 
-->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaba1) 
[0x7fb0e0b5cba1] -->/lib64/libglusterfs.so.0(dict_ref+0x60) 
[0x7fb0f2457c40] ) 0-dict: dict is NULL [Invalid argument]
[2019-01-31 13:44:14.962950] E [MSGID: 101191] 
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to 
dispatch handler
[2019-01-31 13:44:15.038615] W [dict.c:761:dict_ref] 
(-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7c45) 
[0x7fb0e0b49c45] 
-->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaba1) 
[0x7fb0e0b5cba1] -->/lib64/libglusterfs.so.0(dict_ref+0x60) 
[0x7fb0f2457c40] ) 0-dict: dict is NULL [Invalid argument]
[2019-01-31 13:44:15.040956] W [dict.c:761:dict_ref] 
(-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7c45) 
[0x7fb0e0b49c45] 
-->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaba1) 
[0x7fb0e0b5cba1] -->/lib64/libglusterfs.so.0(dict_ref+0x60) 
[0x7fb0f2457c40] ) 0-dict: dict is NULL [Invalid argument]
[2019-01-31 13:44:15.041044] W [dict.c:761:dict_ref] 
(-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7c45) 
[0x7fb0e0b49c45] 
-->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaba1) 
[0x7fb0e0b5cba1] -->/lib64/libglusterfs.so.0(dict_ref+0x60) 
[0x7fb0f2457c40] ) 0-dict: dict is NULL [Invalid argument]
[2019-01-31 13:44:15.041467] W [dict.c:761:dict_ref] 
(-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7c45) 
[0x7fb0e0b49c45] 
-->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaba1) 
[0x7fb0e0b5cba1] -->/lib64/libglusterfs.so.0(dict_ref+0x60) 
[0x7fb0f2457c40] ) 0-dict: dict is NULL [Invalid argument]
[2019-01-31 13:44:15.471018] W [dict.c:761:dict_ref] 
(-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7c45) 
[0x7fb0e0b49c45] 
-->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaba1) 
[0x7fb0e0b5cba1] -->/lib64/libglusterfs.so.0(dict_ref+0x60) 
[0x7fb0f2457c40] ) 0-dict: dict is NULL [Invalid argument]
[2019-01-31 13:44:15.477003] W [dict.c:761:dict_ref] 
(-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7c45) 
[0x7fb0e0b49c45] 
-->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaba1) 
[0x7fb0e0b5cba1] -->/lib64/libglusterfs.so.0(dict_ref+0x60) 
[0x7fb0f2457c40] ) 0-dict: dict is NULL [Invalid argument]
[2019-01-31 13:44:15.482380] W [dict.c:761:dict_ref] 
(-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7c45) 
[0x7fb0e0b49c45] 
-->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaba1) 
[0x7fb0e0b5cba1] -->/lib64/libglusterfs.so.0(dict_ref+0x60) 
[0x7fb0f2457c40] ) 0-dict: dict is NULL [Invalid argument]
[2019-01-31 13:44:15.487047] W [dict.c:761:dict_ref] 
(-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7c45) 
[0x7fb0e0b49c45] 
-->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaba1) 
[0x7fb0e0b5cba1] -->/lib64/libglusterfs.so.0(dict_ref+0x60) 
[0x7fb0f2457c40] ) 0-dict: dict is NULL [Invalid argument]
[2019-01-31 13:44:15.603624] W [dict.c:761:dict_ref] 
(-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7c45) 
[0x7fb0e0b49c45] 
-->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaba1) 
[0x7fb0e0b5cba1] -->/lib64/libglusterfs.so.0(dict_ref+0x60) 
[0x7fb0f2457c40] ) 0-dict: dict is NULL [Invalid argument]
[2019-01-31 13:44:15.607726] W [dict.c:761:dict_ref] 
(-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7c45) 
[0x7fb0e0b49c45] 
-->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaba1) 
[0x7fb0e0b5cba1] -->/lib64/libglusterfs.so.0(dict_ref+0x60) 
[0x7fb0f2457c40] ) 0-dict: dict is NULL [Invalid argument]
[2019-01-31 13:44:15.607906] W [dict.c:761:dict_ref] 
(-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7c45) 
[0x7fb0e0b49c45] 
-->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaba1) 
[0x7fb0e0b5cba1] -->/lib64/libglusterfs.so.0(dict_ref+0x60) 
[0x7fb0f2457c40] ) 0-dict: dict is NULL [Invalid argument]