[Bugs] [Bug 1651246] Failed to dispatch handler
bugzilla at redhat.com
bugzilla at redhat.com
Tue Jan 15 10:43:33 UTC 2019
https://bugzilla.redhat.com/show_bug.cgi?id=1651246
--- Comment #9 from Guillaume Pavese <guillaume.pavese at interact-iv.com> ---
Similar problem on a newly provisioned ovirt 4.3 cluster (centos 7.6, gluster
5.2-1) :
[2019-01-15 09:32:02.558598] I [MSGID: 100030] [glusterfsd.c:2691:main]
0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 5.2 (args:
/usr/sbin/glusterfs --process-name fuse
--volfile-server=ps-inf-int-kvm-fr-306-210.hostics.fr
--volfile-server=10.199.211.7 --volfile-server=10.199.211.5
--volfile-id=/vmstore
/rhev/data-center/mnt/glusterSD/ps-inf-int-kvm-fr-306-210.hostics.fr:_vmstore)
[2019-01-15 09:32:02.566701] I [MSGID: 101190]
[event-epoll.c:622:event_dispatch_epoll_worker] 0-epoll: Started thread with
index 1
[2019-01-15 09:32:02.581138] I [MSGID: 101190]
[event-epoll.c:622:event_dispatch_epoll_worker] 0-epoll: Started thread with
index 2
[2019-01-15 09:32:02.581272] I [MSGID: 114020] [client.c:2354:notify]
0-vmstore-client-0: parent translators are ready, attempting connect on
transport
[2019-01-15 09:32:02.583283] I [MSGID: 114020] [client.c:2354:notify]
0-vmstore-client-1: parent translators are ready, attempting connect on
transport
[2019-01-15 09:32:02.583911] I [rpc-clnt.c:2042:rpc_clnt_reconfig]
0-vmstore-client-0: changing port to 49155 (from 0)
[2019-01-15 09:32:02.585505] I [MSGID: 114020] [client.c:2354:notify]
0-vmstore-client-2: parent translators are ready, attempting connect on
transport
[2019-01-15 09:32:02.587413] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-01-15 09:32:02.587441] E [MSGID: 108006]
[afr-common.c:5314:__afr_handle_child_down_event] 0-vmstore-replicate-0: All
subvolumes are down. Going offline until at least one of them comes back up.
[2019-01-15 09:32:02.587951] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-01-15 09:32:02.588685] I [MSGID: 114046]
[client-handshake.c:1107:client_setvolume_cbk] 0-vmstore-client-0: Connected to
vmstore-client-0, attached to remote volume '/gluster_bricks/vmstore/vmstore'.
[2019-01-15 09:32:02.588708] I [MSGID: 108005]
[afr-common.c:5237:__afr_handle_child_up_event] 0-vmstore-replicate-0:
Subvolume 'vmstore-client-0' came back up; going online.
Final graph:
+------------------------------------------------------------------------------+
1: volume vmstore-client-0
2: type protocol/client
3: option opversion 50000
4: option clnt-lk-version 1
5: option volfile-checksum 0
6: option volfile-key /vmstore
7: option client-version 5.2
8: option process-name fuse
9: option process-uuid
CTX_ID:e5dad97f-5289-4464-9e2f-36e9bb115118-GRAPH_ID:0-PID:39987-HOST:ps-inf-int-kvm-fr-307-210.hostics.fr-PC_NAME:vmstore-client-0-RECON_NO:-0
10: option fops-version 1298437
11: option ping-timeout 30
12: option remote-host 10.199.211.6
13: option remote-subvolume /gluster_bricks/vmstore/vmstore
14: option transport-type socket
15: option transport.address-family inet
16: option filter-O_DIRECT off
17: option transport.tcp-user-timeout 0
18: option transport.socket.keepalive-time 20
19: option transport.socket.keepalive-interval 2
20: option transport.socket.keepalive-count 9
21: option send-gids true
22: end-volume
23:
24: volume vmstore-client-1
25: type protocol/client
26: option ping-timeout 30
27: option remote-host 10.199.211.7
28: option remote-subvolume /gluster_bricks/vmstore/vmstore
29: option transport-type socket
30: option transport.address-family inet
31: option filter-O_DIRECT off
32: option transport.tcp-user-timeout 0
33: option transport.socket.keepalive-time 20
34: option transport.socket.keepalive-interval 2
35: option transport.socket.keepalive-count 9
36: option send-gids true
37: end-volume
38:
39: volume vmstore-client-2
40: type protocol/client
41: option ping-timeout 30
42: option remote-host 10.199.211.5
43: option remote-subvolume /gluster_bricks/vmstore/vmstore
44: option transport-type socket
45: option transport.address-family inet
46: option filter-O_DIRECT off
47: option transport.tcp-user-timeout 0
48: option transport.socket.keepalive-time 20
49: option transport.socket.keepalive-interval 2
50: option transport.socket.keepalive-count 9
51: option send-gids true
52: end-volume
53:
54: volume vmstore-replicate-0
55: type cluster/replicate
56: option afr-pending-xattr
vmstore-client-0,vmstore-client-1,vmstore-client-2
57: option arbiter-count 1
58: option data-self-heal-algorithm full
59: option eager-lock enable
60: option quorum-type auto
61: option choose-local off
62: option shd-max-threads 8
63: option shd-wait-qlength 10000
64: option locking-scheme granular
65: option granular-entry-heal enable
66: option use-compound-fops off
67: subvolumes vmstore-client-0 vmstore-client-1 vmstore-client-2
68: end-volume
69:
70: volume vmstore-dht
71: type cluster/distribute
72: option lock-migration off
73: option force-migration off
74: subvolumes vmstore-replicate-0
75: end-volume
76:
77: volume vmstore-shard
78: type features/shard
79: subvolumes vmstore-dht
80: end-volume
81:
82: volume vmstore-write-behind
83: type performance/write-behind
84: option strict-O_DIRECT on
85: subvolumes vmstore-shard
86: end-volume
87:
88: volume vmstore-readdir-ahead
89: type performance/readdir-ahead
90: option parallel-readdir off
91: option rda-request-size 131072
92: option rda-cache-limit 10MB
93: subvolumes vmstore-write-behind
94: end-volume
95:
96: volume vmstore-open-behind
97: type performance/open-behind
98: subvolumes vmstore-readdir-ahead
99: end-volume
100:
101: volume vmstore-md-cache
102: type performance/md-cache
103: subvolumes vmstore-open-behind
104: end-volume
105:
106: volume vmstore
107: type debug/io-stats
108: option log-level INFO
109: option latency-measurement off
110: option count-fop-hits off
111: subvolumes vmstore-md-cache
112: end-volume
113:
114: volume meta-autoload
115: type meta
116: subvolumes vmstore
117: end-volume
118:
+------------------------------------------------------------------------------+
[2019-01-15 09:32:02.590376] I [rpc-clnt.c:2042:rpc_clnt_reconfig]
0-vmstore-client-2: changing port to 49155 (from 0)
[2019-01-15 09:32:02.592649] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-01-15 09:32:02.593512] I [MSGID: 114046]
[client-handshake.c:1107:client_setvolume_cbk] 0-vmstore-client-2: Connected to
vmstore-client-2, attached to remote volume '/gluster_bricks/vmstore/vmstore'.
[2019-01-15 09:32:02.593528] I [MSGID: 108002] [afr-common.c:5588:afr_notify]
0-vmstore-replicate-0: Client-quorum is met
[2019-01-15 09:32:02.594714] I [fuse-bridge.c:4259:fuse_init] 0-glusterfs-fuse:
FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22
[2019-01-15 09:32:02.594746] I [fuse-bridge.c:4870:fuse_graph_sync] 0-fuse:
switched to graph 0
[2019-01-15 09:32:06.562678] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-01-15 09:32:09.435695] W [dict.c:761:dict_ref]
(-->/usr/lib64/glusterfs/5.2/xlator/performance/open-behind.so(+0x3d7c)
[0x7f5c279cfd7c]
-->/usr/lib64/glusterfs/5.2/xlator/performance/open-behind.so(+0x3bd6)
[0x7f5c279cfbd6] -->/lib64/libglusterfs.so.0(dict_ref+0x5d) [0x7f5c340ae20d] )
0-dict: dict is NULL [Invalid argument]
The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker]
0-epoll: Failed to dispatch handler" repeated 7 times between [2019-01-15
09:32:06.562678] and [2019-01-15 09:32:27.578753]
[2019-01-15 09:32:29.966249] W [glusterfsd.c:1481:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7dd5) [0x7f5c32f1ddd5]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x55af1f5bad45]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55af1f5babbb] ) 0-: received
signum (15), shutting down
[2019-01-15 09:32:29.966265] I [fuse-bridge.c:5897:fini] 0-fuse: Unmounting
'/rhev/data-center/mnt/glusterSD/ps-inf-int-kvm-fr-306-210.hostics.fr:_vmstore'.
[2019-01-15 09:32:29.985157] I [fuse-bridge.c:5134:fuse_thread_proc] 0-fuse:
initating unmount of
/rhev/data-center/mnt/glusterSD/ps-inf-int-kvm-fr-306-210.hostics.fr:_vmstore
[2019-01-15 09:32:29.985434] I [fuse-bridge.c:5902:fini] 0-fuse: Closing fuse
connection to
'/rhev/data-center/mnt/glusterSD/ps-inf-int-kvm-fr-306-210.hostics.fr:_vmstore'.
--
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.
More information about the Bugs
mailing list