[Gluster-users] Lose gnfs connection during test

[Xie Changlong]

Hi all

I use Distributed-Replicate(12 x 2 = 24) hot tier plus 
Distributed-Replicate(36 x (6 + 2) = 288) cold tier with gluster3.8.4 
for performance test. When i set client/server.event-threads as small 
values etc 2, it works ok. But if set client/server.event-threads as big 
values etc 32, the netconnects will always become un-available during 
the test, with following error messages in stree machine.

10712 18:53:28.873 /var/log/messages: Jul 30 18:53:02 localhost_10 
kernel: nfs: server 10.147.4.99 not responding, still trying
10713 18:53:28.873 /var/log/messages: Jul 30 18:53:02 localhost_10 
kernel: nfs: server 10.147.4.99 not responding, still trying
10714 18:53:28.873 /var/log/messages: Jul 30 18:53:02 localhost_10 
kernel: nfs: server 10.147.4.99 not responding, still trying
10715 18:53:28.873 /var/log/messages: Jul 30 18:53:02 localhost_10 
kernel: nfs: server 10.147.4.99 not responding, still trying
10716 18:53:28.873 /var/log/messages: Jul 30 18:53:02 localhost_10 
kernel: nfs: server 10.147.4.99 not responding, still trying
10717 18:53:28.873 /var/log/messages: Jul 30 18:53:02 localhost_10 
kernel: nfs: server 10.147.4.99 not responding, still trying
10718 18:53:28.873 /var/log/messages: Jul 30 18:53:02 localhost_10 
kernel: nfs: server 10.147.4.99 not responding, still trying
10719 18:53:28.873 /var/log/messages: Jul 30 18:53:02 localhost_10 
kernel: nfs: server 10.147.4.99 not responding, still trying
10720 18:53:28.873 /var/log/messages: Jul 30 18:53:02 localhost_10 
kernel: nfs: server 10.147.4.99 not responding, still trying
10721 18:53:28.873 /var/log/messages: Jul 30 18:53:02 localhost_10 
kernel: nfs: server 10.147.4.99 not responding, still trying


Here is the error message in nfs.log for gluster:

  19:26:18.440498] I [rpc-drc.c:689:rpcsvc_drc_init] 0-rpc-service: DRC 
is turned OFF
[2017-07-30 19:26:18.450180] I [glusterfsd-mgmt.c:1620:mgmt_getspec_cbk] 
0-glusterfs: No change in volfile, continuing
[2017-07-30 19:26:18.493551] I [glusterfsd-mgmt.c:1620:mgmt_getspec_cbk] 
0-glusterfs: No change in volfile, continuing
[2017-07-30 19:26:18.545959] I [glusterfsd-mgmt.c:1620:mgmt_getspec_cbk] 
0-glusterfs: No change in volfile, continuing
[2017-07-30 19:42:29.704707] I [glusterfsd-mgmt.c:54:mgmt_cbk_spec] 
0-mgmt: Volume file changed
[2017-07-30 19:42:30.072282] I [glusterfsd-mgmt.c:54:mgmt_cbk_spec] 
0-mgmt: Volume file changed
[2017-07-30 19:42:30.269784] I [glusterfsd-mgmt.c:1620:mgmt_getspec_cbk] 
0-glusterfs: No change in volfile, continuing
[2017-07-30 19:42:30.315577] I [glusterfsd-mgmt.c:1620:mgmt_getspec_cbk] 
0-glusterfs: No change in volfile, continuing
[2017-07-30 19:42:48.473789] I [glusterfsd-mgmt.c:54:mgmt_cbk_spec] 
0-mgmt: Volume file changed
[2017-07-30 19:42:49.035964] I [glusterfsd-mgmt.c:54:mgmt_cbk_spec] 
0-mgmt: Volume file changed
[2017-07-30 19:42:49.128629] I [glusterfsd-mgmt.c:54:mgmt_cbk_spec] 
0-mgmt: Volume file changed
[2017-07-30 19:42:49.510522] I [glusterfsd-mgmt.c:54:mgmt_cbk_spec] 
0-mgmt: Volume file changed
[2017-07-30 19:42:49.598007] I [MSGID: 109086] 
[dht-shared.c:297:dht_parse_decommissioned_bricks] 
0-ectest_vol-tier-dht: decommissioning subvolume ectest_vol-hot-dht
[2017-07-30 19:42:49.598189] I [rpc-drc.c:689:rpcsvc_drc_init] 
0-rpc-service: DRC is turned OFF
[2017-07-30 19:42:49.598601] I [MSGID: 101191] 
[event-epoll.c:659:event_dispatch_epoll_worker] 0-epoll: Exited thread 
with index 20
[2017-07-30 19:42:49.610119] I [glusterfsd-mgmt.c:1620:mgmt_getspec_cbk] 
0-glusterfs: No change in volfile, continuing
[2017-07-30 19:42:49.610143] I [MSGID: 101191] 
[event-epoll.c:659:event_dispatch_epoll_worker] 0-epoll: Exited thread 
with index 32
[2017-07-30 19:42:49.649243] I [glusterfsd-mgmt.c:1620:mgmt_getspec_cbk] 
0-glusterfs: No change in volfile, continuing
[2017-07-30 19:42:49.649274] I [MSGID: 101191] 
[event-epoll.c:659:event_dispatch_epoll_worker] 0-epoll: Exited thread 
with index 18
[2017-07-30 19:42:49.690013] I [glusterfsd-mgmt.c:1620:mgmt_getspec_cbk] 
0-glusterfs: No change in volfile, continuing
[2017-07-30 19:42:56.539540] I [MSGID: 101191] 
[event-epoll.c:659:event_dispatch_epoll_worker] 0-epoll: Exited thread 
with index 31
[2017-07-30 19:42:56.539598] I [MSGID: 101191] 
[event-epoll.c:659:event_dispatch_epoll_worker] 0-epoll: Exited thread 
with index 16
[2017-07-30 19:42:56.539616] I [MSGID: 101191] 
[event-epoll.c:659:event_dispatch_epoll_worker] 0-epoll: Exited thread 
with index 19
[2017-07-30 19:42:56.539669] I [MSGID: 101191] 
[event-epoll.c:659:event_dispatch_epoll_worker] 0-epoll: Exited thread 
with index 24
[2017-07-30 19:42:56.539707] I [MSGID: 101191] 
[event-epoll.c:659:event_dispatch_epoll_worker] 0-epoll: Exited thread 
with index 17
[2017-07-30 19:42:56.539770] I [MSGID: 101191] 
[event-epoll.c:659:event_dispatch_epoll_worker] 0-epoll: Exited thread 
with index 21
[2017-07-30 19:43:03.358143] I [MSGID: 101191] 
[event-epoll.c:659:event_dispatch_epoll_worker] 0-epoll: Exited thread 
with index 29
[2017-07-30 19:43:03.358387] I [MSGID: 101191] 
[event-epoll.c:659:event_dispatch_epoll_worker] 0-epoll: Exited thread 
with index 27
[2017-07-30 19:43:03.359097] I [MSGID: 101191] 
[event-epoll.c:659:event_dispatch_epoll_worker] 0-epoll: Exited thread 
with index 13
[2017-07-30 19:43:03.359166] I [MSGID: 101191] 
[event-epoll.c:659:event_dispatch_epoll_worker] 0-epoll: Exited thread 
with index 30
[2017-07-30 19:43:03.359295] I [MSGID: 101191] 
[event-epoll.c:659:event_dispatch_epoll_worker] 0-epoll: Exited thread 
with index 10
[2017-07-30 19:43:03.359370] I [MSGID: 101191] 
[event-epoll.c:659:event_dispatch_epoll_worker] 0-epoll: Exited thread 
with index 22
[2017-07-30 19:43:03.359382] I [MSGID: 101191] 
[event-epoll.c:659:event_dispatch_epoll_worker] 0-epoll: Exited thread 
with index 25
[2017-07-30 19:43:03.359424] I [MSGID: 101191] 
[event-epoll.c:659:event_dispatch_epoll_worker] 0-epoll: Exited thread 
with index 28
[2017-07-30 19:43:03.359373] I [MSGID: 101191] 
[event-epoll.c:659:event_dispatch_epoll_worker] 0-epoll: Exited thread 
with index 11
[2017-07-30 19:43:03.359454] I [MSGID: 101191] 
[event-epoll.c:659:event_dispatch_epoll_worker] 0-epoll: Exited thread 
with index 9
[2017-07-30 19:43:03.359424] I [MSGID: 101191] 
[event-epoll.c:659:event_dispatch_epoll_worker] 0-epoll: Exited thread 
with index 26
[2017-07-30 19:43:03.359391] I [MSGID: 101191] 
[event-epoll.c:659:event_dispatch_epoll_worker] 0-epoll: Exited thread 
with index 14
[2017-07-30 19:43:03.359486] I [MSGID: 101191] 
[event-epoll.c:659:event_dispatch_epoll_worker] 0-epoll: Exited thread 
with index 23
[2017-07-30 19:43:03.359647] I [MSGID: 101191] 
[event-epoll.c:659:event_dispatch_epoll_worker] 0-epoll: Exited thread 
with index 7
[2017-07-30 19:43:03.360095] I [MSGID: 101191] 
[event-epoll.c:659:event_dispatch_epoll_worker] 0-epoll: Exited thread 
with index 8
[2017-07-30 19:43:03.360364] I [MSGID: 101191] 
[event-epoll.c:659:event_dispatch_epoll_worker] 0-epoll: Exited thread 
with index 15
[2017-07-30 19:43:03.360395] I [MSGID: 101191] 
[event-epoll.c:659:event_dispatch_epoll_worker] 0-epoll: Exited thread 
with index 12


Any idea? Thanks in advance.

-- 
Thanks
     -Xie