<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi Raghavendra,<div class=""><br class=""></div><div class="">Yes, server.event-thread has been changed from 4 to 8.</div><div class="">During last days, I noticed that the error events are still here although they have been considerably reduced.</div><div class=""><br class=""></div><div class="">So, I used grep command against the log files in order to provide you a global vision about the warning, error and critical events appeared today at 06:xx (may be useful I hope).</div><div class="">I collected the info from s06 gluster server, but the behaviour is the the almost the same on the other gluster servers.</div><div class=""><br class=""></div><div class=""><div class=""><b class="">ERRORS: </b></div><div class=""><b class="">CWD: /var/log/glusterfs </b></div><div class=""><b class="">COMMAND: grep " E " *.log |grep "2019-03-13 06:"</b></div><div class=""><br class=""></div><div class="">(I can see a lot of this kind of message in the same period but I'm notifying you only one record for each type of error)</div><div class=""><br class=""></div><div class="">glusterd.log:[2019-03-13 06:12:35.982863] E [MSGID: 101042] [compat.c:569:gf_umount_lazy] 0-management: Lazy unmount of /var/run/gluster/tier2_quota_list/</div><div class=""><br class=""></div><div class="">glustershd.log:[2019-03-13 06:14:28.666562] E [rpc-clnt.c:350:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f4a71ddcebb] (--> /lib64/libgfr</div><div class="">pc.so.0(saved_frames_unwind+0x1de)[0x7f4a71ba1d9e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f4a71ba1ebe] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup</div><div class="">+0x90)[0x7f4a71ba3640] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x2a0)[0x7f4a71ba4130] ))))) 0-tier2-client-55: forced unwinding frame type(GlusterFS 3.3) op(INODELK(29)) </div><div class="">called at 2019-03-13 06:14:14.858441 (xid=0x17fddb50) </div><div class=""><br class=""></div><div class="">glustershd.log:[2019-03-13 06:17:48.883825] E [socket.c:2376:socket_connect_finish] 0-tier2-client-55: connection to 192.168.0.55:49158 failed (Connection timed out); disco</div><div class="">nnecting socket</div><div class="">glustershd.log:[2019-03-13 06:19:58.931798] E [socket.c:2376:socket_connect_finish] 0-tier2-client-55: connection to 192.168.0.55:49158 failed (Connection timed out); disco</div><div class="">nnecting socket</div><div class="">glustershd.log:[2019-03-13 06:22:08.979829] E [socket.c:2376:socket_connect_finish] 0-tier2-client-55: connection to 192.168.0.55:49158 failed (Connection timed out); disco</div><div class="">nnecting socket</div><div class="">glustershd.log:[2019-03-13 06:22:36.226847] E [MSGID: 114031] [client-rpc-fops.c:1508:client3_3_inodelk_cbk] 0-tier2-client-55: remote operation failed [Transport endpoint </div><div class="">is not connected]</div><div class="">glustershd.log:[2019-03-13 06:22:36.306669] E [MSGID: 114031] [client-rpc-fops.c:1508:client3_3_inodelk_cbk] 0-tier2-client-55: remote operation failed [Transport endpoint </div><div class="">is not connected]</div><div class="">glustershd.log:[2019-03-13 06:22:36.385257] E [MSGID: 114031] [client-rpc-fops.c:1508:client3_3_inodelk_cbk] 0-tier2-client-55: remote operation failed [Transport endpoint </div><div class="">is not connected]</div><div class=""><br class=""></div><div class=""><b class="">WARNINGS:</b></div><div class=""><b class="">CWD: /var/log/glusterfs </b></div><div class=""><b class="">COMMAND: grep " W " *.log |grep "2019-03-13 06:"</b></div><div class=""><br class=""></div><div class="">(I can see a lot of this kind of message in the same period but I'm notifying you only one record for each type of warnings)</div><div class=""><br class=""></div><div class="">glustershd.log:[2019-03-13 06:14:28.666772] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-tier2-client-55: remote operation failed. Path: <gfid:b6b35d</div><div class="">0f-f34d-4c25-bbe8-74bde0248d7e> (b6b35d0f-f34d-4c25-bbe8-74bde0248d7e). Key: (null) [Transport endpoint is not connected]</div><div class=""><br class=""></div><div class="">glustershd.log:[2019-03-13 06:14:31.421576] W [MSGID: 122035] [ec-common.c:571:ec_child_select] 0-tier2-disperse-9: Executing operation with some subvolumes unavailable (2)</div><div class=""><br class=""></div><div class="">glustershd.log:[2019-03-13 06:15:31.547417] W [MSGID: 122032] [ec-heald.c:266:ec_shd_index_sweep] 0-tier2-disperse-9: unable to get index-dir on tier2-client-55 [Operation </div><div class="">now in progress]</div><div class=""><br class=""></div><div class="">quota-mount-tier2.log:[2019-03-13 06:12:36.116277] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family' is deprecated, preferred is 'trans</div><div class="">port.address-family', continuing with correction</div><div class="">quota-mount-tier2.log:[2019-03-13 06:12:36.198430] W [MSGID: 101174] [graph.c:363:_log_if_unknown_option] 0-tier2-readdir-ahead: option 'parallel-readdir' is not recognized</div><div class="">quota-mount-tier2.log:[2019-03-13 06:12:37.945007] W [glusterfsd.c:1375:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7f340892be25] -->/usr/sbin/glusterfs(gluste</div><div class="">rfs_sigwaiter+0xe5) [0x55ef010164b5] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55ef0101632b] ) 0-: received signum (15), shutting down</div><div class=""><br class=""></div><div class=""><b class="">CRITICALS:</b></div><div class=""><b class="">CWD: /var/log/glusterfs </b></div><div class=""><b class="">COMMAND: grep " C " *.log |grep "2019-03-13 06:"</b></div><div class=""><br class=""></div><div class="">no critical errors at 06:xx</div><div class="">only one critical error during the day</div><div class=""><br class=""></div><div class=""><b class="">[root@s06 glusterfs]# grep " C " *.log |grep "2019-03-13"</b></div><div class="">glustershd.log:[2019-03-13 02:21:29.126279] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-55: server 192.168.0.55:49158 has not responded in the last 42 seconds, disconnecting.</div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Thank you very much for your help.</div><div class="">Regards,</div><div class="">Mauro<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 12 Mar 2019, at 05:17, Raghavendra Gowdappa <<a href="mailto:rgowdapp@redhat.com" class="">rgowdapp@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Was the suggestion to increase server.event-thread values tried? If yes, what were the results?<br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 11, 2019 at 2:40 PM Mauro Tridici <<a href="mailto:mauro.tridici@cmcc.it" class="">mauro.tridici@cmcc.it</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class="">Dear All,<div class=""><br class=""></div><div class="">do you have any suggestions about the right way to "debug" this issue?</div><div class="">In attachment, the updated logs of “s06" gluster server.</div><div class=""><br class=""></div><div class="">I noticed a lot of intermittent warning and error messages.</div><div class=""><br class=""></div><div class="">Thank you in advance,</div><div class="">Mauro</div><div class=""><br class=""></div><div class=""></div></div><div style="overflow-wrap: break-word;" class=""><div class=""><br class=""><div class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On 4 Mar 2019, at 18:45, Raghavendra Gowdappa <<a href="mailto:rgowdapp@redhat.com" target="_blank" class="">rgowdapp@redhat.com</a>> wrote:</div><br class="gmail-m_1076545541529004495Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class=""><br class=""><a class="gmail_plusreply" id="gmail-m_1076545541529004495plusReplyChip-1" href="mailto:gluster-devel@gluster.org" target="_blank">+Gluster Devel</a>, <a class="gmail_plusreply" id="gmail-m_1076545541529004495plusReplyChip-2" href="mailto:gluster-users@gluster.org" target="_blank">+Gluster-users</a> <br class=""></div><div class=""><br class=""></div><div class="">I would like to point out another issue. Even if what I suggested prevents disconnects, part of the solution would be only symptomatic treatment and doesn't address the root cause of the problem. In most of the ping-timer-expiry issues, the root cause is the increased load on bricks and the inability of bricks to be responsive under high load. So, the actual solution would be doing any or both of the following:</div><div class="">* identify the source of increased load and if possible throttle it. Internal heal processes like self-heal, rebalance, quota heal are known to pump traffic into bricks without much throttling (io-threads _might_ do some throttling, but my understanding is its not sufficient).</div><div class="">* identify the reason for bricks to become unresponsive during load. This may be fixable issues like not enough event-threads to read from network or difficult to fix issues like fsync on backend fs freezing the process or semi fixable issues (in code) like lock contention.</div><div class=""><br class=""></div><div class="">So any genuine effort to fix ping-timer-issues (to be honest most of the times they are not issues related to rpc/network) would involve performance characterization of various subsystems on bricks and clients. Various subsystems can include (but not necessarily limited to), underlying OS/filesystem, glusterfs processes, CPU consumption etc</div><div class=""><br class=""></div><div class="">regards,</div><div class="">Raghavendra<br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 4, 2019 at 9:31 PM Mauro Tridici <<a href="mailto:mauro.tridici@cmcc.it" target="_blank" class="">mauro.tridici@cmcc.it</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="">Thank you, let’s try!<div class="">I will inform you about the effects of the change.</div><div class=""><br class=""></div><div class="">Regards,</div><div class="">Mauro<br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On 4 Mar 2019, at 16:55, Raghavendra Gowdappa <<a href="mailto:rgowdapp@redhat.com" target="_blank" class="">rgowdapp@redhat.com</a>> wrote:</div><br class="gmail-m_1076545541529004495gmail-m_4656060697371208760Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 4, 2019 at 8:54 PM Mauro Tridici <<a href="mailto:mauro.tridici@cmcc.it" target="_blank" class="">mauro.tridici@cmcc.it</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="">Hi Raghavendra,<div class=""><br class=""></div><div class="">thank you for your reply.</div><div class="">Yes, you are right. It is a problem that seems to happen randomly.</div><div class="">At this moment, server.event-threads value is 4. I will try to increase this value to 8. Do you think that it could be a valid value ?</div></div></blockquote><div class=""><br class=""></div><div class="">Yes. We can try with that. You should see at least frequency of ping-timer related disconnects reduce with this value (even if it doesn't eliminate the problem completely).</div><div class=""><br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""><br class=""></div><div class="">Regards,</div><div class="">Mauro</div><div class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On 4 Mar 2019, at 15:36, Raghavendra Gowdappa <<a href="mailto:rgowdapp@redhat.com" target="_blank" class="">rgowdapp@redhat.com</a>> wrote:</div><br class="gmail-m_1076545541529004495gmail-m_4656060697371208760gmail-m_-2079346638131475963Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 4, 2019 at 8:01 PM Nithya Balachandran <<a href="mailto:nbalacha@redhat.com" target="_blank" class="">nbalacha@redhat.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">Hi Mauro,<div class=""><br class=""></div><div class="">It looks like some problem on s06. Are all your other nodes ok? Can you send us the gluster logs from this node?</div><div class=""><br class=""></div><div class=""><a class="gmail_plusreply" id="gmail-m_1076545541529004495gmail-m_4656060697371208760gmail-m_-2079346638131475963gmail-m_-205963745423898280plusReplyChip-0" href="mailto:raghavendra@gluster.com" target="_blank">@Raghavendra G</a> , do you have any idea as to how this can be debugged? Maybe running top ? Or debug brick logs?<br class=""></div></div></blockquote><div class=""><br class=""></div><div class="">If we can reproduce the problem, collecting tcpdump on both ends of connection will help. But, one common problem is these bugs are inconsistently reproducible and hence we may not be able to capture tcpdump at correct intervals. Other than that, we can try to collect some evidence that poller threads were busy (waiting on locks). But, not sure what debug data provides that information.</div><div class=""><br class=""></div><div class="">From what I know, its difficult to collect evidence for this issue and we could only reason about it.</div><div class=""><br class=""></div><div class="">We can try a workaround though - try increasing server.event-threads and see whether ping-timer expiry issues go away with an optimal value. If that's the case, it kind of provides proof for our hypothesis.</div><div class=""><br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Regards,</div><div class="">Nithya</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 4 Mar 2019 at 15:25, Mauro Tridici <<a href="mailto:mauro.tridici@cmcc.it" target="_blank" class="">mauro.tridici@cmcc.it</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="">Hi All,<div class=""><br class=""></div><div class="">some minutes ago I received this message from NAGIOS server </div><div class=""><br class=""></div><div class=""><font size="1" class=""><i class="">***** Nagios *****<br class=""><br class="">Notification Type: PROBLEM<br class=""><br class="">Service: Brick - /gluster/mnt2/brick<br class="">Host: s06<br class="">Address: s06-stg<br class="">State: CRITICAL<br class=""><br class="">Date/Time: Mon Mar 4 10:25:33 CET 2019<br class=""><br class="">Additional Info:<br class="">CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.</i></font></div><div class=""><br class=""></div><div class="">I checked the network, RAM and CPUs usage on s06 node and everything seems to be ok.</div><div class="">No bricks are in error state. In /var/log/messages, I detected again a crash of “check_vol_utili” that I think it is a module used by NRPE executable (that is the NAGIOS client).</div><div class=""><br class=""></div><div class=""><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:15:29 s06 kernel: traps: check_vol_utili[161224] general protection ip:7facffa0a66d sp:7ffe9f4e6fc0 error:0 in libglusterfs.so.0.0.1[7facff9b7000+f7000]</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:15:29 s06 abrt-hook-ccpp: Process 161224 (python2.7) of user 0 killed by SIGSEGV - dumping core</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:15:29 s06 abrt-server: Generating core_backtrace</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:15:29 s06 abrt-server: Error: Unable to open './coredump': No such file or directory</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:16:01 s06 systemd: Created slice User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:16:01 s06 systemd: Starting User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:16:01 s06 systemd: Started Session 201010 of user root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:16:01 s06 systemd: Starting Session 201010 of user root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:16:01 s06 systemd: Removed slice User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:16:01 s06 systemd: Stopping User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:16:24 s06 abrt-server: Duplicate: UUID</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:16:24 s06 abrt-server: DUP_OF_DIR: /var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:16:24 s06 abrt-server: Deleting problem directory ccpp-2019-03-04-10:15:29-161224 (dup of ccpp-2018-09-25-12:27:42-13041)</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:16:24 s06 abrt-server: Generating core_backtrace</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:16:24 s06 abrt-server: Error: Unable to open './coredump': No such file or directory</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:16:24 s06 abrt-server: Cannot notify '/var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041' via uReport: Event 'report_uReport' exited with 1</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:16:24 s06 abrt-hook-ccpp: Process 161391 (python2.7) of user 0 killed by SIGABRT - dumping core</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:16:25 s06 abrt-server: Generating core_backtrace</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:16:25 s06 abrt-server: Error: Unable to open './coredump': No such file or directory</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:17:01 s06 systemd: Created slice User Slice of root.</span></div></div><div class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><br class=""></span></div><div class="">Also, I noticed the following errors that I think are very critical:</div><div class=""><br class=""></div><div class=""><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:21:12 s06 glustershd[20355]: [2019-03-04 09:21:12.954798] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-55: server <a href="http://192.168.0.55:49158/" target="_blank" class="">192.168.0.55:49158</a> has not responded in the last 42 seconds, disconnecting.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:22:01 s06 systemd: Created slice User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:22:01 s06 systemd: Starting User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:22:01 s06 systemd: Started Session 201017 of user root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:22:01 s06 systemd: Starting Session 201017 of user root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:22:01 s06 systemd: Removed slice User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:22:01 s06 systemd: Stopping User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:22:03 s06 glustershd[20355]: [2019-03-04 09:22:03.964120] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-60: server <a href="http://192.168.0.54:49165/" target="_blank" class="">192.168.0.54:49165</a> has not responded in the last 42 seconds, disconnecting.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:23:01 s06 systemd: Created slice User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:23:01 s06 systemd: Starting User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:23:01 s06 systemd: Started Session 201018 of user root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:23:01 s06 systemd: Starting Session 201018 of user root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:23:02 s06 systemd: Removed slice User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:23:02 s06 systemd: Stopping User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:24:01 s06 systemd: Created slice User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:24:01 s06 systemd: Starting User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:24:01 s06 systemd: Started Session 201019 of user root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:24:01 s06 systemd: Starting Session 201019 of user root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:24:01 s06 systemd: Removed slice User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:24:01 s06 systemd: Stopping User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:24:03 s06 glustershd[20355]: [2019-03-04 09:24:03.982502] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-16: server <a href="http://192.168.0.52:49158/" target="_blank" class="">192.168.0.52:49158</a> has not responded in the last 42 seconds, disconnecting.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746109] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-3: server <a href="http://192.168.0.51:49153/" target="_blank" class="">192.168.0.51:49153</a> has not responded in the last 42 seconds, disconnecting.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746215] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-10: server <a href="http://192.168.0.52:49156/" target="_blank" class="">192.168.0.52:49156</a> has not responded in the last 42 seconds, disconnecting.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746260] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-21: server <a href="http://192.168.0.51:49159/" target="_blank" class="">192.168.0.51:49159</a> has not responded in the last 42 seconds, disconnecting.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746296] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-25: server <a href="http://192.168.0.52:49161/" target="_blank" class="">192.168.0.52:49161</a> has not responded in the last 42 seconds, disconnecting.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:24:05 s06 quotad[20374]: [2019-03-04 09:24:05.746413] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-60: server <a href="http://192.168.0.54:49165/" target="_blank" class="">192.168.0.54:49165</a> has not responded in the last 42 seconds, disconnecting.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:24:07 s06 glustershd[20355]: [2019-03-04 09:24:07.982952] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-45: server <a href="http://192.168.0.54:49155/" target="_blank" class="">192.168.0.54:49155</a> has not responded in the last 42 seconds, disconnecting.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:24:18 s06 glustershd[20355]: [2019-03-04 09:24:18.990929] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-25: server <a href="http://192.168.0.52:49161/" target="_blank" class="">192.168.0.52:49161</a> has not responded in the last 42 seconds, disconnecting.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:24:31 s06 glustershd[20355]: [2019-03-04 09:24:31.995781] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-20: server <a href="http://192.168.0.53:49159/" target="_blank" class="">192.168.0.53:49159</a> has not responded in the last 42 seconds, disconnecting.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:25:01 s06 systemd: Created slice User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:25:01 s06 systemd: Starting User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:25:01 s06 systemd: Started Session 201020 of user root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:25:01 s06 systemd: Starting Session 201020 of user root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:25:01 s06 systemd: Removed slice User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:25:01 s06 systemd: Stopping User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:25:57 s06 systemd: Created slice User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:25:57 s06 systemd: Starting User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:25:57 s06 systemd-logind: New session 201021 of user root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:25:57 s06 systemd: Started Session 201021 of user root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:25:57 s06 systemd: Starting Session 201021 of user root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:26:01 s06 systemd: Started Session 201022 of user root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:26:01 s06 systemd: Starting Session 201022 of user root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:26:21 s06 nrpe[162388]: Error: Could not complete SSL handshake with <a href="http://192.168.1.56/" target="_blank" class="">192.168.1.56</a>: 5</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:27:01 s06 systemd: Started Session 201023 of user root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:27:01 s06 systemd: Starting Session 201023 of user root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:28:01 s06 systemd: Started Session 201024 of user root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:28:01 s06 systemd: Starting Session 201024 of user root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:29:01 s06 systemd: Started Session 201025 of user root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:29:01 s06 systemd: Starting Session 201025 of user root.</span></div><div class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><br class=""></span></div><div class=""><span style="font-variant-ligatures:no-common-ligatures" class="">But, unfortunately, I don’t understand why it is happening.</span></div><div class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Now, NAGIOS server shows that s06 status is ok:</span></div><div class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><br class=""></span></div><div class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="1" class=""><i class="">***** Nagios *****<br class=""><br class="">Notification Type: RECOVERY<br class=""><br class="">Service: Brick - /gluster/mnt2/brick<br class="">Host: s06<br class="">Address: s06-stg<br class="">State: OK<br class=""><br class="">Date/Time: Mon Mar 4 10:35:23 CET 2019<br class=""><br class="">Additional Info:<br class="">OK: Brick /gluster/mnt2/brick is up</i></font></span></div><div class=""><br class=""></div><div class="">Nothing is changed from RAM, CPUs, and NETWORK point of view.</div><div class="">/var/log/message file has been updated:</div><div class=""><br class=""></div><div class=""><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:32:01 s06 systemd: Starting Session 201029 of user root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:32:30 s06 glustershd[20355]: [2019-03-04 09:32:30.069082] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-10: server <a href="http://192.168.0.52:49156/" target="_blank" class="">192.168.0.52:49156</a> has not responded in the last 42 seconds, disconnecting.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:32:55 s06 glustershd[20355]: [2019-03-04 09:32:55.074689] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-66: server <a href="http://192.168.0.54:49167/" target="_blank" class="">192.168.0.54:49167</a> has not responded in the last 42 seconds, disconnecting.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:33:01 s06 systemd: Started Session 201030 of user root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:33:01 s06 systemd: Starting Session 201030 of user root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:34:01 s06 systemd: Started Session 201031 of user root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:34:01 s06 systemd: Starting Session 201031 of user root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:35:01 s06 nrpe[162562]: Could not read request from client 192.168.1.56, bailing out...</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:35:01 s06 nrpe[162562]: INFO: SSL Socket Shutdown.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:35:01 s06 systemd: Started Session 201032 of user root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 4 10:35:01 s06 systemd: Starting Session 201032 of user root.</span></div></div><div class=""><br class=""></div><div class="">Could you please help me to understand what it’s happening ?</div><div class="">Thank you in advance.</div><div class=""><br class=""></div><div class="">Rergards,</div><div class="">Mauro</div><div class=""><br class=""></div><div class=""><br class=""><blockquote type="cite" class=""><div class="">On 1 Mar 2019, at 12:17, Mauro Tridici <<a href="mailto:mauro.tridici@cmcc.it" target="_blank" class="">mauro.tridici@cmcc.it</a>> wrote:</div><br class="gmail-m_1076545541529004495gmail-m_4656060697371208760gmail-m_-2079346638131475963gmail-m_-205963745423898280gmail-m_7828862686973368236Apple-interchange-newline"><div class=""><div class=""><div class=""><br class=""></div>Thank you, Milind.<div class="">I executed the instructions you suggested:</div><div class=""><br class=""></div><div class="">- grep “blocked for” /var/log/messages on s06 returns no output (no “blocked” word is detected in messages file);</div><div class="">- in /var/log/messages file I can see this kind of error repeated for a lot of times:</div><div class=""><br class=""></div><div class=""><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">Mar 1 08:43:01 s06 systemd: Starting Session 196071 of user root.</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">Mar 1 08:43:01 s06 systemd: Removed slice User Slice of root.</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">Mar 1 08:43:01 s06 systemd: Stopping User Slice of root.</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">Mar 1 08:43:02 s06 kernel: traps: check_vol_utili[57091] general protection ip:7f88e76ee66d sp:7ffe5a5bcc30 error:0 in libglusterfs.so.0.0.1[7f88e769b000+f7000]</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">Mar 1 08:43:02 s06 abrt-hook-ccpp: Process 57091 (python2.7) of user 0 killed by SIGSEGV - dumping core</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">Mar 1 08:43:02 s06 abrt-server: Generating core_backtrace</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">Mar 1 08:43:02 s06 abrt-server: Error: Unable to open './coredump': No such file or directory</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">Mar 1 08:43:58 s06 abrt-server: Duplicate: UUID</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">Mar 1 08:43:58 s06 abrt-server: DUP_OF_DIR: /var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">Mar 1 08:43:58 s06 abrt-server: Deleting problem directory ccpp-2019-03-01-08:43:02-57091 (dup of ccpp-2018-09-25-12:27:42-13041)</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">Mar 1 08:43:58 s06 dbus[1872]: [system] Activating service name='org.freedesktop.problems' (using servicehelper)</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">Mar 1 08:43:58 s06 dbus[1872]: [system] Successfully activated service 'org.freedesktop.problems'</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">Mar 1 08:43:58 s06 abrt-server: Generating core_backtrace</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">Mar 1 08:43:58 s06 abrt-server: Error: Unable to open './coredump': No such file or directory</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">Mar 1 08:43:58 s06 abrt-server: Cannot notify '/var/tmp/abrt/ccpp-2018-09-25-12:27:42-13041' via uReport: Event 'report_uReport' exited with 1</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">Mar 1 08:44:01 s06 systemd: Created slice User Slice of root.</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">Mar 1 08:44:01 s06 systemd: Starting User Slice of root.</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">Mar 1 08:44:01 s06 systemd: Started Session 196072 of user root.</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">Mar 1 08:44:01 s06 systemd: Starting Session 196072 of user root.</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">Mar 1 08:44:01 s06 systemd: Removed slice User Slice of root.</font></span></div><div class=""><br class=""></div><div class="">- in /var/log/messages file I can see also 4 errors related to other cluster servers:</div><div class=""><br class=""></div><div class=""><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="3" class="">M</font>ar 1 11:05:01 s06 systemd: Starting User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 1 11:05:01 s06 systemd: Started Session 196230 of user root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 1 11:05:01 s06 systemd: Starting Session 196230 of user root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 1 11:05:01 s06 systemd: Removed slice User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 1 11:05:01 s06 systemd: Stopping User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 1 11:05:59 s06 glustershd[70117]: [2019-03-01 10:05:59.347094] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-33: server <a href="http://192.168.0.51:49163/" target="_blank" class="">192.168.0.51:49163</a> has not responded in the last 42 seconds, disconnecting.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 1 11:06:01 s06 systemd: Created slice User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 1 11:06:01 s06 systemd: Starting User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 1 11:06:01 s06 systemd: Started Session 196231 of user root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 1 11:06:01 s06 systemd: Starting Session 196231 of user root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 1 11:06:01 s06 systemd: Removed slice User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 1 11:06:01 s06 systemd: Stopping User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 1 11:06:12 s06 glustershd[70117]: [2019-03-01 10:06:12.351319] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-1: server <a href="http://192.168.0.52:49153/" target="_blank" class="">192.168.0.52:49153</a> has not responded in the last 42 seconds, disconnecting.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 1 11:06:38 s06 glustershd[70117]: [2019-03-01 10:06:38.356920] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-7: server <a href="http://192.168.0.52:49155/" target="_blank" class="">192.168.0.52:49155</a> has not responded in the last 42 seconds, disconnecting.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 1 11:07:01 s06 systemd: Created slice User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 1 11:07:01 s06 systemd: Starting User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 1 11:07:01 s06 systemd: Started Session 196232 of user root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 1 11:07:01 s06 systemd: Starting Session 196232 of user root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 1 11:07:01 s06 systemd: Removed slice User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 1 11:07:01 s06 systemd: Stopping User Slice of root.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 1 11:07:36 s06 glustershd[70117]: [2019-03-01 10:07:36.366259] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-0: server <a href="http://192.168.0.51:49152/" target="_blank" class="">192.168.0.51:49152</a> has not responded in the last 42 seconds, disconnecting.</span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Mar 1 11:08:01 s06 systemd: Created slice User Slice of root.</span></div></div><div class=""><br class=""></div><div class="">No “blocked” word is in /var/log/messages files on other cluster servers.</div><div class="">In attachment, the /var/log/messages file from s06 server.</div><div class=""><br class=""></div><div class="">Thank you in advance,</div><div class="">Mauro</div><div class=""><br class=""></div><div class=""></div></div></div><span id="gmail-m_1076545541529004495gmail-m_4656060697371208760gmail-m_-2079346638131475963gmail-m_-205963745423898280gmail-m_7828862686973368236cid:750DDF09-DBA3-47B1-BDC8-046249D02143@openvpn" class=""><messages.zip></span><div class=""><div class=""><div class=""></div><div class=""><br class=""></div><div class=""><br class=""><blockquote type="cite" class=""><div class="">On 1 Mar 2019, at 11:47, Milind Changire <<a href="mailto:mchangir@redhat.com" target="_blank" class="">mchangir@redhat.com</a>> wrote:</div><br class="gmail-m_1076545541529004495gmail-m_4656060697371208760gmail-m_-2079346638131475963gmail-m_-205963745423898280gmail-m_7828862686973368236Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class="">The traces of very high disk activity on the servers are often found in /var/log/messages</div><div class="">You might want to grep for "blocked for" in /var/log/messages on s06 and correlate the timestamps to confirm the unresponsiveness as reported in gluster client logs.<br class=""></div><div class="">In cases of high disk activity, although the operating system continues to respond to ICMP pings, the processes writing to disks often get blocked to a large flush to the disk which could span beyond 42 seconds and hence result in ping-timer-expiry logs.</div><div class=""><br class=""></div><div class="">As a side note:</div><div class="">If you indeed find gluster processes being blocked in /var/log/messages, you might want to tweak sysctl tunables called vm.dirty_background_ratio or vm.dirty_background_bytes to a smaller value than the existing. Please read up more on those tunables before touching the settings.</div><div class=""><br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Mar 1, 2019 at 4:06 PM Mauro Tridici <<a href="mailto:mauro.tridici@cmcc.it" target="_blank" class="">mauro.tridici@cmcc.it</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""><br class=""></div><div class="">Hi all,</div><div class=""><br class=""></div><div class="">in attachment the client log captured after changing network.ping-timeout option.</div><div class="">I noticed this error involving server 192.168.0.56 (s06)</div><div class=""><br class=""></div><div class="">[2019-03-01 09:23:36.077287] I [rpc-clnt.c:1962:rpc_clnt_reconfig] 0-tier2-client-71: changing ping timeout to 42 (from 0)</div><div class=""><div class="">[2019-03-01 09:23:36.078213] I [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing</div><div class="">[2019-03-01 09:23:36.078432] I [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing</div><div class="">[2019-03-01 09:23:36.092357] I [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing</div><div class="">[2019-03-01 09:23:36.094146] I [glusterfsd-mgmt.c:1894:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing</div><div class="">[2019-03-01 10:06:24.708082] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-tier2-client-50: server <a href="http://192.168.0.56:49156/" target="_blank" class="">192.168.0.56:49156</a> has not responded in the last 42 seconds, disconnecting.</div><div class=""><br class=""></div></div><div class="">I don’t know why it happens, s06 server seems to be reachable.</div><div class=""><br class=""></div><div class=""><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">[athena_login2][/users/home/sysm02/]> telnet 192.168.0.56 49156</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">Trying 192.168.0.56...</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">Connected to 192.168.0.56.</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">Escape character is '^]'.</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">^CConnection closed by foreign host.</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">[athena_login2][/users/home/sysm02/]> ping 192.168.0.56</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">PING 192.168.0.56 (192.168.0.56) 56(84) bytes of data.</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">64 bytes from <a href="http://192.168.0.56/" target="_blank" class="">192.168.0.56</a>: icmp_seq=1 ttl=64 time=0.116 ms</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">64 bytes from <a href="http://192.168.0.56/" target="_blank" class="">192.168.0.56</a>: icmp_seq=2 ttl=64 time=0.101 ms</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><br class=""></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">--- 192.168.0.56 ping statistics ---</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">2 packets transmitted, 2 received, 0% packet loss, time 1528ms</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">rtt min/avg/max/mdev = 0.101/0.108/0.116/0.012 ms</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class=""><br class=""></font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">[athena_login2][/users/home/sysm02/]> telnet 192.168.0.56 49156</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">Trying 192.168.0.56...</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">Connected to 192.168.0.56.</font></span></div><div style="margin:0px;font-stretch:normal;line-height:normal;font-family:"Courier New";color:rgb(170,168,46);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><font size="2" class="">Escape character is '^]'.</font></span></div></div><br class=""><div class=""><div class="">Thank you for your help,</div><div class="">Mauro</div><div class=""><br class=""></div><div class=""></div></div></div><div class=""><div class=""><div class=""></div></div><div class=""><br class=""></div><div class=""><br class=""><blockquote type="cite" class=""><div class="">On 1 Mar 2019, at 10:29, Mauro Tridici <<a href="mailto:mauro.tridici@cmcc.it" target="_blank" class="">mauro.tridici@cmcc.it</a>> wrote:</div><br class="gmail-m_1076545541529004495gmail-m_4656060697371208760gmail-m_-2079346638131475963gmail-m_-205963745423898280gmail-m_7828862686973368236gmail-m_-8269830260017201978Apple-interchange-newline"><div class=""><div class="">Hi all,<div class=""><br class=""></div><div class="">thank you for the explanation.</div><div class="">I just changed network.ping-timeout option to default value (network.ping-timeout=42).</div><div class=""><br class=""></div><div class="">I will check the logs to see if the errors will appear again.</div><div class=""><br class=""></div><div class="">Regards,</div><div class="">Mauro<br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On 1 Mar 2019, at 04:43, Milind Changire <<a href="mailto:mchangir@redhat.com" target="_blank" class="">mchangir@redhat.com</a>> wrote:</div><br class="gmail-m_1076545541529004495gmail-m_4656060697371208760gmail-m_-2079346638131475963gmail-m_-205963745423898280gmail-m_7828862686973368236gmail-m_-8269830260017201978Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class="">network.ping-timeout should not be set to zero for non-glusterd clients.</div><div class="">glusterd is a special case for which ping-timeout is set to zero via /etc/glusterfs/glusterd.vol</div><div class=""><br class=""></div><div class="">Setting network.ping-timeout to zero disables arming of the ping timer for connections. This disables testing the connection for responsiveness and hence avoids proactive fail-over.<br class=""></div><div class=""><br class=""></div><div class="">Please reset network.ping-timeout to a non-zero positive value, eg. 42</div><div class=""><br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Feb 28, 2019 at 5:07 PM Nithya Balachandran <<a href="mailto:nbalacha@redhat.com" target="_blank" class="">nbalacha@redhat.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">Adding Raghavendra and Milind to comment on this.<div class=""><br class=""></div><div class="">What is the effect of setting network.ping-timeout to 0 and should it be set back to 42?</div><div class="">Regards,</div><div class="">Nithya</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, 28 Feb 2019 at 16:01, Mauro Tridici <<a href="mailto:mauro.tridici@cmcc.it" target="_blank" class="">mauro.tridici@cmcc.it</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="">Hi Nithya,<div class=""><br class=""></div><div class="">sorry for the late.</div><div class="">network.ping-timeout has been set to 0 in order to try to solve some timeout problems, but it didn’t help.</div><div class="">I can set it to the default value. </div><div class=""><br class=""></div><div class="">Can I proceed with the change?</div><div class=""><br class=""></div><div class="">Thank you,</div><div class="">Mauro</div><div class=""><br class=""></div><div class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On 28 Feb 2019, at 04:41, Nithya Balachandran <<a href="mailto:nbalacha@redhat.com" target="_blank" class="">nbalacha@redhat.com</a>> wrote:</div><br class="gmail-m_1076545541529004495gmail-m_4656060697371208760gmail-m_-2079346638131475963gmail-m_-205963745423898280gmail-m_7828862686973368236gmail-m_-8269830260017201978gmail-m_-5116677952661970072gmail-m_1695825453356196927gmail-m_6902333806737363462Apple-interchange-newline"><div class=""><div dir="ltr" class="">Hi Mauro,<div class=""><br class=""></div><div class="">Is network.ping-timeout still set to 0. The default value is 42. Is there a particular reason why this was changed?</div><div class=""><br class=""></div><div class="">Regards,</div><div class="">Nithya</div><div class=""><br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 27 Feb 2019 at 21:32, Mauro Tridici <<a href="mailto:mauro.tridici@cmcc.it" target="_blank" class="">mauro.tridici@cmcc.it</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""><br class=""></div><div class="">Hi Xavi,</div><div class=""><br class=""></div><div class="">thank you for the detailed explanation and suggestions.</div><div class="">Yes, transport.listen-backlog option is still set to 1024.</div><div class=""><br class=""></div><div class="">I will check the network and connectivity status using “ping” and “telnet” as soon as the errors will come back again.</div><div class=""><br class=""></div><div class="">Regards,</div><div class="">Mauro</div><br class=""><div class=""><blockquote type="cite" class=""><div class="">Il giorno 27 feb 2019, alle ore 16:42, Xavi Hernandez <<a href="mailto:jahernan@redhat.com" target="_blank" class="">jahernan@redhat.com</a>> ha scritto:</div><br class="gmail-m_1076545541529004495gmail-m_4656060697371208760gmail-m_-2079346638131475963gmail-m_-205963745423898280gmail-m_7828862686973368236gmail-m_-8269830260017201978gmail-m_-5116677952661970072gmail-m_1695825453356196927gmail-m_6902333806737363462gmail-m_-167234932130657844Apple-interchange-newline"><div class=""><div dir="ltr" class="">Hi Mauro,<div class=""><br class=""></div><div class="">those errors say that the mount point is not connected to some of the bricks while executing operations. I see references to 3rd and 6th bricks of several disperse sets, which seem to map to server s06. For some reason, gluster is having troubles connecting from the client machine to that particular server. At the end of the log I see that after long time a reconnect is done to both of them. However little after, other bricks from the s05 get disconnected and a reconnect times out.</div><div class=""><br class=""></div><div class="">That's really odd. It seems like if server/communication is cut to s06 for some time, then restored, and then the same happens to the next server.</div><div class=""><br class=""></div><div class="">If the servers are really online and it's only a communication issue, it explains why server memory and network has increased: if the problem only exists between the client and servers, any write made by the client will automatically mark the file as damaged, since some of the servers have not been updated. Since self-heal runs from the server nodes, they will probably be correctly connected to all bricks, which allows them to heal the just damaged file, which increases memory and network usage.</div><div class=""><br class=""></div>I guess you still have transport.listen-backlog set to 1024, right ?<div class=""><br class=""><div class="">Just to try to identify if the problem really comes from network, can you check if you lose some pings from the client to all of the servers while you are seeing those errors in the log file ?</div><div class=""><br class=""></div><div class="">You can also check if during those errors, you can telnet to the port of the brick from the client.</div><div class=""><br class=""></div><div class="">Xavi</div><div class=""><br class=""></div><div class=""><br class=""></div></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Feb 26, 2019 at 10:17 AM Mauro Tridici <<a href="mailto:mauro.tridici@cmcc.it" target="_blank" class="">mauro.tridici@cmcc.it</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class="">Hi Nithya,</div><div class=""><br class=""></div><div class="">“df -h” operation is not still slow, but no users are using the volume, RAM and NETWORK usage is ok on the client node.</div><div class=""><br class=""></div><div class="">I was worried about this kind of warnings/errors:</div><div class=""><br class=""></div><div class=""><div style="margin:0px;line-height:normal;font-family:"Courier New";color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[2019-02-25 10:59:00.664323] W [MSGID: 122035] [ec-common.c:571:ec_child_select] 0-tier2-disperse-6: Executing operation with some subvolumes unavailable (20)</span></div></div><div class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><br class=""></span></div><div class=""><div style="margin:0px;line-height:normal;font-family:"Courier New";color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[2019-02-26 03:11:35.212603] E [rpc-clnt.c:350:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e5)[0x3d0d410935] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d410b45] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] ))))) 0-tier2-client-50: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2019-02-26 03:10:56.549903 (xid=0x106f1c5)</span></div><div style="margin:0px;line-height:normal;font-family:"Courier New";color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""><br class=""></span></div><div style="margin:0px;line-height:normal;font-family:"Courier New";color:rgb(184,180,59);background-color:rgb(0,0,0)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[2019-02-26 03:13:03.313831] E [socket.c:2376:socket_connect_finish] 0-tier2-client-50: connection to <a href="http://192.168.0.56:49156/" target="_blank" class="">192.168.0.56:49156</a> failed (Timeout della connessione); disconnecting socket</span></div></div><div class=""><br class=""></div><div class="">It seems that some subvolumes are not available and 192.168.0.56 server (s06) is not reachable.</div><div class="">But gluster servers are up&running and bricks are ok.</div><div class=""><br class=""></div><div class="">In attachment the updated tier2.log file.</div><div class=""><br class=""></div><div class=""></div></div><div class=""><div class=""></div><div class=""><br class=""></div><div class="">Thank you.</div><div class="">Regards,</div><div class="">Mauro</div><br class=""><div class=""><blockquote type="cite" class=""><div class="">Il giorno 26 feb 2019, alle ore 04:03, Nithya Balachandran <<a href="mailto:nbalacha@redhat.com" target="_blank" class="">nbalacha@redhat.com</a>> ha scritto:</div><br class="gmail-m_1076545541529004495gmail-m_4656060697371208760gmail-m_-2079346638131475963gmail-m_-205963745423898280gmail-m_7828862686973368236gmail-m_-8269830260017201978gmail-m_-5116677952661970072gmail-m_1695825453356196927gmail-m_6902333806737363462gmail-m_-167234932130657844gmail-m_1905495939815020157Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div dir="ltr" class="">Hi,<div class=""><br class=""></div><div class="">I see a lot of EC messages in the log but they don't seem very serious. Xavi, can you take a look?</div><div class=""><br class=""></div><div class="">The only errors I see are:</div><div class=""><div class=""><font size="1" face="monospace, monospace" class="">[2019-02-25 10:58:45.519871] E [rpc-clnt.c:350:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e5)[0x3d0d410935] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d410b45] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] ))))) 0-tier2-client-50: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2019-02-25 10:57:47.429969 (xid=0xd26fe7)</font></div><div class=""><font size="1" face="monospace, monospace" class="">[2019-02-25 10:58:51.461493] E [rpc-clnt.c:350:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] (--> /usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1e5)[0x3d0d410935] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d410b45] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] ))))) 0-tier2-client-41: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2019-02-25 10:57:47.499174 (xid=0xf47d6a)</font></div><div class=""><font size="1" face="monospace, monospace" class="">[2019-02-25 11:07:57.152874] E [socket.c:2376:socket_connect_finish] 0-tier2-client-70: connection to <a href="http://192.168.0.55:49163/" target="_blank" class="">192.168.0.55:49163</a> failed (Timeout della connessione); disconnecting socket</font></div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Is the df -h operation still slow? If yes, can you take a tcpdump of the client while running df -h and send that across?</div><div class=""><br class=""></div><div class="">Regards,</div><div class="">Nithya</div></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 25 Feb 2019 at 17:27, Mauro Tridici <<a href="mailto:mauro.tridici@cmcc.it" target="_blank" class="">mauro.tridici@cmcc.it</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""><br class=""></div><div class="">Sorry, some minutes after my last mail message, I noticed that “df -h” command hanged for a while before returns the prompt.</div><div class="">Yesterday, everything was ok in the gluster client log, but, today, I see a lot of errors (please, take a look to the attached file).</div><div class=""><br class=""></div><div class="">On the client node, I detected an important RAM e NETWORK usage.</div><div class=""><br class=""></div><div class=""></div></div><div class=""><div class=""></div><div class=""><br class=""></div><div class=""></div></div><div class=""><div class=""></div><div class=""><br class=""></div><div class="">Do you think that the errors have been caused by the client resources usage?</div><div class=""><br class=""></div><div class="">Thank you in advance,</div><div class="">Mauro</div><div class=""><br class=""></div></div></blockquote></div></div></blockquote></div></div></blockquote></div></div></blockquote></div></div></blockquote></div></div></blockquote></div></div></div></blockquote></div></blockquote></div></div></blockquote></div></div></div></div></blockquote></div></div></blockquote></div></div></blockquote></div></div></div></div></blockquote></div><br class=""></div></div></blockquote></div>
</blockquote></div></div>
</div></blockquote></div></div></div></blockquote></div></div></div></blockquote></div><div class=""><div style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none" class=""><br class=""></div>
</div>
<br class=""></div></div></blockquote></div>
</div></blockquote></div><br class="">
<br class=""></div></div></div></blockquote></div>
</div></blockquote></div><br class="">
<br class=""></div></body></html>