<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><br class=""></div><div class="">Hi All,</div><div class=""><br class=""></div><div class=""><div class="">our users are experiencing the following two problems using our gluster storage based on a 12x(4+2) distributed dispersed volume:</div><div class=""><br class=""></div><div class="">1) sometimes, during RSYNC copies executions, they noticed this error message:</div><div class=""><br class=""></div><div class="">rsync: rename "/tier2/CSP/sp1/ftp/lweprsn/.cmcc_CMCC-CM2-v20160423_hindcast_S2005040100_atmos_day_surface_lweprsn_n1-n40.sha256.rD38kX" -> "cmcc_CMCC-CM2-v20160423_hindcast_S2005040100_atmos_day_surface_lweprsn_n1-n40.sha256": Permission denied (13)</div><div class="">rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]</div><div class=""><br class=""></div><div class="">The files seem to be correctly copied but rsync returns this kind of error and stop the sync process.</div></div><div class=""><br class=""></div><div class="">2) if the number of the total rsync sessions against the gluster volume grows, it seems that gluster volume starts suffering some particular workload, directories tree navigation becomes very slow and a lot of timeout errors appear on 192.168.0.54 gluster server (named “s04")</div><div class=""><br class=""></div><div class=""><div class="">I just checked switch, cables, ports and so on, but everything seems to be ok.</div><div class="">I also executed a lot of checks using iperf tool and the network seems to be ok.</div><div class="">I dont’ understand why I see only timeout errors related to s04 host (192.168.0.54). It is up and running…</div><div class=""><br class=""></div><div class="">It seems that rsync copy processes started by the gluster volume users cause some noise (on gluster client log file) during temporary files creation.</div><div class=""><br class=""></div><div class=""><font face="Courier" style="font-size: 12px;" class="">[2019-01-27 02:36:53.488942] E [socket.c:2376:socket_connect_finish] 0-tier2-client-57: connection to 192.168.0.54:49159 failed (Timeout della connessione); disconnecting socke</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">t</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">[2019-01-27 02:53:48.607832] E [rpc-clnt.c:350:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] (--> /usr/lib64/libgfrpc.so.0(saved</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">_frames_unwind+0x1e5)[0x3d0d410935] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">410b45] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] ))))) 0-tier2-client-57: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2019-01-2</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">7 02:53:10.568717 (xid=0x4aeefc8)</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">[2019-01-27 02:55:01.631912] E [socket.c:2376:socket_connect_finish] 0-tier2-client-54: connection to 192.168.0.54:49158 failed (Timeout della connessione); disconnecting socke</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">t</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">[2019-01-27 02:56:05.643880] E [socket.c:2376:socket_connect_finish] 0-tier2-client-54: connection to 192.168.0.54:49158 failed (Timeout della connessione); disconnecting socke</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">t</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">[2019-01-27 02:57:09.653961] E [socket.c:2376:socket_connect_finish] 0-tier2-client-54: connection to 192.168.0.54:49158 failed (Timeout della connessione); disconnecting socke</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">t</font></div><div class=""><font face="Courier" style="font-size: 12px;" class=""><br class=""></font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">[2019-01-27 03:50:56.951088] E [rpc-clnt.c:350:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] (--> /usr/lib64/libgfrpc.so.0(saved</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">_frames_unwind+0x1e5)[0x3d0d410935] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">410b45] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] ))))) 0-tier2-client-54: forced unwinding frame type(GlusterFS 3.3) op(INODELK(29)) called at 2019-01</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">-27 03:39:48.938568 (xid=0x49e54c1)</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">[2019-01-27 03:50:56.951111] E [MSGID: 114031] [client-rpc-fops.c:1508:client3_3_inodelk_cbk] 0-tier2-client-54: remote operation failed [Transport endpoint is not connected]</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">[2019-01-27 03:50:56.951471] E [rpc-clnt.c:350:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] (--> /usr/lib64/libgfrpc.so.0(saved</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">_frames_unwind+0x1e5)[0x3d0d410935] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">410b45] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] ))))) 0-tier2-client-54: forced unwinding frame type(GlusterFS 3.3) op(WRITE(13)) called at 2019-01-2</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">7 03:35:31.328387 (xid=0x49e54bf)</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">[2019-01-27 03:50:56.958357] E [rpc-clnt.c:350:saved_frames_unwind] (--> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x153)[0x3d0cc2f2e3] (--> /usr/lib64/libgfrpc.so.0(saved</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">_frames_unwind+0x1e5)[0x3d0d410935] (--> /usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x3d0d410a7e] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xa5)[0x3d0d</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">410b45] (--> /usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x278)[0x3d0d410e68] ))))) 0-tier2-client-54: forced unwinding frame type(GlusterFS 3.3) op(IPC(47)) called at 2019-01-27 </font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">03:37:49.652707 (xid=0x49e54c0)</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">[2019-01-27 03:50:56.966845] E [MSGID: 114031] [client-rpc-fops.c:435:client3_3_open_cbk] 0-tier2-client-57: remote operation failed. Path: /CSP/sp1/SPS3/sps_201205/.rh2m_6hour</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">ly_sps_201205_017.2012-06.nc.gz.DHNucn (5d0575fb-21c3-45b0-97bf-46d13f3c9262) [Il file esiste]</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">[2019-01-27 03:50:56.967086] E [MSGID: 122034] [ec-common.c:613:ec_child_select] 0-tier2-disperse-9: Insufficient available children for this request (have 0, need 4)</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">[2019-01-27 03:50:56.967224] E [MSGID: 122034] [ec-common.c:613:ec_child_select] 0-tier2-disperse-9: Insufficient available children for this request (have 0, need 4)</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">[2019-01-27 03:50:56.992129] E [MSGID: 114031] [client-rpc-fops.c:435:client3_3_open_cbk] 0-tier2-client-57: remote operation failed. Path: /CSP/sp1/SPS3/sps_201205/.rh2m_6hour</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">ly_sps_201205_017.2012-06.nc.gz.DHNucn (5d0575fb-21c3-45b0-97bf-46d13f3c9262) [Il file esiste]</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">[2019-01-27 03:50:56.995954] E [MSGID: 114031] [client-rpc-fops.c:435:client3_3_open_cbk] 0-tier2-client-57: remote operation failed. Path: /CSP/sp1/SPS3/sps_201205/.rh2m_6hour</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">ly_sps_201205_017.2012-06.nc.gz.DHNucn (5d0575fb-21c3-45b0-97bf-46d13f3c9262) [Il file esiste]</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">[2019-01-27 03:50:56.996126] E [MSGID: 122034] [ec-common.c:613:ec_child_select] 0-tier2-disperse-9: Insufficient available children for this request (have 0, need 4)</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">[2019-01-27 03:50:56.996751] E [MSGID: 122034] [ec-common.c:613:ec_child_select] 0-tier2-disperse-9: Insufficient available children for this request (have 0, need 4)</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">[2019-01-27 03:50:57.001301] E [MSGID: 114031] [client-rpc-fops.c:435:client3_3_open_cbk] 0-tier2-client-57: remote operation failed. Path: /CSP/sp1/SPS3/sps_201205/.rh2m_6hour</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">ly_sps_201205_017.2012-06.nc.gz.DHNucn (5d0575fb-21c3-45b0-97bf-46d13f3c9262) [Il file esiste]</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">[2019-01-27 03:50:57.006375] E [MSGID: 122034] [ec-common.c:613:ec_child_select] 0-tier2-disperse-9: Insufficient available children for this request (have 0, need 4)</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">[2019-01-27 04:02:09.269929] E [socket.c:2376:socket_connect_finish] 0-tier2-client-57: connection to 192.168.0.54:49159 failed (Timeout della connessione); disconnecting socke</font></div><div class=""><font face="Courier" style="font-size: 12px;" class="">t</font></div></div><div class=""><br class=""></div><div class="">I focused my attention on the errors occurred on gluster client during 25th of January 2019.</div><div class="">So, I collected the following log files:</div><div class=""><br class=""></div><div class="">- tier2.old.log.gz, (the client log file covering the period)</div><div class="">- tier2.today.log.gz (the updated client log file reporting similar error related to “renaming” and “timeout” issues)</div><div class="">- memory_usage.pdf, (weekly statistics for memory usage on 192.168.0.54 host from NAGIOS)</div><div class="">- network_usage.pdf, (weekly statistics for network usage on 192.168.0.54 host from NAGIOS)</div><div class="">- cpu_usage.pdf, (weekly statistics for network usage on 192.168.0.54 host from NAGIOS)</div><div class="">- gluster-mnt6-brick.log-20190127, brick 6 (on host 192.168.0.54) log file</div><div class="">- gluster-mnt7-brick.log-20190127, brick 7 (on host 192.168.0.54) log file</div><div class=""><br class=""></div><div class="">I detected the involved bricks greping on “gluster volume status” command output:</div><div class=""><br class=""></div><div class=""><div class="">[root@s04 bricks]# gluster volume status|grep 49158</div><div class="">Brick s02-stg:/gluster/mnt6/brick 49158 0 Y 4147 </div><div class="">Brick s03-stg:/gluster/mnt6/brick 49158 0 Y 4272 </div><div class="">Brick s01-stg:/gluster/mnt7/brick 49158 0 Y 3324 </div><div class="">Brick s04-stg:/gluster/mnt7/brick 49158 0 Y 3787 </div><div class="">Brick s05-stg:/gluster/mnt7/brick 49158 0 Y 3131 </div><div class="">Brick s06-stg:/gluster/mnt7/brick 49158 0 Y 3254 </div></div><div class=""><br class=""></div><div class="">On gluster server s04 (192.168.0.54), in messages log file, I can see a lot of errors like the following one and nothig more</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Jan 25 14:16:22 s04 kernel: traps: check_vol_utili[142140] general protection ip:7f8fcec7866d sp:7ffee3a86a30 error:0 in libglusterfs.so.0.0.1[7f8fcec25000+f7000]</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Jan 25 14:16:22 s04 abrt-hook-ccpp: Process 142140 (python2.7) of user 0 killed by SIGSEGV - dumping core</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Jan 25 14:16:22 s04 abrt-server: Generating core_backtrace</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Jan 25 14:16:22 s04 abrt-server: Error: Unable to open './coredump': No such file or directory</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Jan 25 14:17:00 s04 abrt-server: Duplicate: UUID</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Jan 25 14:17:00 s04 abrt-server: DUP_OF_DIR: /var/tmp/abrt/ccpp-2018-09-25-11:18:20-4471</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Jan 25 14:17:00 s04 abrt-server: Deleting problem directory ccpp-2019-01-25-14:16:22-142140 (dup of ccpp-2018-09-25-11:18:20-4471)</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Jan 25 14:17:00 s04 dbus[1877]: [system] Activating service name='org.freedesktop.problems' (using servicehelper)</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Jan 25 14:17:00 s04 dbus[1877]: [system] Successfully activated service 'org.freedesktop.problems'</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Jan 25 14:17:00 s04 abrt-server: Generating core_backtrace</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Jan 25 14:17:00 s04 abrt-server: Error: Unable to open './coredump': No such file or directory</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Jan 25 14:17:00 s04 abrt-server: Cannot notify '/var/tmp/abrt/ccpp-2018-09-25-11:18:20-4471' via uReport: Event 'report_uReport' exited with 1</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Jan 25 14:17:01 s04 systemd: Created slice User Slice of root.</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Jan 25 14:17:01 s04 systemd: Starting User Slice of root.</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Jan 25 14:17:01 s04 systemd: Started Session 140059 of user root.</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Jan 25 14:17:01 s04 systemd: Starting Session 140059 of user root.</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Jan 25 14:17:01 s04 systemd: Removed slice User Slice of root.</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Jan 25 14:17:01 s04 systemd: Stopping User Slice of root.</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Jan 25 14:18:01 s04 systemd: Created slice User Slice of root.</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Jan 25 14:18:01 s04 systemd: Starting User Slice of root.</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Jan 25 14:18:01 s04 systemd: Started Session 140060 of user root.</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Jan 25 14:18:01 s04 systemd: Starting Session 140060 of user root.</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Jan 25 14:18:01 s04 systemd: Removed slice User Slice of root.</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Jan 25 14:18:01 s04 systemd: Stopping User Slice of root.</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Jan 25 14:18:35 s04 kernel: traps: check_vol_utili[142501] general protection ip:7f119fd7066d sp:7ffc399aa6f0 error:0 in libglusterfs.so.0.0.1[7f119fd1d000+f7000]</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Jan 25 14:18:35 s04 abrt-hook-ccpp: Process 142501 (python2.7) of user 0 killed by SIGSEGV - dumping core</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Jan 25 14:18:35 s04 abrt-server: Generating core_backtrace</span></div><div style="margin: 0px; line-height: normal; font-family: 'Courier New'; color: rgb(184, 180, 59); background-color: rgb(0, 0, 0);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Jan 25 14:18:35 s04 abrt-server: Error: Unable to open './coredump': No such file or directory</span></div></div><div class=""><br class=""></div><div class="">No errors in /var/log/messages log file related to the client machine.</div><div class=""><br class=""></div><div class="">Anyone of us could help me to solve this issue?</div><div class="">Thank you very much in advance,</div><div class="">Mauro</div><div class=""><br class=""></div><div class=""></div></body></html>