Check with top & iotop the load.<div>Especially check the wait for I/O in top.</div><div><br></div><div>Did you check dmesg for any clues ?</div><div><br></div><div>Best Regards,</div><div>Strahil Nikolov<br> <br> <blockquote style="margin: 0 0 20px 0;"> <div style="font-family:Roboto, sans-serif; color:#6D00F6;"> <div>On Thu, Jun 16, 2022 at 22:59, Pat Haley</div><div><phaley@mit.edu> wrote:</div> </div> <div style="padding: 10px 0 0 20px; margin: 10px 0 0 0; border-left: 1px solid #6D00F6;"> <div id="yiv5506566618"><div>
<p><br clear="none">
</p>
<p>Hi Strahil,</p>
<p>I poked around our logs, and found this on the front-end (from
the day & time of the last time we had the issue)</p>
<p><br clear="none">
</p>
<p><font face="monospace">Jun 15 10:51:17 mseas gdata[155485]:
[2022-06-15 14:51:17.263858] C
[rpc-clnt-ping.c:165:rpc_clnt_ping_timer_expired]
0-data-volume-client-2: server 172.16.1.113:49153 has not
responded in the last 42 seconds, disconnecting.<br clear="none">
</font></p>
<p><br clear="none">
</p>
<p>This would indicate that the problem is related. For us,
however, I believe we can reproduce this issue at will (i.e.
simply try to gunzip the same file). Unfortunately I have to go to
a meeting now, but if you have some specific tests you'd like me
to try, I can try them when I get back.</p>
<p>Thanks</p>
<p>Pat</p>
<p><br clear="none">
</p>
<p><br clear="none">
</p>
<div class="yiv5506566618moz-cite-prefix">On 6/16/22 3:07 PM, Strahil Nikolov
wrote:<br clear="none">
</div>
<blockquote type="cite">
</blockquote></div><div>
Pat,
<div><br clear="none">
</div>
<div>
<div>Can you check the cpu and disk performance when the volume
reports the issue?</div>
<div><br clear="none">
</div>
</div>
<div><br clear="none">
</div>
<div>It seems that similar issue was reported
in <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://lists.gluster.org/pipermail/gluster-users/2019-March/035944.html" class="yiv5506566618moz-txt-link-freetext">https://lists.gluster.org/pipermail/gluster-users/2019-March/035944.html</a>
but I don't see a clear solution.</div>
<div>Take a look in the thread and check if it matches your
symptoms.</div>
<div><br clear="none">
</div>
<div><br clear="none">
</div>
<div>Best Regards,</div>
<div>Strahil Nikolov<br clear="none">
<br clear="none">
<blockquote style="margin:0 0 20px 0;">
<div style="font-family:Roboto, sans-serif;color:#6D00F6;">
<div>On Thu, Jun 16, 2022 at 18:14, Pat Haley</div>
<div><a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:phaley@mit.edu" target="_blank" href="mailto:phaley@mit.edu" class="yiv5506566618moz-txt-link-rfc2396E"><phaley@mit.edu></a> wrote:</div>
</div>
<div style="padding:10px 0 0 20px;margin:10px 0 0 0;border-left:1px solid #6D00F6;">
<div id="yiv5506566618">
<div>
<p><br clear="none">
</p>
<p>Hi Strahil,</p>
<p>I poked around again and for brick 3 (where the file
we were testing resides) I only found the same log
file as was at the bottom of my first Email:</p>
<p><br clear="none">
<font face="monospace">---------------------------------------------------<br clear="none">
mseas-data3: bricks/export-sda-brick3.log<br clear="none">
-----------------------------------------<br clear="none">
[2022-06-15 14:50:42.588143] I [MSGID: 115036]
[server.c:552:server_rpc_notify]
0-data-volume-server: disconnecting connection from
mseas.mit.edu-155483-2022/05/13-03:24:14:618694-data-volume-client-2-0-28<br clear="none">
[2022-06-15 14:50:42.588220] I [MSGID: 115013]
[server-helpers.c:294:do_fd_cleanup]
0-data-volume-server: fd cleanup on
/projects/posydon/Acoustics_ASA/MSEAS-ParEq-DO/Save/2D/Test_Cases/RI/DO_NAPE_JASA_Paper/Uncertain_Pekeris_Waveguide_DO_MC<br clear="none">
[2022-06-15 14:50:42.588259] I [MSGID: 115013]
[server-helpers.c:294:do_fd_cleanup]
0-data-volume-server: fd cleanup on
/projects/dri_calypso/PE/2019/Apr09/Ens3R200deg001/pe_out.nc.gz<br clear="none">
[2022-06-15 14:50:42.588288] I [MSGID: 101055]
[client_t.c:420:gf_client_unref]
0-data-volume-server: Shutting down connection
mseas.mit.edu-155483-2022/05/13-03:24:14:618694-data-volume-client-2-0-28<br clear="none">
[2022-06-15 14:50:53.605215] I [MSGID: 115029]
[server-handshake.c:690:server_setvolume]
0-data-volume-server: accepted client from
mseas.mit.edu-155483-2022/05/13-03:24:14:618694-data-volume-client-2-0-29
(version: 3.7.11)<br clear="none">
[2022-06-15 14:50:42.588247] I [MSGID: 115013]
[server-helpers.c:294:do_fd_cleanup]
0-data-volume-server: fd cleanup on
/projects/posydon/Acoustics_ASA/MSEAS-ParEq-DO/Save/2D/Test_Cases/RI/DO_NAPE_JASA_Paper/Uncertain_Pekeris_Waveguide_DO_MC</font><br clear="none">
</p>
<p>Thanks</p>
<p>Pat</p>
<p><br clear="none">
</p>
<div id="yiv5506566618yqt80756" class="yiv5506566618yqt6062330617">
<div class="yiv5506566618moz-cite-prefix">On 6/15/22
6:47 PM, Strahil Nikolov wrote:<br clear="none">
</div>
<blockquote type="cite"> </blockquote>
</div>
</div>
<div id="yiv5506566618yqt04765" class="yiv5506566618yqt6062330617">
<div>
<div id="yiv5506566618">
<div>I agree. It will be very hard to debug.
<div><br clear="none">
</div>
<div>Anything in the brick logs ?</div>
<div><br clear="none">
</div>
<div>I think it's pointless to mention that EL6 is
dead and Gluster v3 is so old that it's worth
considering a migration to a newer setup.</div>
<div><br clear="none">
</div>
<div>Best Regards,</div>
<div>Strahil Nikolov<br clear="none">
<br clear="none">
<blockquote style="margin:0 0 20px 0;">
<div style="font-family:Roboto, sans-serif;color:#6D00F6;">
<div id="yiv5506566618yqtfd58221" class="yiv5506566618yqt6679717770">
<div>On Wed, Jun 15, 2022 at 22:51, Yaniv
Kaul</div>
<div><a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:ykaul@redhat.com" target="_blank" href="mailto:ykaul@redhat.com" class="yiv5506566618moz-txt-link-rfc2396E"><ykaul@redhat.com></a>
wrote:</div>
</div>
</div>
<div id="yiv5506566618yqtfd56990" class="yiv5506566618yqt6679717770">
<div style="padding:10px 0 0 20px;margin:10px 0 0 0;border-left:1px solid #6D00F6;"> ________<br clear="none">
<br clear="none">
<br clear="none">
<br clear="none">
Community Meeting Calendar:<br clear="none">
<br clear="none">
Schedule -<br clear="none">
Every 2nd and 4th Tuesday at 14:30 IST /
09:00 UTC<br clear="none">
Bridge: <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://meet.google.com/cpu-eiue-hvk" class="yiv5506566618moz-txt-link-freetext yiv5506566618moz-txt-link-freetext">https://meet.google.com/cpu-eiue-hvk</a><br clear="none">
Gluster-users mailing list<br clear="none">
<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:Gluster-users@gluster.org" target="_blank" href="mailto:Gluster-users@gluster.org" class="yiv5506566618moz-txt-link-freetext yiv5506566618moz-txt-link-freetext">Gluster-users@gluster.org</a><br clear="none">
<a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://lists.gluster.org/mailman/listinfo/gluster-users" class="yiv5506566618moz-txt-link-freetext yiv5506566618moz-txt-link-freetext">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br clear="none">
</div>
</div>
</blockquote>
</div>
</div>
</div>
<pre class="yiv5506566618moz-signature">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: <a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:phaley@mit.edu" target="_blank" href="mailto:phaley@mit.edu" class="yiv5506566618moz-txt-link-abbreviated yiv5506566618moz-txt-link-freetext">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="http://web.mit.edu/phaley/www/" class="yiv5506566618moz-txt-link-freetext yiv5506566618moz-txt-link-freetext">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</div><div id="yiv5506566618yqtfd71224" class="yiv5506566618yqt0138793630">
</div></div><div id="yiv5506566618yqtfd75825" class="yiv5506566618yqt0138793630">
</div></div><div id="yiv5506566618yqtfd56693" class="yiv5506566618yqt0138793630">
</div></div><div id="yiv5506566618yqtfd94279" class="yiv5506566618yqt0138793630">
</div></blockquote><div id="yiv5506566618yqtfd14360" class="yiv5506566618yqt0138793630">
</div></div><div id="yiv5506566618yqtfd34488" class="yiv5506566618yqt0138793630">
<pre class="yiv5506566618moz-signature">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: <a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:phaley@mit.edu" target="_blank" href="mailto:phaley@mit.edu" class="yiv5506566618moz-txt-link-abbreviated">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="http://web.mit.edu/phaley/www/" class="yiv5506566618moz-txt-link-freetext">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</div></div></div> </div> </blockquote></div>