[Gluster-users] Odd "Transport endpoint is not connected" when trying to gunzip a file

Fri Jun 17 06:18:01 UTC 2022

Check with top & iotop the load.Especially check the wait for I/O in top.
Did you check dmesg for any clues ?
Best Regards,Strahil Nikolov

  On Thu, Jun 16, 2022 at 22:59, Pat Haley<phaley at mit.edu> wrote:    

Hi Strahil,

I poked around our logs, and found this on the front-end (from the day & time of the last time we had the issue)

Jun 15 10:51:17 mseas gdata[155485]: [2022-06-15 14:51:17.263858] C [rpc-clnt-ping.c:165:rpc_clnt_ping_timer_expired] 0-data-volume-client-2: server 172.16.1.113:49153 has not responded in the last 42 seconds, disconnecting.

This would indicate that the problem is related.  For us, however, I believe we can reproduce this issue at will (i.e. simply try to gunzip the same file). Unfortunately I have to go to a meeting now, but if you have some specific tests you'd like me to try, I can try them when I get back.

Thanks

Pat

 On 6/16/22 3:07 PM, Strahil Nikolov wrote:

 Pat,  
   Can you check the cpu and disk  performance when the volume reports the issue? 

  It seems that similar issue was reportedin https://lists.gluster.org/pipermail/gluster-users/2019-March/035944.html but I don't see a clear solution. Take a look in the thread and check if it matches your symptoms. 

  Best Regards, Strahil Nikolov

  On Thu, Jun 16, 2022 at 18:14, Pat Haley <phaley at mit.edu> wrote:     

Hi Strahil,

I poked around again and for brick 3 (where the file we were testing resides)  I only found the same log file as was at the bottom of my first Email:

 ---------------------------------------------------
 mseas-data3:  bricks/export-sda-brick3.log
 -----------------------------------------
 [2022-06-15 14:50:42.588143] I [MSGID: 115036] [server.c:552:server_rpc_notify] 0-data-volume-server: disconnecting connection frommseas.mit.edu-155483-2022/05/13-03:24:14:618694-data-volume-client-2-0-28
 [2022-06-15 14:50:42.588220] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-data-volume-server: fd cleanup on/projects/posydon/Acoustics_ASA/MSEAS-ParEq-DO/Save/2D/Test_Cases/RI/DO_NAPE_JASA_Paper/Uncertain_Pekeris_Waveguide_DO_MC
 [2022-06-15 14:50:42.588259] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-data-volume-server: fd cleanup on /projects/dri_calypso/PE/2019/Apr09/Ens3R200deg001/pe_out.nc.gz
 [2022-06-15 14:50:42.588288] I [MSGID: 101055] [client_t.c:420:gf_client_unref] 0-data-volume-server: Shutting down connectionmseas.mit.edu-155483-2022/05/13-03:24:14:618694-data-volume-client-2-0-28
 [2022-06-15 14:50:53.605215] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-data-volume-server: accepted client frommseas.mit.edu-155483-2022/05/13-03:24:14:618694-data-volume-client-2-0-29 (version: 3.7.11)
 [2022-06-15 14:50:42.588247] I [MSGID: 115013] [server-helpers.c:294:do_fd_cleanup] 0-data-volume-server: fd cleanup on/projects/posydon/Acoustics_ASA/MSEAS-ParEq-DO/Save/2D/Test_Cases/RI/DO_NAPE_JASA_Paper/Uncertain_Pekeris_Waveguide_DO_MC

Thanks

Pat

  On 6/15/22 6:47 PM, Strahil Nikolov wrote:

      I agree. It will be very hard to debug. 
  Anything in the brick logs ? 
  I think it's pointless to mention that EL6 is dead and Gluster v3 is so old that it's worth considering a migration to a newer setup. 
  Best Regards, Strahil Nikolov

   On Wed, Jun 15, 2022 at 22:51, Yaniv Kaul <ykaul at redhat.com> wrote:     ________

 Community Meeting Calendar:

 Schedule -
 Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
 Bridge: https://meet.google.com/cpu-eiue-hvk
 Gluster-users mailing list
 Gluster-users at gluster.org
 https://lists.gluster.org/mailman/listinfo/gluster-users

    -- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley                          Email:  phaley at mit.edu
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301

  -- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley                          Email:  phaley at mit.edu
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20220617/bb403bc5/attachment.html>