<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi Ashish,<div class=""><br class=""></div><div class="">I’m sorry to disturb you again, but I would like to know if you received the log files correctly.</div><div class="">Thank you,</div><div class="">Mauro Tridici</div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">Il giorno 26 set 2017, alle ore 10:19, Mauro Tridici <<a href="mailto:mauro.tridici@cmcc.it" class="">mauro.tridici@cmcc.it</a>> ha scritto:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html charset=us-ascii" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><br class=""></div><div class="">Hi Ashish,</div><div class=""><br class=""></div><div class="">in attachment you can find the gdb output (with bt and thread outputs) and the complete log file until the crash happened.</div><div class=""><br class=""></div><div class="">Thank you for your support.</div><div class="">Mauro Tridici</div><div class=""><br class=""></div><div class=""></div></div><span id="cid:194477FF-67AF-43C6-835F-013810CF7F70@openvpn"><tier2.log-20170924.gz></span><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><meta http-equiv="Content-Type" content="text/html charset=us-ascii" class=""><div class=""></div><div class=""><br class=""></div><div class=""></div></div><span id="cid:EE113972-90DC-4B4F-9CFA-124FDC4C1B22@openvpn"><gdb_bt_output_logs.txt></span><meta http-equiv="Content-Type" content="text/html charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""></div><br class=""><div class=""><blockquote type="cite" class=""><div class="">Il giorno 26 set 2017, alle ore 10:11, Ashish Pandey <<a href="mailto:aspandey@redhat.com" class="">aspandey@redhat.com</a>> ha scritto:</div><br class="Apple-interchange-newline"><div class=""><div class=""><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;" class=""><div class="">Hi,</div><div class=""><br class=""></div><div class="">Following are the command to get the debug info for gluster -<br class=""></div><div class=""><pre class="">gdb /usr/local/sbin/glusterfs <core file path> <br class="">Then on gdb prompt you need to type bt or backtrace <br class="">(gdb) bt<br class="">You can also provide the output of "thread apply all bt".<br class="">(gdb) thread apply all bt <br class=""></pre></div><div class="">Above commands should be executed on client node on which you have mounted the gluster volume and it crashed.<br class=""></div><div class="">However, I am not sure if you have enabled the generation of core dump on your system and have setup path to coredump.</div><div class=""><br class=""></div><div class=""><a href="https://stackoverflow.com/questions/8305866/how-to-analyze-a-programs-core-dump-file-with-gdb" class="">https://stackoverflow.com/questions/8305866/how-to-analyze-a-programs-core-dump-file-with-gdb</a></div><div class=""><a href="https://unix.stackexchange.com/questions/192716/how-to-set-the-core-dump-file-location-and-name" class="">https://unix.stackexchange.com/questions/192716/how-to-set-the-core-dump-file-location-and-name</a></div><div class=""><a href="https://stackoverflow.com/questions/2065912/core-dumped-but-core-file-is-not-in-current-directory" class="">https://stackoverflow.com/questions/2065912/core-dumped-but-core-file-is-not-in-current-directory</a></div><div class=""><br class=""></div><div class="">--<br class=""></div><div class="">Ashish<br class=""></div><div class=""> <br class=""></div><div class=""><br class=""></div><hr id="zwchr" class=""><div style="font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class=""><b class="">From: </b>"Mauro Tridici" <<a href="mailto:mauro.tridici@cmcc.it" class="">mauro.tridici@cmcc.it</a>><br class=""><b class="">To: </b>"Ashish Pandey" <<a href="mailto:aspandey@redhat.com" class="">aspandey@redhat.com</a>><br class=""><b class="">Cc: </b>"Gluster Users" <<a href="mailto:gluster-users@gluster.org" class="">gluster-users@gluster.org</a>><br class=""><b class="">Sent: </b>Tuesday, September 26, 2017 1:21:42 PM<br class=""><b class="">Subject: </b>Re: [Gluster-users] df command shows transport endpoint mount error on gluster client v.3.10.5 + core dump<br class=""><div class=""><br class=""></div><div class="">Hi Ashish,</div><div class=""><br class=""></div><div class="">thank you for your answer.</div><div class="">Do you need complete client log file only or something else in particular?</div><div class="">Unfortunately, I never used “bt” command. Could you please provide me an usage example string?</div><div class=""><br class=""></div><div class="">I will provide all logs you need.</div><div class="">Thank you again,</div><div class="">Mauro</div><br class=""><div class=""><blockquote class=""><div class="">Il giorno 26 set 2017, alle ore 09:30, Ashish Pandey <<a href="mailto:aspandey@redhat.com" class="" target="_blank">aspandey@redhat.com</a>> ha scritto:</div><br class="Apple-interchange-newline"><div class=""><div class=""><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;" class=""><div class="">Hi Mauro,<br class=""></div><div class=""><br class=""></div><div class="">We would require complete log file to debug this issue.<br class=""></div><div class="">Also, could you please provide some more information of the core after attaching to gdb and using command "bt".<br class=""></div><div class=""><br class=""></div><div class="">---<br class=""></div><div class="">Ashish<br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><hr id="zwchr" class=""><div style="font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class=""><b class="">From: </b>"Mauro Tridici" <<a href="mailto:mauro.tridici@cmcc.it" class="" target="_blank">mauro.tridici@cmcc.it</a>><br class=""><b class="">To: </b>"Gluster Users" <<a href="mailto:gluster-users@gluster.org" class="" target="_blank">gluster-users@gluster.org</a>><br class=""><b class="">Sent: </b>Monday, September 25, 2017 5:59:10 PM<br class=""><b class="">Subject: </b>[Gluster-users] df command shows transport endpoint mount error on gluster client v.3.10.5 + core dump<br class=""><div class=""><br class=""></div>Dear Gluster Users,<br class=""><div class=""><br class=""></div>I implemented a distributed disperse 6x(4+2) gluster (v.3.10.5) volume with the following options:<br class=""><div class=""><br class=""></div>[root@s01 tier2]# gluster volume info<br class=""> <br class="">Volume Name: tier2<br class="">Type: Distributed-Disperse<br class="">Volume ID: a28d88c5-3295-4e35-98d4-210b3af9358c<br class="">Status: Started<br class="">Snapshot Count: 0<br class="">Number of Bricks: 6 x (4 + 2) = 36<br class="">Transport-type: tcp<br class="">Bricks:<br class="">Brick1: s01-stg:/gluster/mnt1/brick<br class="">Brick2: s02-stg:/gluster/mnt1/brick<br class="">Brick3: s03-stg:/gluster/mnt1/brick<br class="">Brick4: s01-stg:/gluster/mnt2/brick<br class="">Brick5: s02-stg:/gluster/mnt2/brick<br class="">Brick6: s03-stg:/gluster/mnt2/brick<br class="">Brick7: s01-stg:/gluster/mnt3/brick<br class="">Brick8: s02-stg:/gluster/mnt3/brick<br class="">Brick9: s03-stg:/gluster/mnt3/brick<br class="">Brick10: s01-stg:/gluster/mnt4/brick<br class="">Brick11: s02-stg:/gluster/mnt4/brick<br class="">Brick12: s03-stg:/gluster/mnt4/brick<br class="">Brick13: s01-stg:/gluster/mnt5/brick<br class="">Brick14: s02-stg:/gluster/mnt5/brick<br class="">Brick15: s03-stg:/gluster/mnt5/brick<br class="">Brick16: s01-stg:/gluster/mnt6/brick<br class="">Brick17: s02-stg:/gluster/mnt6/brick<br class="">Brick18: s03-stg:/gluster/mnt6/brick<br class="">Brick19: s01-stg:/gluster/mnt7/brick<br class="">Brick20: s02-stg:/gluster/mnt7/brick<br class="">Brick21: s03-stg:/gluster/mnt7/brick<br class="">Brick22: s01-stg:/gluster/mnt8/brick<br class="">Brick23: s02-stg:/gluster/mnt8/brick<br class="">Brick24: s03-stg:/gluster/mnt8/brick<br class="">Brick25: s01-stg:/gluster/mnt9/brick<br class="">Brick26: s02-stg:/gluster/mnt9/brick<br class="">Brick27: s03-stg:/gluster/mnt9/brick<br class="">Brick28: s01-stg:/gluster/mnt10/brick<br class="">Brick29: s02-stg:/gluster/mnt10/brick<br class="">Brick30: s03-stg:/gluster/mnt10/brick<br class="">Brick31: s01-stg:/gluster/mnt11/brick<br class="">Brick32: s02-stg:/gluster/mnt11/brick<br class="">Brick33: s03-stg:/gluster/mnt11/brick<br class="">Brick34: s01-stg:/gluster/mnt12/brick<br class="">Brick35: s02-stg:/gluster/mnt12/brick<br class="">Brick36: s03-stg:/gluster/mnt12/brick<br class="">Options Reconfigured:<br class="">features.scrub: Active<br class="">features.bitrot: on<br class="">features.inode-quota: on<br class="">features.quota: on<br class="">performance.client-io-threads: on<br class="">cluster.min-free-disk: 10<br class="">cluster.quorum-type: auto<br class="">transport.address-family: inet<br class="">nfs.disable: on<br class="">server.event-threads: 4<br class="">client.event-threads: 4<br class="">cluster.lookup-optimize: on<br class="">performance.readdir-ahead: on<br class="">performance.parallel-readdir: off<br class="">cluster.readdir-optimize: on<br class="">features.cache-invalidation: on<br class="">features.cache-invalidation-timeout: 600<br class="">performance.stat-prefetch: on<br class="">performance.cache-invalidation: on<br class="">performance.md-cache-timeout: 600<br class="">network.inode-lru-limit: 50000<br class=""><a href="http://performance.io/" class="" target="_blank">performance.io</a>-cache: off<br class="">disperse.cpu-extensions: auto<br class=""><a href="http://performance.io/" class="" target="_blank">performance.io</a>-thread-count: 16<br class="">features.quota-deem-statfs: on<br class="">features.default-soft-limit: 90<br class="">cluster.server-quorum-type: server<br class="">cluster.brick-multiplex: on<br class="">cluster.server-quorum-ratio: 51%<br class=""><div class=""><br class=""></div>I also started a long write test (about 69 TB to be written) from different gluster clients.<br class="">One of this client returned the error "Transport Endpoint is not connected" during the rsync copy process.<br class="">A core dump file has been generated by the issue; this is the core dump content:<br class=""><div class=""><br class=""></div>GNU gdb (GDB) Red Hat Enterprise Linux (7.2-50.el6)<br class="">Copyright (C) 2010 Free Software Foundation, Inc.<br class="">License GPLv3+: GNU GPL version 3 or later <<a href="http://gnu.org/licenses/gpl.html" class="" target="_blank">http://gnu.org/licenses/gpl.html</a>><br class="">This is free software: you are free to change and redistribute it.<br class="">There is NO WARRANTY, to the extent permitted by law. Type "show copying"<br class="">and "show warranty" for details.<br class="">This GDB was configured as "x86_64-redhat-linux-gnu".<br class="">For bug reporting instructions, please see:<br class=""><<a href="http://www.gnu.org/software/gdb/bugs/" class="" target="_blank">http://www.gnu.org/software/gdb/bugs/</a>>...<br class="">Missing separate debuginfo for the main executable file<br class="">Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/fb/4d988b681faa09bb74becacd7a24f4186e8185<br class="">[New Thread 15802]<br class="">[New Thread 15804]<br class="">[New Thread 15805]<br class="">[New Thread 6856]<br class="">[New Thread 30432]<br class="">[New Thread 30486]<br class="">[New Thread 1619]<br class="">[New Thread 15806]<br class="">[New Thread 15810]<br class="">[New Thread 30412]<br class="">[New Thread 15809]<br class="">[New Thread 15799]<br class="">[New Thread 30487]<br class="">[New Thread 15795]<br class="">[New Thread 15797]<br class="">[New Thread 15798]<br class="">[New Thread 15800]<br class="">[New Thread 15801]<br class="">Core was generated by `/usr/sbin/glusterfs --volfile-server=s01-stg --volfile-id=/tier2 /tier2'.<br class="">Program terminated with signal 6, Aborted.<br class="">#0 0x00000032a74328a5 in ?? ()<br class="">"/core.15795" is a core file.<br class="">Please specify an executable to debug.<br class=""><div class=""><br class=""></div>No problem detected on the gluster servers, no brick down and so on...<br class=""><div class=""><br class=""></div>Anyone of us experienced the same problem? <br class="">If yes, how did you resolve it?<br class=""><div class=""><br class=""></div>You can find the client "/var/log/messages" and "/var/log/glusterfs" log files content.<br class="">Thank you in advance.<br class="">Mauro Tridici<br class=""><div class=""><br class=""></div>--------------------------------------<br class=""><div class=""><br class=""></div>In /var/log/syslog-ng/messages on the client (OS=centos 6.2, 16 cores, 64GB RAM, gluster client v.3.10.5)<br class=""><div class=""><br class=""></div>Sep 23 10:42:43 login2 tier2[15795]: pending frames:<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(1) op(WRITE)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(1) op(WRITE)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(1) op(WRITE)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(1) op(WRITE)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: frame : type(0) op(0)<br class="">Sep 23 10:42:43 login2 tier2[15795]: patchset: <a href="git://git.gluster.org/glusterfs.git" class="" target="_blank">git://git.gluster.org/glusterfs.git</a><br class="">Sep 23 10:42:43 login2 tier2[15795]: signal received: 6<br class="">Sep 23 10:42:43 login2 tier2[15795]: time of crash: <br class="">Sep 23 10:42:43 login2 tier2[15795]: 2017-09-23 08:42:43<br class="">Sep 23 10:42:43 login2 tier2[15795]: configuration details:<br class="">Sep 23 10:42:43 login2 tier2[15795]: argp 1<br class="">Sep 23 10:42:43 login2 tier2[15795]: backtrace 1<br class="">Sep 23 10:42:43 login2 tier2[15795]: dlfcn 1<br class="">Sep 23 10:42:43 login2 tier2[15795]: libpthread 1<br class="">Sep 23 10:42:43 login2 tier2[15795]: llistxattr 1<br class="">Sep 23 10:42:43 login2 tier2[15795]: setfsid 1<br class="">Sep 23 10:42:43 login2 tier2[15795]: spinlock 1<br class="">Sep 23 10:42:43 login2 tier2[15795]: epoll.h 1<br class="">Sep 23 10:42:43 login2 tier2[15795]: xattr.h 1<br class="">Sep 23 10:42:43 login2 tier2[15795]: st_atim.tv_nsec 1<br class="">Sep 23 10:42:43 login2 tier2[15795]: package-string: glusterfs 3.10.5<br class="">Sep 23 10:42:43 login2 tier2[15795]: ---------<br class="">Sep 23 10:42:43 login2 abrt[8605]: file /usr/sbin/glusterfsd seems to be deleted<br class=""><div class=""><br class=""></div>In /var/log/glusterfs file:<br class=""><div class=""><br class=""></div>pending frames:<br class="">frame : type(1) op(WRITE)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(1) op(WRITE)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(1) op(WRITE)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(1) op(WRITE)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">frame : type(0) op(0)<br class="">patchset: <a href="git://git.gluster.org/glusterfs.git" class="" target="_blank">git://git.gluster.org/glusterfs.git</a><br class="">signal received: 6<br class="">time of crash: <br class="">2017-09-23 08:42:43<br class="">configuration details:<br class="">argp 1<br class="">backtrace 1<br class="">dlfcn 1<br class="">libpthread 1<br class="">llistxattr 1<br class="">setfsid 1<br class="">spinlock 1<br class="">epoll.h 1<br class="">xattr.h 1<br class="">st_atim.tv_nsec 1<br class="">package-string: glusterfs 3.10.5<br class="">/usr/lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0x9c)[0x7fd8d8b8af3c]<br class="">/usr/lib64/libglusterfs.so.0(gf_print_trace+0x338)[0x7fd8d8b96538]<br class="">/lib64/libc.so.6[0x32a7432920]<br class="">/lib64/libc.so.6(gsignal+0x35)[0x32a74328a5]<br class="">/lib64/libc.so.6(abort+0x175)[0x32a7434085]<br class="">/lib64/libc.so.6[0x32a746ffe7]<br class="">/lib64/libc.so.6[0x32a7475916]<br class="">/usr/lib64/glusterfs/3.10.5/xlator/cluster/disperse.so(+0x3181f)[0x7fd8cd51681f]<br class="">/usr/lib64/glusterfs/3.10.5/xlator/cluster/disperse.so(+0x31dad)[0x7fd8cd516dad]<br class="">/usr/lib64/glusterfs/3.10.5/xlator/cluster/disperse.so(+0x12027)[0x7fd8cd4f7027]<br class="">/usr/lib64/glusterfs/3.10.5/xlator/cluster/disperse.so(+0x11dc5)[0x7fd8cd4f6dc5]<br class="">/usr/lib64/glusterfs/3.10.5/xlator/cluster/disperse.so(+0x1475f)[0x7fd8cd4f975f]<br class="">/usr/lib64/glusterfs/3.10.5/xlator/cluster/disperse.so(+0x15178)[0x7fd8cd4fa178]<br class="">/usr/lib64/glusterfs/3.10.5/xlator/cluster/disperse.so(+0x31dc8)[0x7fd8cd516dc8]<br class="">/usr/lib64/glusterfs/3.10.5/xlator/cluster/disperse.so(+0x12027)[0x7fd8cd4f7027]<br class="">/usr/lib64/glusterfs/3.10.5/xlator/cluster/disperse.so(+0x11dc5)[0x7fd8cd4f6dc5]<br class="">/usr/lib64/glusterfs/3.10.5/xlator/cluster/disperse.so(+0x13f24)[0x7fd8cd4f8f24]<br class="">/usr/lib64/glusterfs/3.10.5/xlator/cluster/disperse.so(+0x33e12)[0x7fd8cd518e12]<br class="">/usr/lib64/glusterfs/3.10.5/xlator/protocol/client.so(+0x2c420)[0x7fd8cd793420]<br class="">/usr/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)[0x7fd8d8955ad5]<br class="">/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x195)[0x7fd8d8956c85]<br class="">/usr/lib64/libgfrpc.so.0(rpc_transport_notify+0x28)[0x7fd8d8951d68]<br class="">/usr/lib64/glusterfs/3.10.5/rpc-transport/socket.so(+0x9ccd)[0x7fd8ce5f1ccd]<br class="">/usr/lib64/glusterfs/3.10.5/rpc-transport/socket.so(+0xaffe)[0x7fd8ce5f2ffe]<br class="">/usr/lib64/libglusterfs.so.0(+0x87806)[0x7fd8d8be8806]<br class="">/lib64/libpthread.so.0[0x32a8007851]<br class="">/lib64/libc.so.6(clone+0x6d)[0x32a74e811d]<br class="">_______________________________________________<br class="">Gluster-users mailing list<br class=""><a href="mailto:Gluster-users@gluster.org" class="" target="_blank">Gluster-users@gluster.org</a><br class=""><a href="http://lists.gluster.org/mailman/listinfo/gluster-users" class="">http://lists.gluster.org/mailman/listinfo/gluster-users</a><br class=""></div><div class=""><br class=""></div></div></div></div></blockquote></div><br class="">_______________________________________________<br class="">Gluster-users mailing list<br class=""><a href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class=""><a href="http://lists.gluster.org/mailman/listinfo/gluster-users" class="">http://lists.gluster.org/mailman/listinfo/gluster-users</a></div><div class=""><br class=""></div></div></div></div></blockquote></div><br class="">
<br class=""></div></div></blockquote></div><br class=""><div class="">
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div class=""><br class="Apple-interchange-newline">-------------------------</div><div class="">Mauro Tridici</div><div class=""><br class=""></div><div class="">Fondazione CMCC</div><div class="">CMCC Supercomputing Center</div><div class="">presso Complesso Ecotekne - Università del Salento -</div><div class="">Strada Prov.le Lecce - Monteroni sn</div><div class="">73100 Lecce IT</div><div class=""><a href="http://www.cmcc.it" class="">http://www.cmcc.it</a></div><div class=""><br class=""></div><div class="">mobile: (+39) 327 5630841</div><div class="">email: <a href="mailto:mauro.tridici@cmcc.it" class="">mauro.tridici@cmcc.it</a></div></span></span>
</div>
<br class=""></div></body></html>