<div dir="ltr">Hi Xavi, <div><br></div><div>The OS is Debian 11 with the proxmox kernel. Gluster packages are the official from <a href="http://gluster.org">gluster.org</a> (<a href="https://download.gluster.org/pub/gluster/glusterfs/10/10.3/Debian/bullseye/">https://download.gluster.org/pub/gluster/glusterfs/10/10.3/Debian/bullseye/</a>)</div><div><div><br></div><div>The system logs showed no other issues by the time of the crash, no OOM kill or whatsoever, and no other process was interacting with the gluster mountpoint besides proxmox.</div></div><div><br></div><div>I wasn't running gdb when it crashed, so I don't really know if I can obtain a more detailed trace from logs or if there is a simple way to let it running in the background to see if it happens again (or there is a flag to start the systemd daemon in debug mode).</div><div><br></div><div>Best, </div><div><br><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div style="color:rgb(34,34,34)"><font size="4" face="arial, sans-serif"><b>Angel Docampo</b></font></div><div style="color:rgb(34,34,34)"><a href="https://www.google.com/maps/place/Edificio+de+Oficinas+Euro+3/@41.3755943,2.0730134,17z/data=!3m2!4b1!5s0x12a4997021aad323:0x3e06bf8ae6d68351!4m5!3m4!1s0x12a4997a67bf592f:0x83c2323a9cc2aa4b!8m2!3d41.3755903!4d2.0752021" target="_blank"><img src="https://ci3.googleusercontent.com/mail-sig/AIorK4yfwAc1Ml7oXFmQS6cJWaMeVnZ7xmAkBZPyODZAB9R8us12sFWd19cHxqDJ7CRF-UcvfKFLJNg"></a> <a href="mailto:angel.docampo@eoniantec.com" target="_blank"><img src="https://ci3.googleusercontent.com/mail-sig/AIorK4xhLmETvCmyOlze-bvuD8EJDZ0KgPmtCKnW0ObWzrqFda6zykLG06WgSatNHY2tgyMj_FOg3RY"></a> <a href="tel:+34-93-1592929" target="_blank"><img src="https://ci3.googleusercontent.com/mail-sig/AIorK4wKRh91a3Q-nUQnp1zQ-4rrdeN4FKksw-kDAAzCOg9hOTqSiqNmU2AloNPHrS-QwtOWiFHYHl0"></a></div></div></div></div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">El lun, 21 nov 2022 a las 15:16, Xavi Hernandez (<<a href="mailto:jahernan@redhat.com">jahernan@redhat.com</a>>) escribió:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi Angel,</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Nov 21, 2022 at 2:33 PM Angel Docampo <<a href="mailto:angel.docampo@eoniantec.com" target="_blank">angel.docampo@eoniantec.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Sorry for necrobumping this, but this morning I've suffered this on my Proxmox + GlusterFS cluster. In the log I can see this<div><br><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">[2022-11-21 07:38:00.213620 +0000] I [MSGID: 133017] [shard.c:7275:shard_seek] 11-vmdata-shard: seek called on fbc063cb-874e-475d-b585-f89</span><br>f7518acdd. [Operation not supported]
<br><span style="color:rgb(255,255,255);background-color:rgb(0,0,0)">pending frames</span><span style="color:rgb(0,0,0)">:
</span><br>frame : type(1) op(WRITE)
<br>frame : type(0) op(0)
<br>frame : type(0) op(0)
<br>frame : type(0) op(0)
<br>frame : type(0) op(0)
<br>frame : type(0) op(0)
<br>frame : type(0) op(0)
<br>frame : type(0) op(0)
<br>frame : type(0) op(0)
<br>frame : type(0) op(0)
<br>frame : type(0) op(0)
<br>frame : type(0) op(0)
<br>frame : type(0) op(0)
<br>frame : type(0) op(0)
<br>frame : type(0) op(0)
<br>frame : type(0) op(0)
<br>frame : type(0) op(0)<br>
...</span></div><div><span style="color:rgb(0,0,0);font-family:monospace">frame : type(1) op(FSYNC)</span><br></div><div><span style="font-family:monospace">frame : type(1) op(FSYNC)
<br>frame : type(1) op(FSYNC)
<br>frame : type(1) op(FSYNC)
<br>frame : type(1) op(FSYNC)
<br>frame : type(1) op(FSYNC)
<br>frame : type(1) op(FSYNC)
<br>frame : type(1) op(FSYNC)
<br>frame : type(1) op(FSYNC)
<br>frame : type(1) op(FSYNC)
<br>frame : type(1) op(FSYNC)
<br>frame : type(1) op(FSYNC)
<br>frame : type(1) op(FSYNC)
<br>frame : type(1) op(FSYNC)
<br>frame : type(1) op(FSYNC)
<br>frame : type(1) op(FSYNC)
<br>frame : type(1) op(FSYNC)
<br>frame : type(1) op(FSYNC)
<br>frame : type(1) op(FSYNC)
<br>frame : type(1) op(FSYNC)</span></div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">patchset: git://<a href="http://git.gluster.org/glusterfs.git" target="_blank">git.gluster.org/glusterfs.git</a>
</span><br>signal received: 11
<br>time of crash: <br>2022-11-21 07:38:00 +0000
<br>configuration details:
<br>argp 1
<br>backtrace 1
<br>dlfcn 1
<br>libpthread 1
<br>llistxattr 1
<br>setfsid 1
<br>epoll.h 1
<br>xattr.h 1
<br>st_atim.tv_nsec 1
<br>package-string: glusterfs 10.3
<br>/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x28a54)[0x7f74f286ba54]
<br>/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_trace+0x700)[0x7f74f2873fc0]
<br>/lib/x86_64-linux-gnu/libc.so.6(+0x38d60)[0x7f74f262ed60]
<br>/usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/cluster/disperse.so(+0x37a14)[0x7f74ecfcea14]
<br>/usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/cluster/disperse.so(+0x19414)[0x7f74ecfb0414]
<br>/usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/cluster/disperse.so(+0x16373)[0x7f74ecfad373]
<br>/usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/cluster/disperse.so(+0x21d59)[0x7f74ecfb8d59]
<br>/usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/cluster/disperse.so(+0x22815)[0x7f74ecfb9815]
<br>/usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/cluster/disperse.so(+0x377d9)[0x7f74ecfce7d9]
<br>/usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/cluster/disperse.so(+0x19414)[0x7f74ecfb0414]
<br>/usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/cluster/disperse.so(+0x16373)[0x7f74ecfad373]
<br>/usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/cluster/disperse.so(+0x170f9)[0x7f74ecfae0f9]
<br>/usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/cluster/disperse.so(+0x313bb)[0x7f74ecfc83bb]
<br>/usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/protocol/client.so(+0x48e3a)[0x7f74ed06ce3a]
<br>/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xfccb)[0x7f74f2816ccb]
<br>/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_transport_notify+0x26)[0x7f74f2812646]
<br>/usr/lib/x86_64-linux-gnu/glusterfs/10.3/rpc-transport/socket.so(+0x64c8)[0x7f74ee15f4c8]
<br>/usr/lib/x86_64-linux-gnu/glusterfs/10.3/rpc-transport/socket.so(+0xd38c)[0x7f74ee16638c]
<br>/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x7971d)[0x7f74f28bc71d]
<br>/lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7)[0x7f74f27d2ea7]
<br>/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f74f26f2aef]
<br>---------<br>
</span><font face="arial, sans-serif">The mount point wasn't accessible with the "Tr<span style="color:rgb(0,0,0)">ansport endpoint is not connected" message and it was shown like this.</span><br></font></div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0)">d????????? ? ? ? ? ? </span><span style="font-weight:bold;color:rgb(84,84,255)">vmdata</span><br><span style="color:rgb(0,0,0)">
</span><br></span><font face="arial, sans-serif">I had to stop all the VMs on that proxmox node, then stop the gluster daemon to ummount de directory, and after starting the daemon and re-mounting, all was working again.</font></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace">My gluster volume info returns this</span></div><div><span style="font-family:monospace"> <br>Volume Name: vmdata
<br>Type: Distributed-Disperse
<br>Volume ID: cace5aa4-b13a-4750-8736-aa179c2485e1
<br>Status: Started
<br>Snapshot Count: 0
<br>Number of Bricks: 2 x (2 + 1) = 6
<br>Transport-type: tcp
<br>Bricks:
<br>Brick1: g01:/data/brick1/brick
<br>Brick2: g02:/data/brick2/brick
<br>Brick3: g03:/data/brick1/brick
<br>Brick4: g01:/data/brick2/brick
<br>Brick5: g02:/data/brick1/brick
<br>Brick6: g03:/data/brick2/brick
<br>Options Reconfigured:
<br>nfs.disable: on
<br>transport.address-family: inet
<br>storage.fips-mode-rchecksum: on
<br>features.shard: enable
<br>features.shard-block-size: 256MB
<br>performance.read-ahead: off
<br>performance.quick-read: off
<br>performance.io-cache: off
<br>server.event-threads: 2
<br>client.event-threads: 3
<br>performance.client-io-threads: on
<br>performance.stat-prefetch: off
<br>dht.force-readdirp: off
<br>performance.force-readdirp: off
<br>network.remote-dio: on
<br>features.cache-invalidation: on
<br>performance.parallel-readdir: on
<br>performance.readdir-ahead: on<br>
<br></span><font face="arial, sans-serif">Xavi, do you think the open-behind off setting can help somehow? I did try to understand what it does (with no luck), and if it could impact the performance of my VMs (I've the setup you know so well ;))</font><div>I would like to avoid more crashings like this, version 10.3 of gluster was working since two weeks ago, quite well until this morning.</div></div></div></div></blockquote><div><br></div><div>I don't think disabling open-behind will have any visible effect on performance. Open-behind is only useful for small files when the workload is mostly open + read + close, and quick-read is also enabled (which is not your case). The only effect it will have is that the latency "saved" during open is "paid" on the next operation sent to the file, so the total overall latency should be the same. Additionally, VM workload doesn't open files frequently, so it shouldn't matter much in any case.</div><div><br></div><div>That said, I'm not sure if the problem is the same in your case. Based on the stack of the crash, it seems an issue inside the disperse module.</div><div><br></div><div>What OS are you using ? are you using official packages ? if so, which ones ?</div><div><br></div><div>Is it possible to provide a backtrace from gdb ?</div><div><br></div><div>Regards,</div><div><br></div><div>Xavi</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><br></div><div><div><div dir="ltr"><div dir="ltr"><div style="color:rgb(34,34,34)"><font size="4" face="arial, sans-serif"><b>Angel Docampo</b></font></div><div style="color:rgb(34,34,34)"><a href="https://www.google.com/maps/place/Edificio+de+Oficinas+Euro+3/@41.3755943,2.0730134,17z/data=!3m2!4b1!5s0x12a4997021aad323:0x3e06bf8ae6d68351!4m5!3m4!1s0x12a4997a67bf592f:0x83c2323a9cc2aa4b!8m2!3d41.3755903!4d2.0752021" target="_blank"><img src="https://ci3.googleusercontent.com/mail-sig/AIorK4yfwAc1Ml7oXFmQS6cJWaMeVnZ7xmAkBZPyODZAB9R8us12sFWd19cHxqDJ7CRF-UcvfKFLJNg"></a> <a href="mailto:angel.docampo@eoniantec.com" target="_blank"><img src="https://ci3.googleusercontent.com/mail-sig/AIorK4xhLmETvCmyOlze-bvuD8EJDZ0KgPmtCKnW0ObWzrqFda6zykLG06WgSatNHY2tgyMj_FOg3RY"></a> <a href="tel:+34-93-1592929" target="_blank"><img src="https://ci3.googleusercontent.com/mail-sig/AIorK4wKRh91a3Q-nUQnp1zQ-4rrdeN4FKksw-kDAAzCOg9hOTqSiqNmU2AloNPHrS-QwtOWiFHYHl0"></a></div></div></div></div><br></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">El vie, 19 mar 2021 a las 2:10, David Cunningham (<<a href="mailto:dcunningham@voisonics.com" target="_blank">dcunningham@voisonics.com</a>>) escribió:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi Xavi,</div><div><br></div><div>Thank you for that information. We'll look at upgrading it.</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 12 Mar 2021 at 05:20, Xavi Hernandez <<a href="mailto:jahernan@redhat.com" target="_blank">jahernan@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi David,</div><div><br></div><div>with so little information it's hard to tell, but given that there are several OPEN and UNLINK operations, it could be related to an already fixed bug (in recent versions) in open-behind.</div><div><br></div><div>You can try disabling open-behind with this command:</div><div><br></div><div> <font face="monospace"># gluster volume set <volname> open-behind off</font></div><div><font face="monospace"><br></font></div><div><font face="arial, sans-serif">But given the version you are using is very old and unmaintained, I would recommend you to upgrade to 8.x at least.</font></div><div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">Regards,</font></div><div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">Xavi</font></div><div><font face="arial, sans-serif"><br></font></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Mar 10, 2021 at 5:10 AM David Cunningham <<a href="mailto:dcunningham@voisonics.com" target="_blank">dcunningham@voisonics.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hello,</div><div><br></div><div>We have a GlusterFS 5.13 server which also mounts itself with the native FUSE client. Recently the FUSE mount crashed and we found the following in the syslog. There isn't anything logged in mnt-glusterfs.log for that time. After killing all processes with a file handle open on the filesystem we were able to unmount and then remount the filesystem successfully.<br></div><div><br></div><div>Would anyone have advice on how to debug this crash? Thank you in advance!<br></div><div><br></div><div>Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: pending frames:<br>Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: frame : type(0) op(0)<br>Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: frame : type(0) op(0)<br>Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: frame : type(1) op(UNLINK)<br>Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: frame : type(1) op(UNLINK)<br>Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: frame : type(1) op(OPEN)<br>Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: message repeated 3355 times: [ frame : type(1) op(OPEN)]<br>Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: frame : type(1) op(OPEN)<br>Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: message repeated 6965 times: [ frame : type(1) op(OPEN)]<br>Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: frame : type(1) op(OPEN)<br>Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: message repeated 4095 times: [ frame : type(1) op(OPEN)]<br>Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: frame : type(0) op(0)<br>Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: patchset: git://<a href="http://git.gluster.org/glusterfs.git" target="_blank">git.gluster.org/glusterfs.git</a><br>Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: signal received: 11<br>Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: time of crash:<br>Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: 2021-03-09 03:12:31<br>Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: configuration details:<br>Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: argp 1<br>Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: backtrace 1<br>Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: dlfcn 1<br>Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: libpthread 1<br>Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: llistxattr 1<br>Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: setfsid 1<br>Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: spinlock 1<br>Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: epoll.h 1<br>Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: xattr.h 1<br>Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: st_atim.tv_nsec 1<br>Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: package-string: glusterfs 5.13<br>Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: ---------<br>...<br>Mar 9 05:13:50 voip1 systemd[1]: glusterfssharedstorage.service: Main process exited, code=killed, status=11/SEGV<br>Mar 9 05:13:50 voip1 systemd[1]: glusterfssharedstorage.service: Failed with result 'signal'.<br>...<br>Mar 9 05:13:54 voip1 systemd[1]: glusterfssharedstorage.service: Service hold-off time over, scheduling restart.<br>Mar 9 05:13:54 voip1 systemd[1]: glusterfssharedstorage.service: Scheduled restart job, restart counter is at 2.<br>Mar 9 05:13:54 voip1 systemd[1]: Stopped Mount glusterfs sharedstorage.<br>Mar 9 05:13:54 voip1 systemd[1]: Starting Mount glusterfs sharedstorage...<br>Mar 9 05:13:54 voip1 mount-shared-storage.sh[20520]: ERROR: Mount point does not exist<br>Mar 9 05:13:54 voip1 mount-shared-storage.sh[20520]: Please specify a mount point<br>Mar 9 05:13:54 voip1 mount-shared-storage.sh[20520]: Usage:<br>Mar 9 05:13:54 voip1 mount-shared-storage.sh[20520]: man 8 /sbin/mount.glusterfs</div><div><br>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>David Cunningham, Voisonics Limited<br><a href="http://voisonics.com/" target="_blank">http://voisonics.com/</a><br>USA: +1 213 221 1092<br>New Zealand: +64 (0)28 2558 3782</div></div></div></div></div></div></div></div></div></div></div></div></div>
________<br>
<br>
<br>
<br>
Community Meeting Calendar:<br>
<br>
Schedule -<br>
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" rel="noreferrer" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>David Cunningham, Voisonics Limited<br><a href="http://voisonics.com/" target="_blank">http://voisonics.com/</a><br>USA: +1 213 221 1092<br>New Zealand: +64 (0)28 2558 3782</div></div></div></div></div></div></div></div></div></div></div>
________<br>
<br>
<br>
<br>
Community Meeting Calendar:<br>
<br>
Schedule -<br>
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" rel="noreferrer" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div>
</blockquote></div></div>
</blockquote></div>