[Gluster-users] GlusterFS mount crash
Angel Docampo
angel.docampo at eoniantec.com
Fri Dec 2 16:30:32 UTC 2022
Yes, I'm on 10.3 on a brand new installation (i.e.: no upgrade or
whatsoever)
Ok, I've finally read how to get core dumps on Debian. Soft limits are 0 by
default, so no core dumps can be generated. I've set up soft ulimit to
unlimited and killed a test process with a SIGSEGV signal, then I was able
to see the core dump. Great!
I've moved all the qcow2 files to another location but not destroyed the
volume or deleted any files, I will create some test VMs there and leave
this a few days to see if the dump is generated properly next time,
hopefully soon enough.
Thank you Xavi.
*Angel Docampo*
<https://www.google.com/maps/place/Edificio+de+Oficinas+Euro+3/@41.3755943,2.0730134,17z/data=!3m2!4b1!5s0x12a4997021aad323:0x3e06bf8ae6d68351!4m5!3m4!1s0x12a4997a67bf592f:0x83c2323a9cc2aa4b!8m2!3d41.3755903!4d2.0752021>
<angel.docampo at eoniantec.com> <+34-93-1592929>
El jue, 1 dic 2022 a las 18:02, Xavi Hernandez (<jahernan at redhat.com>)
escribió:
> I'm not so sure the problem is with sharding. Basically it's saying that
> seek is not supported, which means that something between shard and the
> bricks doesn't support it. DHT didn't support seek before 10.3, but if I'm
> not wrong you are already using 10.3, so the message is weird. But in any
> case this shouldn't cause a crash. The stack trace seems to indicate that
> the crash happens inside disperse, but without a core dump there's little
> more that I can do.
>
>
>
> On Thu, Dec 1, 2022 at 5:27 PM Angel Docampo <angel.docampo at eoniantec.com>
> wrote:
>
>> Well, that last more time, but it crashed once again, same node, same
>> mountpoint... fortunately, I've moved preventively all the VMs to the
>> underlying ZFS filesystem those past days, so none of them have been
>> affected this time...
>>
>> dmesg show this
>> [2022-12-01 15:49:54] INFO: task iou-wrk-637144:946532 blocked for more
>> than 120 seconds.
>> [2022-12-01 15:49:54] Tainted: P IO 5.15.74-1-pve #1
>> [2022-12-01 15:49:54] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
>> disables this message.
>> [2022-12-01 15:49:54] task:iou-wrk-637144 state:D stack: 0
>> pid:946532 ppid: 1 flags:0x00004000
>> [2022-12-01 15:49:54] Call Trace:
>> [2022-12-01 15:49:54] <TASK>
>> [2022-12-01 15:49:54] __schedule+0x34e/0x1740
>> [2022-12-01 15:49:54] ? kmem_cache_free+0x271/0x290
>> [2022-12-01 15:49:54] ? mempool_free_slab+0x17/0x20
>> [2022-12-01 15:49:54] schedule+0x69/0x110
>> [2022-12-01 15:49:54] rwsem_down_write_slowpath+0x231/0x4f0
>> [2022-12-01 15:49:54] ? ttwu_queue_wakelist+0x40/0x1c0
>> [2022-12-01 15:49:54] down_write+0x47/0x60
>> [2022-12-01 15:49:54] fuse_file_write_iter+0x1a3/0x430
>> [2022-12-01 15:49:54] ? apparmor_file_permission+0x70/0x170
>> [2022-12-01 15:49:54] io_write+0xf6/0x330
>> [2022-12-01 15:49:54] ? update_cfs_group+0x9c/0xc0
>> [2022-12-01 15:49:54] ? dequeue_entity+0xd8/0x490
>> [2022-12-01 15:49:54] io_issue_sqe+0x401/0x1fc0
>> [2022-12-01 15:49:54] ? lock_timer_base+0x3b/0xd0
>> [2022-12-01 15:49:54] io_wq_submit_work+0x76/0xd0
>> [2022-12-01 15:49:54] io_worker_handle_work+0x1a7/0x5f0
>> [2022-12-01 15:49:54] io_wqe_worker+0x2c0/0x360
>> [2022-12-01 15:49:54] ? finish_task_switch.isra.0+0x7e/0x2b0
>> [2022-12-01 15:49:54] ? io_worker_handle_work+0x5f0/0x5f0
>> [2022-12-01 15:49:54] ? io_worker_handle_work+0x5f0/0x5f0
>> [2022-12-01 15:49:54] ret_from_fork+0x1f/0x30
>> [2022-12-01 15:49:54] RIP: 0033:0x0
>> [2022-12-01 15:49:54] RSP: 002b:0000000000000000 EFLAGS: 00000207
>> [2022-12-01 15:49:54] RAX: 0000000000000000 RBX: 0000000000000011 RCX:
>> 0000000000000000
>> [2022-12-01 15:49:54] RDX: 0000000000000001 RSI: 0000000000000001 RDI:
>> 0000000000000120
>> [2022-12-01 15:49:54] RBP: 0000000000000120 R08: 0000000000000001 R09:
>> 00000000000000f0
>> [2022-12-01 15:49:54] R10: 00000000000000f8 R11: 00000001239a4128 R12:
>> ffffffffffffdb90
>> [2022-12-01 15:49:54] R13: 0000000000000001 R14: 0000000000000001 R15:
>> 0000000000000100
>> [2022-12-01 15:49:54] </TASK>
>>
>> My gluster volume log shows plenty of error like this
>> The message "I [MSGID: 133017] [shard.c:7275:shard_seek] 0-vmdata-shard:
>> seek called on 73f0ad95-f7e3-4a68-8d08-9f7e03182baa. [Operation not
>> supported]" repeated 1564 times between [2022-12-01 00:20:09.578233 +0000]
>> and [2022-12-01 00:22:09.436927 +0000]
>> [2022-12-01 00:22:09.516269 +0000] I [MSGID: 133017]
>> [shard.c:7275:shard_seek] 0-vmdata-shard: seek called on
>> 73f0ad95-f7e3-4a68-8d08-9f7e03182baa. [Operation not supported]
>>
>> and of this
>> [2022-12-01 09:05:08.525867 +0000] I [MSGID: 133017]
>> [shard.c:7275:shard_seek] 0-vmdata-shard: seek called on
>> 3ed993c4-bbb5-4938-86e9-6d22b8541e8e. [Operation not supported]
>>
>> Then simply the same
>> pending frames:
>> frame : type(0) op(0)
>> frame : type(0) op(0)
>> frame : type(0) op(0)
>> frame : type(0) op(0)
>> frame : type(1) op(FSYNC)
>> frame : type(0) op(0)
>> frame : type(0) op(0)
>> frame : type(0) op(0)
>> frame : type(0) op(0)
>> frame : type(0) op(0)
>> frame : type(0) op(0)
>> frame : type(0) op(0)
>> frame : type(0) op(0)
>> frame : type(0) op(0)
>> frame : type(0) op(0)
>> frame : type(0) op(0)
>> frame : type(0) op(0)
>> frame : type(0) op(0)
>> frame : type(0) op(0)
>> patchset: git://git.gluster.org/glusterfs.git
>> signal received: 11
>> time of crash:
>> 2022-12-01 14:45:14 +0000
>> configuration details:
>> argp 1
>> backtrace 1
>> dlfcn 1
>> libpthread 1
>> llistxattr 1
>> setfsid 1
>> epoll.h 1
>> xattr.h 1
>> st_atim.tv_nsec 1
>> package-string: glusterfs 10.3
>> /lib/x86_64-linux-gnu/libglusterfs.so.0(+0x28a54)[0x7f1e23db3a54]
>> /lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_trace+0x700)[0x7f1e23dbbfc0]
>>
>> /lib/x86_64-linux-gnu/libc.so.6(+0x38d60)[0x7f1e23b76d60]
>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/cluster/disperse.so(+0x37a14)[0x7f1e200e9a14]
>>
>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/cluster/disperse.so(+0x19414)[0x7f1e200cb414]
>>
>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/cluster/disperse.so(+0xd072)[0x7f1e200bf072]
>>
>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/performance/readdir-ahead.so(+0x316d)[0x7f1e200a316d]
>>
>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/cluster/distribute.so(+0x5bdd4)[0x7f1e197aadd4]
>>
>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/features/shard.so(+0x1e69c)[0x7f1e2008b69c]
>>
>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/features/shard.so(+0x16551)[0x7f1e20083551]
>>
>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/features/shard.so(+0x25abf)[0x7f1e20092abf]
>>
>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/features/shard.so(+0x25d21)[0x7f1e20092d21]
>>
>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/features/shard.so(+0x167be)[0x7f1e200837be]
>>
>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/features/shard.so(+0x1c178)[0x7f1e20089178]
>>
>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/features/utime.so(+0x7804)[0x7f1e20064804]
>>
>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/performance/write-behind.so(+0x8164)[0x7f1e2004e164]
>>
>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/performance/write-behind.so(+0x9228)[0x7f1e2004f228]
>>
>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/performance/write-behind.so(+0x9a4d)[0x7f1e2004fa4d]
>>
>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/features/utime.so(+0x29e5)[0x7f1e2005f9e5]
>>
>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/features/shard.so(+0x12e59)[0x7f1e2007fe59]
>>
>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/features/shard.so(+0xc2c6)[0x7f1e200792c6]
>>
>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/cluster/distribute.so(+0x69e90)[0x7f1e197b8e90]
>>
>> /lib/x86_64-linux-gnu/libglusterfs.so.0(default_fxattrop_cbk+0x125)[0x7f1e23e27515]
>>
>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/cluster/disperse.so(+0x2421c)[0x7f1e200d621c]
>>
>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/cluster/disperse.so(+0x19414)[0x7f1e200cb414]
>>
>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/cluster/disperse.so(+0x16373)[0x7f1e200c8373]
>>
>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/cluster/disperse.so(+0x170f9)[0x7f1e200c90f9]
>>
>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/cluster/disperse.so(+0x1f929)[0x7f1e200d1929]
>>
>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/protocol/client.so(+0x469c2)[0x7f1e201859c2]
>>
>> /lib/x86_64-linux-gnu/libgfrpc.so.0(+0xfccb)[0x7f1e23d5eccb]
>> /lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_transport_notify+0x26)[0x7f1e23d5a646]
>>
>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/rpc-transport/socket.so(+0x64c8)[0x7f1e202784c8]
>>
>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/rpc-transport/socket.so(+0xd38c)[0x7f1e2027f38c]
>>
>> /lib/x86_64-linux-gnu/libglusterfs.so.0(+0x7971d)[0x7f1e23e0471d]
>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7)[0x7f1e23d1aea7]
>> /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f1e23c3aa2f]
>>
>> I'm still unable to gather any core dump.
>>
>> I can barely read something intelligible from all of this, but it's
>> clearly something happening with sharding here. So I'm going to empty all
>> the volume and destroy it completely and re-create another volume without
>> sharding, and see what happens
>>
>> *Angel Docampo*
>>
>> <https://www.google.com/maps/place/Edificio+de+Oficinas+Euro+3/@41.3755943,2.0730134,17z/data=!3m2!4b1!5s0x12a4997021aad323:0x3e06bf8ae6d68351!4m5!3m4!1s0x12a4997a67bf592f:0x83c2323a9cc2aa4b!8m2!3d41.3755903!4d2.0752021>
>> <angel.docampo at eoniantec.com> <+34-93-1592929>
>>
>>
>> El vie, 25 nov 2022 a las 19:08, Angel Docampo (<
>> angel.docampo at eoniantec.com>) escribió:
>>
>>> I did also notice about that loop0... AFAIK, I wasn't using any loop
>>> device, at least consciously.
>>> After looking for the same messages at the other gluster/proxmox nodes,
>>> I saw no trace of it.
>>> Then I saw on that node, there is a single LXC container, which disk is
>>> living on the glusterfs, and effectively, is using ext4.
>>> After the crash of today, I was unable to boot it up again, and the logs
>>> became silent, I did just try to boot it up, and immediately appeared this
>>> on dmesg
>>> [2022-11-25 18:04:18] loop0: detected capacity change from 0 to
>>> 16777216
>>> [2022-11-25 18:04:18] EXT4-fs (loop0): error loading journal
>>> [2022-11-25 18:05:26] loop0: detected capacity change from 0 to
>>> 16777216
>>> [2022-11-25 18:05:26] EXT4-fs (loop0): INFO: recovery required on
>>> readonly filesystem
>>> [2022-11-25 18:05:26] EXT4-fs (loop0): write access unavailable, cannot
>>> proceed (try mounting with noload)
>>>
>>> And the LXC container didn't boot up. I've manually moved the LXC
>>> container to the underlying ZFS where gluster lives, and the LXC booted up
>>> and the dmesg log shows
>>> [2022-11-25 18:24:06] loop0: detected capacity change from 0 to
>>> 16777216
>>> [2022-11-25 18:24:06] EXT4-fs warning (device loop0):
>>> ext4_multi_mount_protect:326: MMP interval 42 higher than expected, please
>>> wait.
>>> [2022-11-25 18:24:50] EXT4-fs (loop0): mounted filesystem with ordered
>>> data mode. Opts: (null). Quota mode: none.
>>>
>>> So, to recapitulate:
>>> - the loop device on the host relies on the LXC, is not surprising, but
>>> I didn't know it.
>>> - the LXC container had a lot of I/O issues just before the two crashes,
>>> the crash from today, and the crash 4 days ago, this Monday
>>> - as side note, this gluster is in production since last Thursday, so
>>> the first crash was exactly 4 days since this LXC was started with the
>>> storage on the gluster, and exactly 4 days after, it crashed again.
>>> - this crashes began to happen since the upgrade to gluster 10.3,
>>> because it was working just fine with former versions of gluster (from 3.X
>>> to 9.X), and from proxmox 5.X to proxmox 7.1, when all the issues begun,
>>> now I'm on proxmox 7.2.
>>> - underlying ZFS where gluster is, has no ZIL or ZLOG (it had before the
>>> upgrade to gluster 10.3, but as I had to re-create the gluster, I decided
>>> not to add them because all my disks are SSD, so there is no need to add
>>> any of those), I've added them to test if the LXC container caused the same
>>> issues, it did, so they don't seem to make any difference.
>>> - there are more loop0 I/O errors on the dmesg besides the days of the
>>> crash, but there are just "one" error per day, and not all days, but the
>>> days gluster mountpoint become inaccessible, there are tens of errors per
>>> millisecond just before the crash
>>>
>>> I'm going to get rid of that LXC, as now I'm migrating from VMs to K8s
>>> (living in a VM cluster inside proxmox), I was ready to convert this as
>>> well, now is a must.
>>>
>>> I don't know if anyone at gluster can replicate this scenario (proxmox +
>>> gluster distributed disperse + LXC on a gluster directory), to see if it
>>> can be reproducible. I know this must be a corner case, just wondering why
>>> stopped working, if it is a bug on GlusterFS 10.3, a bug in LXC or in
>>> Proxmox 7.1 upwards (where I'm going to post this now, but Proxmox probably
>>> won't be interested as they explicitly suggest mounting glusterfs with the
>>> gluster client, and not to map a directory where gluster is mounted via
>>> fstab)
>>>
>>> Thank you a lot Xavi, I will monitor dmesg to make sure all those loop
>>> errors disappear, and hopefully I won't have a crash next Tuesday. :)
>>>
>>> *Angel Docampo*
>>>
>>> <https://www.google.com/maps/place/Edificio+de+Oficinas+Euro+3/@41.3755943,2.0730134,17z/data=!3m2!4b1!5s0x12a4997021aad323:0x3e06bf8ae6d68351!4m5!3m4!1s0x12a4997a67bf592f:0x83c2323a9cc2aa4b!8m2!3d41.3755903!4d2.0752021>
>>> <angel.docampo at eoniantec.com> <+34-93-1592929>
>>>
>>>
>>> El vie, 25 nov 2022 a las 13:25, Xavi Hernandez (<jahernan at redhat.com>)
>>> escribió:
>>>
>>>> What is "loop0" it seems it's having some issue. Does it point to a
>>>> Gluster file ?
>>>>
>>>> I also see that there's an io_uring thread in D state. If that one
>>>> belongs to Gluster, it may explain why systemd was unable to generate a
>>>> core dump (all threads need to be stopped to generate a core dump, but a
>>>> thread blocked inside the kernel cannot be stopped).
>>>>
>>>> If you are using io_uring in Gluster, maybe you can disable it to see
>>>> if it's related.
>>>>
>>>> Xavi
>>>>
>>>> On Fri, Nov 25, 2022 at 11:39 AM Angel Docampo <
>>>> angel.docampo at eoniantec.com> wrote:
>>>>
>>>>> Well, just happened again, the same server, the same mountpoint.
>>>>>
>>>>> I'm unable to get the core dumps, coredumpctl says there are no core
>>>>> dumps, it would be funny if I wasn't the one suffering it, but
>>>>> systemd-coredump service crashed as well
>>>>> ● systemd-coredump at 0-3199871-0.service - Process Core Dump (PID
>>>>> 3199871/UID 0)
>>>>> Loaded: loaded (/lib/systemd/system/systemd-coredump at .service;
>>>>> static)
>>>>> Active: failed (Result: timeout) since Fri 2022-11-25 10:54:59
>>>>> CET; 39min ago
>>>>> TriggeredBy: ● systemd-coredump.socket
>>>>> Docs: man:systemd-coredump(8)
>>>>> Process: 3199873 ExecStart=/lib/systemd/systemd-coredump
>>>>> (code=killed, signal=TERM)
>>>>> Main PID: 3199873 (code=killed, signal=TERM)
>>>>> CPU: 15ms
>>>>>
>>>>> Nov 25 10:49:59 pve02 systemd[1]: Started Process Core Dump (PID
>>>>> 3199871/UID 0).
>>>>> Nov 25 10:54:59 pve02 systemd[1]: systemd-coredump at 0-3199871-0.service:
>>>>> Service reached runtime time limit. Stopping.
>>>>> Nov 25 10:54:59 pve02 systemd[1]: systemd-coredump at 0-3199871-0.service:
>>>>> Failed with result 'timeout'.
>>>>>
>>>>>
>>>>> I just saw the exception on dmesg,
>>>>> [2022-11-25 10:50:08] INFO: task kmmpd-loop0:681644 blocked for more
>>>>> than 120 seconds.
>>>>> [2022-11-25 10:50:08] Tainted: P IO 5.15.60-2-pve
>>>>> #1
>>>>> [2022-11-25 10:50:08] "echo 0 >
>>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>> [2022-11-25 10:50:08] task:kmmpd-loop0 state:D stack: 0
>>>>> pid:681644 ppid: 2 flags:0x00004000
>>>>> [2022-11-25 10:50:08] Call Trace:
>>>>> [2022-11-25 10:50:08] <TASK>
>>>>> [2022-11-25 10:50:08] __schedule+0x33d/0x1750
>>>>> [2022-11-25 10:50:08] ? bit_wait+0x70/0x70
>>>>> [2022-11-25 10:50:08] schedule+0x4e/0xc0
>>>>> [2022-11-25 10:50:08] io_schedule+0x46/0x80
>>>>> [2022-11-25 10:50:08] bit_wait_io+0x11/0x70
>>>>> [2022-11-25 10:50:08] __wait_on_bit+0x31/0xa0
>>>>> [2022-11-25 10:50:08] out_of_line_wait_on_bit+0x8d/0xb0
>>>>> [2022-11-25 10:50:08] ? var_wake_function+0x30/0x30
>>>>> [2022-11-25 10:50:08] __wait_on_buffer+0x34/0x40
>>>>> [2022-11-25 10:50:08] write_mmp_block+0x127/0x180
>>>>> [2022-11-25 10:50:08] kmmpd+0x1b9/0x430
>>>>> [2022-11-25 10:50:08] ? write_mmp_block+0x180/0x180
>>>>> [2022-11-25 10:50:08] kthread+0x127/0x150
>>>>> [2022-11-25 10:50:08] ? set_kthread_struct+0x50/0x50
>>>>> [2022-11-25 10:50:08] ret_from_fork+0x1f/0x30
>>>>> [2022-11-25 10:50:08] </TASK>
>>>>> [2022-11-25 10:50:08] INFO: task iou-wrk-1511979:3200401 blocked for
>>>>> more than 120 seconds.
>>>>> [2022-11-25 10:50:08] Tainted: P IO 5.15.60-2-pve
>>>>> #1
>>>>> [2022-11-25 10:50:08] "echo 0 >
>>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>> [2022-11-25 10:50:08] task:iou-wrk-1511979 state:D stack: 0
>>>>> pid:3200401 ppid: 1 flags:0x00004000
>>>>> [2022-11-25 10:50:08] Call Trace:
>>>>> [2022-11-25 10:50:08] <TASK>
>>>>> [2022-11-25 10:50:08] __schedule+0x33d/0x1750
>>>>> [2022-11-25 10:50:08] schedule+0x4e/0xc0
>>>>> [2022-11-25 10:50:08] rwsem_down_write_slowpath+0x231/0x4f0
>>>>> [2022-11-25 10:50:08] down_write+0x47/0x60
>>>>> [2022-11-25 10:50:08] fuse_file_write_iter+0x1a3/0x430
>>>>> [2022-11-25 10:50:08] ? apparmor_file_permission+0x70/0x170
>>>>> [2022-11-25 10:50:08] io_write+0xfb/0x320
>>>>> [2022-11-25 10:50:08] ? put_dec+0x1c/0xa0
>>>>> [2022-11-25 10:50:08] io_issue_sqe+0x401/0x1fc0
>>>>> [2022-11-25 10:50:08] io_wq_submit_work+0x76/0xd0
>>>>> [2022-11-25 10:50:08] io_worker_handle_work+0x1a7/0x5f0
>>>>> [2022-11-25 10:50:08] io_wqe_worker+0x2c0/0x360
>>>>> [2022-11-25 10:50:08] ? finish_task_switch.isra.0+0x7e/0x2b0
>>>>> [2022-11-25 10:50:08] ? io_worker_handle_work+0x5f0/0x5f0
>>>>> [2022-11-25 10:50:08] ? io_worker_handle_work+0x5f0/0x5f0
>>>>> [2022-11-25 10:50:08] ret_from_fork+0x1f/0x30
>>>>> [2022-11-25 10:50:08] RIP: 0033:0x0
>>>>> [2022-11-25 10:50:08] RSP: 002b:0000000000000000 EFLAGS: 00000216
>>>>> ORIG_RAX: 00000000000001aa
>>>>> [2022-11-25 10:50:08] RAX: 0000000000000000 RBX: 00007fdb1efef640
>>>>> RCX: 00007fdd59f872e9
>>>>> [2022-11-25 10:50:08] RDX: 0000000000000000 RSI: 0000000000000001
>>>>> RDI: 0000000000000011
>>>>> [2022-11-25 10:50:08] RBP: 0000000000000000 R08: 0000000000000000
>>>>> R09: 0000000000000008
>>>>> [2022-11-25 10:50:08] R10: 0000000000000000 R11: 0000000000000216
>>>>> R12: 000055662e5bd268
>>>>> [2022-11-25 10:50:08] R13: 000055662e5bd320 R14: 000055662e5bd260
>>>>> R15: 0000000000000000
>>>>> [2022-11-25 10:50:08] </TASK>
>>>>> [2022-11-25 10:52:08] INFO: task kmmpd-loop0:681644 blocked for more
>>>>> than 241 seconds.
>>>>> [2022-11-25 10:52:08] Tainted: P IO 5.15.60-2-pve
>>>>> #1
>>>>> [2022-11-25 10:52:08] "echo 0 >
>>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>> [2022-11-25 10:52:08] task:kmmpd-loop0 state:D stack: 0
>>>>> pid:681644 ppid: 2 flags:0x00004000
>>>>> [2022-11-25 10:52:08] Call Trace:
>>>>> [2022-11-25 10:52:08] <TASK>
>>>>> [2022-11-25 10:52:08] __schedule+0x33d/0x1750
>>>>> [2022-11-25 10:52:08] ? bit_wait+0x70/0x70
>>>>> [2022-11-25 10:52:08] schedule+0x4e/0xc0
>>>>> [2022-11-25 10:52:08] io_schedule+0x46/0x80
>>>>> [2022-11-25 10:52:08] bit_wait_io+0x11/0x70
>>>>> [2022-11-25 10:52:08] __wait_on_bit+0x31/0xa0
>>>>> [2022-11-25 10:52:08] out_of_line_wait_on_bit+0x8d/0xb0
>>>>> [2022-11-25 10:52:08] ? var_wake_function+0x30/0x30
>>>>> [2022-11-25 10:52:08] __wait_on_buffer+0x34/0x40
>>>>> [2022-11-25 10:52:08] write_mmp_block+0x127/0x180
>>>>> [2022-11-25 10:52:08] kmmpd+0x1b9/0x430
>>>>> [2022-11-25 10:52:08] ? write_mmp_block+0x180/0x180
>>>>> [2022-11-25 10:52:08] kthread+0x127/0x150
>>>>> [2022-11-25 10:52:08] ? set_kthread_struct+0x50/0x50
>>>>> [2022-11-25 10:52:08] ret_from_fork+0x1f/0x30
>>>>> [2022-11-25 10:52:08] </TASK>
>>>>> [2022-11-25 10:52:08] INFO: task iou-wrk-1511979:3200401 blocked for
>>>>> more than 241 seconds.
>>>>> [2022-11-25 10:52:08] Tainted: P IO 5.15.60-2-pve
>>>>> #1
>>>>> [2022-11-25 10:52:08] "echo 0 >
>>>>> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>>>> [2022-11-25 10:52:08] task:iou-wrk-1511979 state:D stack: 0
>>>>> pid:3200401 ppid: 1 flags:0x00004000
>>>>> [2022-11-25 10:52:08] Call Trace:
>>>>> [2022-11-25 10:52:08] <TASK>
>>>>> [2022-11-25 10:52:08] __schedule+0x33d/0x1750
>>>>> [2022-11-25 10:52:08] schedule+0x4e/0xc0
>>>>> [2022-11-25 10:52:08] rwsem_down_write_slowpath+0x231/0x4f0
>>>>> [2022-11-25 10:52:08] down_write+0x47/0x60
>>>>> [2022-11-25 10:52:08] fuse_file_write_iter+0x1a3/0x430
>>>>> [2022-11-25 10:52:08] ? apparmor_file_permission+0x70/0x170
>>>>> [2022-11-25 10:52:08] io_write+0xfb/0x320
>>>>> [2022-11-25 10:52:08] ? put_dec+0x1c/0xa0
>>>>> [2022-11-25 10:52:08] io_issue_sqe+0x401/0x1fc0
>>>>> [2022-11-25 10:52:08] io_wq_submit_work+0x76/0xd0
>>>>> [2022-11-25 10:52:08] io_worker_handle_work+0x1a7/0x5f0
>>>>> [2022-11-25 10:52:08] io_wqe_worker+0x2c0/0x360
>>>>> [2022-11-25 10:52:08] ? finish_task_switch.isra.0+0x7e/0x2b0
>>>>> [2022-11-25 10:52:08] ? io_worker_handle_work+0x5f0/0x5f0
>>>>> [2022-11-25 10:52:08] ? io_worker_handle_work+0x5f0/0x5f0
>>>>> [2022-11-25 10:52:08] ret_from_fork+0x1f/0x30
>>>>> [2022-11-25 10:52:08] RIP: 0033:0x0
>>>>> [2022-11-25 10:52:08] RSP: 002b:0000000000000000 EFLAGS: 00000216
>>>>> ORIG_RAX: 00000000000001aa
>>>>> [2022-11-25 10:52:08] RAX: 0000000000000000 RBX: 00007fdb1efef640
>>>>> RCX: 00007fdd59f872e9
>>>>> [2022-11-25 10:52:08] RDX: 0000000000000000 RSI: 0000000000000001
>>>>> RDI: 0000000000000011
>>>>> [2022-11-25 10:52:08] RBP: 0000000000000000 R08: 0000000000000000
>>>>> R09: 0000000000000008
>>>>> [2022-11-25 10:52:08] R10: 0000000000000000 R11: 0000000000000216
>>>>> R12: 000055662e5bd268
>>>>> [2022-11-25 10:52:08] R13: 000055662e5bd320 R14: 000055662e5bd260
>>>>> R15: 0000000000000000
>>>>> [2022-11-25 10:52:08] </TASK>
>>>>> [2022-11-25 10:52:12] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:52:12] print_req_error: 7 callbacks suppressed
>>>>> [2022-11-25 10:52:12] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:52:12] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:52:12] EXT4-fs error (device loop0): kmmpd:179: comm
>>>>> kmmpd-loop0: Error writing to MMP block
>>>>> [2022-11-25 10:52:12] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:52:12] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:52:12] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:52:18] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:52:18] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:52:18] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:52:18] loop: Write error at byte offset 4490452992,
>>>>> length 4096.
>>>>> [2022-11-25 10:52:18] loop: Write error at byte offset 4490457088,
>>>>> length 4096.
>>>>> [2022-11-25 10:52:18] blk_update_request: I/O error, dev loop0,
>>>>> sector 8770416 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:52:18] blk_update_request: I/O error, dev loop0,
>>>>> sector 8770424 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:52:18] Aborting journal on device loop0-8.
>>>>> [2022-11-25 10:52:18] loop: Write error at byte offset 4429185024,
>>>>> length 4096.
>>>>> [2022-11-25 10:52:18] blk_update_request: I/O error, dev loop0,
>>>>> sector 8650752 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:52:18] blk_update_request: I/O error, dev loop0,
>>>>> sector 8650752 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:52:18] Buffer I/O error on dev loop0, logical block
>>>>> 1081344, lost sync page write
>>>>> [2022-11-25 10:52:18] JBD2: Error -5 detected when updating journal
>>>>> superblock for loop0-8.
>>>>> [2022-11-25 10:52:23] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:52:23] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:52:23] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:52:28] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:52:28] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:52:28] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:52:33] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:52:33] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:52:33] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:52:38] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:52:38] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:52:38] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:52:43] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:52:43] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:52:43] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:52:48] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:52:48] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:52:48] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:52:53] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:52:53] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:52:53] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:52:59] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:52:59] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:52:59] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:53:04] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:53:04] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:53:04] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:53:09] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:53:09] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:53:09] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:53:14] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:53:14] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:53:14] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:53:19] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:53:19] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:53:19] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:53:24] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:53:24] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:53:24] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:53:29] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:53:29] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:53:29] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:53:34] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:53:34] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:53:34] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:53:40] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:53:40] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:53:40] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:53:45] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:53:45] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:53:45] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:53:50] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:53:50] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:53:50] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:53:55] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:53:55] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:53:55] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:54:00] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:54:00] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:54:00] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:54:05] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:54:05] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:54:05] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:54:10] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:54:10] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:54:10] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:54:15] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:54:15] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:54:15] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:54:21] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:54:21] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:54:21] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:54:26] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:54:26] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:54:26] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:54:31] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:54:31] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:54:31] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:54:36] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:54:36] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:54:36] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:54:41] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:54:41] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:54:41] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:54:46] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:54:46] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:54:46] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:54:51] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:54:51] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:54:51] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:54:56] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:54:56] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:54:56] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:55:01] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:55:01] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:55:01] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:55:04] EXT4-fs error (device loop0):
>>>>> ext4_journal_check_start:83: comm burp: Detected aborted journal
>>>>> [2022-11-25 10:55:04] loop: Write error at byte offset 0, length
>>>>> 4096.
>>>>> [2022-11-25 10:55:04] blk_update_request: I/O error, dev loop0,
>>>>> sector 0 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:55:04] blk_update_request: I/O error, dev loop0,
>>>>> sector 0 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:55:04] Buffer I/O error on dev loop0, logical block 0,
>>>>> lost sync page write
>>>>> [2022-11-25 10:55:04] EXT4-fs (loop0): I/O error while writing
>>>>> superblock
>>>>> [2022-11-25 10:55:04] EXT4-fs (loop0): Remounting filesystem
>>>>> read-only
>>>>> [2022-11-25 10:55:07] loop: Write error at byte offset 37908480,
>>>>> length 4096.
>>>>> [2022-11-25 10:55:07] blk_update_request: I/O error, dev loop0,
>>>>> sector 74040 op 0x1:(WRITE) flags 0x3800 phys_seg 1 prio class 0
>>>>> [2022-11-25 10:55:07] Buffer I/O error on dev loop0, logical block
>>>>> 9255, lost sync page write
>>>>> [2022-11-25 10:57:14] blk_update_request: I/O error, dev loop0,
>>>>> sector 16390368 op 0x0:(READ) flags 0x80700 phys_seg 6 prio class 0
>>>>> [2022-11-25 11:03:45] device tap136i0 entered promiscuous mode
>>>>>
>>>>> I don't know if it is relevant somehow or it is unrelated to
>>>>> glusterfs, but the consequences are the mountpoint crashes, I'm forced to
>>>>> lazy unmount it and remount it back. Then restart all the VMs on there,
>>>>> unfortunately, this time several have the hard disk corrupted and now I'm
>>>>> restoring them from the backup.
>>>>>
>>>>> Any tip?
>>>>>
>>>>> *Angel Docampo*
>>>>>
>>>>> <https://www.google.com/maps/place/Edificio+de+Oficinas+Euro+3/@41.3755943,2.0730134,17z/data=!3m2!4b1!5s0x12a4997021aad323:0x3e06bf8ae6d68351!4m5!3m4!1s0x12a4997a67bf592f:0x83c2323a9cc2aa4b!8m2!3d41.3755903!4d2.0752021>
>>>>> <angel.docampo at eoniantec.com> <+34-93-1592929>
>>>>>
>>>>>
>>>>> El mar, 22 nov 2022 a las 12:31, Angel Docampo (<
>>>>> angel.docampo at eoniantec.com>) escribió:
>>>>>
>>>>>> I've taken a look into all possible places they should be, and I
>>>>>> couldn't find it anywhere. Some people say the dump file is generated where
>>>>>> the application is running... well, I don't know where to look then, and I
>>>>>> hope they hadn't been generated on the failed mountpoint.
>>>>>>
>>>>>> As Debian 11 has systemd, I've installed systemd-coredump, so in the
>>>>>> case a new crash happens, at least I will have the exact location and tool
>>>>>> (coredumpctl) to find them and will install then the debug symbols, which
>>>>>> is particularly tricky on debian. But I need to wait to happen again, now
>>>>>> the tool says there isn't any core dump on the system.
>>>>>>
>>>>>> Thank you, Xavi, if this happens again (let's hope it won't), I will
>>>>>> report back.
>>>>>>
>>>>>> Best regards!
>>>>>>
>>>>>> *Angel Docampo*
>>>>>>
>>>>>> <https://www.google.com/maps/place/Edificio+de+Oficinas+Euro+3/@41.3755943,2.0730134,17z/data=!3m2!4b1!5s0x12a4997021aad323:0x3e06bf8ae6d68351!4m5!3m4!1s0x12a4997a67bf592f:0x83c2323a9cc2aa4b!8m2!3d41.3755903!4d2.0752021>
>>>>>> <angel.docampo at eoniantec.com> <+34-93-1592929>
>>>>>>
>>>>>>
>>>>>> El mar, 22 nov 2022 a las 10:45, Xavi Hernandez (<jahernan at redhat.com>)
>>>>>> escribió:
>>>>>>
>>>>>>> The crash seems related to some problem in ec xlator, but I don't
>>>>>>> have enough information to determine what it is. The crash should have
>>>>>>> generated a core dump somewhere in the system (I don't know where Debian
>>>>>>> keeps the core dumps). If you find it, you should be able to open it using
>>>>>>> this command (make sure debug symbols package is also installed before
>>>>>>> running it):
>>>>>>>
>>>>>>> # gdb /usr/sbin/glusterfs <path to core dump>
>>>>>>>
>>>>>>> And then run this command:
>>>>>>>
>>>>>>> # bt -full
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Xavi
>>>>>>>
>>>>>>> On Tue, Nov 22, 2022 at 9:41 AM Angel Docampo <
>>>>>>> angel.docampo at eoniantec.com> wrote:
>>>>>>>
>>>>>>>> Hi Xavi,
>>>>>>>>
>>>>>>>> The OS is Debian 11 with the proxmox kernel. Gluster packages are
>>>>>>>> the official from gluster.org (
>>>>>>>> https://download.gluster.org/pub/gluster/glusterfs/10/10.3/Debian/bullseye/
>>>>>>>> )
>>>>>>>>
>>>>>>>> The system logs showed no other issues by the time of the crash, no
>>>>>>>> OOM kill or whatsoever, and no other process was interacting with the
>>>>>>>> gluster mountpoint besides proxmox.
>>>>>>>>
>>>>>>>> I wasn't running gdb when it crashed, so I don't really know if I
>>>>>>>> can obtain a more detailed trace from logs or if there is a simple way to
>>>>>>>> let it running in the background to see if it happens again (or there is a
>>>>>>>> flag to start the systemd daemon in debug mode).
>>>>>>>>
>>>>>>>> Best,
>>>>>>>>
>>>>>>>> *Angel Docampo*
>>>>>>>>
>>>>>>>> <https://www.google.com/maps/place/Edificio+de+Oficinas+Euro+3/@41.3755943,2.0730134,17z/data=!3m2!4b1!5s0x12a4997021aad323:0x3e06bf8ae6d68351!4m5!3m4!1s0x12a4997a67bf592f:0x83c2323a9cc2aa4b!8m2!3d41.3755903!4d2.0752021>
>>>>>>>> <angel.docampo at eoniantec.com> <+34-93-1592929>
>>>>>>>>
>>>>>>>>
>>>>>>>> El lun, 21 nov 2022 a las 15:16, Xavi Hernandez (<
>>>>>>>> jahernan at redhat.com>) escribió:
>>>>>>>>
>>>>>>>>> Hi Angel,
>>>>>>>>>
>>>>>>>>> On Mon, Nov 21, 2022 at 2:33 PM Angel Docampo <
>>>>>>>>> angel.docampo at eoniantec.com> wrote:
>>>>>>>>>
>>>>>>>>>> Sorry for necrobumping this, but this morning I've suffered this
>>>>>>>>>> on my Proxmox + GlusterFS cluster. In the log I can see this
>>>>>>>>>>
>>>>>>>>>> [2022-11-21 07:38:00.213620 +0000] I [MSGID: 133017]
>>>>>>>>>> [shard.c:7275:shard_seek] 11-vmdata-shard: seek called on
>>>>>>>>>> fbc063cb-874e-475d-b585-f89
>>>>>>>>>> f7518acdd. [Operation not supported]
>>>>>>>>>> pending frames:
>>>>>>>>>> frame : type(1) op(WRITE)
>>>>>>>>>> frame : type(0) op(0)
>>>>>>>>>> frame : type(0) op(0)
>>>>>>>>>> frame : type(0) op(0)
>>>>>>>>>> frame : type(0) op(0)
>>>>>>>>>> frame : type(0) op(0)
>>>>>>>>>> frame : type(0) op(0)
>>>>>>>>>> frame : type(0) op(0)
>>>>>>>>>> frame : type(0) op(0)
>>>>>>>>>> frame : type(0) op(0)
>>>>>>>>>> frame : type(0) op(0)
>>>>>>>>>> frame : type(0) op(0)
>>>>>>>>>> frame : type(0) op(0)
>>>>>>>>>> frame : type(0) op(0)
>>>>>>>>>> frame : type(0) op(0)
>>>>>>>>>> frame : type(0) op(0)
>>>>>>>>>> frame : type(0) op(0)
>>>>>>>>>> ...
>>>>>>>>>> frame : type(1) op(FSYNC)
>>>>>>>>>> frame : type(1) op(FSYNC)
>>>>>>>>>> frame : type(1) op(FSYNC)
>>>>>>>>>> frame : type(1) op(FSYNC)
>>>>>>>>>> frame : type(1) op(FSYNC)
>>>>>>>>>> frame : type(1) op(FSYNC)
>>>>>>>>>> frame : type(1) op(FSYNC)
>>>>>>>>>> frame : type(1) op(FSYNC)
>>>>>>>>>> frame : type(1) op(FSYNC)
>>>>>>>>>> frame : type(1) op(FSYNC)
>>>>>>>>>> frame : type(1) op(FSYNC)
>>>>>>>>>> frame : type(1) op(FSYNC)
>>>>>>>>>> frame : type(1) op(FSYNC)
>>>>>>>>>> frame : type(1) op(FSYNC)
>>>>>>>>>> frame : type(1) op(FSYNC)
>>>>>>>>>> frame : type(1) op(FSYNC)
>>>>>>>>>> frame : type(1) op(FSYNC)
>>>>>>>>>> frame : type(1) op(FSYNC)
>>>>>>>>>> frame : type(1) op(FSYNC)
>>>>>>>>>> frame : type(1) op(FSYNC)
>>>>>>>>>> patchset: git://git.gluster.org/glusterfs.git
>>>>>>>>>> signal received: 11
>>>>>>>>>> time of crash:
>>>>>>>>>> 2022-11-21 07:38:00 +0000
>>>>>>>>>> configuration details:
>>>>>>>>>> argp 1
>>>>>>>>>> backtrace 1
>>>>>>>>>> dlfcn 1
>>>>>>>>>> libpthread 1
>>>>>>>>>> llistxattr 1
>>>>>>>>>> setfsid 1
>>>>>>>>>> epoll.h 1
>>>>>>>>>> xattr.h 1
>>>>>>>>>> st_atim.tv_nsec 1
>>>>>>>>>> package-string: glusterfs 10.3
>>>>>>>>>> /lib/x86_64-linux-gnu/libglusterfs.so.0(+0x28a54)[0x7f74f286ba54]
>>>>>>>>>> /lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_trace+0x700)[0x7f74f2873fc0]
>>>>>>>>>>
>>>>>>>>>> /lib/x86_64-linux-gnu/libc.so.6(+0x38d60)[0x7f74f262ed60]
>>>>>>>>>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/cluster/disperse.so(+0x37a14)[0x7f74ecfcea14]
>>>>>>>>>>
>>>>>>>>>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/cluster/disperse.so(+0x19414)[0x7f74ecfb0414]
>>>>>>>>>>
>>>>>>>>>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/cluster/disperse.so(+0x16373)[0x7f74ecfad373]
>>>>>>>>>>
>>>>>>>>>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/cluster/disperse.so(+0x21d59)[0x7f74ecfb8d59]
>>>>>>>>>>
>>>>>>>>>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/cluster/disperse.so(+0x22815)[0x7f74ecfb9815]
>>>>>>>>>>
>>>>>>>>>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/cluster/disperse.so(+0x377d9)[0x7f74ecfce7d9]
>>>>>>>>>>
>>>>>>>>>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/cluster/disperse.so(+0x19414)[0x7f74ecfb0414]
>>>>>>>>>>
>>>>>>>>>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/cluster/disperse.so(+0x16373)[0x7f74ecfad373]
>>>>>>>>>>
>>>>>>>>>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/cluster/disperse.so(+0x170f9)[0x7f74ecfae0f9]
>>>>>>>>>>
>>>>>>>>>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/cluster/disperse.so(+0x313bb)[0x7f74ecfc83bb]
>>>>>>>>>>
>>>>>>>>>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/xlator/protocol/client.so(+0x48e3a)[0x7f74ed06ce3a]
>>>>>>>>>>
>>>>>>>>>> /lib/x86_64-linux-gnu/libgfrpc.so.0(+0xfccb)[0x7f74f2816ccb]
>>>>>>>>>> /lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_transport_notify+0x26)[0x7f74f2812646]
>>>>>>>>>>
>>>>>>>>>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/rpc-transport/socket.so(+0x64c8)[0x7f74ee15f4c8]
>>>>>>>>>>
>>>>>>>>>> /usr/lib/x86_64-linux-gnu/glusterfs/10.3/rpc-transport/socket.so(+0xd38c)[0x7f74ee16638c]
>>>>>>>>>>
>>>>>>>>>> /lib/x86_64-linux-gnu/libglusterfs.so.0(+0x7971d)[0x7f74f28bc71d]
>>>>>>>>>> /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7)[0x7f74f27d2ea7]
>>>>>>>>>> /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f74f26f2aef]
>>>>>>>>>> ---------
>>>>>>>>>> The mount point wasn't accessible with the "Transport endpoint
>>>>>>>>>> is not connected" message and it was shown like this.
>>>>>>>>>> d????????? ? ? ? ? ? vmdata
>>>>>>>>>>
>>>>>>>>>> I had to stop all the VMs on that proxmox node, then stop the
>>>>>>>>>> gluster daemon to ummount de directory, and after starting the daemon and
>>>>>>>>>> re-mounting, all was working again.
>>>>>>>>>>
>>>>>>>>>> My gluster volume info returns this
>>>>>>>>>>
>>>>>>>>>> Volume Name: vmdata
>>>>>>>>>> Type: Distributed-Disperse
>>>>>>>>>> Volume ID: cace5aa4-b13a-4750-8736-aa179c2485e1
>>>>>>>>>> Status: Started
>>>>>>>>>> Snapshot Count: 0
>>>>>>>>>> Number of Bricks: 2 x (2 + 1) = 6
>>>>>>>>>> Transport-type: tcp
>>>>>>>>>> Bricks:
>>>>>>>>>> Brick1: g01:/data/brick1/brick
>>>>>>>>>> Brick2: g02:/data/brick2/brick
>>>>>>>>>> Brick3: g03:/data/brick1/brick
>>>>>>>>>> Brick4: g01:/data/brick2/brick
>>>>>>>>>> Brick5: g02:/data/brick1/brick
>>>>>>>>>> Brick6: g03:/data/brick2/brick
>>>>>>>>>> Options Reconfigured:
>>>>>>>>>> nfs.disable: on
>>>>>>>>>> transport.address-family: inet
>>>>>>>>>> storage.fips-mode-rchecksum: on
>>>>>>>>>> features.shard: enable
>>>>>>>>>> features.shard-block-size: 256MB
>>>>>>>>>> performance.read-ahead: off
>>>>>>>>>> performance.quick-read: off
>>>>>>>>>> performance.io-cache: off
>>>>>>>>>> server.event-threads: 2
>>>>>>>>>> client.event-threads: 3
>>>>>>>>>> performance.client-io-threads: on
>>>>>>>>>> performance.stat-prefetch: off
>>>>>>>>>> dht.force-readdirp: off
>>>>>>>>>> performance.force-readdirp: off
>>>>>>>>>> network.remote-dio: on
>>>>>>>>>> features.cache-invalidation: on
>>>>>>>>>> performance.parallel-readdir: on
>>>>>>>>>> performance.readdir-ahead: on
>>>>>>>>>>
>>>>>>>>>> Xavi, do you think the open-behind off setting can help somehow?
>>>>>>>>>> I did try to understand what it does (with no luck), and if it could impact
>>>>>>>>>> the performance of my VMs (I've the setup you know so well ;))
>>>>>>>>>> I would like to avoid more crashings like this, version 10.3 of
>>>>>>>>>> gluster was working since two weeks ago, quite well until this morning.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I don't think disabling open-behind will have any visible effect
>>>>>>>>> on performance. Open-behind is only useful for small files when the
>>>>>>>>> workload is mostly open + read + close, and quick-read is also enabled
>>>>>>>>> (which is not your case). The only effect it will have is that the latency
>>>>>>>>> "saved" during open is "paid" on the next operation sent to the file, so
>>>>>>>>> the total overall latency should be the same. Additionally, VM workload
>>>>>>>>> doesn't open files frequently, so it shouldn't matter much in any case.
>>>>>>>>>
>>>>>>>>> That said, I'm not sure if the problem is the same in your case.
>>>>>>>>> Based on the stack of the crash, it seems an issue inside the disperse
>>>>>>>>> module.
>>>>>>>>>
>>>>>>>>> What OS are you using ? are you using official packages ? if so,
>>>>>>>>> which ones ?
>>>>>>>>>
>>>>>>>>> Is it possible to provide a backtrace from gdb ?
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Xavi
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> *Angel Docampo*
>>>>>>>>>>
>>>>>>>>>> <https://www.google.com/maps/place/Edificio+de+Oficinas+Euro+3/@41.3755943,2.0730134,17z/data=!3m2!4b1!5s0x12a4997021aad323:0x3e06bf8ae6d68351!4m5!3m4!1s0x12a4997a67bf592f:0x83c2323a9cc2aa4b!8m2!3d41.3755903!4d2.0752021>
>>>>>>>>>> <angel.docampo at eoniantec.com> <+34-93-1592929>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> El vie, 19 mar 2021 a las 2:10, David Cunningham (<
>>>>>>>>>> dcunningham at voisonics.com>) escribió:
>>>>>>>>>>
>>>>>>>>>>> Hi Xavi,
>>>>>>>>>>>
>>>>>>>>>>> Thank you for that information. We'll look at upgrading it.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, 12 Mar 2021 at 05:20, Xavi Hernandez <
>>>>>>>>>>> jahernan at redhat.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi David,
>>>>>>>>>>>>
>>>>>>>>>>>> with so little information it's hard to tell, but given that
>>>>>>>>>>>> there are several OPEN and UNLINK operations, it could be related to an
>>>>>>>>>>>> already fixed bug (in recent versions) in open-behind.
>>>>>>>>>>>>
>>>>>>>>>>>> You can try disabling open-behind with this command:
>>>>>>>>>>>>
>>>>>>>>>>>> # gluster volume set <volname> open-behind off
>>>>>>>>>>>>
>>>>>>>>>>>> But given the version you are using is very old and
>>>>>>>>>>>> unmaintained, I would recommend you to upgrade to 8.x at least.
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>>
>>>>>>>>>>>> Xavi
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Mar 10, 2021 at 5:10 AM David Cunningham <
>>>>>>>>>>>> dcunningham at voisonics.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>
>>>>>>>>>>>>> We have a GlusterFS 5.13 server which also mounts itself with
>>>>>>>>>>>>> the native FUSE client. Recently the FUSE mount crashed and we found the
>>>>>>>>>>>>> following in the syslog. There isn't anything logged in mnt-glusterfs.log
>>>>>>>>>>>>> for that time. After killing all processes with a file handle open on the
>>>>>>>>>>>>> filesystem we were able to unmount and then remount the filesystem
>>>>>>>>>>>>> successfully.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Would anyone have advice on how to debug this crash? Thank you
>>>>>>>>>>>>> in advance!
>>>>>>>>>>>>>
>>>>>>>>>>>>> Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: pending frames:
>>>>>>>>>>>>> Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: frame : type(0) op(0)
>>>>>>>>>>>>> Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: frame : type(0) op(0)
>>>>>>>>>>>>> Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: frame : type(1)
>>>>>>>>>>>>> op(UNLINK)
>>>>>>>>>>>>> Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: frame : type(1)
>>>>>>>>>>>>> op(UNLINK)
>>>>>>>>>>>>> Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: frame : type(1)
>>>>>>>>>>>>> op(OPEN)
>>>>>>>>>>>>> Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: message repeated
>>>>>>>>>>>>> 3355 times: [ frame : type(1) op(OPEN)]
>>>>>>>>>>>>> Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: frame : type(1)
>>>>>>>>>>>>> op(OPEN)
>>>>>>>>>>>>> Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: message repeated
>>>>>>>>>>>>> 6965 times: [ frame : type(1) op(OPEN)]
>>>>>>>>>>>>> Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: frame : type(1)
>>>>>>>>>>>>> op(OPEN)
>>>>>>>>>>>>> Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: message repeated
>>>>>>>>>>>>> 4095 times: [ frame : type(1) op(OPEN)]
>>>>>>>>>>>>> Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: frame : type(0) op(0)
>>>>>>>>>>>>> Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: patchset: git://
>>>>>>>>>>>>> git.gluster.org/glusterfs.git
>>>>>>>>>>>>> Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: signal received: 11
>>>>>>>>>>>>> Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: time of crash:
>>>>>>>>>>>>> Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: 2021-03-09 03:12:31
>>>>>>>>>>>>> Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: configuration
>>>>>>>>>>>>> details:
>>>>>>>>>>>>> Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: argp 1
>>>>>>>>>>>>> Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: backtrace 1
>>>>>>>>>>>>> Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: dlfcn 1
>>>>>>>>>>>>> Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: libpthread 1
>>>>>>>>>>>>> Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: llistxattr 1
>>>>>>>>>>>>> Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: setfsid 1
>>>>>>>>>>>>> Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: spinlock 1
>>>>>>>>>>>>> Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: epoll.h 1
>>>>>>>>>>>>> Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: xattr.h 1
>>>>>>>>>>>>> Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: st_atim.tv_nsec 1
>>>>>>>>>>>>> Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: package-string:
>>>>>>>>>>>>> glusterfs 5.13
>>>>>>>>>>>>> Mar 9 05:12:31 voip1 mnt-glusterfs[2932]: ---------
>>>>>>>>>>>>> ...
>>>>>>>>>>>>> Mar 9 05:13:50 voip1 systemd[1]:
>>>>>>>>>>>>> glusterfssharedstorage.service: Main process exited, code=killed,
>>>>>>>>>>>>> status=11/SEGV
>>>>>>>>>>>>> Mar 9 05:13:50 voip1 systemd[1]:
>>>>>>>>>>>>> glusterfssharedstorage.service: Failed with result 'signal'.
>>>>>>>>>>>>> ...
>>>>>>>>>>>>> Mar 9 05:13:54 voip1 systemd[1]:
>>>>>>>>>>>>> glusterfssharedstorage.service: Service hold-off time over, scheduling
>>>>>>>>>>>>> restart.
>>>>>>>>>>>>> Mar 9 05:13:54 voip1 systemd[1]:
>>>>>>>>>>>>> glusterfssharedstorage.service: Scheduled restart job, restart counter is
>>>>>>>>>>>>> at 2.
>>>>>>>>>>>>> Mar 9 05:13:54 voip1 systemd[1]: Stopped Mount glusterfs
>>>>>>>>>>>>> sharedstorage.
>>>>>>>>>>>>> Mar 9 05:13:54 voip1 systemd[1]: Starting Mount glusterfs
>>>>>>>>>>>>> sharedstorage...
>>>>>>>>>>>>> Mar 9 05:13:54 voip1 mount-shared-storage.sh[20520]: ERROR:
>>>>>>>>>>>>> Mount point does not exist
>>>>>>>>>>>>> Mar 9 05:13:54 voip1 mount-shared-storage.sh[20520]: Please
>>>>>>>>>>>>> specify a mount point
>>>>>>>>>>>>> Mar 9 05:13:54 voip1 mount-shared-storage.sh[20520]: Usage:
>>>>>>>>>>>>> Mar 9 05:13:54 voip1 mount-shared-storage.sh[20520]: man 8
>>>>>>>>>>>>> /sbin/mount.glusterfs
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> David Cunningham, Voisonics Limited
>>>>>>>>>>>>> http://voisonics.com/
>>>>>>>>>>>>> USA: +1 213 221 1092
>>>>>>>>>>>>> New Zealand: +64 (0)28 2558 3782
>>>>>>>>>>>>> ________
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Community Meeting Calendar:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Schedule -
>>>>>>>>>>>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>>>>>>>>>>>> Bridge: https://meet.google.com/cpu-eiue-hvk
>>>>>>>>>>>>> Gluster-users mailing list
>>>>>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> David Cunningham, Voisonics Limited
>>>>>>>>>>> http://voisonics.com/
>>>>>>>>>>> USA: +1 213 221 1092
>>>>>>>>>>> New Zealand: +64 (0)28 2558 3782
>>>>>>>>>>> ________
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Community Meeting Calendar:
>>>>>>>>>>>
>>>>>>>>>>> Schedule -
>>>>>>>>>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>>>>>>>>>> Bridge: https://meet.google.com/cpu-eiue-hvk
>>>>>>>>>>> Gluster-users mailing list
>>>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>>>>
>>>>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20221202/cf5c1ee5/attachment.html>
More information about the Gluster-users
mailing list