[Gluster-devel] gluster 10.3: task glfs_fusenoti blocked for more than 120 seconds

[Mohit Agrawal]

I don't think the issue is on gluster side, it seems the issue is on kernel
side (possible deadlock in fuse_reverse_inval_entry)
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=bda9a71980e083699a0360963c0135657b73f47a

On Tue, May 2, 2023 at 5:48 PM Hu Bert <revirii at googlemail.com> wrote:

> Good morning,
>
> we've recently had some strange message in /var/log/syslog.
>
> System:
> debian bullseye, kernel 5.10.0-21-amd64 and 5.10.0-22-amd64
> gluster 10.3
>
> The message look like:
>
> Apr 27 13:30:18 piggy kernel: [24287.715229] INFO: task
> glfs_fusenoti:2787 blocked for more than 120 seconds.
> Apr 27 13:30:18 piggy kernel: [24287.715327]       Not tainted
> 5.10.0-22-amd64 #1 Debian 5.10.178-3
> Apr 27 13:30:18 piggy kernel: [24287.715419] "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Apr 27 13:30:18 piggy kernel: [24287.715575] task:glfs_fusenoti
> state:D stack:    0 pid: 2787 ppid:     1 flags:0x00000000
> Apr 27 13:30:18 piggy kernel: [24287.715734] Call Trace:
> Apr 27 13:30:18 piggy kernel: [24287.715847]  __schedule+0x282/0x870
> Apr 27 13:30:18 piggy kernel: [24287.715959]  schedule+0x46/0xb0
> Apr 27 13:30:18 piggy kernel: [24287.716073]
> rwsem_down_write_slowpath+0x257/0x4d0
> Apr 27 13:30:18 piggy kernel: [24287.716194]
> fuse_reverse_inval_entry+0x3b/0x1e0 [fuse]
>
> etc. Full excerpt here: https://pastebin.com/6gDHgh16
>
> This may cause our application to "hang". In the latest release i read
> something about fuse:
> https://docs.gluster.org/en/latest/release-notes/10.4/
> "Fix fuse concurrency problems"
>
> but i checked the tickets and wasn't able to find something.
>
> However, i upgraded gluster client 10.3 -> 10.4, hoping that this
> fixes the hang-issue.
>
> Has anyone seen these messages before?
>
>
> Best regards,
> Hubert
> -------
>
> Community Meeting Calendar:
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://meet.google.com/cpu-eiue-hvk
>
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20230502/f748ed4c/attachment.html>