[Gluster-users] blocking process on FUSE mount in directory which is using quota

[Nithya Balachandran]

Hi,

Please provide the following:

   1. gluster volume info
   2. statedump of the fuse process when it hangs


Thanks,
Nithya

On 9 August 2018 at 18:24, mabi <mabi at protonmail.ch> wrote:

> Hello,
>
> I recently upgraded my GlusterFS replica 2+1 (aribter) to version 3.12.12
> and now I see a weird behaviour on my client (using FUSE mount) where I
> have processes (PHP 5.6 FPM) trying to access a specific directory and then
> the process blocks. I can't kill the process either, not even with kill -9.
> I need to reboot the machine in order to get rid of these blocked processes.
>
> This directory has one particularity compared to the other directories it
> is that it has reached it's quota soft-limit as you can see here in the
> output of gluster volume quota list:
>
>                   Path                   Hard-limit  Soft-limit      Used
> Available  Soft-limit exceeded? Hard-limit exceeded?
> ------------------------------------------------------------
> -------------------------------------------------------------------
> /directory                          100.0GB     80%(80.0GB)   90.5GB
>  9.5GB             Yes                   No
>
> That does not mean that it is the quota's fault but it might be a hint
> where to start looking for... And by the way can someone explain me what
> the soft-limit does? or does it not do anything special?
>
> Here is an the linux stack of a blocking process on that directory which
> happened with a simple "ls -la":
>
> [Thu Aug  9 14:21:07 2018] INFO: task ls:2272 blocked for more than 120
> seconds.
> [Thu Aug  9 14:21:07 2018]       Not tainted 3.16.0-4-amd64 #1
> [Thu Aug  9 14:21:07 2018] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> disables this message.
> [Thu Aug  9 14:21:07 2018] ls              D ffff88017ef93200     0  2272
>  2268 0x00000004
> [Thu Aug  9 14:21:07 2018]  ffff88017653f490 0000000000000286
> 0000000000013200 ffff880174d7bfd8
> [Thu Aug  9 14:21:07 2018]  0000000000013200 ffff88017653f490
> ffff8800eeb3d5f0 ffff8800fefac800
> [Thu Aug  9 14:21:07 2018]  ffff880174d7bbe0 ffff8800eeb3d6d0
> ffff8800fefac800 ffff8800ffe1e1c0
> [Thu Aug  9 14:21:07 2018] Call Trace:
> [Thu Aug  9 14:21:07 2018]  [<ffffffffa00dc50d>] ?
> __fuse_request_send+0xbd/0x270 [fuse]
> [Thu Aug  9 14:21:07 2018]  [<ffffffff810abce0>] ?
> prepare_to_wait_event+0xf0/0xf0
> [Thu Aug  9 14:21:07 2018]  [<ffffffffa00e0791>] ?
> fuse_dentry_revalidate+0x181/0x300 [fuse]
> [Thu Aug  9 14:21:07 2018]  [<ffffffff811b944e>] ? lookup_fast+0x25e/0x2b0
> [Thu Aug  9 14:21:07 2018]  [<ffffffff811bacc5>] ?
> path_lookupat+0x155/0x780
> [Thu Aug  9 14:21:07 2018]  [<ffffffff81195715>] ?
> kmem_cache_alloc+0x75/0x480
> [Thu Aug  9 14:21:07 2018]  [<ffffffffa00dfca9>] ?
> fuse_getxattr+0xe9/0x150 [fuse]
> [Thu Aug  9 14:21:07 2018]  [<ffffffff811bb316>] ?
> filename_lookup+0x26/0xc0
> [Thu Aug  9 14:21:07 2018]  [<ffffffff811bf594>] ?
> user_path_at_empty+0x54/0x90
> [Thu Aug  9 14:21:07 2018]  [<ffffffff81193e08>] ?
> kmem_cache_free+0xd8/0x210
> [Thu Aug  9 14:21:07 2018]  [<ffffffff811bf59f>] ?
> user_path_at_empty+0x5f/0x90
> [Thu Aug  9 14:21:07 2018]  [<ffffffff811b3d46>] ? vfs_fstatat+0x46/0x90
> [Thu Aug  9 14:21:07 2018]  [<ffffffff811b421d>] ? SYSC_newlstat+0x1d/0x40
> [Thu Aug  9 14:21:07 2018]  [<ffffffff811d34b8>] ? SyS_lgetxattr+0x58/0x80
> [Thu Aug  9 14:21:07 2018]  [<ffffffff81525d0d>] ?
> system_call_fast_compare_end+0x10/0x15
>
>
> My 3 gluster nodes are all Debian 9 and my client Debian 8.
>
> Let me know if you need more information.
>
> Best regards,
> Mabi
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180809/4e9c0625/attachment.html>