<div dir="ltr">Hi,<div><br></div><div>Please provide the following:</div><div><ol><li>gluster volume info<br></li><li>statedump of the fuse process when it hangs</li></ol></div><div><br></div><div>Thanks,</div><div>Nithya</div></div><div class="gmail_extra"><br><div class="gmail_quote">On 9 August 2018 at 18:24, mabi <span dir="ltr"><<a href="mailto:mabi@protonmail.ch" target="_blank">mabi@protonmail.ch</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello,<br>
<br>
I recently upgraded my GlusterFS replica 2+1 (aribter) to version 3.12.12 and now I see a weird behaviour on my client (using FUSE mount) where I have processes (PHP 5.6 FPM) trying to access a specific directory and then the process blocks. I can't kill the process either, not even with kill -9. I need to reboot the machine in order to get rid of these blocked processes.<br>
<br>
This directory has one particularity compared to the other directories it is that it has reached it's quota soft-limit as you can see here in the output of gluster volume quota list:<br>
<br>
Path Hard-limit Soft-limit Used Available Soft-limit exceeded? Hard-limit exceeded?<br>
------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-------<br>
/directory 100.0GB 80%(80.0GB) 90.5GB 9.5GB Yes No<br>
<br>
That does not mean that it is the quota's fault but it might be a hint where to start looking for... And by the way can someone explain me what the soft-limit does? or does it not do anything special?<br>
<br>
Here is an the linux stack of a blocking process on that directory which happened with a simple "ls -la":<br>
<br>
[Thu Aug 9 14:21:07 2018] INFO: task ls:2272 blocked for more than 120 seconds.<br>
[Thu Aug 9 14:21:07 2018] Not tainted 3.16.0-4-amd64 #1<br>
[Thu Aug 9 14:21:07 2018] "echo 0 > /proc/sys/kernel/hung_task_<wbr>timeout_secs" disables this message.<br>
[Thu Aug 9 14:21:07 2018] ls D ffff88017ef93200 0 2272 2268 0x00000004<br>
[Thu Aug 9 14:21:07 2018] ffff88017653f490 0000000000000286 0000000000013200 ffff880174d7bfd8<br>
[Thu Aug 9 14:21:07 2018] 0000000000013200 ffff88017653f490 ffff8800eeb3d5f0 ffff8800fefac800<br>
[Thu Aug 9 14:21:07 2018] ffff880174d7bbe0 ffff8800eeb3d6d0 ffff8800fefac800 ffff8800ffe1e1c0<br>
[Thu Aug 9 14:21:07 2018] Call Trace:<br>
[Thu Aug 9 14:21:07 2018] [<ffffffffa00dc50d>] ? __fuse_request_send+0xbd/0x270 [fuse]<br>
[Thu Aug 9 14:21:07 2018] [<ffffffff810abce0>] ? prepare_to_wait_event+0xf0/<wbr>0xf0<br>
[Thu Aug 9 14:21:07 2018] [<ffffffffa00e0791>] ? fuse_dentry_revalidate+0x181/<wbr>0x300 [fuse]<br>
[Thu Aug 9 14:21:07 2018] [<ffffffff811b944e>] ? lookup_fast+0x25e/0x2b0<br>
[Thu Aug 9 14:21:07 2018] [<ffffffff811bacc5>] ? path_lookupat+0x155/0x780<br>
[Thu Aug 9 14:21:07 2018] [<ffffffff81195715>] ? kmem_cache_alloc+0x75/0x480<br>
[Thu Aug 9 14:21:07 2018] [<ffffffffa00dfca9>] ? fuse_getxattr+0xe9/0x150 [fuse]<br>
[Thu Aug 9 14:21:07 2018] [<ffffffff811bb316>] ? filename_lookup+0x26/0xc0<br>
[Thu Aug 9 14:21:07 2018] [<ffffffff811bf594>] ? user_path_at_empty+0x54/0x90<br>
[Thu Aug 9 14:21:07 2018] [<ffffffff81193e08>] ? kmem_cache_free+0xd8/0x210<br>
[Thu Aug 9 14:21:07 2018] [<ffffffff811bf59f>] ? user_path_at_empty+0x5f/0x90<br>
[Thu Aug 9 14:21:07 2018] [<ffffffff811b3d46>] ? vfs_fstatat+0x46/0x90<br>
[Thu Aug 9 14:21:07 2018] [<ffffffff811b421d>] ? SYSC_newlstat+0x1d/0x40<br>
[Thu Aug 9 14:21:07 2018] [<ffffffff811d34b8>] ? SyS_lgetxattr+0x58/0x80<br>
[Thu Aug 9 14:21:07 2018] [<ffffffff81525d0d>] ? system_call_fast_compare_end+<wbr>0x10/0x15<br>
<br>
<br>
My 3 gluster nodes are all Debian 9 and my client Debian 8.<br>
<br>
Let me know if you need more information.<br>
<br>
Best regards,<br>
Mabi<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br>
</blockquote></div><br></div>