<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <p>A while back we found an issue with PHP around file locks.</p>
    <p><a class="moz-txt-link-freetext" href="https://bugs.php.net/bug.php?id=53076">https://bugs.php.net/bug.php?id=53076</a></p>
    <p>This would leave hanging processes that we could only kill by
      forcibly remounting the filesystems.</p>
    <p>Its not really a PHP bug but they are not willing to fix the code
      to deal with Glusters idiosyncrasies.<br>
    </p>
    <br>
    <div class="moz-cite-prefix">On 08/09/2018 09:10 AM, Nithya
      Balachandran wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CAOUCJ=gOk_tcJ--o_KB1znViLcF+sP7U3sy+xYTk22CP-Bt9Gw@mail.gmail.com">
      <div dir="ltr">Hi,
        <div><br>
        </div>
        <div>Please provide the following:</div>
        <div>
          <ol>
            <li>gluster volume info<br>
            </li>
            <li>statedump of the fuse process when it hangs</li>
          </ol>
        </div>
        <div><br>
        </div>
        <div>Thanks,</div>
        <div>Nithya</div>
      </div>
      <div class="gmail_extra"><br>
        <div class="gmail_quote">On 9 August 2018 at 18:24, mabi <span
            dir="ltr">&lt;<a href="mailto:mabi@protonmail.ch"
              target="_blank" moz-do-not-send="true">mabi@protonmail.ch</a>&gt;</span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0
            .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello,<br>
            <br>
            I recently upgraded my GlusterFS replica 2+1 (aribter) to
            version 3.12.12 and now I see a weird behaviour on my client
            (using FUSE mount) where I have processes (PHP 5.6 FPM)
            trying to access a specific directory and then the process
            blocks. I can't kill the process either, not even with kill
            -9. I need to reboot the machine in order to get rid of
            these blocked processes.<br>
            <br>
            This directory has one particularity compared to the other
            directories it is that it has reached it's quota soft-limit
            as you can see here in the output of gluster volume quota
            list:<br>
            <br>
                              Path                   Hard-limit 
            Soft-limit      Used  Available  Soft-limit exceeded?
            Hard-limit exceeded?<br>
            ------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-------<br>
            /directory                          100.0GB     80%(80.0GB) 
             90.5GB   9.5GB             Yes                   No<br>
            <br>
            That does not mean that it is the quota's fault but it might
            be a hint where to start looking for... And by the way can
            someone explain me what the soft-limit does? or does it not
            do anything special?<br>
            <br>
            Here is an the linux stack of a blocking process on that
            directory which happened with a simple "ls -la":<br>
            <br>
            [Thu Aug  9 14:21:07 2018] INFO: task ls:2272 blocked for
            more than 120 seconds.<br>
            [Thu Aug  9 14:21:07 2018]       Not tainted 3.16.0-4-amd64
            #1<br>
            [Thu Aug  9 14:21:07 2018] "echo 0 &gt;
            /proc/sys/kernel/hung_task_<wbr>timeout_secs" disables this
            message.<br>
            [Thu Aug  9 14:21:07 2018] ls              D
            ffff88017ef93200     0  2272   2268 0x00000004<br>
            [Thu Aug  9 14:21:07 2018]  ffff88017653f490
            0000000000000286 0000000000013200 ffff880174d7bfd8<br>
            [Thu Aug  9 14:21:07 2018]  0000000000013200
            ffff88017653f490 ffff8800eeb3d5f0 ffff8800fefac800<br>
            [Thu Aug  9 14:21:07 2018]  ffff880174d7bbe0
            ffff8800eeb3d6d0 ffff8800fefac800 ffff8800ffe1e1c0<br>
            [Thu Aug  9 14:21:07 2018] Call Trace:<br>
            [Thu Aug  9 14:21:07 2018]  [&lt;ffffffffa00dc50d&gt;] ?
            __fuse_request_send+0xbd/0x270 [fuse]<br>
            [Thu Aug  9 14:21:07 2018]  [&lt;ffffffff810abce0&gt;] ?
            prepare_to_wait_event+0xf0/<wbr>0xf0<br>
            [Thu Aug  9 14:21:07 2018]  [&lt;ffffffffa00e0791&gt;] ?
            fuse_dentry_revalidate+0x181/<wbr>0x300 [fuse]<br>
            [Thu Aug  9 14:21:07 2018]  [&lt;ffffffff811b944e&gt;] ?
            lookup_fast+0x25e/0x2b0<br>
            [Thu Aug  9 14:21:07 2018]  [&lt;ffffffff811bacc5&gt;] ?
            path_lookupat+0x155/0x780<br>
            [Thu Aug  9 14:21:07 2018]  [&lt;ffffffff81195715&gt;] ?
            kmem_cache_alloc+0x75/0x480<br>
            [Thu Aug  9 14:21:07 2018]  [&lt;ffffffffa00dfca9&gt;] ?
            fuse_getxattr+0xe9/0x150 [fuse]<br>
            [Thu Aug  9 14:21:07 2018]  [&lt;ffffffff811bb316&gt;] ?
            filename_lookup+0x26/0xc0<br>
            [Thu Aug  9 14:21:07 2018]  [&lt;ffffffff811bf594&gt;] ?
            user_path_at_empty+0x54/0x90<br>
            [Thu Aug  9 14:21:07 2018]  [&lt;ffffffff81193e08&gt;] ?
            kmem_cache_free+0xd8/0x210<br>
            [Thu Aug  9 14:21:07 2018]  [&lt;ffffffff811bf59f&gt;] ?
            user_path_at_empty+0x5f/0x90<br>
            [Thu Aug  9 14:21:07 2018]  [&lt;ffffffff811b3d46&gt;] ?
            vfs_fstatat+0x46/0x90<br>
            [Thu Aug  9 14:21:07 2018]  [&lt;ffffffff811b421d&gt;] ?
            SYSC_newlstat+0x1d/0x40<br>
            [Thu Aug  9 14:21:07 2018]  [&lt;ffffffff811d34b8&gt;] ?
            SyS_lgetxattr+0x58/0x80<br>
            [Thu Aug  9 14:21:07 2018]  [&lt;ffffffff81525d0d&gt;] ?
            system_call_fast_compare_end+<wbr>0x10/0x15<br>
            <br>
            <br>
            My 3 gluster nodes are all Debian 9 and my client Debian 8.<br>
            <br>
            Let me know if you need more information.<br>
            <br>
            Best regards,<br>
            Mabi<br>
            ______________________________<wbr>_________________<br>
            Gluster-users mailing list<br>
            <a href="mailto:Gluster-users@gluster.org"
              moz-do-not-send="true">Gluster-users@gluster.org</a><br>
            <a
              href="https://lists.gluster.org/mailman/listinfo/gluster-users"
              rel="noreferrer" target="_blank" moz-do-not-send="true">https://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br>
          </blockquote>
        </div>
        <br>
      </div>
      <!--'"--><br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="https://lists.gluster.org/mailman/listinfo/gluster-users">https://lists.gluster.org/mailman/listinfo/gluster-users</a></pre>
    </blockquote>
    <br>
    <pre class="moz-signature" cols="72">-- 
Alvin Starr                   ||   land:  (905)513-7688
Netvel Inc.                   ||   Cell:  (416)806-0133
<a class="moz-txt-link-abbreviated" href="mailto:alvin@netvel.net">alvin@netvel.net</a>              ||

</pre>
  </body>
</html>