<div dir="ltr"><div dir="ltr"><br></div><div dir="ltr">Hi All,</div><div dir="ltr"><br><div><div>There is a good chance that, the inode on which unref came has already been</div><div> zero refed and added to the purge list. This can happen when inode table is</div><div> being destroyed (glfs_fini is something which destroys the inode table).</div><div>    </div><div> Consider a directory &#39;a&#39; which has a file &#39;b&#39;. Now as part of inode table destruction</div><div> zero refing of inodes does not happen from leaf to the root. It happens in the order</div><div> inodes are present in the list. So, in this example, the dentry of &#39;b&#39; would have its</div><div> parent set to the inode of &#39;a&#39;. So if &#39;a&#39; gets zero refed first (as part of</div><div> inode table cleanup) and then &#39;b&#39; has to zero refed, then dentry_unset is called on</div><div> the dentry of &#39;b&#39; and it further goes on to call inode_unref on b&#39;s parent which is</div><div> &#39;a&#39;. In this situation, GF_ASSERT would be called as the refcount of &#39;a&#39; has been</div><div> already set to zero.</div></div><div><br></div><div>Below is a snippet of the core file generated from such ASSERT call in one of the regression test runs.</div><div><br></div><div>&quot;</div><div><pre class="gmail-console-output" style="box-sizing:border-box;white-space:pre-wrap;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:14px"><span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>No symbol table info available.
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>#1  0x00007f2a539fc8f8 in abort () from /lib64/libc.so.6
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>No symbol table info available.
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>#2  0x00007f2a539f4026 in __assert_fail_base () from /lib64/libc.so.6
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>No symbol table info available.
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>#3  0x00007f2a539f40d2 in __assert_fail () from /lib64/libc.so.6
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>No symbol table info available.
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>#4  0x00007f2a553e3208 in __inode_unref (inode=0x7f2a3c05edf8, clear=false) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/inode.c:483
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>        index = 0
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>        this = 0x7f2a3c03e840
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>        nlookup = 0
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>        __PRETTY_FUNCTION__ = &quot;__inode_unref&quot;
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>#5  0x00007f2a553e2745 in __dentry_unset (dentry=0x7f2a3c064e48) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/inode.c:212
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>No locals.
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>#6  0x00007f2a553e308a in __inode_retire (inode=0x7f2a3c05ebc8) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/inode.c:442
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>        dentry = 0x7f2a3c064e48
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>        t = 0x7f2a3c064398
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>#7  0x00007f2a553e392f in __inode_ref_reduce_by_n (inode=0x7f2a3c05ebc8, nref=0) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/inode.c:708
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>        nlookup = 0
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>        __PRETTY_FUNCTION__ = &quot;__inode_ref_reduce_by_n&quot;
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>#8  0x00007f2a553e61d5 in inode_table_destroy (inode_table=0x7f2a28007f90) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/inode.c:1867
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>        trav = 0x7f2a3c05ebc8
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>        __FUNCTION__ = &quot;inode_table_destroy&quot;
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>#9  0x00007f2a553e600c in inode_table_destroy_all (ctx=0x7f2a3c001170) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/inode.c:1791
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>        trav_graph = 0x7f2a240041a0
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>        tmp = 0x7f2a3c0013c8
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>        tree = 0x7f2a24022af0
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>        inode_table = 0x7f2a28007f90
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>#10 0x00007f2a46a83390 in pub_glfs_fini (fs=0x7f2a3c000ff0) at /home/jenkins/root/workspace/centos7-regression/api/src/glfs.c:1346
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>        ret = 0
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>        countdown = 98
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>        subvol = 0x7f2a24022af0
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>        ctx = 0x7f2a3c001170
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>        graph = 0x7f2a240041a0
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>        call_pool = 0x7f2a3c000df0
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>        fs_init = 1
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>        err = 0
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>        old_THIS = 0x7f2a380084c0
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>        __FUNCTION__ = &quot;pub_glfs_fini&quot;</pre><pre class="gmail-console-output" style="box-sizing:border-box;white-space:pre-wrap;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:14px">&quot;</pre><pre class="gmail-console-output" style="box-sizing:border-box;white-space:pre-wrap;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:14px"><br></pre></div><div>IIUC the solution for it would be to add a flag in the inode table which tells whether the</div><div>cleanup of inode table has started or not. Do not call GF_ASSRT (inode-&gt;ref) if the inode</div><div>table to which the inode being unrefed belongs to, is already getting cleaned up.</div><div><br></div><div>A patch [1] has been submitted for review with the change mentioned above.</div><div><br></div><div>[1] <a href="https://review.gluster.org/#/c/glusterfs/+/22650/">https://review.gluster.org/#/c/glusterfs/+/22650/</a></div><div><br></div><div>Regards,</div><div>Raghavendra</div></div></div>