<div dir="ltr"><div dir="ltr"><br></div><div dir="ltr">Hi All,</div><div dir="ltr"><br><div><div>There is a good chance that, the inode on which unref came has already been</div><div> zero refed and added to the purge list. This can happen when inode table is</div><div> being destroyed (glfs_fini is something which destroys the inode table).</div><div> </div><div> Consider a directory 'a' which has a file 'b'. Now as part of inode table destruction</div><div> zero refing of inodes does not happen from leaf to the root. It happens in the order</div><div> inodes are present in the list. So, in this example, the dentry of 'b' would have its</div><div> parent set to the inode of 'a'. So if 'a' gets zero refed first (as part of</div><div> inode table cleanup) and then 'b' has to zero refed, then dentry_unset is called on</div><div> the dentry of 'b' and it further goes on to call inode_unref on b's parent which is</div><div> 'a'. In this situation, GF_ASSERT would be called as the refcount of 'a' has been</div><div> already set to zero.</div></div><div><br></div><div>Below is a snippet of the core file generated from such ASSERT call in one of the regression test runs.</div><div><br></div><div>"</div><div><pre class="gmail-console-output" style="box-sizing:border-box;white-space:pre-wrap;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:14px"><span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>No symbol table info available.
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>#1 0x00007f2a539fc8f8 in abort () from /lib64/libc.so.6
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>No symbol table info available.
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>#2 0x00007f2a539f4026 in __assert_fail_base () from /lib64/libc.so.6
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>No symbol table info available.
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>#3 0x00007f2a539f40d2 in __assert_fail () from /lib64/libc.so.6
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>No symbol table info available.
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>#4 0x00007f2a553e3208 in __inode_unref (inode=0x7f2a3c05edf8, clear=false) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/inode.c:483
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span> index = 0
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span> this = 0x7f2a3c03e840
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span> nlookup = 0
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span> __PRETTY_FUNCTION__ = "__inode_unref"
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>#5 0x00007f2a553e2745 in __dentry_unset (dentry=0x7f2a3c064e48) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/inode.c:212
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>No locals.
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>#6 0x00007f2a553e308a in __inode_retire (inode=0x7f2a3c05ebc8) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/inode.c:442
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span> dentry = 0x7f2a3c064e48
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span> t = 0x7f2a3c064398
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>#7 0x00007f2a553e392f in __inode_ref_reduce_by_n (inode=0x7f2a3c05ebc8, nref=0) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/inode.c:708
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span> nlookup = 0
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span> __PRETTY_FUNCTION__ = "__inode_ref_reduce_by_n"
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>#8 0x00007f2a553e61d5 in inode_table_destroy (inode_table=0x7f2a28007f90) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/inode.c:1867
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span> trav = 0x7f2a3c05ebc8
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span> __FUNCTION__ = "inode_table_destroy"
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>#9 0x00007f2a553e600c in inode_table_destroy_all (ctx=0x7f2a3c001170) at /home/jenkins/root/workspace/centos7-regression/libglusterfs/src/inode.c:1791
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span> trav_graph = 0x7f2a240041a0
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span> tmp = 0x7f2a3c0013c8
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span> tree = 0x7f2a24022af0
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span> inode_table = 0x7f2a28007f90
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span>#10 0x00007f2a46a83390 in pub_glfs_fini (fs=0x7f2a3c000ff0) at /home/jenkins/root/workspace/centos7-regression/api/src/glfs.c:1346
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span> ret = 0
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span> countdown = 98
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span> subvol = 0x7f2a24022af0
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span> ctx = 0x7f2a3c001170
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span> graph = 0x7f2a240041a0
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span> call_pool = 0x7f2a3c000df0
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span> fs_init = 1
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span> err = 0
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span> old_THIS = 0x7f2a380084c0
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">14:39:49</b> </span> __FUNCTION__ = "pub_glfs_fini"</pre><pre class="gmail-console-output" style="box-sizing:border-box;white-space:pre-wrap;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:14px">"</pre><pre class="gmail-console-output" style="box-sizing:border-box;white-space:pre-wrap;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:14px"><br></pre></div><div>IIUC the solution for it would be to add a flag in the inode table which tells whether the</div><div>cleanup of inode table has started or not. Do not call GF_ASSRT (inode->ref) if the inode</div><div>table to which the inode being unrefed belongs to, is already getting cleaned up.</div><div><br></div><div>A patch [1] has been submitted for review with the change mentioned above.</div><div><br></div><div>[1] <a href="https://review.gluster.org/#/c/glusterfs/+/22650/">https://review.gluster.org/#/c/glusterfs/+/22650/</a></div><div><br></div><div>Regards,</div><div>Raghavendra</div></div></div>