<html xmlns="http://www.w3.org/1999/xhtml" xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office"><head><!--[if gte mso 9]><xml><o:OfficeDocumentSettings><o:AllowPNG/><o:PixelsPerInch>96</o:PixelsPerInch></o:OfficeDocumentSettings></xml><![endif]--></head><body>
What is the uptime of the affected node ?<br>There is a similar error reported in <a href="https://access.redhat.com/solutions/5518661" target="_blank">https://access.redhat.com/solutions/5518661</a> which could indicate a possible problem in a memory area named ‘lru’ .<div>Have you noticed any ECC errors in dmesg/IPMI of the system ?<div><br></div><div>At least I would reboot the node and run hardware diagnostics to check that everything is fine.<br><div><div><div id="temp-enhancr-placeholder" class="enhancr-placeholder-medium" data-size="medium"></div><span style="-webkit-text-size-adjust: auto;"><br></span>Best Regards,<div>Strahil Nikolov <br><br><div class="yahoo-signature"><a href="https://mail.onelink.me/107872968?pid=nativeplacement&c=Global_Acquisition_YMktg_315_Internal_EmailSignature&af_sub1=Acquisition&af_sub2=Global_YMktg&af_sub3=&af_sub4=100000604&af_sub5=EmailSignature__Static_">Sent from Yahoo Mail for iPhone</a><br></div><br><p class="yahoo-quoted-begin" style="font-size: 15px; color: #715FFA; padding-top: 15px; margin-top: 0">On Tuesday, July 25, 2023, 4:31 AM, W Kern <wkmail@bneit.com> wrote:</p><blockquote class="iosymail"><div dir="ltr">we have an older 2+1 arbiter gluster cluster running 6.10 on Ubuntu18LTS<br></div><div dir="ltr"><br></div><div dir="ltr">It has run beautifully for years. Only occaisionally needing attention <br></div><div dir="ltr">as drives have died, etc<br></div><div dir="ltr"><br></div><div dir="ltr">Each peer has two volumes. G1 and G2 with a shared 'gluster' network.<br></div><div dir="ltr"><br></div><div dir="ltr">Since July 1st one of the peers for one volume is spewing the logfile <br></div><div dir="ltr">/var-lib-G1.log with the following errors.<br></div><div dir="ltr"><br></div><div dir="ltr">The volume (G2) is not showing this nor are there issue with other peer <br></div><div dir="ltr">and the arbiter for the G1 volume.<br></div><div dir="ltr"><br></div><div dir="ltr">So its one machine with one volume that has the problem. There have <br></div><div dir="ltr">been NO issues with the volumes themselves.<br></div><div dir="ltr"><br></div><div dir="ltr">It simply a matter of the the logfiles generating GBs of entries every <br></div><div dir="ltr">hour (which is how we noticed it when we started running out of log space).<br></div><div dir="ltr"><br></div><div dir="ltr">According to google there are mentions of this error, but that it was <br></div><div dir="ltr">fixed in the 6.x series. I can find no other mentions.<br></div><div dir="ltr"><br></div><div dir="ltr">I have tried restarting glusterd with no change. there doesn't seem to <br></div><div dir="ltr">be any hardware issues.<br></div><div dir="ltr"><br></div><div dir="ltr">I am wondering if perhaps this is an XFS file corruption issue and if I <br></div><div dir="ltr">were to unmount the Gluster run xfs_repair and bring it back, that would <br></div><div dir="ltr">solve the issue.<br></div><div dir="ltr"><br></div><div dir="ltr">Any other suggestions?<br></div><div dir="ltr"><br></div><div dir="ltr">[2023-07-21 18:51:38.260507] W [inode.c:1638:inode_table_prune] <br></div><div dir="ltr">(-->/usr/lib/x86_64-linux-gnu/glusterfs/6.10/xlator/features/shard.so(+0x21b47) <br></div><div dir="ltr">[0x7fb261c13b47] <br></div><div dir="ltr">-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_unref+0x36) <br></div><div dir="ltr">[0x7fb26947f416] <br></div><div dir="ltr">-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x3337a) <br></div><div dir="ltr">[0x7fb26947f37a] ) 0-GLB1image-shard: Empty inode lru list found but <br></div><div dir="ltr">with (-2) lru_size<br></div><div dir="ltr">[2023-07-21 18:51:38.261231] W [inode.c:1638:inode_table_prune] <br></div><div dir="ltr">(-->/usr/lib/x86_64-linux-gnu/glusterfs/6.10/xlator/mount/fuse.so(+0xba51) <br></div><div dir="ltr">[0x7fb266cdca51] <br></div><div dir="ltr">-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_unref+0x36) <br></div><div dir="ltr">[0x7fb26947f416] <br></div><div dir="ltr">-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x3337a) <br></div><div dir="ltr">[0x7fb26947f37a] ) 0-fuse: Empty inode lru list found but with (-2) lru_size<br></div><div dir="ltr">[2023-07-21 18:51:38.261377] W [inode.c:1638:inode_table_prune] <br></div><div dir="ltr">(-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(loc_wipe+0x12) <br></div><div dir="ltr">[0x7fb26946bd72] <br></div><div dir="ltr">-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_unref+0x36) <br></div><div dir="ltr">[0x7fb26947f416] <br></div><div dir="ltr">-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x3337a) <br></div><div dir="ltr">[0x7fb26947f37a] ) 0-GLB1image-shard: Empty inode lru list found but <br></div><div dir="ltr">with (-2) lru_size<br></div><div dir="ltr">[2023-07-21 18:51:38.261806] W [inode.c:1638:inode_table_prune] <br></div><div dir="ltr">(-->/usr/lib/x86_64-linux-gnu/glusterfs/6.10/xlator/cluster/replicate.so(+0x5ca57) <br></div><div dir="ltr">[0x7fb26213ba57] <br></div><div dir="ltr">-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_unref+0x36) <br></div><div dir="ltr">[0x7fb26947f416] <br></div><div dir="ltr">-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x3337a) <br></div><div dir="ltr">[0x7fb26947f37a] ) 0-GLB1image-replicate-0: Empty inode lru list found <br></div><div dir="ltr">but with (-2) lru_size<br></div><div dir="ltr">[2023-07-21 18:51:38.261933] W [inode.c:1638:inode_table_prune] <br></div><div dir="ltr">(-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(fd_unref+0x1ef) <br></div><div dir="ltr">[0x7fb269495eaf] <br></div><div dir="ltr">-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_unref+0x36) <br></div><div dir="ltr">[0x7fb26947f416] <br></div><div dir="ltr">-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x3337a) <br></div><div dir="ltr">[0x7fb26947f37a] ) 0-GLB1image-client-1: Empty inode lru list found but <br></div><div dir="ltr">with (-2) lru_size<br></div><div dir="ltr">[2023-07-21 18:51:38.262684] W [inode.c:1638:inode_table_prune] <br></div><div dir="ltr">(-->/usr/lib/x86_64-linux-gnu/glusterfs/6.10/xlator/cluster/replicate.so(+0x5ca57) <br></div><div dir="ltr">[0x7fb26213ba57] <br></div><div dir="ltr">-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_unref+0x36) <br></div><div dir="ltr">[0x7fb26947f416] <br></div><div dir="ltr">-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x3337a) <br></div><div dir="ltr">[0x7fb26947f37a] ) 0-GLB1image-replicate-0: Empty inode lru list found <br></div><div dir="ltr">but with (-2) lru_size<br></div><div dir="ltr"><br></div><div dir="ltr">-wk<br></div><div dir="ltr"><br></div><div dir="ltr">________<br></div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr">Community Meeting Calendar:<br></div><div dir="ltr"><br></div><div dir="ltr">Schedule -<br></div><div dir="ltr">Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br></div><div dir="ltr">Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br></div><div dir="ltr">Gluster-users mailing list<br></div><div dir="ltr"><a ymailto="mailto:Gluster-users@gluster.org" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br></div><div dir="ltr"><a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br></div><blockquote></blockquote></blockquote></div></div></div></div></div>
</body></html>