<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Well as I indicated a day or so later. in the RESOLVED subject
addition.<br>
</p>
<p>unmounting the folder, and running xfs_repair seemed to solve the
problem.</p>
<p>No issues noted in dmesg. Uptime was almost 300 days.</p>
<p>We shall see if the problem returns.</p>
<p>-wk<br>
</p>
<div class="moz-cite-prefix">On 7/25/23 1:22 PM, Strahil Nikolov
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:952194860.2834250.1690316537802@mail.yahoo.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<!--[if gte mso 9]><xml><o:OfficeDocumentSettings><o:AllowPNG/><o:PixelsPerInch>96</o:PixelsPerInch></o:OfficeDocumentSettings></xml><![endif]-->
What is the uptime of the affected node ?<br>
There is a similar error reported in <a
href="https://access.redhat.com/solutions/5518661"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://access.redhat.com/solutions/5518661</a>
which could indicate a possible problem in a memory area named
‘lru’ .
<div>Have you noticed any ECC errors in dmesg/IPMI of the system ?
<div><br>
</div>
<div>At least I would reboot the node and run hardware
diagnostics to check that everything is fine.<br>
<div>
<div><span style="-webkit-text-size-adjust: auto;"><br>
</span>Best Regards,
<div>Strahil Nikolov <br>
<br>
<div class="yahoo-signature"><a
href="https://mail.onelink.me/107872968?pid=nativeplacement&c=Global_Acquisition_YMktg_315_Internal_EmailSignature&af_sub1=Acquisition&af_sub2=Global_YMktg&af_sub3=&af_sub4=100000604&af_sub5=EmailSignature__Static_"
moz-do-not-send="true">Sent from Yahoo Mail for
iPhone</a><br>
</div>
<br>
<p class="yahoo-quoted-begin" style="font-size: 15px;
color: #715FFA; padding-top: 15px; margin-top: 0">On
Tuesday, July 25, 2023, 4:31 AM, W Kern
<a class="moz-txt-link-rfc2396E" href="mailto:wkmail@bneit.com"><wkmail@bneit.com></a> wrote:</p>
<blockquote class="iosymail">
<div dir="ltr">we have an older 2+1 arbiter gluster
cluster running 6.10 on Ubuntu18LTS<br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">It has run beautifully for years. Only
occaisionally needing attention <br>
</div>
<div dir="ltr">as drives have died, etc<br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Each peer has two volumes. G1 and G2
with a shared 'gluster' network.<br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Since July 1st one of the peers for one
volume is spewing the logfile <br>
</div>
<div dir="ltr">/var-lib-G1.log with the following
errors.<br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">The volume (G2) is not showing this nor
are there issue with other peer <br>
</div>
<div dir="ltr">and the arbiter for the G1 volume.<br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">So its one machine with one volume that
has the problem. There have <br>
</div>
<div dir="ltr">been NO issues with the volumes
themselves.<br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">It simply a matter of the the logfiles
generating GBs of entries every <br>
</div>
<div dir="ltr">hour (which is how we noticed it when
we started running out of log space).<br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">According to google there are mentions
of this error, but that it was <br>
</div>
<div dir="ltr">fixed in the 6.x series. I can find no
other mentions.<br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">I have tried restarting glusterd with
no change. there doesn't seem to <br>
</div>
<div dir="ltr">be any hardware issues.<br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">I am wondering if perhaps this is an
XFS file corruption issue and if I <br>
</div>
<div dir="ltr">were to unmount the Gluster run
xfs_repair and bring it back, that would <br>
</div>
<div dir="ltr">solve the issue.<br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Any other suggestions?<br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">[2023-07-21 18:51:38.260507] W
[inode.c:1638:inode_table_prune] <br>
</div>
<div dir="ltr">(-->/usr/lib/x86_64-linux-gnu/glusterfs/6.10/xlator/features/shard.so(+0x21b47)
<br>
</div>
<div dir="ltr">[0x7fb261c13b47] <br>
</div>
<div dir="ltr">-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_unref+0x36)
<br>
</div>
<div dir="ltr">[0x7fb26947f416] <br>
</div>
<div dir="ltr">-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x3337a)
<br>
</div>
<div dir="ltr">[0x7fb26947f37a] ) 0-GLB1image-shard:
Empty inode lru list found but <br>
</div>
<div dir="ltr">with (-2) lru_size<br>
</div>
<div dir="ltr">[2023-07-21 18:51:38.261231] W
[inode.c:1638:inode_table_prune] <br>
</div>
<div dir="ltr">(-->/usr/lib/x86_64-linux-gnu/glusterfs/6.10/xlator/mount/fuse.so(+0xba51)
<br>
</div>
<div dir="ltr">[0x7fb266cdca51] <br>
</div>
<div dir="ltr">-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_unref+0x36)
<br>
</div>
<div dir="ltr">[0x7fb26947f416] <br>
</div>
<div dir="ltr">-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x3337a)
<br>
</div>
<div dir="ltr">[0x7fb26947f37a] ) 0-fuse: Empty inode
lru list found but with (-2) lru_size<br>
</div>
<div dir="ltr">[2023-07-21 18:51:38.261377] W
[inode.c:1638:inode_table_prune] <br>
</div>
<div dir="ltr">(-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(loc_wipe+0x12)
<br>
</div>
<div dir="ltr">[0x7fb26946bd72] <br>
</div>
<div dir="ltr">-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_unref+0x36)
<br>
</div>
<div dir="ltr">[0x7fb26947f416] <br>
</div>
<div dir="ltr">-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x3337a)
<br>
</div>
<div dir="ltr">[0x7fb26947f37a] ) 0-GLB1image-shard:
Empty inode lru list found but <br>
</div>
<div dir="ltr">with (-2) lru_size<br>
</div>
<div dir="ltr">[2023-07-21 18:51:38.261806] W
[inode.c:1638:inode_table_prune] <br>
</div>
<div dir="ltr">(-->/usr/lib/x86_64-linux-gnu/glusterfs/6.10/xlator/cluster/replicate.so(+0x5ca57)
<br>
</div>
<div dir="ltr">[0x7fb26213ba57] <br>
</div>
<div dir="ltr">-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_unref+0x36)
<br>
</div>
<div dir="ltr">[0x7fb26947f416] <br>
</div>
<div dir="ltr">-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x3337a)
<br>
</div>
<div dir="ltr">[0x7fb26947f37a] )
0-GLB1image-replicate-0: Empty inode lru list found
<br>
</div>
<div dir="ltr">but with (-2) lru_size<br>
</div>
<div dir="ltr">[2023-07-21 18:51:38.261933] W
[inode.c:1638:inode_table_prune] <br>
</div>
<div dir="ltr">(-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(fd_unref+0x1ef)
<br>
</div>
<div dir="ltr">[0x7fb269495eaf] <br>
</div>
<div dir="ltr">-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_unref+0x36)
<br>
</div>
<div dir="ltr">[0x7fb26947f416] <br>
</div>
<div dir="ltr">-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x3337a)
<br>
</div>
<div dir="ltr">[0x7fb26947f37a] )
0-GLB1image-client-1: Empty inode lru list found but
<br>
</div>
<div dir="ltr">with (-2) lru_size<br>
</div>
<div dir="ltr">[2023-07-21 18:51:38.262684] W
[inode.c:1638:inode_table_prune] <br>
</div>
<div dir="ltr">(-->/usr/lib/x86_64-linux-gnu/glusterfs/6.10/xlator/cluster/replicate.so(+0x5ca57)
<br>
</div>
<div dir="ltr">[0x7fb26213ba57] <br>
</div>
<div dir="ltr">-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_unref+0x36)
<br>
</div>
<div dir="ltr">[0x7fb26947f416] <br>
</div>
<div dir="ltr">-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x3337a)
<br>
</div>
<div dir="ltr">[0x7fb26947f37a] )
0-GLB1image-replicate-0: Empty inode lru list found
<br>
</div>
<div dir="ltr">but with (-2) lru_size<br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">-wk<br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">________<br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Community Meeting Calendar:<br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Schedule -<br>
</div>
<div dir="ltr">Every 2nd and 4th Tuesday at 14:30 IST
/ 09:00 UTC<br>
</div>
<div dir="ltr">Bridge: <a
href="https://meet.google.com/cpu-eiue-hvk"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://meet.google.com/cpu-eiue-hvk</a><br>
</div>
<div dir="ltr">Gluster-users mailing list<br>
</div>
<div dir="ltr"><a
ymailto="mailto:Gluster-users@gluster.org"
href="mailto:Gluster-users@gluster.org"
moz-do-not-send="true"
class="moz-txt-link-freetext">Gluster-users@gluster.org</a><br>
</div>
<div dir="ltr"><a
href="https://lists.gluster.org/mailman/listinfo/gluster-users"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</body>
</html>