<html xmlns="http://www.w3.org/1999/xhtml" xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office"><head><!--[if gte mso 9]><xml><o:OfficeDocumentSettings><o:AllowPNG/><o:PixelsPerInch>96</o:PixelsPerInch></o:OfficeDocumentSettings></xml><![endif]--></head><body>
Hi Liam,<div><br></div><div>Have you tried to convert the volume into replica 2 , wipe the arbiter brick , recreate it and then convert back to ‘replica 3 arbiter 1’ mode ?</div><div><br></div><div>Best Regards,</div><div>Strahil Nikolov <br><br><p class="yahoo-quoted-begin" style="font-size: 15px; color: #715FFA; padding-top: 15px; margin-top: 0">On Wednesday, July 12, 2023, 1:31 PM, Liam Smith <liam.smith@ek.co> wrote:</p><blockquote class="iosymail"><div id="yiv7898679786">
<style type="text/css">#yiv7898679786 P {margin-top:0;margin-bottom:0;}</style>
<div dir="ltr"><div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0, 0, 0);" class="yiv7898679786elementToProof">
Hi,</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0, 0, 0);" class="yiv7898679786elementToProof">
<br>
</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0, 0, 0);" class="yiv7898679786elementToProof">
We're still seeing this issue, we can see entries like the below on the servers connecting to gluster in the gluster client logs in case that helps:</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0, 0, 0);" class="yiv7898679786elementToProof yiv7898679786ContentPasted0">
<blockquote itemscope="" itemtype="https://schemas.microsoft.com/QuotedText" style="border-left:3px solid rgb(200, 200, 200);border-top-color:rgb(200, 200, 200);border-right-color:rgb(200, 200, 200);border-bottom-color:rgb(200, 200, 200);padding-left:1ex;margin-left:0.8ex;">
[2023-07-06 22:59:47.029036 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-gv1-client-7: remote operation failed. [{path=<gfid:55bca2d2-ae47-4
<div class="yiv7898679786ContentPasted0">57c-82f2-bcea5947f558>}, {gfid=55bca2d2-ae47-457c-82f2-bcea5947f558}, {errno=2}, {error=No such file or directory}]
</div>
<div class="yiv7898679786ContentPasted0">[2023-07-06 22:59:47.029136 +0000] W [MSGID: 114031] [client-rpc-fops_v2.c:2561:client4_0_lookup_cbk] 0-gv1-client-5: remote operation failed. [{path=<gfid:55bca2d2-ae47-4</div>
<div class="yiv7898679786ContentPasted0">57c-82f2-bcea5947f558>}, {gfid=55bca2d2-ae47-457c-82f2-bcea5947f558}, {errno=2}, {error=No such file or directory}]
</div>
<div class="yiv7898679786ContentPasted0">[2023-07-06 22:59:47.038752 +0000] E [MSGID: 133021] [shard.c:3822:shard_delete_shards] 0-gv1-shard: Failed to clean up shards of gfid 55bca2d2-ae47-457c-82f2-bcea5947f558</div>
[No such file or directory]</blockquote>
</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0, 0, 0);" class="yiv7898679786elementToProof yiv7898679786ContentPasted0">
Thanks,</div>
<div>
<div id="yiv7898679786Signature">
<div>
<div dir="ltr" style="font-size:1px;direction:ltr;">
<table cellpadding="0" cellspacing="0" border="0" style="width:100%;font-size:1px;">
<tbody>
<tr style="font-size:0;">
<td align="left" style="vertical-align:top;">
<table cellpadding="0" cellspacing="0" border="0" style="font-size:0;">
<tbody>
<tr style="font-size:0;">
<td colspan="2" style="padding:0;"> </td>
</tr>
<tr style="font-size:0;">
<td align="left" style="padding:0 0 5px;vertical-align:top;">
<table cellpadding="0" cellspacing="0" border="0" style="font-size:0;">
<tbody>
<tr style="font-size:0;">
<td align="left" style="vertical-align:top;">
<table cellpadding="0" cellspacing="0" border="0" style="font-size:0;">
<tbody>
<tr style="font-size:0;">
<td align="left" style="padding:0 0 11px;vertical-align:top;">
<table cellpadding="0" cellspacing="0" border="0" style="font-size:0px;font-style:normal;font-weight:700;white-space:nowrap;color:rgb(3, 90, 255);">
<tbody>
<tr style="font-size:14.67px;">
<td align="left" style="vertical-align:top;font-family:Arial;">Liam Smith<span style="font-family:remialcxesans;font-size:1px;line-height:1px;color:rgb(255, 255, 255);"><span style="font-family:'template-ra4AhfxhEey0egBQ8nUoZA';"></span><span style="font-family:'zone-1';"></span><span style="font-family:'zones-AQ';"></span></span></td>
</tr>
<tr style="font-size:14.67px;font-weight:400;color:rgb(0, 0, 1);">
<td align="left" style="vertical-align:top;font-family:Arial;">Linux Systems Support Engineer, Scholar</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
<td style="padding:0 0 5px;"> </td>
</tr>
<tr style="font-size:0;">
<td colspan="2" align="left" style="padding:0;vertical-align:top;">
<table cellpadding="0" cellspacing="0" border="0" style="font-size:0;line-height:normal;">
<tbody>
<tr style="font-size:0;">
<td align="left" style="padding:0 0 5px;vertical-align:top;"><a rel="nofollow noopener noreferrer" target="_blank" href="https://www.ek.co/" id="yiv7898679786LPlnk689713" style="text-decoration:none;"><img width="168.07" height="49" border="0" alt="" style="width:168.07px;min-width:168.07px;max-width:168.07px;min-height:49px;min-height:49px;max-height:49px;font-size:0;" src="cid:TlXe7WOUgNqP2CZvQaQY" yahoo_partid="5"></a></td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</div>
<div id="yiv7898679786appendonsend"></div>
<hr style="display:inline-block;width:98%;" tabindex="-1">
<div id="yiv7898679786divRplyFwdMsg" dir="ltr"><b>From:</b> Strahil Nikolov <hunter86_bg@yahoo.com><br>
<b>Sent:</b> 05 July 2023 11:59<br>
<b>To:</b> Liam Smith <liam.smith@ek.co>; gluster-users@gluster.org <gluster-users@gluster.org>; Gluster Devel <gluster-devel@gluster.org><br>
<b>Subject:</b> Re: [Gluster-users] remove_me files building up
<div> </div>
</div>
<div>
<div class="yiv7898679786x_mc-ip-hide">
<div style="color:#000000;font-size:12px;text-align:left;font-family:Helvetica, Arial, sans-serif;">
<strong>
<table style="width:100%;background:#fff8c4 no-repeat 10px 50%;border:1px solid #f2c779;font-family:Arial, Helvetica, sans-serif;border-radius:5px;padding:1px;">
<tbody>
<tr>
<td>
<p style="text-align:left;color:black;line-height:1;"><strong><span style="color:rgb(255,0,0);font-size:13px;">CAUTION:</span><span style="font-size:13px;"> </span></strong><span style="font-size:13px;">This e-mail originates from outside of Ekco. Do not click
links or attachments unless you recognise the sender. </span></p>
</td>
</tr>
</tbody>
</table>
</strong><br>
</div>
<hr>
</div>
Adding Gluster Devel list.
<div><br>
</div>
<div>Best Regards,</div>
<div>Strahil Nikolov <br>
<div>
<div><br>
<br>
<div class="yiv7898679786x_yahoo-signature"><a rel="nofollow noopener noreferrer" target="_blank" href="https://mail.onelink.me/107872968?pid=nativeplacement&c=Global_Acquisition_YMktg_315_Internal_EmailSignature&af_sub1=Acquisition&af_sub2=Global_YMktg&af_sub3=&af_sub4=100000604&af_sub5=EmailSignature__Static_">Sent from Yahoo Mail for iPhone</a><br>
</div>
<br>
<p style="font-size:15px;color:#715FFA;padding-top:15px;margin-top:0;" class="yiv7898679786x_yahoo-quoted-begin">
On Wednesday, July 5, 2023, 12:41 PM, Liam Smith <liam.smith@ek.co> wrote:</p>
<br>
<br>
<hr>
<p style="font-family:'Courier New';margin-top:0px;">
The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure.
</p>
<br>
<br>
<blockquote class="yiv7898679786x_iosymail">
<div id="yiv7898679786x_yiv4442503694"><style type="text/css">
<!--
#yiv7898679786 #yiv7898679786x_yiv4442503694 p
{margin-top:0;margin-bottom:0;}
-->
</style>
<div dir="ltr">
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694elementToProof">
Hi Strahil,</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694elementToProof">
<br>
</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694elementToProof">
This is the output from the commands:</div>
<blockquote itemscope="" itemtype="https://schemas.microsoft.com/QuotedText" style="border-left:3px solid rgb(200,200,200);border-top-color:rgb(200,200,200);border-right-color:rgb(200,200,200);border-bottom-color:rgb(200,200,200);padding-left:1ex;margin-left:0.8ex;">
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694">
<span style="margin:0px;background-color:rgb(40,40,40);" class="yiv7898679786x_yiv4442503694ContentPasted0 yiv7898679786x_yiv4442503694ContentPasted1">root@uk3-prod-gfs-arb-01:~# du -h -x -d 1 /data/glusterfs/gv1/brick1/brick</span>
<div style="margin:0px;background-color:rgb(40,40,40);" class="yiv7898679786x_yiv4442503694ContentPasted0 yiv7898679786x_yiv4442503694ContentPasted1">
2.2G /data/glusterfs/gv1/brick1/brick/.glusterfs</div>
<div style="margin:0px;background-color:rgb(40,40,40);" class="yiv7898679786x_yiv4442503694ContentPasted0 yiv7898679786x_yiv4442503694ContentPasted1">
24M /data/glusterfs/gv1/brick1/brick/scalelite-recordings</div>
<div style="margin:0px;background-color:rgb(40,40,40);" class="yiv7898679786x_yiv4442503694ContentPasted0 yiv7898679786x_yiv4442503694ContentPasted1">
16K /data/glusterfs/gv1/brick1/brick/mytute</div>
<div style="margin:0px;background-color:rgb(40,40,40);" class="yiv7898679786x_yiv4442503694ContentPasted0 yiv7898679786x_yiv4442503694ContentPasted1">
18M /data/glusterfs/gv1/brick1/brick/.shard</div>
<div style="margin:0px;background-color:rgb(40,40,40);" class="yiv7898679786x_yiv4442503694ContentPasted0 yiv7898679786x_yiv4442503694ContentPasted1">
0 /data/glusterfs/gv1/brick1/brick/.glusterfs-anonymous-inode-d3d1fdec-7df9-4f71-b9fc-660d12c2a046</div>
<span style="margin:0px;background-color:rgb(40,40,40);" class="yiv7898679786x_yiv4442503694ContentPasted0 yiv7898679786x_yiv4442503694ContentPasted1">2.3G /data/glusterfs/gv1/brick1/brick</span></div>
</blockquote>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694elementToProof yiv7898679786x_yiv4442503694ContentPasted0">
<blockquote itemscope="" itemtype="https://schemas.microsoft.com/QuotedText" style="border-left:3px solid rgb(200,200,200);border-top-color:rgb(200,200,200);border-right-color:rgb(200,200,200);border-bottom-color:rgb(200,200,200);padding-left:1ex;margin-left:0.8ex;">
root@uk3-prod-gfs-arb-01:~# du -h -x -d 1 /data/glusterfs/gv1/brick3/brick
<div class="yiv7898679786x_yiv4442503694ContentPasted0">11G /data/glusterfs/gv1/brick3/brick/.glusterfs</div>
<div class="yiv7898679786x_yiv4442503694ContentPasted0">15M /data/glusterfs/gv1/brick3/brick/scalelite-recordings</div>
<div class="yiv7898679786x_yiv4442503694ContentPasted0">460K /data/glusterfs/gv1/brick3/brick/mytute</div>
<div class="yiv7898679786x_yiv4442503694ContentPasted0">151M /data/glusterfs/gv1/brick3/brick/.shard</div>
<div class="yiv7898679786x_yiv4442503694ContentPasted0">0 /data/glusterfs/gv1/brick3/brick/.glusterfs-anonymous-inode-d3d1fdec-7df9-4f71-b9fc-660d12c2a046</div>
<div class="yiv7898679786x_yiv4442503694ContentPasted0">11G /data/glusterfs/gv1/brick3/brick</div>
</blockquote>
<blockquote itemscope="" itemtype="https://schemas.microsoft.com/QuotedText" style="border-left:3px solid rgb(200,200,200);border-top-color:rgb(200,200,200);border-right-color:rgb(200,200,200);border-bottom-color:rgb(200,200,200);padding-left:1ex;margin-left:0.8ex;">
<div class="yiv7898679786x_yiv4442503694ContentPasted0">root@uk3-prod-gfs-arb-01:~# du -h -x -d 1 /data/glusterfs/gv1/brick2/brick</div>
<div class="yiv7898679786x_yiv4442503694ContentPasted0">12G /data/glusterfs/gv1/brick2/brick/.glusterfs</div>
<div class="yiv7898679786x_yiv4442503694ContentPasted0">110M /data/glusterfs/gv1/brick2/brick/scalelite-recordings</div>
<div class="yiv7898679786x_yiv4442503694ContentPasted0">3.1M /data/glusterfs/gv1/brick2/brick/mytute</div>
<div class="yiv7898679786x_yiv4442503694ContentPasted0">169M /data/glusterfs/gv1/brick2/brick/.shard</div>
<div class="yiv7898679786x_yiv4442503694ContentPasted0">0 /data/glusterfs/gv1/brick2/brick/.glusterfs-anonymous-inode-d3d1fdec-7df9-4f71-b9fc-660d12c2a046</div>
<div class="yiv7898679786x_yiv4442503694ContentPasted0">12G /data/glusterfs/gv1/brick2/brick</div>
</blockquote>
<div class="yiv7898679786x_yiv4442503694ContentPasted0">Also, this is the du -sh output for the specific directory that appears to be taking up space on brick 3:</div>
<blockquote itemscope="" itemtype="https://schemas.microsoft.com/QuotedText" style="border-left:3px solid rgb(200,200,200);border-top-color:rgb(200,200,200);border-right-color:rgb(200,200,200);border-bottom-color:rgb(200,200,200);padding-left:1ex;margin-left:0.8ex;">
<div class="yiv7898679786x_yiv4442503694ContentPasted0">root@uk3-prod-gfs-arb-01:~# du -sh /data/glusterfs/gv1/brick3/brick/.shard/.remove_me/</div>
<div class="yiv7898679786x_yiv4442503694ContentPasted0">10G /data/glusterfs/gv1/brick3/brick/.shard/.remove_me/</div>
</blockquote>
<br>
</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694elementToProof yiv7898679786x_yiv4442503694ContentPasted0">
The gluster package version on all the servers is 11.0, and I believe they were upgraded from 7.2>8.6>11.0; the op.version was only changed after the 11.0 upgrade.</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694elementToProof yiv7898679786x_yiv4442503694ContentPasted0">
<br>
</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694elementToProof yiv7898679786x_yiv4442503694ContentPasted0">
The archival job deletes the files, files shouldn't be overwritten at any point as it's new, unique files being generated everyday.</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694elementToProof yiv7898679786x_yiv4442503694ContentPasted0">
<br>
</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694elementToProof yiv7898679786x_yiv4442503694ContentPasted0">
Thanks,</div>
<div>
<div id="yiv7898679786x_yiv4442503694Signature">
<div>
<div dir="ltr" style="font-size:1px;direction:ltr;">
<table cellpadding="0" cellspacing="0" border="0" style="width:100%;font-size:1px;">
<tbody>
<tr style="font-size:0;">
<td align="left" style="vertical-align:top;">
<table cellpadding="0" cellspacing="0" border="0" style="font-size:0;">
<tbody>
<tr style="font-size:0;">
<td colspan="2" style="padding:0;"> </td>
</tr>
<tr style="font-size:0;">
<td align="left" style="padding:0 0 5px;vertical-align:top;">
<table cellpadding="0" cellspacing="0" border="0" style="font-size:0;">
<tbody>
<tr style="font-size:0;">
<td align="left" style="vertical-align:top;">
<table cellpadding="0" cellspacing="0" border="0" style="font-size:0;">
<tbody>
<tr style="font-size:0;">
<td align="left" style="padding:0 0 11px;vertical-align:top;">
<table cellpadding="0" cellspacing="0" border="0" style="font-size:0px;font-style:normal;font-weight:700;white-space:nowrap;color:rgb(3,90,255);">
<tbody>
<tr style="font-size:14.67px;">
<td align="left" style="vertical-align:top;font-family:Arial;">Liam Smith<span style="font-family:remialcxesans;font-size:1px;line-height:1px;color:rgb(255,255,255);"><span style="font-family:'template-ra4AhfxhEey0egBQ8nUoZA';"></span><span style="font-family:'zone-1';"></span><span style="font-family:'zones-AQ';"></span></span></td>
</tr>
<tr style="font-size:14.67px;font-weight:400;color:rgb(0,0,1);">
<td align="left" style="vertical-align:top;font-family:Arial;">Linux Systems Support Engineer, Scholar</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
<td style="padding:0 0 5px;"> </td>
</tr>
<tr style="font-size:0;">
<td colspan="2" align="left" style="padding:0;vertical-align:top;">
<table cellpadding="0" cellspacing="0" border="0" style="font-size:0;line-height:normal;">
<tbody>
<tr style="font-size:0;">
<td align="left" style="padding:0 0 5px;vertical-align:top;"><a rel="nofollow noopener noreferrer" target="_blank" href="https://www.ek.co" id="yiv7898679786x_yiv4442503694LPlnk689713" style="text-decoration:none;"><img width="168.07" height="49" border="0" alt="" style="width:168.07px;min-width:168.07px;max-width:168.07px;min-height:49px;min-height:49px;max-height:49px;font-size:0;" src="cid:lcnl13YPjmjr0oIzn8Rp" yahoo_partid="2"></a></td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</div>
<div id="yiv7898679786x_yiv4442503694appendonsend"></div>
<hr tabindex="-1" style="display:inline-block;width:98%;">
<div id="yiv7898679786x_yiv4442503694divRplyFwdMsg" dir="ltr"><b>From:</b> Strahil Nikolov <hunter86_bg@yahoo.com><br>
<b>Sent:</b> 04 July 2023 17:47<br>
<b>To:</b> Liam Smith <liam.smith@ek.co>; gluster-users@gluster.org <gluster-users@gluster.org><br>
<b>Subject:</b> Re: [Gluster-users] remove_me files building up
<div> </div>
</div>
<div>
<div class="yiv7898679786x_yiv4442503694x_mc-ip-hide">
<div style="color:#000000;font-size:12px;text-align:left;font-family:Helvetica, Arial, sans-serif;">
<strong>
<table style="width:100%;background:#fff8c4 no-repeat 10px 50%;border:1px solid #f2c779;font-family:Arial, Helvetica, sans-serif;border-radius:5px;padding:1px;">
<tbody>
<tr>
<td>
<p style="text-align:left;color:black;line-height:1;"><strong><span style="color:rgb(255,0,0);font-size:13px;">CAUTION:</span><span style="font-size:13px;"> </span></strong><span style="font-size:13px;">This e-mail originates from outside of Ekco. Do not click
links or attachments unless you recognise the sender. </span></p>
</td>
</tr>
</tbody>
</table>
</strong><br>
</div>
<hr>
</div>
Thanks for the clarification.
<div><br>
</div>
<div>That behaviour is quite weird as arbiter bricks should hold only metadata.</div>
<div><br>
</div>
<div>What does the following show o<span style="">n host </span><span style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:16px;">uk3-prod-gfs-arb-01:</span></div>
<div><span style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:16px;"></span></div>
<div><span style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:16px;"><br>
</span></div>
<div><span style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:16px;">du -h -x -d 1 </span><span style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:16px;">/data/glusterfs/gv1/brick1/brick</span></div>
<div><span style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:16px;">du -h -x -d 1 </span><span style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:16px;">/data/glusterfs/gv1/brick3/brick</span></div>
<div><span style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:16px;">du -h -x -d 1 /</span><span style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:16px;">data/glusterfs/gv1/brick2/brick</span></div>
<div><span style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:16px;"><br>
</span></div>
<div><span style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:16px;"><br>
</span></div>
<div><span style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:16px;">If indeed the shards are taking space - that is a really strange situation.</span></div>
<div><span style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:16px;">From which version did you upgrade and which one is now ? </span><span style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:16px;">I assume all gluster TSP members
(the servers) have the same version, but it’s nice to double check.</span></div>
<div><span style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:16px;"><br>
</span></div>
<div><span style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:16px;">Does the archival job actually deletes the original files after being processed or the workload keeps overriding the existing files ?</span></div>
<div><span style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:16px;"><br>
</span></div>
<div><span style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:16px;">Best Regards,</span></div>
<div><span style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:16px;">Strahil Nikolov </span></div>
<div><br>
</div>
<div><br>
</div>
<div><br>
<br>
<br>
<div class="yiv7898679786x_yiv4442503694x_yahoo-signature"><a rel="nofollow noopener noreferrer" target="_blank" href="https://mail.onelink.me/107872968?pid=nativeplacement&c=Global_Acquisition_YMktg_315_Internal_EmailSignature&af_sub1=Acquisition&af_sub2=Global_YMktg&af_sub3=&af_sub4=100000604&af_sub5=EmailSignature__Static_">Sent from Yahoo Mail for iPhone</a><br>
</div>
<br>
<p style="font-size:15px;color:#715FFA;padding-top:15px;margin-top:0;" class="yiv7898679786x_yiv4442503694x_yahoo-quoted-begin">
On Tuesday, July 4, 2023, 6:50 PM, Liam Smith <liam.smith@ek.co> wrote:</p>
<br>
<br>
<hr>
<p style="font-family:'Courier New';margin-top:0px;">The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure.
</p>
<br>
<br>
<blockquote class="yiv7898679786x_yiv4442503694x_iosymail">
<div id="yiv7898679786x_yiv4442503694x_yiv0069265236"><style type="text/css">
<!--
#yiv7898679786 #yiv7898679786x_yiv4442503694 #yiv7898679786x_yiv4442503694x_yiv0069265236 p
{margin-top:0;margin-bottom:0;}
-->
</style>
<div dir="ltr">
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236elementToProof">
Hi Strahil,</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236elementToProof">
<br>
</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236elementToProof">
We're using gluster to act as a share for an application to temporarily process and store files, before they're then archived off over night.</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236elementToProof">
<br>
</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236elementToProof">
The issue we're seeing isn't with the inodes running out of space, but the actual disk space on the arb server running low.</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236elementToProof">
<br>
</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236elementToProof">
This is the <code>df -h</code> output for the bricks on the arb server:</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236ContentPasted0">
/dev/sdd1 15G 12G 3.3G 79% /data/glusterfs/gv1/brick3
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236ContentPasted0">/dev/sdc1 15G 2.8G 13G 19% /data/glusterfs/gv1/brick1</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236ContentPasted0">/dev/sde1 15G 14G 1.6G 90% /data/glusterfs/gv1/brick2</div>
<br>
</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236ContentPasted0">
And this is the <code>df -hi</code> output for the bricks on the arb server:</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236ContentPasted0 yiv7898679786x_yiv4442503694x_yiv0069265236ContentPasted1">
/dev/sdd1 7.5M 2.7M 4.9M 35% /data/glusterfs/gv1/brick3
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236ContentPasted1">/dev/sdc1 7.5M 643K 6.9M 9% /data/glusterfs/gv1/brick1</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236ContentPasted1">/dev/sde1 6.1M 3.0M 3.1M 49% /data/glusterfs/gv1/brick2</div>
<br>
</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236ContentPasted0 yiv7898679786x_yiv4442503694x_yiv0069265236ContentPasted1">
So the inode usage appears to be fine, but we're seeing that the actual disk usage keeps increasing on the bricks despite it being the arbiter.</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236ContentPasted0 yiv7898679786x_yiv4442503694x_yiv0069265236ContentPasted1">
<br>
</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236ContentPasted0 yiv7898679786x_yiv4442503694x_yiv0069265236ContentPasted1">
The actual issue appears to be that files under <code>/data/glusterfs/gv1/brick3/brick/.shard/.remove_me/</code> and
<span style="display:inline!important;background-color:rgb(40,40,40);" class="yiv7898679786x_yiv4442503694x_yiv0069265236ContentPasted3">
<code>/data/glusterfs/gv1/brick2/brick/.shard/.remove_me/</code> </span><span style="display:inline!important;background-color:rgb(40,40,40);" class="yiv7898679786x_yiv4442503694x_yiv0069265236ContentPasted2"></span>are being retained, even when the original files are
deleted from the data nodes.</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236ContentPasted0 yiv7898679786x_yiv4442503694x_yiv0069265236ContentPasted1">
<br>
</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236ContentPasted0 yiv7898679786x_yiv4442503694x_yiv0069265236ContentPasted1">
For reference, I've attached disk usage graphs for brick 3 over the past two weeks; one is a graph from a data node, the other from the arb.</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236ContentPasted0 yiv7898679786x_yiv4442503694x_yiv0069265236ContentPasted1">
<br>
</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236ContentPasted0 yiv7898679786x_yiv4442503694x_yiv0069265236ContentPasted1">
As you can see, the disk usage of the data node builds throughout the day, but then an archival job clears space down. However, on the arb, we see the disk space increasing in the same sort of trend, but it's never cleared down like the data node.</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236ContentPasted0 yiv7898679786x_yiv4442503694x_yiv0069265236ContentPasted1">
<br>
</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236ContentPasted0 yiv7898679786x_yiv4442503694x_yiv0069265236ContentPasted1">
Hopefully this clarifies the issue, we're a bit confused as to why this is occurring and whether this is actually intended behaviour or potentially a bug, so any advice is greatly appreciated.</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236ContentPasted0 yiv7898679786x_yiv4442503694x_yiv0069265236ContentPasted1">
<br>
</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236ContentPasted0 yiv7898679786x_yiv4442503694x_yiv0069265236ContentPasted1">
Thanks,</div>
<div>
<div id="yiv7898679786x_yiv4442503694x_yiv0069265236Signature">
<div>
<div dir="ltr" style="font-size:1px;direction:ltr;">
<table cellpadding="0" cellspacing="0" border="0" style="width:100%;font-size:1px;">
<tbody>
<tr style="font-size:0;">
<td align="left" style="vertical-align:top;">
<table cellpadding="0" cellspacing="0" border="0" style="font-size:0;">
<tbody>
<tr style="font-size:0;">
<td colspan="2" style="padding:0;"> </td>
</tr>
<tr style="font-size:0;">
<td align="left" style="padding:0 0 5px;vertical-align:top;">
<table cellpadding="0" cellspacing="0" border="0" style="font-size:0;">
<tbody>
<tr style="font-size:0;">
<td align="left" style="vertical-align:top;">
<table cellpadding="0" cellspacing="0" border="0" style="font-size:0;">
<tbody>
<tr style="font-size:0;">
<td align="left" style="padding:0 0 11px;vertical-align:top;">
<table cellpadding="0" cellspacing="0" border="0" style="font-size:0px;font-style:normal;font-weight:700;white-space:nowrap;color:rgb(3,90,255);">
<tbody>
<tr style="font-size:14.67px;">
<td align="left" style="vertical-align:top;font-family:Arial;">Liam Smith<span style="font-family:remialcxesans;font-size:1px;line-height:1px;color:rgb(255,255,255);"><span style="font-family:'template-ra4AhfxhEey0egBQ8nUoZA';"></span><span style="font-family:'zone-1';"></span><span style="font-family:'zones-AQ';"></span></span></td>
</tr>
<tr style="font-size:14.67px;font-weight:400;color:rgb(0,0,1);">
<td align="left" style="vertical-align:top;font-family:Arial;">Linux Systems Support Engineer, Scholar</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
<td style="padding:0 0 5px;"> </td>
</tr>
<tr style="font-size:0;">
<td colspan="2" align="left" style="padding:0;vertical-align:top;"><br>
<br>
<hr>
<p style="font-family:'Courier New';margin-top:0px;">The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure.
</p>
<br>
<br>
<table cellpadding="0" cellspacing="0" border="0" style="font-size:0;line-height:normal;">
<tbody>
<tr style="font-size:0;">
<td align="left" style="padding:0 0 5px;vertical-align:top;"><a rel="nofollow noopener noreferrer" target="_blank" href="https://www.ek.co" id="yiv7898679786x_yiv4442503694x_yiv0069265236LPlnk689713" style="text-decoration:none;"><img width="168.07" height="49" border="0" alt="" style="width:168.07px;min-width:168.07px;max-width:168.07px;min-height:49px;min-height:49px;max-height:49px;font-size:0;" src="cid:7QFshWXAsZCLimJByBZh" yahoo_partid="3"></a></td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</div>
<div id="yiv7898679786x_yiv4442503694x_yiv0069265236appendonsend"></div>
<hr tabindex="-1" style="display:inline-block;width:98%;">
<div id="yiv7898679786x_yiv4442503694x_yiv0069265236divRplyFwdMsg" dir="ltr"><b>From:</b> Strahil Nikolov <hunter86_bg@yahoo.com><br>
<b>Sent:</b> 04 July 2023 15:51<br>
<b>To:</b> Liam Smith <liam.smith@ek.co>; gluster-users@gluster.org <gluster-users@gluster.org><br>
<b>Subject:</b> Re: [Gluster-users] remove_me files building up
<div> </div>
</div>
<div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_mc-ip-hide">
<div style="color:#000000;font-size:12px;text-align:left;font-family:Helvetica, Arial, sans-serif;">
<strong>
<table style="width:100%;background:#fff8c4 no-repeat 10px 50%;border:1px solid #f2c779;font-family:Arial, Helvetica, sans-serif;border-radius:5px;padding:1px;">
<tbody>
<tr>
<td>
<p style="text-align:left;color:black;line-height:1;"><strong><span style="color:rgb(255,0,0);font-size:13px;">CAUTION:</span><span style="font-size:13px;"> </span></strong><span style="font-size:13px;">This e-mail originates from outside of Ekco. Do not click
links or attachments unless you recognise the sender. </span></p>
</td>
</tr>
</tbody>
</table>
</strong><br>
</div>
<hr>
</div>
<div>Hi Liam,</div>
<div><br>
</div>
I saw that your XFS uses ‘<span style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:16px;">imaxpct=25’ which for an arbiter brick is a little bit low.</span>
<div><span style=""><br>
</span></div>
<div><span style="">If you have free space on the bricks, increase the maxpct to a bigger value, like:</span></div>
<div><span style="">xfs_growfs -m 80 /path/to/brick</span></div>
<div><span style="">That will set 80% of the Filesystem for inodes, which you can verify with df -i /brick/path (compare before and after).</span><span style=""> Th</span><span style="">is way</span><span style=""> you won’t run out of inodes in the future.</span></div>
<div><span style=""><br>
</span></div>
<div><span style="">Of course, always test that on non Prod first.</span></div>
<div><span style=""><br>
</span></div>
<div><span style="">Are you using the volume for VM disk storage domain ? What is your main workload ?</span></div>
<div><span style=""><br>
</span></div>
<div><span style="">Best Regards,</span></div>
<div><span style="">Strahil Nikolov <br>
</span><br>
<br>
<br>
<p style="font-size:15px;color:#715FFA;padding-top:15px;margin-top:0;" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yahoo-quoted-begin">
On Tuesday, July 4, 2023, 2:12 PM, Liam Smith <liam.smith@ek.co> wrote:</p>
<blockquote class="yiv7898679786x_yiv4442503694x_yiv0069265236x_iosymail">
<div id="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153"><style type="text/css">
<!--
#yiv7898679786 #yiv7898679786x_yiv4442503694 #yiv7898679786x_yiv4442503694x_yiv0069265236 #yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153 p
{margin-top:0;margin-bottom:0;}
-->
</style>
<div dir="ltr">
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153elementToProof">
Hi,</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153elementToProof">
<br clear="none">
</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153elementToProof">
Thanks for your response, please find the xfs_info for each brick on the arbiter below:</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted0 yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted1 yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted2 yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted3">
<blockquote itemscope="" itemtype="https://schemas.microsoft.com/QuotedText" style="border-left:3px solid rgb(200,200,200);border-top-color:rgb(200,200,200);border-right-color:rgb(200,200,200);border-bottom-color:rgb(200,200,200);padding-left:1ex;margin-left:0.8ex;">
root@uk3-prod-gfs-arb-01:~# xfs_info /data/glusterfs/gv1/brick1
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted0">meta-data=/dev/sdc1 isize=512 agcount=31, agsize=131007 blks</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted0"> = sectsz=512 attr=2, projid32bit=1</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted0"> = crc=1 finobt=1, sparse=1, rmapbt=0</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted0"> = reflink=1</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted0">data = bsize=4096 blocks=3931899, imaxpct=25</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted0"> = sunit=0 swidth=0 blks</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted0">naming =version 2 bsize=4096 ascii-ci=0, ftype=1</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted0">log =internal log bsize=4096 blocks=2560, version=2</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted0"> = sectsz=512 sunit=0 blks, lazy-count=1</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted0">realtime =none extsz=4096 blocks=0, rtextents=0</div>
</blockquote>
<blockquote itemscope="" itemtype="https://schemas.microsoft.com/QuotedText" style="border-left:3px solid rgb(200,200,200);border-top-color:rgb(200,200,200);border-right-color:rgb(200,200,200);border-bottom-color:rgb(200,200,200);padding-left:1ex;margin-left:0.8ex;">
root@uk3-prod-gfs-arb-01:~# xfs_info /data/glusterfs/gv1/brick2
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted1">meta-data=/dev/sde1 isize=512 agcount=13, agsize=327616 blks</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted1"> = sectsz=512 attr=2, projid32bit=1</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted1"> = crc=1 finobt=1, sparse=1, rmapbt=0</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted1"> = reflink=1</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted1">data = bsize=4096 blocks=3931899, imaxpct=25</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted1"> = sunit=0 swidth=0 blks</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted1">naming =version 2 bsize=4096 ascii-ci=0, ftype=1</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted1">log =internal log bsize=4096 blocks=2560, version=2</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted1"> = sectsz=512 sunit=0 blks, lazy-count=1</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted1">realtime =none extsz=4096 blocks=0, rtextents=0</div>
</blockquote>
<blockquote itemscope="" itemtype="https://schemas.microsoft.com/QuotedText" style="border-left:3px solid rgb(200,200,200);border-top-color:rgb(200,200,200);border-right-color:rgb(200,200,200);border-bottom-color:rgb(200,200,200);padding-left:1ex;margin-left:0.8ex;">
root@uk3-prod-gfs-arb-01:~# xfs_info /data/glusterfs/gv1/brick3
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted2">meta-data=/dev/sdd1 isize=512 agcount=13, agsize=327616 blks</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted2"> = sectsz=512 attr=2, projid32bit=1</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted2"> = crc=1 finobt=1, sparse=1, rmapbt=0</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted2"> = reflink=1</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted2">data = bsize=4096 blocks=3931899, imaxpct=25</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted2"> = sunit=0 swidth=0 blks</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted2">naming =version 2 bsize=4096 ascii-ci=0, ftype=1</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted2">log =internal log bsize=4096 blocks=2560, version=2</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted2"> = sectsz=512 sunit=0 blks, lazy-count=1</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted2">realtime =none extsz=4096 blocks=0, rtextents=0</div>
</blockquote>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted0 yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted1 yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted2 yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted3">
<br clear="none">
</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted0 yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted1 yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted2 yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted3">
I've also copied below some df output from the arb server:</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted0 yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted1 yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted2 yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted3 yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted4 yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted5">
<blockquote itemscope="" itemtype="https://schemas.microsoft.com/QuotedText" style="border-left:3px solid rgb(200,200,200);border-top-color:rgb(200,200,200);border-right-color:rgb(200,200,200);border-bottom-color:rgb(200,200,200);padding-left:1ex;margin-left:0.8ex;">
root@uk3-prod-gfs-arb-01:~# df -hi
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted4">Filesystem Inodes IUsed IFree IUse% Mounted on</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted4">udev 992K 473 991K 1% /dev</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted4">tmpfs 995K 788 994K 1% /run</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted4">/dev/sda1 768K 105K 664K 14% /</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted4">tmpfs 995K 3 995K 1% /dev/shm</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted4">tmpfs 995K 4 995K 1% /run/lock</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted4">tmpfs 995K 18 995K 1% /sys/fs/cgroup</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted4">/dev/sdb1 128K 113 128K 1% /var/lib/glusterd</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted4">/dev/sdd1 7.5M 2.6M 5.0M 35% /data/glusterfs/gv1/brick3</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted4">/dev/sdc1 7.5M 600K 7.0M 8% /data/glusterfs/gv1/brick1</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted4">/dev/sde1 6.4M 2.9M 3.5M 46% /data/glusterfs/gv1/brick2</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted4">uk1-prod-gfs-01:/gv1 150M 6.5M 144M 5% /mnt/gfs</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted4">tmpfs 995K 21 995K 1% /run/user/1004</div>
</blockquote>
<blockquote itemscope="" itemtype="https://schemas.microsoft.com/QuotedText" style="border-left:3px solid rgb(200,200,200);border-top-color:rgb(200,200,200);border-right-color:rgb(200,200,200);border-bottom-color:rgb(200,200,200);padding-left:1ex;margin-left:0.8ex;">
root@uk3-prod-gfs-arb-01:~# df -h
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted5">Filesystem Size Used Avail Use% Mounted on</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted5">udev 3.9G 0 3.9G 0% /dev</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted5">tmpfs 796M 916K 795M 1% /run</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted5">/dev/sda1 12G 3.9G 7.3G 35% /</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted5">tmpfs 3.9G 8.0K 3.9G 1% /dev/shm</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted5">tmpfs 5.0M 0 5.0M 0% /run/lock</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted5">tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted5">/dev/sdb1 2.0G 456K 1.9G 1% /var/lib/glusterd</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted5">/dev/sdd1 15G 12G 3.5G 78% /data/glusterfs/gv1/brick3</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted5">/dev/sdc1 15G 2.6G 13G 18% /data/glusterfs/gv1/brick1</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted5">/dev/sde1 15G 14G 1.8G 89% /data/glusterfs/gv1/brick2</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted5">uk1-prod-gfs-01:/gv1 300G 139G 162G 47% /mnt/gfs</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted5">tmpfs 796M 0 796M 0% /run/user/1004</div>
</blockquote>
</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted0 yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted1 yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted2 yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted3">
<br clear="none">
</div>
Something I forgot to mention in my initial message is that the opversion was upgraded from 70200 to 100000, which seems as though it could have been a trigger for the issue as well.</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted0 yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted1 yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted2 yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted3">
<br clear="none">
</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted0 yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted1 yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted2 yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153ContentPasted3">
Thanks,</div>
<div>
<div id="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153Signature">
<div>
<div dir="ltr" style="font-size:1px;direction:ltr;">
<table cellpadding="0" cellspacing="0" border="0" style="width:100%;font-size:1px;">
<tbody>
<tr style="font-size:0;">
<td align="left" colspan="1" rowspan="1" style="vertical-align:top;">
<table cellpadding="0" cellspacing="0" border="0" style="font-size:0;">
<tbody>
<tr style="font-size:0;">
<td colspan="2" rowspan="1" style="padding:0;"> </td>
</tr>
<tr style="font-size:0;">
<td align="left" colspan="1" rowspan="1" style="padding:0 0 5px;vertical-align:top;">
<table cellpadding="0" cellspacing="0" border="0" style="font-size:0;">
<tbody>
<tr style="font-size:0;">
<td align="left" colspan="1" rowspan="1" style="vertical-align:top;">
<table cellpadding="0" cellspacing="0" border="0" style="font-size:0;">
<tbody>
<tr style="font-size:0;">
<td align="left" colspan="1" rowspan="1" style="padding:0 0 11px;vertical-align:top;">
<table cellpadding="0" cellspacing="0" border="0" style="font-size:0px;font-style:normal;font-weight:700;white-space:nowrap;color:rgb(3,90,255);">
<tbody>
<tr style="font-size:14.67px;">
<td align="left" colspan="1" rowspan="1" style="vertical-align:top;font-family:Arial;">
Liam Smith<span style="font-family:remialcxesans;font-size:1px;line-height:1px;color:rgb(255,255,255);"><span style="font-family:'template-ra4AhfxhEey0egBQ8nUoZA';"></span><span style="font-family:'zone-1';"></span><span style="font-family:'zones-AQ';"></span></span></td>
</tr>
<tr style="font-size:14.67px;font-weight:400;color:rgb(0,0,1);">
<td align="left" colspan="1" rowspan="1" style="vertical-align:top;font-family:Arial;">
Linux Systems Support Engineer, Scholar</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
<td colspan="1" rowspan="1" style="padding:0 0 5px;"> </td>
</tr>
<tr style="font-size:0;">
<td align="left" colspan="2" rowspan="1" style="padding:0;vertical-align:top;">
<table cellpadding="0" cellspacing="0" border="0" style="font-size:0;line-height:normal;">
<tbody>
<tr style="font-size:0;">
<td align="left" colspan="1" rowspan="1" style="padding:0 0 5px;vertical-align:top;">
<a rel="nofollow noopener noreferrer" shape="rect" id="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153LPlnk689713" target="_blank" href="https://www.ek.co" style="text-decoration:none;"><img width="168.07" height="49" border="0" alt="" style="width:168.07px;min-width:168.07px;max-width:168.07px;min-height:49px;min-height:49px;max-height:49px;font-size:0;" src="cid:IPIMjUP96PPE6N51Udhx" yahoo_partid="4"></a></td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</div>
<div id="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153appendonsend"></div>
<hr tabindex="-1" style="display:inline-block;width:98%;">
<div dir="ltr" id="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153divRplyFwdMsg"><b>From:</b> Strahil Nikolov <hunter86_bg@yahoo.com><br clear="none">
<b>Sent:</b> 03 July 2023 18:28<br clear="none">
<b>To:</b> Liam Smith <liam.smith@ek.co>; gluster-users@gluster.org <gluster-users@gluster.org><br clear="none">
<b>Subject:</b> Re: [Gluster-users] remove_me files building up
<div> </div>
</div>
<div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_mc-ip-hide">
<div style="color:#000000;font-size:12px;text-align:left;font-family:Helvetica, Arial, sans-serif;">
<strong></strong>
<table style="width:100%;background:#fff8c4 no-repeat 10px 50%;border:1px solid #f2c779;font-family:Arial, Helvetica, sans-serif;border-radius:5px;padding:1px;">
<tbody>
<tr>
<td colspan="1" rowspan="1">
<p style="text-align:left;color:black;line-height:1;"><strong><span style="color:rgb(255,0,0);font-size:13px;">CAUTION:</span><span style="font-size:13px;"> </span></strong><span style="font-size:13px;">This e-mail originates from outside of Ekco. Do not click
links or attachments unless you recognise the sender. </span></p>
</td>
</tr>
</tbody>
</table>
<br clear="none">
</div>
<hr>
</div>
Hi,
<div><br clear="none">
</div>
<div>you mentioned that the arbiter bricks run out of inodes.</div>
<div>Are you using XFS ?</div>
<div>Can you provide the xfs_info of each brick ?</div>
<div><br clear="none">
</div>
<div>Best Regards,</div>
<div>Strahil Nikolov <br clear="none">
<br clear="none">
<br clear="none">
<hr>
<p style="font-family:'Courier New';margin-top:0px;">The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure.
</p>
<div id="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153yqtfd02980" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153yqt0084106110">
<br clear="none">
<br clear="none">
<br clear="none">
<blockquote style="margin:0 0 20px 0;">
<div style="font-family:Roboto, sans-serif;color:#6D00F6;">
<div>On Sat, Jul 1, 2023 at 19:41, Liam Smith</div>
<div><liam.smith@ek.co> wrote:</div>
</div>
<div style="padding:10px 0 0 20px;margin:10px 0 0 0;border-left:1px solid #6D00F6;">
<div id="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085">
<div dir="ltr">
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085elementToProof">
Hi,</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085elementToProof">
<br clear="none">
</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085elementToProof">
We're running a cluster with two data nodes and one arbiter, and have sharding enabled.</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085elementToProof">
<br clear="none">
</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085elementToProof">
We had an issue a while back where one of the server's crashed, we got the server back up and running and ensured that all healing entries cleared, and also increased the server spec (CPU/Mem) as this seemed to be the potential cause.</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085elementToProof">
<br clear="none">
</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted0 yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted1">
Since then however, we've seen some strange behaviour, whereby a lot of 'remove_me' files are building up under `/data/glusterfs/gv1/brick2/brick/.shard/.remove_me/` and `/data/glusterfs/gv1/brick3/brick/.shard/.remove_me/`. <span style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;">This
is causing the arbiter to run out of space on brick2 and brick3, as the remove_me files are constantly increasing.</span></div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted0 yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted1">
<span style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;"><br clear="none">
</span></div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted0 yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted1">
brick1 appears to be fine, the disk usage increases throughout the day and drops down in line with the trend of the brick on the data nodes. We see the disk usage increase and drop throughout the day on the data nodes for brick2 and brick3 as well, but while
the arbiter follows the same trend of the disk usage increasing, it doesn't drop at any point.</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted0 yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted1">
<br clear="none">
</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted0 yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted1">
This is the output of some gluster commands, occasional heal entries come and go:</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted5">
<blockquote itemscope="" itemtype="https://schemas.microsoft.com/QuotedText" style="border-left:3px solid rgb(200,200,200);border-top-color:rgb(200,200,200);border-right-color:rgb(200,200,200);border-bottom-color:rgb(200,200,200);padding-left:1ex;margin-left:0.8ex;">
</blockquote>
<blockquote itemscope="" itemtype="https://schemas.microsoft.com/QuotedText" style="border-left:3px solid rgb(200,200,200);border-top-color:rgb(200,200,200);border-right-color:rgb(200,200,200);border-bottom-color:rgb(200,200,200);padding-left:1ex;margin-left:0.8ex;">
root@uk3-prod-gfs-arb-01:~# gluster volume info gv1
<div><br clear="none" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Volume Name: gv1</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Type: Distributed-Replicate</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Volume ID: d3d1fdec-7df9-4f71-b9fc-660d12c2a046</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Status: Started</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Snapshot Count: 0</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Number of Bricks: 3 x (2 + 1) = 9</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Transport-type: tcp</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Bricks:</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Brick1: uk1-prod-gfs-01:/data/glusterfs/gv1/brick1/brick</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Brick2: uk2-prod-gfs-01:/data/glusterfs/gv1/brick1/brick</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Brick3: uk3-prod-gfs-arb-01:/data/glusterfs/gv1/brick1/brick (arbiter)</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Brick4: uk1-prod-gfs-01:/data/glusterfs/gv1/brick3/brick</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Brick5: uk2-prod-gfs-01:/data/glusterfs/gv1/brick3/brick</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Brick6: uk3-prod-gfs-arb-01:/data/glusterfs/gv1/brick3/brick (arbiter)</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Brick7: uk1-prod-gfs-01:/data/glusterfs/gv1/brick2/brick</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Brick8: uk2-prod-gfs-01:/data/glusterfs/gv1/brick2/brick</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Brick9: uk3-prod-gfs-arb-01:/data/glusterfs/gv1/brick2/brick (arbiter)</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Options Reconfigured:</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
cluster.entry-self-heal: on</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
cluster.metadata-self-heal: on</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
cluster.data-self-heal: on</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
performance.client-io-threads: off</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
storage.fips-mode-rchecksum: on</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
transport.address-family: inet</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
cluster.lookup-optimize: off</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
performance.readdir-ahead: off</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
cluster.readdir-optimize: off</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
cluster.self-heal-daemon: enable</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
features.shard: enable</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
features.shard-block-size: 512MB</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
cluster.min-free-disk: 10%</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
cluster.use-anonymous-inode: yes</div>
</blockquote>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085elementToProof">
<br clear="none">
</div>
<blockquote itemscope="" itemtype="https://schemas.microsoft.com/QuotedText" style="border-left:3px solid rgb(200,200,200);border-top-color:rgb(200,200,200);border-right-color:rgb(200,200,200);border-bottom-color:rgb(200,200,200);padding-left:1ex;margin-left:0.8ex;">
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
<span style="">root@uk3-prod-gfs-arb-01:~# gluster peer status</span><br clear="none">
</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Number of Peers: 2</div>
<div><br clear="none" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Hostname: uk2-prod-gfs-01</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Uuid: 2fdfa4a2-195d-4cc5-937c-f48466e76149</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
State: Peer in Cluster (Connected)</div>
<div><br clear="none" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Hostname: uk1-prod-gfs-01</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Uuid: 43ec93d1-ad83-4103-aea3-80ded0903d88</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
State: Peer in Cluster (Connected)</div>
</blockquote>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085elementToProof">
<br clear="none">
</div>
<blockquote itemscope="" itemtype="https://schemas.microsoft.com/QuotedText" style="border-left:3px solid rgb(200,200,200);border-top-color:rgb(200,200,200);border-right-color:rgb(200,200,200);border-bottom-color:rgb(200,200,200);padding-left:1ex;margin-left:0.8ex;">
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
<span style="">root@uk3-prod-gfs-arb-01:~# gluster volume heal gv1 info</span><br clear="none">
</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Brick uk1-prod-gfs-01:/data/glusterfs/gv1/brick1/brick</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
<gfid:5b57e1f6-3e3d-4334-a0db-b2560adae6d1></div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Status: Connected</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Number of entries: 1</div>
<div><br clear="none" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Brick uk2-prod-gfs-01:/data/glusterfs/gv1/brick1/brick</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Status: Connected</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Number of entries: 0</div>
<div><br clear="none" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Brick uk3-prod-gfs-arb-01:/data/glusterfs/gv1/brick1/brick</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Status: Connected</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Number of entries: 0</div>
<div><br clear="none" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Brick uk1-prod-gfs-01:/data/glusterfs/gv1/brick3/brick</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Status: Connected</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Number of entries: 0</div>
<div><br clear="none" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Brick uk2-prod-gfs-01:/data/glusterfs/gv1/brick3/brick</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Status: Connected</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Number of entries: 0</div>
<div><br clear="none" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Brick uk3-prod-gfs-arb-01:/data/glusterfs/gv1/brick3/brick</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Status: Connected</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Number of entries: 0</div>
<div><br clear="none" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Brick uk1-prod-gfs-01:/data/glusterfs/gv1/brick2/brick</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Status: Connected</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Number of entries: 0</div>
<div><br clear="none" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Brick uk2-prod-gfs-01:/data/glusterfs/gv1/brick2/brick</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
<gfid:6ba9c472-9232-4b45-b12f-a1232d6f4627></div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
/.shard/.remove_me</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
<gfid:0f042518-248d-426a-93f4-cfaa92b6ef3e></div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Status: Connected</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Number of entries: 3</div>
<div><br clear="none" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Brick uk3-prod-gfs-arb-01:/data/glusterfs/gv1/brick2/brick</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
<gfid:6ba9c472-9232-4b45-b12f-a1232d6f4627></div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
/.shard/.remove_me</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
<gfid:0f042518-248d-426a-93f4-cfaa92b6ef3e></div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Status: Connected</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted4">
Number of entries: 3</div>
</blockquote>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted5">
<br clear="none">
</div>
<blockquote itemscope="" itemtype="https://schemas.microsoft.com/QuotedText" style="border-left:3px solid rgb(200,200,200);border-top-color:rgb(200,200,200);border-right-color:rgb(200,200,200);border-bottom-color:rgb(200,200,200);padding-left:1ex;margin-left:0.8ex;">
root@uk3-prod-gfs-arb-01:~# gluster volume get all cluster.op-version
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted5">
Option Value</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted5">
------ -----</div>
<div class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted5">
cluster.op-version 100000</div>
</blockquote>
</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted5">
<br clear="none">
</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted0 yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted1">
We're not sure if this is a potential bug or if something's corrupted that we don't have visibility of, so any pointers/suggestions about how to approach this would be appreciated. </div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted0 yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted1">
<br clear="none">
</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted0 yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted1">
Thanks,</div>
<div style="font-family:Calibri, Arial, Helvetica, sans-serif;font-size:12pt;color:rgb(0,0,0);" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085elementToProof yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted0 yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085ContentPasted1">
Liam</div>
<div>
<div id="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085Signature">
<div>
<div dir="ltr" style="font-size:1px;direction:ltr;">
<table id="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153x_yiv8312530085tableSelected2" cellpadding="0" cellspacing="0" border="0" style="width:100%;font-size:1px;">
<tbody>
<tr style="font-size:0;">
<td align="left" colspan="1" rowspan="1" style="vertical-align:top;"><br clear="none">
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</div>
<br clear="none">
<br clear="none">
<hr>
<p style="font-family:'Courier New';margin-top:0px;">The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure.
</p>
<br clear="none">
<br clear="none">
</div>
</div>
________<br clear="none">
<br clear="none">
<br clear="none">
<br clear="none">
Community Meeting Calendar:<br clear="none">
<br clear="none">
Schedule -<br clear="none">
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br clear="none">
Bridge: <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://meet.google.com/cpu-eiue-hvk">
https://meet.google.com/cpu-eiue-hvk</a><br clear="none">
Gluster-users mailing list<br clear="none">
<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:Gluster-users@gluster.org" target="_blank" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br clear="none">
<a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://lists.gluster.org/mailman/listinfo/gluster-users">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br clear="none">
</div>
</blockquote>
</div>
</div>
<div id="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153yqtfd64290" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153yqt0084106110">
</div>
</div>
<div id="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153yqtfd83570" class="yiv7898679786x_yiv4442503694x_yiv0069265236x_yiv8784601153yqt0084106110">
</div>
</div>
</div>
<blockquote></blockquote>
</blockquote>
</div>
</div>
</div>
</div>
<blockquote></blockquote>
</blockquote>
</div>
</div>
</div>
</div>
<blockquote></blockquote>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div><blockquote></blockquote></blockquote></div>
</body></html>