<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head><body dir="ltr"><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; --darkreader-inline-color: var(--darkreader-text--darkColor_rgb_0__0__0_, #ebdbb2); color: rgb(0, 0, 0);" data-darkreader-inline-color="" class="elementToProof">
Hi,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; --darkreader-inline-color: var(--darkreader-text--darkColor_rgb_0__0__0_, #ebdbb2); color: rgb(0, 0, 0);" data-darkreader-inline-color="" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; --darkreader-inline-color: var(--darkreader-text--darkColor_rgb_0__0__0_, #ebdbb2); color: rgb(0, 0, 0);" data-darkreader-inline-color="" class="elementToProof">
We're running a cluster with two data nodes and one arbiter, and have sharding enabled.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; --darkreader-inline-color: var(--darkreader-text--darkColor_rgb_0__0__0_, #ebdbb2); color: rgb(0, 0, 0);" data-darkreader-inline-color="" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; --darkreader-inline-color: var(--darkreader-text--darkColor_rgb_0__0__0_, #ebdbb2); color: rgb(0, 0, 0);" data-darkreader-inline-color="" class="elementToProof">
We had an issue a while back where one of the server's crashed, we got the server back up and running and ensured that all healing entries cleared, and also increased the server spec (CPU/Mem) as this seemed to be the potential cause.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; --darkreader-inline-color: var(--darkreader-text--darkColor_rgb_0__0__0_, #ebdbb2); color: rgb(0, 0, 0);" data-darkreader-inline-color="" class="elementToProof">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; --darkreader-inline-color: var(--darkreader-text--darkColor_rgb_0__0__0_, #ebdbb2); color: rgb(0, 0, 0);" data-darkreader-inline-color="" class="elementToProof ContentPasted0 ContentPasted1">
Since then however, we've seen some strange behaviour, whereby a lot of 'remove_me' files are building up under `/data/glusterfs/gv1/brick2/brick/.shard/.remove_me/` and `/data/glusterfs/gv1/brick3/brick/.shard/.remove_me/`. <span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; --darkreader-inline-bgcolor: var(--darkreader-bg--darkreader-bg--neutralPrimarySurface);" data-darkreader-inline-bgcolor="">This
is causing the arbiter to run out of space on brick2 and brick3, as the remove_me files are constantly increasing.</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; --darkreader-inline-color: var(--darkreader-text--darkColor_rgb_0__0__0_, #ebdbb2); color: rgb(0, 0, 0);" data-darkreader-inline-color="" class="elementToProof ContentPasted0 ContentPasted1">
<span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; --darkreader-inline-bgcolor: var(--darkreader-bg--darkreader-bg--neutralPrimarySurface);" data-darkreader-inline-bgcolor=""><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; --darkreader-inline-color: var(--darkreader-text--darkColor_rgb_0__0__0_, #ebdbb2); color: rgb(0, 0, 0);" data-darkreader-inline-color="" class="elementToProof ContentPasted0 ContentPasted1">
brick1 appears to be fine, the disk usage increases throughout the day and drops down in line with the trend of the brick on the data nodes. We see the disk usage increase and drop throughout the day on the data nodes for brick2 and brick3 as well, but while
the arbiter follows the same trend of the disk usage increasing, it doesn't drop at any point.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; --darkreader-inline-color: var(--darkreader-text--darkColor_rgb_0__0__0_, #ebdbb2); color: rgb(0, 0, 0);" data-darkreader-inline-color="" class="elementToProof ContentPasted0 ContentPasted1">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; --darkreader-inline-color: var(--darkreader-text--darkColor_rgb_0__0__0_, #ebdbb2); color: rgb(0, 0, 0);" data-darkreader-inline-color="" class="elementToProof ContentPasted0 ContentPasted1">
This is the output of some gluster commands, occasional heal entries come and go:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; --darkreader-inline-color: var(--darkreader-text--darkColor_rgb_0__0__0_, #ebdbb2); color: rgb(0, 0, 0);" data-darkreader-inline-color="" class="elementToProof ContentPasted5">
<blockquote itemscope="" itemtype="https://schemas.microsoft.com/QuotedText" style="border-left: 3px solid rgb(200, 200, 200); border-top-color: rgb(200, 200, 200); border-right-color: rgb(200, 200, 200); border-bottom-color: rgb(200, 200, 200); padding-left: 1ex; margin-left: 0.8ex; --darkreader-inline-border-left: #444444; --darkreader-inline-border-top: #444444; --darkreader-inline-border-right: #444444; --darkreader-inline-border-bottom: #444444;" data-darkreader-inline-border-left="" data-darkreader-inline-border-top="" data-darkreader-inline-border-right="" data-darkreader-inline-border-bottom="">
</blockquote>
<blockquote itemscope="" itemtype="https://schemas.microsoft.com/QuotedText" style="border-left: 3px solid rgb(200, 200, 200); border-top-color: rgb(200, 200, 200); border-right-color: rgb(200, 200, 200); border-bottom-color: rgb(200, 200, 200); padding-left: 1ex; margin-left: 0.8ex; --darkreader-inline-border-left: #444444; --darkreader-inline-border-top: #444444; --darkreader-inline-border-right: #444444; --darkreader-inline-border-bottom: #444444;" data-darkreader-inline-border-left="" data-darkreader-inline-border-top="" data-darkreader-inline-border-right="" data-darkreader-inline-border-bottom="">
root@uk3-prod-gfs-arb-01:~# gluster volume info gv1
<div><br class="ContentPasted4">
</div>
<div class="ContentPasted4">Volume Name: gv1</div>
<div class="ContentPasted4">Type: Distributed-Replicate</div>
<div class="ContentPasted4">Volume ID: d3d1fdec-7df9-4f71-b9fc-660d12c2a046</div>
<div class="ContentPasted4">Status: Started</div>
<div class="ContentPasted4">Snapshot Count: 0</div>
<div class="ContentPasted4">Number of Bricks: 3 x (2 + 1) = 9</div>
<div class="ContentPasted4">Transport-type: tcp</div>
<div class="ContentPasted4">Bricks:</div>
<div class="ContentPasted4">Brick1: uk1-prod-gfs-01:/data/glusterfs/gv1/brick1/brick</div>
<div class="ContentPasted4">Brick2: uk2-prod-gfs-01:/data/glusterfs/gv1/brick1/brick</div>
<div class="ContentPasted4">Brick3: uk3-prod-gfs-arb-01:/data/glusterfs/gv1/brick1/brick (arbiter)</div>
<div class="ContentPasted4">Brick4: uk1-prod-gfs-01:/data/glusterfs/gv1/brick3/brick</div>
<div class="ContentPasted4">Brick5: uk2-prod-gfs-01:/data/glusterfs/gv1/brick3/brick</div>
<div class="ContentPasted4">Brick6: uk3-prod-gfs-arb-01:/data/glusterfs/gv1/brick3/brick (arbiter)</div>
<div class="ContentPasted4">Brick7: uk1-prod-gfs-01:/data/glusterfs/gv1/brick2/brick</div>
<div class="ContentPasted4">Brick8: uk2-prod-gfs-01:/data/glusterfs/gv1/brick2/brick</div>
<div class="ContentPasted4">Brick9: uk3-prod-gfs-arb-01:/data/glusterfs/gv1/brick2/brick (arbiter)</div>
<div class="ContentPasted4">Options Reconfigured:</div>
<div class="ContentPasted4">cluster.entry-self-heal: on</div>
<div class="ContentPasted4">cluster.metadata-self-heal: on</div>
<div class="ContentPasted4">cluster.data-self-heal: on</div>
<div class="ContentPasted4">performance.client-io-threads: off</div>
<div class="ContentPasted4">storage.fips-mode-rchecksum: on</div>
<div class="ContentPasted4">transport.address-family: inet</div>
<div class="ContentPasted4">cluster.lookup-optimize: off</div>
<div class="ContentPasted4">performance.readdir-ahead: off</div>
<div class="ContentPasted4">cluster.readdir-optimize: off</div>
<div class="ContentPasted4">cluster.self-heal-daemon: enable</div>
<div class="ContentPasted4">features.shard: enable</div>
<div class="ContentPasted4">features.shard-block-size: 512MB</div>
<div class="ContentPasted4">cluster.min-free-disk: 10%</div>
<div class="ContentPasted4">cluster.use-anonymous-inode: yes</div>
</blockquote>
<div class="elementToProof"><br>
</div>
<blockquote itemscope="" itemtype="https://schemas.microsoft.com/QuotedText" style="border-left: 3px solid rgb(200, 200, 200); border-top-color: rgb(200, 200, 200); border-right-color: rgb(200, 200, 200); border-bottom-color: rgb(200, 200, 200); padding-left: 1ex; margin-left: 0.8ex; --darkreader-inline-border-left: #444444; --darkreader-inline-border-top: #444444; --darkreader-inline-border-right: #444444; --darkreader-inline-border-bottom: #444444;" data-darkreader-inline-border-left="" data-darkreader-inline-border-top="" data-darkreader-inline-border-right="" data-darkreader-inline-border-bottom="">
<div class="ContentPasted4"><span style="">root@uk3-prod-gfs-arb-01:~# gluster peer status</span><br>
</div>
<div class="ContentPasted4">Number of Peers: 2</div>
<div><br class="ContentPasted4">
</div>
<div class="ContentPasted4">Hostname: uk2-prod-gfs-01</div>
<div class="ContentPasted4">Uuid: 2fdfa4a2-195d-4cc5-937c-f48466e76149</div>
<div class="ContentPasted4">State: Peer in Cluster (Connected)</div>
<div><br class="ContentPasted4">
</div>
<div class="ContentPasted4">Hostname: uk1-prod-gfs-01</div>
<div class="ContentPasted4">Uuid: 43ec93d1-ad83-4103-aea3-80ded0903d88</div>
<div class="ContentPasted4">State: Peer in Cluster (Connected)</div>
</blockquote>
<div class="elementToProof"><br>
</div>
<blockquote itemscope="" itemtype="https://schemas.microsoft.com/QuotedText" style="border-left: 3px solid rgb(200, 200, 200); border-top-color: rgb(200, 200, 200); border-right-color: rgb(200, 200, 200); border-bottom-color: rgb(200, 200, 200); padding-left: 1ex; margin-left: 0.8ex; --darkreader-inline-border-left: #444444; --darkreader-inline-border-top: #444444; --darkreader-inline-border-right: #444444; --darkreader-inline-border-bottom: #444444;" data-darkreader-inline-border-left="" data-darkreader-inline-border-top="" data-darkreader-inline-border-right="" data-darkreader-inline-border-bottom="">
<div class="ContentPasted4"><span style="">root@uk3-prod-gfs-arb-01:~# gluster volume heal gv1 info</span><br>
</div>
<div class="ContentPasted4">Brick uk1-prod-gfs-01:/data/glusterfs/gv1/brick1/brick</div>
<div class="ContentPasted4"><gfid:5b57e1f6-3e3d-4334-a0db-b2560adae6d1></div>
<div class="ContentPasted4">Status: Connected</div>
<div class="ContentPasted4">Number of entries: 1</div>
<div><br class="ContentPasted4">
</div>
<div class="ContentPasted4">Brick uk2-prod-gfs-01:/data/glusterfs/gv1/brick1/brick</div>
<div class="ContentPasted4">Status: Connected</div>
<div class="ContentPasted4">Number of entries: 0</div>
<div><br class="ContentPasted4">
</div>
<div class="ContentPasted4">Brick uk3-prod-gfs-arb-01:/data/glusterfs/gv1/brick1/brick</div>
<div class="ContentPasted4">Status: Connected</div>
<div class="ContentPasted4">Number of entries: 0</div>
<div><br class="ContentPasted4">
</div>
<div class="ContentPasted4">Brick uk1-prod-gfs-01:/data/glusterfs/gv1/brick3/brick</div>
<div class="ContentPasted4">Status: Connected</div>
<div class="ContentPasted4">Number of entries: 0</div>
<div><br class="ContentPasted4">
</div>
<div class="ContentPasted4">Brick uk2-prod-gfs-01:/data/glusterfs/gv1/brick3/brick</div>
<div class="ContentPasted4">Status: Connected</div>
<div class="ContentPasted4">Number of entries: 0</div>
<div><br class="ContentPasted4">
</div>
<div class="ContentPasted4">Brick uk3-prod-gfs-arb-01:/data/glusterfs/gv1/brick3/brick</div>
<div class="ContentPasted4">Status: Connected</div>
<div class="ContentPasted4">Number of entries: 0</div>
<div><br class="ContentPasted4">
</div>
<div class="ContentPasted4">Brick uk1-prod-gfs-01:/data/glusterfs/gv1/brick2/brick</div>
<div class="ContentPasted4">Status: Connected</div>
<div class="ContentPasted4">Number of entries: 0</div>
<div><br class="ContentPasted4">
</div>
<div class="ContentPasted4">Brick uk2-prod-gfs-01:/data/glusterfs/gv1/brick2/brick</div>
<div class="ContentPasted4"><gfid:6ba9c472-9232-4b45-b12f-a1232d6f4627></div>
<div class="ContentPasted4">/.shard/.remove_me</div>
<div class="ContentPasted4"><gfid:0f042518-248d-426a-93f4-cfaa92b6ef3e></div>
<div class="ContentPasted4">Status: Connected</div>
<div class="ContentPasted4">Number of entries: 3</div>
<div><br class="ContentPasted4">
</div>
<div class="ContentPasted4">Brick uk3-prod-gfs-arb-01:/data/glusterfs/gv1/brick2/brick</div>
<div class="ContentPasted4"><gfid:6ba9c472-9232-4b45-b12f-a1232d6f4627></div>
<div class="ContentPasted4">/.shard/.remove_me</div>
<div class="ContentPasted4"><gfid:0f042518-248d-426a-93f4-cfaa92b6ef3e></div>
<div class="ContentPasted4">Status: Connected</div>
<div class="ContentPasted4">Number of entries: 3</div>
</blockquote>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; --darkreader-inline-color: var(--darkreader-text--darkColor_rgb_0__0__0_, #ebdbb2); color: rgb(0, 0, 0);" data-darkreader-inline-color="" class="elementToProof ContentPasted5">
<br>
</div>
<blockquote itemscope="" itemtype="https://schemas.microsoft.com/QuotedText" style="border-left: 3px solid rgb(200, 200, 200); border-top-color: rgb(200, 200, 200); border-right-color: rgb(200, 200, 200); border-bottom-color: rgb(200, 200, 200); padding-left: 1ex; margin-left: 0.8ex; --darkreader-inline-border-left: #444444; --darkreader-inline-border-top: #444444; --darkreader-inline-border-right: #444444; --darkreader-inline-border-bottom: #444444;" data-darkreader-inline-border-left="" data-darkreader-inline-border-top="" data-darkreader-inline-border-right="" data-darkreader-inline-border-bottom="">
root@uk3-prod-gfs-arb-01:~# gluster volume get all cluster.op-version
<div class="ContentPasted5">Option Value</div>
<div class="ContentPasted5">------ -----</div>
<div class="ContentPasted5">cluster.op-version 100000</div>
</blockquote>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; --darkreader-inline-color: var(--darkreader-text--darkColor_rgb_0__0__0_, #ebdbb2); color: rgb(0, 0, 0);" data-darkreader-inline-color="" class="elementToProof ContentPasted5">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; --darkreader-inline-color: var(--darkreader-text--darkColor_rgb_0__0__0_, #ebdbb2); color: rgb(0, 0, 0);" data-darkreader-inline-color="" class="elementToProof ContentPasted0 ContentPasted1">
We're not sure if this is a potential bug or if something's corrupted that we don't have visibility of, so any pointers/suggestions about how to approach this would be appreciated. </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; --darkreader-inline-color: var(--darkreader-text--darkColor_rgb_0__0__0_, #ebdbb2); color: rgb(0, 0, 0);" data-darkreader-inline-color="" class="elementToProof ContentPasted0 ContentPasted1">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; --darkreader-inline-color: var(--darkreader-text--darkColor_rgb_0__0__0_, #ebdbb2); color: rgb(0, 0, 0);" data-darkreader-inline-color="" class="elementToProof ContentPasted0 ContentPasted1">
Thanks,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; --darkreader-inline-color: var(--darkreader-text--darkColor_rgb_0__0__0_, #ebdbb2); color: rgb(0, 0, 0);" data-darkreader-inline-color="" class="elementToProof ContentPasted0 ContentPasted1">
Liam</div>
<div>
<div id="Signature">
<div>
<div dir="ltr" style="font-size:1px; direction:ltr">
<table cellpadding="0" cellspacing="0" border="0" style="width:100%; font-size:1px" id="tableSelected2">
<tbody>
<tr style="font-size:0">
<td align="left" style="vertical-align:top"><br>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</div>
<br>
<br>
<hr>
<p style="font-family: 'Courier New';margin-top: 0px">
The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential and/or privileged information and may be legally protected from disclosure.
</p>
<br>
<br>
</body></html>