<div dir="ltr"><div dir="ltr">Hi Dan,</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jul 12, 2021 at 2:20 PM Dan Thomson <<a href="mailto:dthomson@triumf.ca">dthomson@triumf.ca</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi gluster users,<br>
<br>
I'm having an issue that I'm hoping to get some help with on a<br>
dispersed volume (EC: 2x(4+2)) that's causing me some headaches. This is<br>
on a cluster running Gluster 6.9 on CentOS 7.<br>
<br>
At some point in the last week, writes to one of my bricks have started<br>
failing due to an "No Space Left on Device" error:<br>
<br>
[2021-07-06 16:08:57.261307] E [MSGID: 115067] [server-rpc-fops_v2.c:1373:server4_writev_cbk] 0-gluster-01-server: 1853436561: WRITEV -2 (f2d6f2f8-4fd7-4692-bd60-23124897be54), client: CTX_ID:648a7383-46c8-4ed7-a921-acafc90bec1a-GRAPH_ID:4-PID:19471-HOST:rhevh08.mgmt.triumf.ca-PC_NAME:gluster-01-client-5-RECON_NO:-5, error-xlator: gluster-01-posix [No space left on device]<br>
<br>
The disk is quite full (listed as 100% on the server), but does have<br>
some writable room left:<br>
<br>
/dev/mapper/vg--brick1-brick1 11T 11T 97G 100% /data/glusterfs/gluster-01/brick1<br>
<br>
however, I'm not sure if the amount of disk space used on the physical<br>
drive is the true cause of the "No Space Left on Device" errors anyway.<br>
I can still manually write to this brick outside of Gluster, so it seems<br>
like the operating system isn't preventing the writes from happening.<br></blockquote><div><br></div><div>As Strahil has said, you are probably hitting the minimum space reserved by Gluster. You can try those options. However I don't recommend keeping bricks above 90% utilization. All filesystems, including XFS, tend to degrade performance when available space is limited. If the brick's filesystem works worse, Gluster performance will also drop.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
During my investigation, I noticed that one .glusterfs paths on the problem<br>
server is using up much more space than it is on the other servers. I can't<br>
quite figure out why that might be, or how that happened. I'm wondering<br>
if there's any advice on what the cause might've been.<br>
<br>
I had done some package updates on this server with the issue and not on the<br>
other servers. This included the kernel version, but didn't include the Gluster<br>
packages. So possibly this, or the reboot to load the new kernel may<br>
have caused a problem. I have scripts on my gluster machines to nicely kill<br>
all of the brick processes before rebooting, so I'm not leaning towards<br>
an abrupt shutdown being the cause, but it's a possibility.<br>
<br>
I'm also looking for advice on how to safely remove the problem file and<br>
rebuild it from the other Gluster peers. I've seen some documentation on<br>
this, but I'm a little nervous about corrupting the volume if I<br>
misunderstand the process. I'm not free to take the volume or cluster down and<br>
do maintenance at this point, but that might be something I'll have to consider<br>
if it's my only option.<br>
<br>
For reference, here's the comparison of the same path that seems to be<br>
taking up extra space on one of the hosts:<br>
<br>
1: 26G /data/gluster-01/brick1/vol/.glusterfs/99/56<br>
2: 26G /data/gluster-01/brick1/vol/.glusterfs/99/56<br>
3: 26G /data/gluster-01/brick1/vol/.glusterfs/99/56<br>
4: 26G /data/gluster-01/brick1/vol/.glusterfs/99/56<br>
5: 26G /data/gluster-01/brick1/vol/.glusterfs/99/56<br>
6: 3.0T /data/gluster-01/brick1/vol/.glusterfs/99/56<br></blockquote><div><br></div><div>This is not normal at all. In a dispersed volume all bricks should use roughly the same used space.</div><div><br></div><div>Can you provide the output of the following commands:</div><div><br></div><div><font face="monospace"> # gluster volume info <volname></font></div><div><font face="monospace"> # gluster volume status <volname></font></div><div><br></div><div>Also provide the output of this command from all bricks:</div><div><br></div><div> # ls -ls /data/gluster-01/brick1/vol/.glusterfs/99/56</div><div><br></div><div>Regards,</div><div><br></div><div>Xavi</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Any and all advice is appreciated.<br>
<br>
Thanks!<br>
--<br>
<br>
Daniel Thomson<br>
DevOps Engineer<br>
t +1 604 222 7428<br>
<a href="mailto:dthomson@triumf.ca" target="_blank">dthomson@triumf.ca</a><br>
TRIUMF Canada's particle accelerator centre<br>
<a href="http://www.triumf.ca" rel="noreferrer" target="_blank">www.triumf.ca</a> @TRIUMFLab<br>
4004 Wesbrook Mall<br>
Vancouver BC V6T 2A3 Canada<br>
________<br>
<br>
<br>
<br>
Community Meeting Calendar:<br>
<br>
Schedule -<br>
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" rel="noreferrer" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div></div>