<div dir="ltr"><div>This appears to be exactly the issue. After testing with xfs mounted with the option 'allocsize=64k' and even trying ext4, the extraneous filesystem usage disappears. Removing 'allocsize=64k' returns the additional space usage. The linked article also makes reference to /proc/sys/fs/xfs/speculative_prealloc_lifetime to show the lifetime of the allocation which matches with behavior I was seeing.</div><div><br></div><div>Thank you for this most excellent find!<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Feb 9, 2022 at 10:01 AM Xavi Hernandez <<a href="mailto:jahernan@redhat.com">jahernan@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>this problem is most likely caused by the XFS speculative preallocation (<a href="https://linux-xfs.oss.sgi.narkive.com/jjjfnyI1/faq-xfs-speculative-preallocation" target="_blank">https://linux-xfs.oss.sgi.narkive.com/jjjfnyI1/faq-xfs-speculative-preallocation</a>)</div><div><br></div><div>Regards,</div><div><br></div><div>Xavi</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Feb 5, 2022 at 10:19 AM Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com" target="_blank">hunter86_bg@yahoo.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div> It seems quite odd.<br>I'm adding the devel list,as it looks like a bug - but it could be a feature ;)<br><br>Best Regards,<br>Strahil Nikolov<br><br><br> </div> <div style="margin:10px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div style="font-family:"Helvetica Neue",Helvetica,Arial,sans-serif;font-size:13px;color:rgb(38,40,42)"> <div>----- Препратено съобщение -----</div> <div><b>От:</b> Fox <<a href="mailto:foxxz.net@gmail.com" target="_blank">foxxz.net@gmail.com</a>></div><div><b>До:</b> Gluster Users <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>></div><div><b>Изпратено:</b> събота, 5 февруари 2022 г., 05:39:36 Гринуич+2</div><div><b>Тема:</b> Re: [Gluster-users] Distributed-Disperse Shard Behavior</div><div><br></div> <div><div id="gmail-m_2761059194694091256gmail-m_-4800094665969601885yiv6958131709"><div><div dir="ltr">I tried setting the shard size to 512MB. It slightly improved the space utilization during creation - not quite double space utilization. And I didn't run out of space creating a file that occupied 6gb of the 8gb volume (and I even tried 7168MB just fine). See attached command line log.</div><br clear="none"><div id="gmail-m_2761059194694091256gmail-m_-4800094665969601885yiv6958131709yqt05344"><div><div dir="ltr">On Fri, Feb 4, 2022 at 6:59 PM Strahil Nikolov <<a rel="nofollow noopener noreferrer" shape="rect" href="mailto:hunter86_bg@yahoo.com" target="_blank">hunter86_bg@yahoo.com</a>> wrote:<br clear="none"></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">It sounds like a bug to me.<div id="gmail-m_2761059194694091256gmail-m_-4800094665969601885yiv6958131709gmail-m_4951758407816662440yMail_cursorElementTracker_1644018926578">In virtualization sharding is quite common (yet, on replica volumes) and I have never observed such behavior.</div><div id="gmail-m_2761059194694091256gmail-m_-4800094665969601885yiv6958131709gmail-m_4951758407816662440yMail_cursorElementTracker_1644018982688">Can you increase the shard size to 512M and check if the situation is better ?</div><div id="gmail-m_2761059194694091256gmail-m_-4800094665969601885yiv6958131709gmail-m_4951758407816662440yMail_cursorElementTracker_1644018991946">Also, share the volume info.</div><div id="gmail-m_2761059194694091256gmail-m_-4800094665969601885yiv6958131709gmail-m_4951758407816662440yMail_cursorElementTracker_1644019002232"><br clear="none"></div><div id="gmail-m_2761059194694091256gmail-m_-4800094665969601885yiv6958131709gmail-m_4951758407816662440yMail_cursorElementTracker_1644018982884">Best Regards,</div><div id="gmail-m_2761059194694091256gmail-m_-4800094665969601885yiv6958131709gmail-m_4951758407816662440yMail_cursorElementTracker_1644018986574">Strahil Nikolov<br clear="none"> <br clear="none"> <blockquote style="margin:0px 0px 20px"> <div style="font-family:Roboto,sans-serif;color:rgb(109,0,246)"> <div>On Fri, Feb 4, 2022 at 22:32, Fox</div><div><<a rel="nofollow noopener noreferrer" shape="rect" href="mailto:foxxz.net@gmail.com" target="_blank">foxxz.net@gmail.com</a>> wrote:</div> </div> <div style="padding:10px 0px 0px 20px;margin:10px 0px 0px;border-left:1px solid rgb(109,0,246)"> <div id="gmail-m_2761059194694091256gmail-m_-4800094665969601885yiv6958131709gmail-m_4951758407816662440yiv2116146258"><div dir="ltr"><div>Using gluster v10.1 and creating a Distributed-Dispersed volume with sharding enabled.</div><div><br clear="none"></div><div>I create a 2gb file on the volume using the 'dd' tool. The file size shows 2gb with 'ls'. However, 'df' shows 4gb of space utilized on the volume. After several minutes the volume utilization drops to the 2gb I would expect.</div><div><br clear="none"></div><div>This is repeatable for different large file sizes and different disperse/redundancy brick configurations.<br clear="none"></div><div><br clear="none"></div><div>I've also encountered a situation, as configured above, where I utilize close to full disk capacity and am momentarily unable to delete the file.</div><div><br clear="none"></div><div>I have attached a command line log of an example of above using a set of test VMs setup in a glusterfs cluster.</div><div><br clear="none"></div><div><div></div><div>Is this initial 2x space utilization anticipated behavior for sharding?</div><div><br clear="none"></div><div>It would mean that I can never create a file bigger than half my volume size as I get an I/O error with no space left on disk.</div></div></div></div>________<br clear="none"><br clear="none"><br clear="none"><br clear="none">Community Meeting Calendar:<br clear="none"><br clear="none">Schedule -<br clear="none">Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br clear="none">Bridge: <a rel="nofollow noopener noreferrer" shape="rect" href="https://meet.google.com/cpu-eiue-hvk" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br clear="none">Gluster-users mailing list<br clear="none"><a rel="nofollow noopener noreferrer" shape="rect" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br clear="none"><a rel="nofollow noopener noreferrer" shape="rect" href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br clear="none"> </div> </blockquote></div></blockquote></div></div></div></div><div id="gmail-m_2761059194694091256gmail-m_-4800094665969601885yqt81243">________<br clear="none"><br clear="none"><br clear="none"><br clear="none">Community Meeting Calendar:<br clear="none"><br clear="none">Schedule -<br clear="none">Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br clear="none">Bridge: <a shape="rect" href="https://meet.google.com/cpu-eiue-hvk" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br clear="none">Gluster-users mailing list<br clear="none"><a shape="rect" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br clear="none"><a shape="rect" href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br clear="none"></div></div> </div> </div>________<br>
<br>
<br>
<br>
Community Meeting Calendar:<br>
<br>
Schedule -<br>
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" rel="noreferrer" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div>
-------<br>
<br>
Community Meeting Calendar:<br>
Schedule -<br>
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" rel="noreferrer" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br>
<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-devel</a><br>
<br>
</blockquote></div>