<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Very similar to what we noticed too.<div class=""><br class=""></div><div class="">I suspect it's something to do with the metadata or xattrs stored on the filesystem that gives quite different results from the actual file sizes.</div><div class=""><br class=""><div class="">
<div dir="auto" style="color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">--<br class="">Sam McLeod (protoporpoise on IRC)</div><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><a href="https://smcleod.net" class="">https://smcleod.net</a><br class="">https://twitter.com/s_mcleod<br class=""><br class="">Words are my own opinions and do not necessarily represent those of my employer or partners.</div></div></div></div>
</div>
<div><br class=""><blockquote type="cite" class=""><div class="">On 31 Jan 2018, at 2:14 pm, Freer, Eva B. <<a href="mailto:freereb@ornl.gov" class="">freereb@ornl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="WordSection1" style="page: WordSection1; font-family: Helvetica; font-size: 13px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);"><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: Tahoma, sans-serif; color: rgb(46, 116, 181);" class="">Sam,<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: Tahoma, sans-serif; color: rgb(46, 116, 181);" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: Tahoma, sans-serif; color: rgb(46, 116, 181);" class="">For du –sh on my newer volume, the result is 161T. The sum of the Used space in the df –h output for all the bricks is ~163T. Close enough for me to believe everything is there. The total for used space in the df –h of the mountpoint it 83T, roughly half what is used.<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: Tahoma, sans-serif; color: rgb(46, 116, 181);" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: Tahoma, sans-serif; color: rgb(46, 116, 181);" class="">Relevant lines from df –h on server-A:<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: Consolas, sans-serif;" class="">Filesystem Size Used Avail Use% Mounted on<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: Consolas, sans-serif;" class="">/dev/sda1 59T 42T 17T 72% /bricks/data_A1<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: Consolas, sans-serif;" class="">/dev/sdb1 59T 45T 14T 77% /bricks/data_A2<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: Consolas, sans-serif;" class="">/dev/sdd1 59T 39M 59T 1% /bricks/data_A4<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: Consolas, sans-serif;" class="">/dev/sdc1 59T 1.9T 57T 4% /bricks/data_A3<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: Consolas, sans-serif;" class="">server-A:/dataeng 350T 83T 268T 24% /dataeng<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: Tahoma, sans-serif; color: rgb(46, 116, 181);" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: Tahoma, sans-serif; color: rgb(46, 116, 181);" class="">And on server-B:<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: Consolas, sans-serif;" class="">Filesystem Size Used Avail Use% Mounted on<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: Consolas, sans-serif;" class="">/dev/sdb1 59T 34T 25T 58% /bricks/data_B2<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: Consolas, sans-serif;" class="">/dev/sdc1 59T 2.0T 57T 4% /bricks/data_B3<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: Consolas, sans-serif;" class="">/dev/sdd1 59T 39M 59T 1% /bricks/data_B4<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: Consolas, sans-serif;" class="">/dev/sda1 59T 38T 22T 64% /bricks/data_B1<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: Consolas, sans-serif;" class="">server-B:/dataeng 350T 83T 268T 24% /dataeng<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: Tahoma, sans-serif; color: rgb(46, 116, 181);" class=""><o:p class=""> </o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: Tahoma, sans-serif; color: rgb(46, 116, 181);" class="">Eva Freer</span><span style="font-size: 12pt; font-family: Tahoma, sans-serif; color: rgb(68, 114, 196);" class=""><o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: Tahoma, sans-serif; color: rgb(46, 116, 181);" class=""><o:p class=""> </o:p></span></div><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0in 0in;" class=""><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class=""><span style="font-size: 12pt;" class="">From:<span class="Apple-converted-space"> </span></span></b><span style="font-size: 12pt;" class="">Sam McLeod <<a href="mailto:mailinglists@smcleod.net" class="">mailinglists@smcleod.net</a>><br class=""><b class="">Date:<span class="Apple-converted-space"> </span></b>Tuesday, January 30, 2018 at 9:43 PM<br class=""><b class="">To:<span class="Apple-converted-space"> </span></b>Eva Freer <<a href="mailto:freereb@ornl.gov" class="">freereb@ornl.gov</a>><br class=""><b class="">Cc:<span class="Apple-converted-space"> </span></b>"<a href="mailto:gluster-users@gluster.org" class="">gluster-users@gluster.org</a>" <<a href="mailto:gluster-users@gluster.org" class="">gluster-users@gluster.org</a>>, "Greene, Tami McFarlin" <<a href="mailto:greenet@ornl.gov" class="">greenet@ornl.gov</a>><br class=""><b class="">Subject:<span class="Apple-converted-space"> </span></b>Re: [Gluster-users] df does not show full volume capacity after update to 3.12.4<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">We noticed something similar.<span class="Apple-converted-space"> </span><o:p class=""></o:p></div><div class=""><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Out of interest, does du -sh . show the same size?<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">--<br class="">Sam McLeod (protoporpoise on IRC)<o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""><a href="https://smcleod.net/" style="color: purple; text-decoration: underline;" class="">https://smcleod.net</a><br class=""><a href="https://twitter.com/s_mcleod" class="">https://twitter.com/s_mcleod</a><br class=""><br class="">Words are my own opinions and do not necessarily represent those of my employer or partners.<o:p class=""></o:p></span></div></div></div></div></div></div><div class=""><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><br class=""><br class=""><o:p class=""></o:p></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="" type="cite"><div class=""><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">On 31 Jan 2018, at 12:47 pm, Freer, Eva B. <<a href="mailto:freereb@ornl.gov" style="color: purple; text-decoration: underline;" class="">freereb@ornl.gov</a>> wrote:<o:p class=""></o:p></div></div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div class=""><div class=""><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;" class=""><span style="font-size: 12pt; font-family: Tahoma, sans-serif; color: rgb(46, 116, 181);" class="">After OS update to CentOS 7.4 or RedHat 6.9 and update to Gluster 3.12.4, the ‘df’ command shows only part of the available space on the mount point for multi-brick volumes. All nodes are at 3.12.4. This occurs on both servers and clients.<span class="apple-converted-space"> </span></span><span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;" class=""><span style="font-size: 12pt; font-family: Tahoma, sans-serif; color: rgb(46, 116, 181);" class=""> </span><span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;" class=""><span style="font-size: 12pt; font-family: Tahoma, sans-serif; color: rgb(46, 116, 181);" class="">We have 2 different server configurations.</span><span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;" class=""><span style="font-size: 12pt; font-family: Tahoma, sans-serif; color: rgb(46, 116, 181);" class=""> </span><span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;" class=""><span style="font-size: 12pt; font-family: Tahoma, sans-serif; color: rgb(46, 116, 181);" class="">Configuration 1: A distributed volume of 8 bricks with 4 on each server. The initial configuration had 4 bricks of 59TB each with 2 on each server. Prior to the update to CentOS 7.4 and gluster 3.12.4, ‘df’ correctly showed the size for the volume as 233TB. After the update, we added 2 bricks with 1 on each server, but the output of ‘df’ still only listed 233TB for the volume. We added 2 more bricks, again with 1 on each server. The output of ‘df’ now shows 350TB, but the aggregate of 8 – 59TB bricks should be ~466TB.</span><span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;" class=""><span style="font-size: 12pt; font-family: Tahoma, sans-serif; color: rgb(46, 116, 181);" class=""> </span><span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;" class=""><span style="font-size: 12pt; font-family: Tahoma, sans-serif; color: rgb(46, 116, 181);" class="">Configuration 2: A distributed, replicated volume with 9 bricks on each server for a total of ~350TB of storage. After the server update to RHEL 6.9 and gluster 3.12.4, the volume now shows as having 50TB with ‘df’. No changes were made to this volume after the update.</span><span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;" class=""><span style="font-size: 12pt; font-family: Tahoma, sans-serif; color: rgb(46, 116, 181);" class=""> </span><span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;" class=""><span style="font-size: 12pt; font-family: Tahoma, sans-serif; color: rgb(46, 116, 181);" class="">In both cases, examining the bricks shows that the space and files are still there, just not reported correctly with ‘df’. All machines have been rebooted and the problem persists.</span><span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;" class=""><span style="font-size: 12pt; font-family: Tahoma, sans-serif; color: rgb(46, 116, 181);" class=""> </span><span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;" class=""><span style="font-size: 12pt; font-family: Tahoma, sans-serif; color: rgb(46, 116, 181);" class="">Any help/advice you can give on this would be greatly appreciated.</span><span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;" class=""><span style="font-size: 12pt; font-family: Tahoma, sans-serif; color: rgb(68, 114, 196);" class=""> </span><span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;" class=""><span style="font-size: 12pt; font-family: Tahoma, sans-serif; color: rgb(68, 114, 196);" class="">Thanks in advance.</span><span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div></div><div class=""><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif; background-color: white;" class=""><span style="font-size: 12pt; font-family: Tahoma, sans-serif; color: rgb(68, 114, 196);" class="">Eva Freer<br class=""><br class=""><br class=""></span><span style="font-size: 12pt;" class=""><o:p class=""></o:p></span></div></div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 10pt; font-family: Helvetica, sans-serif; background-color: white; background-position: initial initial; background-repeat: initial initial;" class="">_______________________________________________</span><span style="font-size: 10pt; font-family: Helvetica, sans-serif;" class=""><br class=""><span style="background-color: white; background-position: initial initial; background-repeat: initial initial;" class="">Gluster-users mailing list</span><br class=""></span><a href="mailto:Gluster-users@gluster.org" style="color: purple; text-decoration: underline;" class=""><span style="font-size: 10pt; font-family: Helvetica, sans-serif; color: rgb(149, 79, 114); background-color: white; background-position: initial initial; background-repeat: initial initial;" class="">Gluster-users@gluster.org</span></a><span style="font-size: 10pt; font-family: Helvetica, sans-serif;" class=""><br class=""></span><a href="http://lists.gluster.org/mailman/listinfo/gluster-users" style="color: purple; text-decoration: underline;" class=""><span style="font-size: 10pt; font-family: Helvetica, sans-serif; color: rgb(149, 79, 114); background-color: white; background-position: initial initial; background-repeat: initial initial;" class="">http://lists.gluster.org/mailman/listinfo/gluster-users</span></a><o:p class=""></o:p></div></div></blockquote></div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div></div></div></blockquote></div><br class=""></div></body></html>