<div dir="ltr">I found this on the mailing list:<div><i><span style="font-size:12.8px">I found the issue.</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">The CentOS 7 RPMs, upon upgrade, modifies the .vol files. Among other things, it adds "option shared-brick-count \d", using the number of bricks in the volume.</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">This gives you an average free space per brick, instead of total free space in the volume.</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">When I create a new volume, the value of "shared-brick-count" is "1".</span><br style="font-size:12.8px"><br style="font-size:12.8px"><span style="font-size:12.8px">find /var/lib/glusterd/vols -type f|xargs sed -i -e 's/option shared-brick-count [0-9]*/option shared-brick-count 1/g'</span></i><br></div><div><i><span style="font-size:12.8px"><br></span></i></div><div><i><span style="font-size:12.8px"><br></span></i></div><div><i><span style="font-size:12.8px"><br></span></i></div><div><span style="font-size:12.8px">Eva, can you send me the contents of the /var/lib/glusterd/<volname> folder from any one node so I can confirm if this is the problem?</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Regards,</span></div><div><span style="font-size:12.8px">Nithya</span></div><div><span style="font-size:12.8px"><br></span></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 31 January 2018 at 10:47, Nithya Balachandran <span dir="ltr"><<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Eva,<div><br></div><div>One more question. What version of gluster were you running before the upgrade?</div><div><br></div><div>Thanks,</div><div>Nithya</div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On 31 January 2018 at 09:52, Nithya Balachandran <span dir="ltr"><<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Eva,<div><br></div><div>Can you send us the following:</div><div><br></div><div>gluster volume info</div><div>gluster volume status</div><div><br></div><div>The log files and tcpdump for df on a fresh mount point for that volume.</div><div><br></div><div>Thanks,</div><div>Nithya</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="m_5998552805180114428h5">On 31 January 2018 at 07:17, Freer, Eva B. <span dir="ltr"><<a href="mailto:freereb@ornl.gov" target="_blank">freereb@ornl.gov</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="m_5998552805180114428h5">
<div bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="m_5998552805180114428m_8360380693207930883m_708523669856371683WordSection1">
<p class="MsoNormal"><span style="font-family:"Tahoma",sans-serif;color:#2e74b5">After OS update to CentOS 7.4 or RedHat 6.9 and update to Gluster 3.12.4, the ‘df’ command shows only part of the available space on the mount point for multi-brick volumes. All
nodes are at 3.12.4. This occurs on both servers and clients. <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Tahoma",sans-serif;color:#2e74b5"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Tahoma",sans-serif;color:#2e74b5">We have 2 different server configurations.
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Tahoma",sans-serif;color:#2e74b5"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Tahoma",sans-serif;color:#2e74b5">Configuration 1: A distributed volume of 8 bricks with 4 on each server. The initial configuration had 4 bricks of 59TB each with 2 on each server. Prior to the update to CentOS
7.4 and gluster 3.12.4, ‘df’ correctly showed the size for the volume as 233TB. After the update, we added 2 bricks with 1 on each server, but the output of ‘df’ still only listed 233TB for the volume. We added 2 more bricks, again with 1 on each server. The
output of ‘df’ now shows 350TB, but the aggregate of 8 – 59TB bricks should be ~466TB.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Tahoma",sans-serif;color:#2e74b5"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Tahoma",sans-serif;color:#2e74b5">Configuration 2: A distributed, replicated volume with 9 bricks on each server for a total of ~350TB of storage. After the server update to RHEL 6.9 and gluster 3.12.4, the volume
now shows as having 50TB with ‘df’. No changes were made to this volume after the update.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Tahoma",sans-serif;color:#2e74b5"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Tahoma",sans-serif;color:#2e74b5">In both cases, examining the bricks shows that the space and files are still there, just not reported correctly with ‘df’. All machines have been rebooted and the problem persists.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Tahoma",sans-serif;color:#2e74b5"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Tahoma",sans-serif;color:#2e74b5">Any help/advice you can give on this would be greatly appreciated.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Tahoma",sans-serif;color:#4472c4"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-family:"Tahoma",sans-serif;color:#4472c4">Thanks in advance.<span class="m_5998552805180114428m_8360380693207930883HOEnZb"><font color="#888888"><u></u><u></u></font></span></span></p><span class="m_5998552805180114428m_8360380693207930883HOEnZb"><font color="#888888">
<p class="MsoNormal"><span style="font-family:"Tahoma",sans-serif;color:#4472c4">Eva Freer<br>
<br>
</span><u></u><u></u></p>
</font></span></div>
</div>
<br></div></div><span>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br></span></blockquote></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>