<div dir="ltr">Btw, I've now noticed at least 5 variations in toggling binary option values. Are they all interchangeable, or will using the wrong value not work in some cases?<div><br></div><div>yes/no</div><div>true/false</div><div>True/False</div><div>on/off</div><div>enable/disable</div><div><br></div><div>It's quite a confusing/inconsistent practice, especially given that many options will accept any value without erroring out/validation.</div><div class="gmail_extra"><br clear="all"><div><div class="m_-5940135115956573066gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><br>Sincerely,<br>Artem<br><br>--<br>Founder, <a href="http://www.androidpolice.com" target="_blank">Android Police</a>, <a href="http://www.apkmirror.com/" style="font-size:12.8000001907349px" target="_blank">APK Mirror</a><span style="font-size:12.8000001907349px">, Illogical Robot LLC</span></div><div dir="ltr"><a href="http://beerpla.net/" target="_blank">beerpla.net</a> | <a href="https://plus.google.com/+ArtemRussakovskii" target="_blank">+ArtemRussakovskii</a> | <a href="http://twitter.com/ArtemR" target="_blank">@ArtemR</a><br></div></div></div></div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Tue, Apr 17, 2018 at 11:22 PM, Artem Russakovskii <span dir="ltr"><<a href="mailto:archon810@gmail.com" target="_blank">archon810@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks for the link. Looking at the status of that doc, it isn't quite ready yet, and there's no mention of the option.<div><br></div><div>Does it mean that whatever is ready now in 4.0.1 is incomplete but can be enabled via granular-entry-heal=on, and when it is complete, it'll become the default and the flag will simply go away?<div><br></div><div>Is there any risk enabling the option now in 4.0.1?</div></div></div><div class="gmail_extra"><span><br clear="all"><div><div class="m_-5940135115956573066m_7089802003934311205gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><br>Sincerely,<br>Artem<br><br>--<br>Founder, <a href="http://www.androidpolice.com" target="_blank">Android Police</a>, <a href="http://www.apkmirror.com/" style="font-size:12.8000001907349px" target="_blank">APK Mirror</a><span style="font-size:12.8000001907349px">, Illogical Robot LLC</span></div><div dir="ltr"><a href="http://beerpla.net/" target="_blank">beerpla.net</a> | <a href="https://plus.google.com/+ArtemRussakovskii" target="_blank">+ArtemRussakovskii</a> | <a href="http://twitter.com/ArtemR" target="_blank">@ArtemR</a><br></div></div></div></div></div></div></div></div></div></div></div>
<br></span><div><div class="m_-5940135115956573066h5"><div class="gmail_quote">On Tue, Apr 17, 2018 at 11:16 PM, Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><span>
<p><br>
</p>
<br>
<div class="m_-5940135115956573066m_7089802003934311205m_1780620483197147473moz-cite-prefix">On 04/18/2018 10:35 AM, Artem
Russakovskii wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi Ravi,
<div><br>
</div>
<div>Could you please expand on how these would help?</div>
<div><br>
</div>
<div>By forcing full here, we move the logic from the CPU to
network, thus decreasing CPU utilization, is that right?</div>
</div>
</blockquote></span>
Yes, 'diff' employs the rchecksum FOP which does a sha256 checksum
which can consume CPU. So yes it is sort of shifting the load from
CPU to the network. But if your average file size is small, it would
make sense to copy the entire file instead of computing checksums.<span><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div> This is assuming the CPU and disk utilization are caused
by the differ and not by lstat and other calls or something.</div>
<div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Option:
cluster.data-self-heal-algorit<wbr>hm<br>
Default Value: (null)<br>
Description: Select between "full", "diff". The "full"
algorithm copies the entire file from source to sink. The
"diff" algorithm copies to sink only those blocks whose
checksums don't match with those of source. If no option is
configured the option is chosen dynamically as follows: If
the file does not exist on one of the sinks or empty file
exists or if the source file size is about the same as page
size the entire file will be read and written i.e "full"
algo, otherwise "diff" algo is chosen.</blockquote>
</div>
<div><br>
</div>
<div>I really have no idea what this means and how/why it would
help. Any more info on this option?</div>
</div>
</blockquote>
<br>
</span><a class="m_-5940135115956573066m_7089802003934311205m_1780620483197147473moz-txt-link-freetext" href="https://github.com/gluster/glusterfs-specs/blob/master/done/GlusterFS%203.8/granular-entry-self-healing.md" target="_blank">https://github.com/gluster/glu<wbr>sterfs-specs/blob/master/done/<wbr>GlusterFS%203.8/granular-entry<wbr>-self-healing.md</a>
should help.<br>
Regards,<br>
Ravi<div><div class="m_-5940135115956573066m_7089802003934311205h5"><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Option:
cluster.granular-entry-heal<br>
Default Value: no<br>
Description: If this option is enabled, self-heal will
resort to granular way of recording changelogs and doing
entry self-heal.</blockquote>
</div>
<div><br>
</div>
<div>Thank you.</div>
</div>
<div class="gmail_extra"><br clear="all">
<div>
<div class="m_-5940135115956573066m_7089802003934311205m_1780620483197147473gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr"><br>
Sincerely,<br>
Artem<br>
<br>
--<br>
Founder, <a href="http://www.androidpolice.com" target="_blank">Android
Police</a>, <a href="http://www.apkmirror.com/" style="font-size:12.8000001907349px" target="_blank">APK
Mirror</a><span style="font-size:12.8000001907349px">,
Illogical Robot LLC</span></div>
<div dir="ltr"><a href="http://beerpla.net/" target="_blank">beerpla.net</a>
| <a href="https://plus.google.com/+ArtemRussakovskii" target="_blank">+ArtemRussakovskii</a>
| <a href="http://twitter.com/ArtemR" target="_blank">@ArtemR</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">On Tue, Apr 17, 2018 at 9:58 PM,
Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>
<div class="m_-5940135115956573066m_7089802003934311205m_1780620483197147473h5"> <br>
<div class="m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854moz-cite-prefix">On
04/18/2018 10:14 AM, Artem Russakovskii wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Following up here on a related and
very serious for us issue.
<div><br>
</div>
<div>I took down one of the 4 replicate gluster
servers for maintenance today. There are 2
gluster volumes totaling about 600GB. Not that
much data. After the server comes back online,
it starts auto healing and pretty much all
operations on gluster freeze for many minutes.</div>
<div><br>
</div>
<div>For example, I was trying to run an ls -alrt
in a folder with 7300 files, and it took a good
15-20 minutes before returning.</div>
<div><br>
</div>
<div>During this time, I can see iostat show 100%
utilization on the brick, heal status takes many
minutes to return, glusterfsd uses up tons of
CPU (I saw it spike to 600%). gluster already
has massive performance issues for me, but
healing after a 4-hour downtime is on another
level of bad perf.</div>
<div><br>
</div>
<div>For example, this command took many minutes
to run:</div>
<div><br>
</div>
<div>
<div>gluster volume heal androidpolice_data3
info summary</div>
<div>Brick nexus2:/mnt/nexus2_block4/andr<wbr>oidpolice_data3</div>
<div>Status: Connected</div>
<div>Total Number of entries: 91</div>
<div>Number of entries in heal pending: 90</div>
<div>Number of entries in split-brain: 0</div>
<div>Number of entries possibly healing: 1</div>
<div><br>
</div>
<div>Brick forge:/mnt/forge_block4/androi<wbr>dpolice_data3</div>
<div>Status: Connected</div>
<div>Total Number of entries: 87</div>
<div>Number of entries in heal pending: 86</div>
<div>Number of entries in split-brain: 0</div>
<div>Number of entries possibly healing: 1</div>
<div><br>
</div>
<div>Brick hive:/mnt/hive_block4/androidp<wbr>olice_data3</div>
<div>Status: Connected</div>
<div>Total Number of entries: 87</div>
<div>Number of entries in heal pending: 86</div>
<div>Number of entries in split-brain: 0</div>
<div>Number of entries possibly healing: 1</div>
<div><br>
</div>
<div>Brick citadel:/mnt/citadel_block4/an<wbr>droidpolice_data3</div>
<div>Status: Connected</div>
<div>Total Number of entries: 0</div>
<div>Number of entries in heal pending: 0</div>
<div>Number of entries in split-brain: 0</div>
<div>Number of entries possibly healing: 0</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Statistics showed a diminishing number of
failed heals:</div>
<div>...</div>
<div>
<div>Ending time of crawl: Tue Apr 17 21:13:08
2018</div>
<div><br>
</div>
<div>Type of crawl: INDEX</div>
<div>No. of entries healed: 2</div>
<div>No. of entries in split-brain: 0</div>
<div>No. of heal failed entries: 102</div>
<div><br>
</div>
<div>Starting time of crawl: Tue Apr 17 21:13:09
2018</div>
<div><br>
</div>
<div>Ending time of crawl: Tue Apr 17 21:14:30
2018</div>
<div><br>
</div>
<div>Type of crawl: INDEX</div>
<div>No. of entries healed: 4</div>
<div>No. of entries in split-brain: 0</div>
<div>No. of heal failed entries: 91</div>
<div><br>
</div>
<div>Starting time of crawl: Tue Apr 17 21:14:31
2018</div>
<div><br>
</div>
<div>Ending time of crawl: Tue Apr 17 21:15:34
2018</div>
<div><br>
</div>
<div>Type of crawl: INDEX</div>
<div>No. of entries healed: 0</div>
<div>No. of entries in split-brain: 0</div>
<div>No. of heal failed entries: 88</div>
</div>
<div>...</div>
<div><br>
</div>
<div>Eventually, everything heals and goes back to
at least where the roof isn't on fire anymore.</div>
<div><br>
</div>
<div>The server stats and volume options were
given in one of the previous replies to this
thread.</div>
<div><br>
</div>
<div>Any ideas or things I could run and show the
output of to help diagnose? I'm also very open
to working with someone on the team on a live
debugging session if there's interest.</div>
</div>
</blockquote>
<br>
</div>
</div>
It is likely that self-heal is causing the CPU spike due
to the flood of lookups/ locks and checksum fops that the
self-heal-daemon sends to the bricks.<br>
There's a script to control shd's cpu usage using cgroups.
That should help in regulating self-heal traffic: <a class="m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854moz-txt-link-freetext" href="https://review.gluster.org/#/c/18404/" target="_blank">https://review.gluster.org/#/c<wbr>/18404/</a>
(see extras/control-cpu-load.sh)<br>
Other self-heal related volume options that you could
change are setting 'cluster.data-self-heal-algori<wbr>thm'
to 'full' and 'granular-entry-heal' to 'enable'. `gluster
volume set help` should give you more information about
these options.<br>
Thanks,<br>
Ravi
<div>
<div class="m_-5940135115956573066m_7089802003934311205m_1780620483197147473h5"><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>Thank you.</div>
</div>
<div class="gmail_extra"><br clear="all">
<div>
<div class="m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr"><br>
Sincerely,<br>
Artem<br>
<br>
--<br>
Founder, <a href="http://www.androidpolice.com" target="_blank">Android
Police</a>, <a href="http://www.apkmirror.com/" style="font-size:12.8000001907349px" target="_blank">APK
Mirror</a><span style="font-size:12.8000001907349px">,
Illogical Robot LLC</span></div>
<div dir="ltr"><a href="http://beerpla.net/" target="_blank">beerpla.net</a>
| <a href="https://plus.google.com/+ArtemRussakovskii" target="_blank">+ArtemRussakovskii</a>
| <a href="http://twitter.com/ArtemR" target="_blank">@ArtemR</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">On Tue, Apr 10, 2018 at
9:56 AM, Artem Russakovskii <span dir="ltr"><<a href="mailto:archon810@gmail.com" target="_blank">archon810@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi Vlad,
<div><br>
</div>
<div>I actually saw that post already and
even asked a question 4 days ago (<a href="https://serverfault.com/questions/517775/glusterfs-direct-i-o-mode#comment1172497_540917" target="_blank">https://serverfault.com/quest<wbr>ions/517775/glusterfs-direct-i<wbr>-o-mode#comment1172497_540917</a>)<wbr>.
The accepted answer also seems to go
against your suggestion to enable
direct-io-mode as it says it should be
disabled for better performance when used
just for file accesses.</div>
<div><br>
</div>
<div>It'd be great if someone from the
Gluster team chimed in about this thread.</div>
</div>
<div class="gmail_extra"><span><br clear="all">
<div>
<div class="m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854m_3369104992988144420gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr"><br>
Sincerely,<br>
Artem<br>
<br>
--<br>
Founder, <a href="http://www.androidpolice.com" target="_blank">Android
Police</a>, <a href="http://www.apkmirror.com/" style="font-size:12.8000001907349px" target="_blank">APK
Mirror</a><span style="font-size:12.8000001907349px">,
Illogical Robot
LLC</span></div>
<div dir="ltr"><a href="http://beerpla.net/" target="_blank">beerpla.net</a>
| <a href="https://plus.google.com/+ArtemRussakovskii" target="_blank">+ArtemRussakovskii</a>
| <a href="http://twitter.com/ArtemR" target="_blank">@ArtemR</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</span>
<div>
<div class="m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854h5">
<div class="gmail_quote">On Tue, Apr 10,
2018 at 7:01 AM, Vlad Kopylov <span dir="ltr"><<a href="mailto:vladkopy@gmail.com" target="_blank">vladkopy@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>Wish I knew or was able to
get detailed description of
those options myself.<br>
</div>
<div>here is direct-io-mode <a href="https://serverfault.com/questions/517775/glusterfs-direct-i-o-mode" target="_blank">https://serverfault.com/questi<wbr>ons/517775/glusterfs-direct-i-<wbr>o-mode</a><br>
</div>
Same as you I ran tests on a
large volume of files, finding
that main delays are in
attribute calls, ending up with
those mount options to add
performance.<br>
</div>
<div>I discovered those options
through basically googling this
user list with people sharing
their tests.<br>
</div>
Not sure I would share your
optimism, and rather then going up
I downgraded to 3.12 and have no
dir view issue now. Though I had
to recreate the cluster and had to
re-add bricks with existing data.<br>
</div>
<div class="m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854m_3369104992988144420HOEnZb">
<div class="m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854m_3369104992988144420h5">
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Tue, Apr 10, 2018 at 1:47
AM, Artem Russakovskii <span dir="ltr"><<a href="mailto:archon810@gmail.com" target="_blank">archon810@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi Vlad,
<div><br>
</div>
<div>I'm using only
localhost: mounts.</div>
<div><br>
</div>
<div>Can you please
explain what effect
each option has on
performance issues
shown in my posts?
"negative-timeout=10,attribute<wbr>-timeout=30,fopen-keep-cache,d<wbr>irect-io-mode=enable,fetch-att<wbr>empts=5"
From what I remember,
direct-io-mode=enable
didn't make a
difference in my
tests, but I suppose I
can try again. The
explanations about
direct-io-mode are
quite confusing on the
web in various guides,
saying enabling it
could make performance
worse in some
situations and better
in others due to OS
file cache.</div>
<div><br>
</div>
<div>There are also
these gluster volume
settings, adding to
the confusion:</div>
<div>
<div>Option:
performance.strict-o-direct</div>
<div>Default Value:
off</div>
<div>Description: This
option when set to
off, ignores the
O_DIRECT flag.</div>
<div><br>
</div>
<div>Option:
performance.nfs.strict-o-direc<wbr>t</div>
<div>Default Value:
off</div>
<div>Description: This
option when set to
off, ignores the
O_DIRECT flag.</div>
</div>
<div><br>
</div>
<div>Re: 4.0. I moved to
4.0 after finding out
that it fixes the
disappearing dirs bug
related to
cluster.readdir-optimize
if you remember (<a href="http://lists.gluster.org/pipermail/gluster-users/2018-April/033830.html" target="_blank">http://lists.gluster.org/pipe<wbr>rmail/gluster-users/2018-April<wbr>/033830.html</a>).
I was already on 3.13
by then, and 4.0
resolved the issue.
It's been stable for
me so far, thankfully.</div>
<div class="gmail_extra"><span><br clear="all">
<div>
<div class="m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854m_3369104992988144420m_-2442537929453809111m_3104514395859481521m_-9168214684418515114gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr"><br>
Sincerely,<br>
Artem<br>
<br>
--<br>
Founder, <a href="http://www.androidpolice.com" target="_blank">Android Police</a>, <a href="http://www.apkmirror.com/" style="font-size:12.8000001907349px" target="_blank">APK
Mirror</a><span style="font-size:12.8000001907349px">, Illogical Robot LLC</span></div>
<div dir="ltr"><a href="http://beerpla.net/" target="_blank">beerpla.net</a>
| <a href="https://plus.google.com/+ArtemRussakovskii" target="_blank">+ArtemRussakovskii</a> | <a href="http://twitter.com/ArtemR" target="_blank">@ArtemR</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</span>
<div>
<div class="m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854m_3369104992988144420m_-2442537929453809111h5">
<div class="gmail_quote">On
Mon, Apr 9, 2018
at 10:38 PM,
Vlad Kopylov <span dir="ltr"><<a href="mailto:vladkopy@gmail.com" target="_blank">vladkopy@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>you
definitely
need mount
options to
/etc/fstab<br>
</div>
use ones from
here <a href="http://lists.gluster.org/pipermail/gluster-users/2018-April/033811.html" target="_blank">http://lists.gluster.org/piper<wbr>mail/gluster-users/2018-April/<wbr>033811.html</a><br>
<br>
</div>
<div>I went on
with using
local mounts
to achieve
performance as
well<br>
</div>
<div><br>
</div>
Also, 3.12 or
3.10 branches
would be
preferable for
production <br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">
<div>
<div class="m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854m_3369104992988144420m_-2442537929453809111m_3104514395859481521m_-9168214684418515114h5">On
Fri, Apr 6,
2018 at 4:12
AM, Artem
Russakovskii <span dir="ltr"><<a href="mailto:archon810@gmail.com" target="_blank">archon810@gmail.com</a>></span>
wrote:<br>
</div>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div class="m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854m_3369104992988144420m_-2442537929453809111m_3104514395859481521m_-9168214684418515114h5">
<div dir="ltr">Hi
again,
<div><br>
</div>
<div>I'd like
to expand on
the
performance
issues and
plead for
help. Here's
one case which
shows these
odd hiccups: <a href="https://i.imgur.com/CXBPjTK.gifv" target="_blank">https://i.imgur.com/C<wbr>XBPjTK.gifv</a>.
<div><br>
</div>
<div>In this
GIF where I
switch <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">back
and forth</span>
between copy
operations on
2 servers, I'm
copying a 10GB
dir full of
.apk and image
files.</div>
<div><br>
</div>
<div>On server
"hive" I'm
copying
straight from
the main disk
to an attached
volume block
(xfs). As you
can see, the
transfers are
relatively
speedy and
don't hiccup.</div>
<div>On server
"citadel" I'm
copying the
same set of
data to a
4-replicate
gluster which
uses block
storage as a
brick. As you
can see,
performance is
much worse,
and there are
frequent
pauses for
many seconds
where nothing
seems to be
happening -
just freezes.<br>
</div>
<div><br>
</div>
<div>All 4
servers have
the same
specs, and all
of them have
performance
issues with
gluster and no
such issues
when raw xfs
block storage
is used.</div>
<div><br>
</div>
<div>hive has
long finished
copying the
data, while
citadel is
barely
chugging along
and is
expected to
take probably
half an hour
to an hour. I
have over 1TB
of data to
migrate, at
which point if
we went live,
I'm not even
sure gluster
would be able
to keep up
instead of
bringing the
machines and
services down.</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Here's
the cluster
config, though
it didn't seem
to make any
difference
performance-wise
before I
applied the
customizations
vs after.</div>
<div><br>
</div>
<div>
<div>Volume
Name:
apkmirror_data1</div>
<div>Type:
Replicate</div>
<div>Volume
ID:
11ecee7e-d4f8-497a-9994-ceb144<wbr>d6841e</div>
<div>Status:
Started</div>
<div>Snapshot
Count: 0</div>
<div>Number of
Bricks: 1 x 4
= 4</div>
<div>Transport-type:
tcp</div>
<div>Bricks:</div>
<div>Brick1:
nexus2:/mnt/nexus2_block1/apkm<wbr>irror_data1</div>
<div>Brick2:
forge:/mnt/forge_block1/apkmir<wbr>ror_data1</div>
<div>Brick3:
hive:/mnt/hive_block1/apkmirro<wbr>r_data1</div>
<div>Brick4:
citadel:/mnt/citadel_block1/ap<wbr>kmirror_data1</div>
<div>Options
Reconfigured:</div>
<div>cluster.quorum-count:
1</div>
<div>cluster.quorum-type:
fixed</div>
<div>network.ping-timeout:
5</div>
<div>network.remote-dio:
enable</div>
<div>performance.rda-cache-limit:
256MB</div>
<div>performance.readdir-ahead:
on</div>
<div>performance.parallel-readdir:
on</div>
<div>network.inode-lru-limit:
500000</div>
<div>performance.md-cache-timeout:
600</div>
<div>performance.cache-invalidation<wbr>:
on</div>
<div>performance.stat-prefetch:
on</div>
<div>features.cache-invalidation-ti<wbr>meout:
600</div>
<div>features.cache-invalidation:
on</div>
<div>cluster.readdir-optimize:
on</div>
<div>performance.io-thread-count:
32</div>
<div>server.event-threads:
4</div>
<div>client.event-threads:
4</div>
<div>performance.read-ahead:
off</div>
<div>cluster.lookup-optimize:
on</div>
<div>performance.cache-size:
1GB</div>
<div>cluster.self-heal-daemon:
enable</div>
<div>transport.address-family:
inet</div>
<div>nfs.disable:
on</div>
<div>performance.client-io-threads:
on</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>The
mounts are
done as
follows in
/etc/fstab:</div>
<div>/dev/disk/by-id/scsi-0Linode_V<wbr>olume_citadel_block1
/mnt/citadel_block1 xfs defaults 0 2<br>
</div>
<div>localhost:/apkmirror_data1
/mnt/apkmirror_data1 glusterfs defaults,_netdev 0 0<br>
</div>
<div><br>
</div>
<div>I'm
really not
sure if
direct-io-mode
mount tweaks
would do
anything here,
what the value
should be set
to, and what
it is by
default.</div>
<div><br>
</div>
<div>The OS is
OpenSUSE 42.3,
64-bit. 80GB
of RAM, 20
CPUs, hosted
by Linode.</div>
<div><br>
</div>
<div>I'd
really
appreciate any
help in the
matter. </div>
<div><br>
</div>
<div>Thank
you.</div>
</div>
</div>
<div class="gmail_extra"><span><br clear="all">
<div>
<div class="m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854m_3369104992988144420m_-2442537929453809111m_3104514395859481521m_-9168214684418515114m_-7640979976631609660m_7374879952101670984gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr"><br>
Sincerely,<br>
Artem<br>
<br>
--<br>
Founder, <a href="http://www.androidpolice.com" target="_blank">Android Police</a>, <a href="http://www.apkmirror.com/" style="font-size:12.8000001907349px" target="_blank">APK
Mirror</a><span style="font-size:12.8000001907349px">, Illogical Robot LLC</span></div>
<div dir="ltr"><a href="http://beerpla.net/" target="_blank">beerpla.net</a>
| <a href="https://plus.google.com/+ArtemRussakovskii" target="_blank">+ArtemRussakovskii</a> | <a href="http://twitter.com/ArtemR" target="_blank">@ArtemR</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</span>
<div>
<div class="m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854m_3369104992988144420m_-2442537929453809111m_3104514395859481521m_-9168214684418515114m_-7640979976631609660h5">
<div class="gmail_quote">On
Thu, Apr 5,
2018 at 11:13
PM, Artem
Russakovskii <span dir="ltr"><<a href="mailto:archon810@gmail.com" target="_blank">archon810@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi,
<div><br>
</div>
<div>I'm
trying to
squeeze
performance
out of gluster
on 4 80GB RAM
20-CPU
machines where
Gluster runs
on attached
block storage
(Linode) in (4
replicate
bricks), and
so far
everything I
tried results
in sub-optimal
performance.</div>
<div><br>
</div>
<div>There are
many files -
mostly images,
several
million - and
many
operations
take minutes,
copying
multiple files
(even if
they're small)
suddenly
freezes up for
seconds at a
time, then
continues,
iostat
frequently
shows large
r_await and
w_awaits with
100%
utilization
for the
attached block
device, etc.</div>
<div><br>
</div>
<div>But
anyway, there
are many
guides out
there for
small-file
performance
improvements,
but more
explanation is
needed, and I
think more
tweaks should
be possible.<br>
</div>
<div><br>
</div>
<div>My
question today
is
about performance.cache-size.
Is this a size
of cache in
RAM? If so,
how do I view
the current
cache size to
see if it gets
full and I
should
increase its
size? Is it
advisable to
bump it up if
I have many
tens of gigs
of RAM free? </div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>More
generally, in
the last 2
months since I
first started
working with
gluster and
set a
production
system live,
I've been
feeling
frustrated
because
Gluster has a
lot of
poorly-documented
and confusing
options. I
really wish
documentation
could be
improved with
examples and
better
explanations.</div>
<div><br>
</div>
<div>Specifically,
it'd be
absolutely
amazing if the
docs offered a
strategy for
setting each
value and ways
of determining
more optimal
values. For
example,
for performance.cache-size,
if it said
something like
"run command
abc to see
your current
cache size,
and if it's
hurting, up
it, but be
aware that
it's limited
by RAM," it'd
be already a
huge
improvement to
the docs. And
so on with
other options.</div>
<div>
<div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br>
</div>
<div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br>
</div>
<div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br>
</div>
<div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">The
gluster team
is quite
helpful on
this mailing
list, but in a
reactive
rather than
proactive way.
Perhaps it's
tunnel vision
once you've
worked on a
project for so
long where
less technical
explanations
and even
proper
documentation
of options
takes a back
seat, but I
encourage you
to be more
proactive
about helping
us understand
and optimize
Gluster.</div>
<div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br>
</div>
<div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">Thank
you.</div>
<div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br>
</div>
<div>
<div class="m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854m_3369104992988144420m_-2442537929453809111m_3104514395859481521m_-9168214684418515114m_-7640979976631609660m_7374879952101670984m_1420630595765814996gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">Sincerely,<br>
Artem<br>
<br>
--<br>
Founder, <a href="http://www.androidpolice.com" target="_blank">Android Police</a>, <a href="http://www.apkmirror.com/" style="font-size:12.8px" target="_blank">APK
Mirror</a><span style="font-size:12.8px">, Illogical Robot LLC</span></div>
<div dir="ltr"><a href="http://beerpla.net/" target="_blank">beerpla.net</a>
| <a href="https://plus.google.com/+ArtemRussakovskii" target="_blank">+ArtemRussakovskii</a> | <a href="http://twitter.com/ArtemR" target="_blank">@ArtemR</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
<br>
</div>
</div>
<span>______________________________<wbr>_________________<br>
Gluster-users
mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
</span></blockquote>
</div>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854mimeAttachmentHeader"></fieldset>
<br>
<pre>______________________________<wbr>_________________
Gluster-users mailing list
<a class="m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a class="m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854moz-txt-link-freetext" href="http://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</div></div></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div></div>