<div dir="ltr">OK, thank you. I'll try that.<div><br></div><div>The reason I was confused about its status is these things in the doc:</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">How To Test<br>TBD.<br>Documentation<br>TBD<br>Status<br>Design complete. Implementation done. The only thing pending is the compounding of two fops in shd code.</blockquote></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><br>Sincerely,<br>Artem<br><br>--<br>Founder, <a href="http://www.androidpolice.com" target="_blank">Android Police</a>, <a href="http://www.apkmirror.com/" style="font-size:12.8000001907349px" target="_blank">APK Mirror</a><span style="font-size:12.8000001907349px">, Illogical Robot LLC</span></div><div dir="ltr"><a href="http://beerpla.net/" target="_blank">beerpla.net</a> | <a href="https://plus.google.com/+ArtemRussakovskii" target="_blank">+ArtemRussakovskii</a> | <a href="http://twitter.com/ArtemR" target="_blank">@ArtemR</a><br></div></div></div></div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Tue, Apr 17, 2018 at 11:49 PM, Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p><br>
</p>
<br>
<div class="m_229972322048706121moz-cite-prefix">On 04/18/2018 11:59 AM, Artem
Russakovskii wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Btw, I've now noticed at least 5 variations in
toggling binary option values. Are they all interchangeable, or
will using the wrong value not work in some cases?
<div><br>
</div>
<div>yes/no</div>
<div>true/false</div>
<div>True/False</div>
<div>on/off</div>
<div>enable/disable</div>
<div><br>
</div>
<div>It's quite a confusing/inconsistent practice, especially
given that many options will accept any value without erroring
out/validation.</div>
</div>
</blockquote>
<br>
All these options are okay.<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra"><br clear="all">
<div>
<div class="m_229972322048706121m_-5940135115956573066gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr"><br>
Sincerely,<br>
Artem<br>
<br>
--<br>
Founder, <a href="http://www.androidpolice.com" target="_blank">Android
Police</a>, <a href="http://www.apkmirror.com/" style="font-size:12.8000001907349px" target="_blank">APK
Mirror</a><span style="font-size:12.8000001907349px">,
Illogical Robot LLC</span></div>
<div dir="ltr"><a href="http://beerpla.net/" target="_blank">beerpla.net</a>
| <a href="https://plus.google.com/+ArtemRussakovskii" target="_blank">+ArtemRussakovskii</a>
| <a href="http://twitter.com/ArtemR" target="_blank">@ArtemR</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">On Tue, Apr 17, 2018 at 11:22 PM,
Artem Russakovskii <span dir="ltr"><<a href="mailto:archon810@gmail.com" target="_blank">archon810@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Thanks for the link. Looking at the status
of that doc, it isn't quite ready yet, and there's no
mention of the option.</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
<br>
No, this is a completed feature available since 3.8 IIRC. You can
use it safely. There is a difference in how to enable it though.
Instead of using 'gluster volume set ...', you need to use 'gluster
volume heal <volname> granular-entry-heal enable' to turn it
on. If there are no pending heals, it will run successfully.
Otherwise you need to wait until heals are over (i.e. heal info
shows zero entries). Just follow what the CLI says and you should be
fine.<br>
<br>
-Ravi<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div><br>
</div>
<div>Does it mean that whatever is ready now in 4.0.1 is
incomplete but can be enabled via
granular-entry-heal=on, and when it is complete, it'll
become the default and the flag will simply go away?
<div><br>
</div>
<div>Is there any risk enabling the option now in
4.0.1?</div>
</div>
</div>
<div class="gmail_extra"><span><br clear="all">
<div>
<div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr"><br>
Sincerely,<br>
Artem<br>
<br>
--<br>
Founder, <a href="http://www.androidpolice.com" target="_blank">Android
Police</a>, <a href="http://www.apkmirror.com/" style="font-size:12.8000001907349px" target="_blank">APK
Mirror</a><span style="font-size:12.8000001907349px">,
Illogical Robot LLC</span></div>
<div dir="ltr"><a href="http://beerpla.net/" target="_blank">beerpla.net</a>
| <a href="https://plus.google.com/+ArtemRussakovskii" target="_blank">+ArtemRussakovskii</a>
| <a href="http://twitter.com/ArtemR" target="_blank">@ArtemR</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</span>
<div>
<div class="m_229972322048706121m_-5940135115956573066h5">
<div class="gmail_quote">On Tue, Apr 17, 2018 at
11:16 PM, Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><span>
<p><br>
</p>
<br>
<div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473moz-cite-prefix">On
04/18/2018 10:35 AM, Artem Russakovskii
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi Ravi,
<div><br>
</div>
<div>Could you please expand on how
these would help?</div>
<div><br>
</div>
<div>By forcing full here, we move the
logic from the CPU to network, thus
decreasing CPU utilization, is that
right?</div>
</div>
</blockquote>
</span> Yes, 'diff' employs the rchecksum FOP
which does a sha256 checksum which can
consume CPU. So yes it is sort of shifting the
load from CPU to the network. But if your
average file size is small, it would make
sense to copy the entire file instead of
computing checksums.<span><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div> This is assuming the CPU and disk
utilization are caused by the differ
and not by lstat and other calls or
something.</div>
<div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Option:
cluster.data-self-heal-algorit<wbr>hm<br>
Default Value: (null)<br>
Description: Select between "full",
"diff". The "full" algorithm copies
the entire file from source to sink.
The "diff" algorithm copies to sink
only those blocks whose checksums
don't match with those of source. If
no option is configured the option
is chosen dynamically as follows: If
the file does not exist on one of
the sinks or empty file exists or if
the source file size is about the
same as page size the entire file
will be read and written i.e "full"
algo, otherwise "diff" algo is
chosen.</blockquote>
</div>
<div><br>
</div>
<div>I really have no idea what this
means and how/why it would help. Any
more info on this option?</div>
</div>
</blockquote>
<br>
</span><a class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473moz-txt-link-freetext" href="https://github.com/gluster/glusterfs-specs/blob/master/done/GlusterFS%203.8/granular-entry-self-healing.md" target="_blank">https://github.com/gluster/glu<wbr>sterfs-specs/blob/master/done/<wbr>GlusterFS%203.8/granular-entry<wbr>-self-healing.md</a>
should help.<br>
Regards,<br>
Ravi
<div>
<div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205h5"><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Option:
cluster.granular-entry-heal<br>
Default Value: no<br>
Description: If this option is
enabled, self-heal will resort to
granular way of recording
changelogs and doing entry
self-heal.</blockquote>
</div>
<div><br>
</div>
<div>Thank you.</div>
</div>
<div class="gmail_extra"><br clear="all">
<div>
<div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr"><br>
Sincerely,<br>
Artem<br>
<br>
--<br>
Founder, <a href="http://www.androidpolice.com" target="_blank">Android Police</a>, <a href="http://www.apkmirror.com/" style="font-size:12.8000001907349px" target="_blank">APK
Mirror</a><span style="font-size:12.8000001907349px">, Illogical Robot LLC</span></div>
<div dir="ltr"><a href="http://beerpla.net/" target="_blank">beerpla.net</a>
| <a href="https://plus.google.com/+ArtemRussakovskii" target="_blank">+ArtemRussakovskii</a> | <a href="http://twitter.com/ArtemR" target="_blank">@ArtemR</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">On Tue, Apr
17, 2018 at 9:58 PM, Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div>
<div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473h5">
<br>
<div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854moz-cite-prefix">On
04/18/2018 10:14 AM, Artem
Russakovskii wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Following
up here on a related and
very serious for us
issue.
<div><br>
</div>
<div>I took down one of
the 4 replicate
gluster servers for
maintenance today.
There are 2 gluster
volumes totaling about
600GB. Not that much
data. After the server
comes back online, it
starts auto healing
and pretty much all
operations on gluster
freeze for many
minutes.</div>
<div><br>
</div>
<div>For example, I was
trying to run an ls
-alrt in a folder with
7300 files, and it
took a good 15-20
minutes before
returning.</div>
<div><br>
</div>
<div>During this time, I
can see iostat show
100% utilization on
the brick, heal status
takes many minutes to
return, glusterfsd
uses up tons of CPU (I
saw it spike to 600%).
gluster already has
massive performance
issues for me, but
healing after a 4-hour
downtime is on another
level of bad perf.</div>
<div><br>
</div>
<div>For example, this
command took many
minutes to run:</div>
<div><br>
</div>
<div>
<div>gluster volume
heal
androidpolice_data3
info summary</div>
<div>Brick
nexus2:/mnt/nexus2_block4/andr<wbr>oidpolice_data3</div>
<div>Status: Connected</div>
<div>Total Number of
entries: 91</div>
<div>Number of entries
in heal pending: 90</div>
<div>Number of entries
in split-brain: 0</div>
<div>Number of entries
possibly healing: 1</div>
<div><br>
</div>
<div>Brick
forge:/mnt/forge_block4/androi<wbr>dpolice_data3</div>
<div>Status: Connected</div>
<div>Total Number of
entries: 87</div>
<div>Number of entries
in heal pending: 86</div>
<div>Number of entries
in split-brain: 0</div>
<div>Number of entries
possibly healing: 1</div>
<div><br>
</div>
<div>Brick
hive:/mnt/hive_block4/androidp<wbr>olice_data3</div>
<div>Status: Connected</div>
<div>Total Number of
entries: 87</div>
<div>Number of entries
in heal pending: 86</div>
<div>Number of entries
in split-brain: 0</div>
<div>Number of entries
possibly healing: 1</div>
<div><br>
</div>
<div>Brick
citadel:/mnt/citadel_block4/an<wbr>droidpolice_data3</div>
<div>Status: Connected</div>
<div>Total Number of
entries: 0</div>
<div>Number of entries
in heal pending: 0</div>
<div>Number of entries
in split-brain: 0</div>
<div>Number of entries
possibly healing: 0</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Statistics showed a
diminishing number of
failed heals:</div>
<div>...</div>
<div>
<div>Ending time of
crawl: Tue Apr 17
21:13:08 2018</div>
<div><br>
</div>
<div>Type of crawl:
INDEX</div>
<div>No. of entries
healed: 2</div>
<div>No. of entries in
split-brain: 0</div>
<div>No. of heal
failed entries: 102</div>
<div><br>
</div>
<div>Starting time of
crawl: Tue Apr 17
21:13:09 2018</div>
<div><br>
</div>
<div>Ending time of
crawl: Tue Apr 17
21:14:30 2018</div>
<div><br>
</div>
<div>Type of crawl:
INDEX</div>
<div>No. of entries
healed: 4</div>
<div>No. of entries in
split-brain: 0</div>
<div>No. of heal
failed entries: 91</div>
<div><br>
</div>
<div>Starting time of
crawl: Tue Apr 17
21:14:31 2018</div>
<div><br>
</div>
<div>Ending time of
crawl: Tue Apr 17
21:15:34 2018</div>
<div><br>
</div>
<div>Type of crawl:
INDEX</div>
<div>No. of entries
healed: 0</div>
<div>No. of entries in
split-brain: 0</div>
<div>No. of heal
failed entries: 88</div>
</div>
<div>...</div>
<div><br>
</div>
<div>Eventually,
everything heals and
goes back to at least
where the roof isn't
on fire anymore.</div>
<div><br>
</div>
<div>The server stats
and volume options
were given in one of
the previous replies
to this thread.</div>
<div><br>
</div>
<div>Any ideas or things
I could run and show
the output of to help
diagnose? I'm also
very open to working
with someone on the
team on a live
debugging session if
there's interest.</div>
</div>
</blockquote>
<br>
</div>
</div>
It is likely that self-heal is
causing the CPU spike due to the
flood of lookups/ locks and
checksum fops that the
self-heal-daemon sends to the
bricks.<br>
There's a script to control
shd's cpu usage using cgroups.
That should help in regulating
self-heal traffic: <a class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854moz-txt-link-freetext" href="https://review.gluster.org/#/c/18404/" target="_blank">https://review.gluster.org/#/c<wbr>/18404/</a>
(see extras/control-cpu-load.sh)<br>
Other self-heal related volume
options that you could change
are setting
'cluster.data-self-heal-algori<wbr>thm'
to 'full' and
'granular-entry-heal' to
'enable'. `gluster volume set
help` should give you more
information about these options.<br>
Thanks,<br>
Ravi
<div>
<div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473h5"><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>Thank you.</div>
</div>
<div class="gmail_extra"><br clear="all">
<div>
<div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr"><br>
Sincerely,<br>
Artem<br>
<br>
--<br>
Founder, <a href="http://www.androidpolice.com" target="_blank">Android Police</a>, <a href="http://www.apkmirror.com/" style="font-size:12.8000001907349px" target="_blank">APK
Mirror</a><span style="font-size:12.8000001907349px">, Illogical Robot LLC</span></div>
<div dir="ltr"><a href="http://beerpla.net/" target="_blank">beerpla.net</a>
| <a href="https://plus.google.com/+ArtemRussakovskii" target="_blank">+ArtemRussakovskii</a> | <a href="http://twitter.com/ArtemR" target="_blank">@ArtemR</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">On
Tue, Apr 10, 2018 at
9:56 AM, Artem
Russakovskii <span dir="ltr"><<a href="mailto:archon810@gmail.com" target="_blank">archon810@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi
Vlad,
<div><br>
</div>
<div>I actually
saw that post
already and even
asked a question
4 days ago (<a href="https://serverfault.com/questions/517775/glusterfs-direct-i-o-mode#comment1172497_540917" target="_blank">https://serverfault.com/quest<wbr>ions/517775/glusterfs-direct-i<wbr>-o-mode#comment1172497_540917</a>)<wbr>.
The accepted
answer also
seems to go
against your
suggestion to
enable
direct-io-mode
as it says it
should be
disabled for
better
performance when
used just for
file accesses.</div>
<div><br>
</div>
<div>It'd be great
if someone from
the Gluster team
chimed in about
this thread.</div>
</div>
<div class="gmail_extra"><span><br clear="all">
<div>
<div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854m_3369104992988144420gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr"><br>
Sincerely,<br>
Artem<br>
<br>
--<br>
Founder, <a href="http://www.androidpolice.com" target="_blank">Android Police</a>, <a href="http://www.apkmirror.com/" style="font-size:12.8000001907349px" target="_blank">APK
Mirror</a><span style="font-size:12.8000001907349px">, Illogical Robot LLC</span></div>
<div dir="ltr"><a href="http://beerpla.net/" target="_blank">beerpla.net</a>
| <a href="https://plus.google.com/+ArtemRussakovskii" target="_blank">+ArtemRussakovskii</a> | <a href="http://twitter.com/ArtemR" target="_blank">@ArtemR</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</span>
<div>
<div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854h5">
<div class="gmail_quote">On
Tue, Apr 10,
2018 at 7:01
AM, Vlad
Kopylov <span dir="ltr"><<a href="mailto:vladkopy@gmail.com" target="_blank">vladkopy@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>Wish I
knew or was
able to get
detailed
description of
those options
myself.<br>
</div>
<div>here is
direct-io-mode
<a href="https://serverfault.com/questions/517775/glusterfs-direct-i-o-mode" target="_blank">https://serverfault.com/questi<wbr>ons/517775/glusterfs-direct-i-<wbr>o-mode</a><br>
</div>
Same as you I
ran tests on a
large volume
of files,
finding that
main delays
are in
attribute
calls, ending
up with those
mount options
to add
performance.<br>
</div>
<div>I
discovered
those options
through
basically
googling this
user list with
people sharing
their tests.<br>
</div>
Not sure I
would share
your optimism,
and rather
then going up
I downgraded
to 3.12 and
have no dir
view issue
now. Though I
had to
recreate the
cluster and
had to re-add
bricks with
existing data.<br>
</div>
<div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854m_3369104992988144420HOEnZb">
<div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854m_3369104992988144420h5">
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Tue, Apr 10,
2018 at 1:47
AM, Artem
Russakovskii <span dir="ltr"><<a href="mailto:archon810@gmail.com" target="_blank">archon810@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi
Vlad,
<div><br>
</div>
<div>I'm using
only
localhost:
mounts.</div>
<div><br>
</div>
<div>Can you
please explain
what effect
each option
has on
performance
issues shown
in my posts?
"negative-timeout=10,attribute<wbr>-timeout=30,fopen-keep-cache,d<wbr>irect-io-mode=enable,fetch-att<wbr>empts=5"
From what I
remember,
direct-io-mode=enable
didn't make a
difference in
my tests, but
I suppose I
can try again.
The
explanations
about
direct-io-mode
are quite
confusing on
the web in
various
guides, saying
enabling it
could make
performance
worse in some
situations and
better in
others due to
OS file cache.</div>
<div><br>
</div>
<div>There are
also these
gluster volume
settings,
adding to the
confusion:</div>
<div>
<div>Option:
performance.strict-o-direct</div>
<div>Default
Value: off</div>
<div>Description:
This option
when set to
off, ignores
the O_DIRECT
flag.</div>
<div><br>
</div>
<div>Option:
performance.nfs.strict-o-direc<wbr>t</div>
<div>Default
Value: off</div>
<div>Description:
This option
when set to
off, ignores
the O_DIRECT
flag.</div>
</div>
<div><br>
</div>
<div>Re: 4.0.
I moved to 4.0
after finding
out that it
fixes the
disappearing
dirs bug
related to
cluster.readdir-optimize
if you
remember (<a href="http://lists.gluster.org/pipermail/gluster-users/2018-April/033830.html" target="_blank">http://lists.gluster.org/pipe<wbr>rmail/gluster-users/2018-April<wbr>/033830.html</a>).
I was already
on 3.13 by
then, and 4.0
resolved the
issue. It's
been stable
for me so far,
thankfully.</div>
<div class="gmail_extra"><span><br clear="all">
<div>
<div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854m_3369104992988144420m_-2442537929453809111m_3104514395859481521m_-9168214684418515114gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr"><br>
Sincerely,<br>
Artem<br>
<br>
--<br>
Founder, <a href="http://www.androidpolice.com" target="_blank">Android Police</a>, <a href="http://www.apkmirror.com/" style="font-size:12.8000001907349px" target="_blank">APK
Mirror</a><span style="font-size:12.8000001907349px">, Illogical Robot LLC</span></div>
<div dir="ltr"><a href="http://beerpla.net/" target="_blank">beerpla.net</a>
| <a href="https://plus.google.com/+ArtemRussakovskii" target="_blank">+ArtemRussakovskii</a> | <a href="http://twitter.com/ArtemR" target="_blank">@ArtemR</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</span>
<div>
<div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854m_3369104992988144420m_-2442537929453809111h5">
<div class="gmail_quote">On
Mon, Apr 9,
2018 at 10:38
PM, Vlad
Kopylov <span dir="ltr"><<a href="mailto:vladkopy@gmail.com" target="_blank">vladkopy@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>you
definitely
need mount
options to
/etc/fstab<br>
</div>
use ones from
here <a href="http://lists.gluster.org/pipermail/gluster-users/2018-April/033811.html" target="_blank">http://lists.gluster.org/piper<wbr>mail/gluster-users/2018-April/<wbr>033811.html</a><br>
<br>
</div>
<div>I went on
with using
local mounts
to achieve
performance as
well<br>
</div>
<div><br>
</div>
Also, 3.12 or
3.10 branches
would be
preferable for
production <br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">
<div>
<div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854m_3369104992988144420m_-2442537929453809111m_3104514395859481521m_-9168214684418515114h5">On
Fri, Apr 6,
2018 at 4:12
AM, Artem
Russakovskii <span dir="ltr"><<a href="mailto:archon810@gmail.com" target="_blank">archon810@gmail.com</a>></span>
wrote:<br>
</div>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854m_3369104992988144420m_-2442537929453809111m_3104514395859481521m_-9168214684418515114h5">
<div dir="ltr">Hi
again,
<div><br>
</div>
<div>I'd like
to expand on
the
performance
issues and
plead for
help. Here's
one case which
shows these
odd hiccups: <a href="https://i.imgur.com/CXBPjTK.gifv" target="_blank">https://i.imgur.com/C<wbr>XBPjTK.gifv</a>.
<div><br>
</div>
<div>In this
GIF where I
switch <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">back
and forth</span>
between copy
operations on
2 servers, I'm
copying a 10GB
dir full of
.apk and image
files.</div>
<div><br>
</div>
<div>On server
"hive" I'm
copying
straight from
the main disk
to an attached
volume block
(xfs). As you
can see, the
transfers are
relatively
speedy and
don't hiccup.</div>
<div>On server
"citadel" I'm
copying the
same set of
data to a
4-replicate
gluster which
uses block
storage as a
brick. As you
can see,
performance is
much worse,
and there are
frequent
pauses for
many seconds
where nothing
seems to be
happening -
just freezes.<br>
</div>
<div><br>
</div>
<div>All 4
servers have
the same
specs, and all
of them have
performance
issues with
gluster and no
such issues
when raw xfs
block storage
is used.</div>
<div><br>
</div>
<div>hive has
long finished
copying the
data, while
citadel is
barely
chugging along
and is
expected to
take probably
half an hour
to an hour. I
have over 1TB
of data to
migrate, at
which point if
we went live,
I'm not even
sure gluster
would be able
to keep up
instead of
bringing the
machines and
services down.</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Here's
the cluster
config, though
it didn't seem
to make any
difference
performance-wise
before I
applied the
customizations
vs after.</div>
<div><br>
</div>
<div>
<div>Volume
Name:
apkmirror_data1</div>
<div>Type:
Replicate</div>
<div>Volume
ID:
11ecee7e-d4f8-497a-9994-ceb144<wbr>d6841e</div>
<div>Status:
Started</div>
<div>Snapshot
Count: 0</div>
<div>Number of
Bricks: 1 x 4
= 4</div>
<div>Transport-type:
tcp</div>
<div>Bricks:</div>
<div>Brick1:
nexus2:/mnt/nexus2_block1/apkm<wbr>irror_data1</div>
<div>Brick2:
forge:/mnt/forge_block1/apkmir<wbr>ror_data1</div>
<div>Brick3:
hive:/mnt/hive_block1/apkmirro<wbr>r_data1</div>
<div>Brick4:
citadel:/mnt/citadel_block1/ap<wbr>kmirror_data1</div>
<div>Options
Reconfigured:</div>
<div>cluster.quorum-count:
1</div>
<div>cluster.quorum-type:
fixed</div>
<div>network.ping-timeout:
5</div>
<div>network.remote-dio:
enable</div>
<div>performance.rda-cache-limit:
256MB</div>
<div>performance.readdir-ahead:
on</div>
<div>performance.parallel-readdir:
on</div>
<div>network.inode-lru-limit:
500000</div>
<div>performance.md-cache-timeout:
600</div>
<div>performance.cache-invalidation<wbr>:
on</div>
<div>performance.stat-prefetch:
on</div>
<div>features.cache-invalidation-ti<wbr>meout:
600</div>
<div>features.cache-invalidation:
on</div>
<div>cluster.readdir-optimize:
on</div>
<div>performance.io-thread-count:
32</div>
<div>server.event-threads:
4</div>
<div>client.event-threads:
4</div>
<div>performance.read-ahead:
off</div>
<div>cluster.lookup-optimize:
on</div>
<div>performance.cache-size:
1GB</div>
<div>cluster.self-heal-daemon:
enable</div>
<div>transport.address-family:
inet</div>
<div>nfs.disable:
on</div>
<div>performance.client-io-threads:
on</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>The
mounts are
done as
follows in
/etc/fstab:</div>
<div>/dev/disk/by-id/scsi-0Linode_V<wbr>olume_citadel_block1
/mnt/citadel_block1 xfs defaults 0 2<br>
</div>
<div>localhost:/apkmirror_data1
/mnt/apkmirror_data1 glusterfs defaults,_netdev 0 0<br>
</div>
<div><br>
</div>
<div>I'm
really not
sure if
direct-io-mode
mount tweaks
would do
anything here,
what the value
should be set
to, and what
it is by
default.</div>
<div><br>
</div>
<div>The OS is
OpenSUSE 42.3,
64-bit. 80GB
of RAM, 20
CPUs, hosted
by Linode.</div>
<div><br>
</div>
<div>I'd
really
appreciate any
help in the
matter. </div>
<div><br>
</div>
<div>Thank
you.</div>
</div>
</div>
<div class="gmail_extra"><span><br clear="all">
<div>
<div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854m_3369104992988144420m_-2442537929453809111m_3104514395859481521m_-9168214684418515114m_-7640979976631609660m_7374879952101670984gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr"><br>
Sincerely,<br>
Artem<br>
<br>
--<br>
Founder, <a href="http://www.androidpolice.com" target="_blank">Android Police</a>, <a href="http://www.apkmirror.com/" style="font-size:12.8000001907349px" target="_blank">APK
Mirror</a><span style="font-size:12.8000001907349px">, Illogical Robot LLC</span></div>
<div dir="ltr"><a href="http://beerpla.net/" target="_blank">beerpla.net</a>
| <a href="https://plus.google.com/+ArtemRussakovskii" target="_blank">+ArtemRussakovskii</a> | <a href="http://twitter.com/ArtemR" target="_blank">@ArtemR</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</span>
<div>
<div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854m_3369104992988144420m_-2442537929453809111m_3104514395859481521m_-9168214684418515114m_-7640979976631609660h5">
<div class="gmail_quote">On
Thu, Apr 5,
2018 at 11:13
PM, Artem
Russakovskii <span dir="ltr"><<a href="mailto:archon810@gmail.com" target="_blank">archon810@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi,
<div><br>
</div>
<div>I'm
trying to
squeeze
performance
out of gluster
on 4 80GB RAM
20-CPU
machines where
Gluster runs
on attached
block storage
(Linode) in (4
replicate
bricks), and
so far
everything I
tried results
in sub-optimal
performance.</div>
<div><br>
</div>
<div>There are
many files -
mostly images,
several
million - and
many
operations
take minutes,
copying
multiple files
(even if
they're small)
suddenly
freezes up for
seconds at a
time, then
continues,
iostat
frequently
shows large
r_await and
w_awaits with
100%
utilization
for the
attached block
device, etc.</div>
<div><br>
</div>
<div>But
anyway, there
are many
guides out
there for
small-file
performance
improvements,
but more
explanation is
needed, and I
think more
tweaks should
be possible.<br>
</div>
<div><br>
</div>
<div>My
question today
is
about performance.cache-size.
Is this a size
of cache in
RAM? If so,
how do I view
the current
cache size to
see if it gets
full and I
should
increase its
size? Is it
advisable to
bump it up if
I have many
tens of gigs
of RAM free? </div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>More
generally, in
the last 2
months since I
first started
working with
gluster and
set a
production
system live,
I've been
feeling
frustrated
because
Gluster has a
lot of
poorly-documented
and confusing
options. I
really wish
documentation
could be
improved with
examples and
better
explanations.</div>
<div><br>
</div>
<div>Specifically,
it'd be
absolutely
amazing if the
docs offered a
strategy for
setting each
value and ways
of determining
more optimal
values. For
example,
for performance.cache-size,
if it said
something like
"run command
abc to see
your current
cache size,
and if it's
hurting, up
it, but be
aware that
it's limited
by RAM," it'd
be already a
huge
improvement to
the docs. And
so on with
other options.</div>
<div>
<div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br>
</div>
<div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br>
</div>
<div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br>
</div>
<div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">The
gluster team
is quite
helpful on
this mailing
list, but in a
reactive
rather than
proactive way.
Perhaps it's
tunnel vision
once you've
worked on a
project for so
long where
less technical
explanations
and even
proper
documentation
of options
takes a back
seat, but I
encourage you
to be more
proactive
about helping
us understand
and optimize
Gluster.</div>
<div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br>
</div>
<div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial">Thank
you.</div>
<div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial"><br>
</div>
<div>
<div class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854m_3369104992988144420m_-2442537929453809111m_3104514395859481521m_-9168214684418515114m_-7640979976631609660m_7374879952101670984m_1420630595765814996gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">Sincerely,<br>
Artem<br>
<br>
--<br>
Founder, <a href="http://www.androidpolice.com" target="_blank">Android Police</a>, <a href="http://www.apkmirror.com/" style="font-size:12.8px" target="_blank">APK
Mirror</a><span style="font-size:12.8px">, Illogical Robot LLC</span></div>
<div dir="ltr"><a href="http://beerpla.net/" target="_blank">beerpla.net</a>
| <a href="https://plus.google.com/+ArtemRussakovskii" target="_blank">+ArtemRussakovskii</a> | <a href="http://twitter.com/ArtemR" target="_blank">@ArtemR</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
<br>
</div>
</div>
<span>______________________________<wbr>_________________<br>
Gluster-users
mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
</span></blockquote>
</div>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854mimeAttachmentHeader"></fieldset>
<br>
<pre>______________________________<wbr>_________________
Gluster-users mailing list
<a class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a class="m_229972322048706121m_-5940135115956573066m_7089802003934311205m_1780620483197147473m_979837676701406854moz-txt-link-freetext" href="http://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<br>
</div>
</blockquote></div><br></div>