<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p><br>
</p>
<div class="moz-cite-prefix">在 2019/11/22 17:43, Barak Sason Rofman
写道:<br>
</div>
<blockquote type="cite"
cite="mid:CAFx2kQ11R4rbtBYFHwMRsjhku=2c9hG32CNYTHHBmRyc0Me5sQ@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>Thank you for your input Atin and Xie Changlong.<br>
<br>
</div>
Regarding log ordering - my initial thought was to do
it offline using a dedicated too. Should be straight
forward, as the logs have time stamp composed of
seconds and microseconds, so ordering them using this
value is definitely possible.<br>
</div>
This is actually one of the main reasons I wanted to
bring this up for discussion - will it be fine with the
community to run a dedicated tool to reorder the logs
offline?<br>
</div>
Reordering the logs offline will allow us to gain the most
performance improvement, as keeping the logs order online
will have some cost (probably through stronger
synchronization).<br>
</div>
<div>Moreover, we can take log viewing one step further and
maybe create some GUI system (JAVA based?) to view and
handle logs (e.g. one window to show the combined order
logs, other windows to show logs per thread etc').<br>
</div>
<div><br>
</div>
Regarding the test method - my initial testing was done by
removing all logging from regression. Modifying the method
"skip_logging" to return 'true' in all cases seems to remove
most of the logs (though not all, "to be on the safe side",
really removing all logging related methods is probably even
better).<br>
</div>
</div>
</div>
</blockquote>
<p><br>
</p>
<p>Thanks for sharing, i'll go back to your method and do some perf
test : )<br>
</p>
<p><br>
</p>
<blockquote type="cite"
cite="mid:CAFx2kQ11R4rbtBYFHwMRsjhku=2c9hG32CNYTHHBmRyc0Me5sQ@mail.gmail.com">
<div dir="ltr">
<div>As regression tests use mostly single-node tests, some
additional testing was needed. I've written a couple of very
basic scripts to create large number of files / big files,
read / write to / from them, move them around and perform some
other basic functionality.<br>
</div>
I'd actually be glad to test this in some 'real world' use cases
- if you have specific use cases that you use frequently, we can
model them and benchmark against - this will likely offer an
even more accurate benchmark.<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Nov 22, 2019 at 7:27
AM Xie Changlong <<a href="mailto:zgrep@139.com"
moz-do-not-send="true">zgrep@139.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p><br>
</p>
<div>在 2019/11/21 21:04, Barak Sason Rofman 写道:<br>
</div>
<blockquote type="cite">
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"
id="gmail-m_2174678472873504521gmail-docs-internal-guid-2211c633-7fff-c4eb-83dd-3e2c2e691e75"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">I see two design / implementation problems with that mechanism:</span></p>
<ol style="margin-top:0px;margin-bottom:0px">
<li dir="ltr" style="list-style-type:decimal;font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">The mutex that guards the log file is likely under constant contention.</span></p></li>
<li dir="ltr" style="list-style-type:decimal;font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">The fact that each worker thread perform the IO by himself, thus slowing his "real" work.</span></p></li>
</ol>
<br>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Initial tests, done by <b>removing logging from the regression testing, shows an improvement of about 20% in run time</b>. This indicates we’re taking a pretty heavy performance hit just because of the logging activity.</span></p>
<br>
</blockquote>
<p>Hi Barak Sason Rofman. Amazing perf improvement! Could
show me the detail test method ? <br>
</p>
<p>Thanks</p>
<p> -Xie<br>
</p>
<blockquote type="cite">
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">In addition to these problems, the logging module is due for an upgrade:</span></p>
<ol style="margin-top:0px;margin-bottom:0px">
<li dir="ltr" style="list-style-type:decimal;font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">There are dozens of APIs in the logger, much of them are deprecated - this makes it very hard for new developers to keep evolving the project.</span></p></li>
<li dir="ltr" style="list-style-type:decimal;font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">One of the key points for Gluster-X, presented in October at Bangalore, is the switch to a structured logging all across gluster.</span></p></li>
</ol>
</blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<br>
-- <br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div style="margin-left:44px">
<div
style="margin-bottom:0px;margin-left:0px;padding-bottom:5px">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div><font face="RedHatText, sans-serif"><span
style="font-size:14px;text-transform:capitalize"><b>Barak Sason Rofman</b></span></font><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div style="margin-left:44px">
<div
style="margin-bottom:0px;margin-left:0px;padding-bottom:5px">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<p
style="margin:0px;font-size:14px;color:rgb(0,0,0);font-family:overpass,sans-serif;font-weight:bold;padding:0px;text-transform:uppercase"><span
style="font-family:RedHatText,sans-serif;font-size:12px;text-transform:capitalize;font-weight:normal;color:rgb(34,34,34)">Gluster
Storage </span><span
style="font-family:RedHatText,sans-serif;font-size:12px;text-transform:capitalize;font-weight:normal;color:rgb(34,34,34)">Development</span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div style="margin-left:44px">
<div
style="margin-bottom:0px;margin-left:0px;padding-bottom:5px">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<p style="margin:0px 0px
4px;font-size:12px;font-family:RedHatText,sans-serif"><a
href="https://www.redhat.com/"
style="color:rgb(0,136,206);margin:0px"
target="_blank"
moz-do-not-send="true">Red
Hat Israel</a></p>
<div style="margin-bottom:4px">
<p
style="margin:0px;font-size:12px;font-family:RedHatText,sans-serif">34
Jerusalem rd. Ra'anana, 43501</p>
</div>
<p
style="margin:0px;font-size:12px;font-family:RedHatText,sans-serif"><span
style="margin:0px;padding:0px">bsasonro<a
href="mailto:adi@redhat.com"
style="color:rgb(0,0,0);margin:0px"
target="_blank"
moz-do-not-send="true">@redhat.com</a>
</span> T: <u>+972-9-7692304</u><br>
M: <u>+972-52-4326355</u></p>
<div style="margin-top:12px">
<table border="0">
<tbody>
<tr>
<td width="100px"><a
href="https://red.ht/sig"
style="color:rgb(17,85,204)" target="_blank" moz-do-not-send="true"><img
src="https://static.redhat.com/libs/redhat/brand-assets/latest/corp/logo.png"
moz-do-not-send="true"
width="90" height="auto"></a></td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</body>
</html>