[Gluster-devel] [Gluster-Maintainers] Modifying gluster's logging mechanism

Sun Dec 1 09:19:51 UTC 2019

Greetings all and thank you for your contribution to this discussion.

Amar and Sankarshan, regarding your comments:

> * First of all, when we looked at log very early in project's life, our
> idea was based mostly on kernel logs (/var/log/messages). We decided, as a
> file-system, because it is very active with I/Os and should run for years
> together without failing, there should be NO log messages when the system
> is healthy, which should be 99%+ time.
>
> * Now, if there are no logs when everything is healthy, and most of the
> things are healthy 99% of the time, naturally the focus was not
> 'performance' of logging infra, but the correctness. This is where, the
> strict ordering through locks to preserve the timestamps of logs, and have
> it organized came by.
>
 I believe this can be said about pretty much every software - nothing is
designed to break (except maybe *piñatas*?).
I am curious though - when the project had begun, there were no INFO levels
logs? These logs are logged even when everything is healthy.
In any case, we can set the history aside and focus on the present, and I
believe many components were upgraded and improved over time and it looks
like the logger was left behind, and is seems there are consequences for
this now.

That is interesting observation. For this alone, can we have an option to
> disable all logging during regression? That would fasten up things for
> normal runs immediately.

We had an idea that could have improved CI time, but because the test
module is built in such a way that each test starts gluster by itself, our
idea was not feasible as it would require modification to all the tests (we
wanted to run tests with no logging, and do log only when a test fails, on
the re-run).
Anyway, IMO, turning off logging does not help us at all handle the core
problem, and I also believe it may send a wrong message to our users ("our
logging is not good enough so we decide to get rid of it instead of
improving it").
And I fully agree with Sankarshan's comment:

> If having quicker regression runs is the objective, then perhaps we should
> not look at turning off logging to accomplish them. Instead, there ought to
> be additional aspects which can be reviewed with turning off logging being
> the last available option.
>

> Please make sure, we have minimum dependency on this project, and also the
> install steps are easy for your project. One of the reasons Gluster
> succeeded is because it is very easy to install (and uninstall), without
> much dependencies. I personally would like to keep that behavior same.
>
I did not mean for it to be a dependency for the project, I suggested that
as an alternative logging solution that would be fully integrated into
gluster.

I am in agreement with Sankarshan, who responded in another thread about
> tools like EFK/ELK stack for centralized logging. By moving to centralized
> logging, and having proper structure for logging, the initial goals we had
> wouldn't be broken, and we will have better performance even when we are
> logging in DEBUG mode.
>
I do not know enough about EFK/ELK to base an opinion, but I understand
from your comments that this is considered standard  components for logging
etc'.
I'm not against this at all but I do need to investigate more before I
could share my thoughts on the technical aspects.

[Amar] This, in my opinion is a good reason to undertake this activity.
> Mainly because we should be having our logging infra similar with other
> tools, and one shouldn't be having a learning curve to understand gluster's
> logging.

[Sankarshan] The original proposal from Barak has merit for planning
> towards an alpha form to be available.
>
I'm very happy to hear this, as it gives me confidence in my actions, thank
you.

Unfortunately, I've been informed that I'm not to pursue this initiative at
the current time, as there are other, higher, priorities that requires
resources and it was decided not to dedicate them for this initiative.
I do hope that when the time is right, we could revisit this discussion and
see how we can advance on that front.
If anyone has insight he'd like to share, I'd be happy to discuss it.

Thank you all very much,

On Wed, Nov 27, 2019 at 4:33 AM Sankarshan Mukhopadhyay <
sankarshan.mukhopadhyay at gmail.com> wrote:

>
>
> On Wed, Nov 27, 2019 at 7:44 AM Amar Tumballi <amarts at gmail.com> wrote:
>
>> Hi Barak,
>>
>> My replies inline.
>>
>> On Thu, Nov 21, 2019 at 6:34 PM Barak Sason Rofman <bsasonro at redhat.com>
>> wrote:
>>
>>>
>>>
> [snip]
>
>
>>
>>> Initial tests, done by *removing logging from the regression testing,
>>> shows an improvement of about 20% in run time*. This indicates we’re
>>> taking a pretty heavy performance hit just because of the logging activity.
>>>
>>>
>> That is interesting observation. For this alone, can we have an option to
>> disable all logging during regression? That would fasten up things for
>> normal runs immediately.
>>
>
> If having quicker regression runs is the objective, then perhaps we should
> not look at turning off logging to accomplish them. Instead, there ought to
> be additional aspects which can be reviewed with turning off logging being
> the last available option.
> _______________________________________________
>
> Community Meeting Calendar:
>
> APAC Schedule -
> Every 2nd and 4th Tuesday at 11:30 AM IST
> Bridge: https://bluejeans.com/441850968
>
>
> NA/EMEA Schedule -
> Every 1st and 3rd Tuesday at 01:00 PM EDT
> Bridge: https://bluejeans.com/441850968
>
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
>

-- 
*Barak Sason Rofman*

Gluster Storage Development

Red Hat Israel <https://www.redhat.com/>

34 Jerusalem rd. Ra'anana, 43501

bsasonro at redhat.com <adi at redhat.com>    T: *+972-9-7692304*
M: *+972-52-4326355*
<https://red.ht/sig>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20191201/d3e32a0b/attachment.html>