[Gluster-devel] Modifying gluster's logging mechanism
Xie Changlong
zgrep at 139.com
Fri Nov 22 09:54:42 UTC 2019
在 2019/11/22 17:43, Barak Sason Rofman 写道:
> Thank you for your input Atin and Xie Changlong.
>
> Regarding log ordering - my initial thought was to do it offline using
> a dedicated too. Should be straight forward, as the logs have time
> stamp composed of seconds and microseconds, so ordering them using
> this value is definitely possible.
> This is actually one of the main reasons I wanted to bring this up for
> discussion - will it be fine with the community to run a dedicated
> tool to reorder the logs offline?
> Reordering the logs offline will allow us to gain the most performance
> improvement, as keeping the logs order online will have some cost
> (probably through stronger synchronization).
> Moreover, we can take log viewing one step further and maybe create
> some GUI system (JAVA based?) to view and handle logs (e.g. one window
> to show the combined order logs, other windows to show logs per thread
> etc').
>
> Regarding the test method - my initial testing was done by removing
> all logging from regression. Modifying the method "skip_logging" to
> return 'true' in all cases seems to remove most of the logs (though
> not all, "to be on the safe side", really removing all logging related
> methods is probably even better).
Thanks for sharing, i'll go back to your method and do some perf test : )
> As regression tests use mostly single-node tests, some additional
> testing was needed. I've written a couple of very basic scripts to
> create large number of files / big files, read / write to / from them,
> move them around and perform some other basic functionality.
> I'd actually be glad to test this in some 'real world' use cases - if
> you have specific use cases that you use frequently, we can model them
> and benchmark against - this will likely offer an even more accurate
> benchmark.
>
> On Fri, Nov 22, 2019 at 7:27 AM Xie Changlong <zgrep at 139.com
> <mailto:zgrep at 139.com>> wrote:
>
>
> 在 2019/11/21 21:04, Barak Sason Rofman 写道:
>>
>> I see two design / implementation problems with that mechanism:
>>
>> 1.
>>
>> The mutex that guards the log file is likely under constant
>> contention.
>>
>> 2.
>>
>> The fact that each worker thread perform the IO by himself,
>> thus slowing his "real" work.
>>
>>
>> Initial tests, done by *removing logging from the regression
>> testing, shows an improvement of about 20% in run time*. This
>> indicates we’re taking a pretty heavy performance hit just
>> because of the logging activity.
>>
>>
> Hi Barak Sason Rofman. Amazing perf improvement! Could show me
> the detail test method ?
>
> Thanks
>
> -Xie
>
>> In addition to these problems, the logging module is due for an
>> upgrade:
>>
>> 1.
>>
>> There are dozens of APIs in the logger, much of them are
>> deprecated - this makes it very hard for new developers to
>> keep evolving the project.
>>
>> 2.
>>
>> One of the key points for Gluster-X, presented in October at
>> Bangalore, is the switch to a structured logging all across
>> gluster.
>>
>
>
> --
> *Barak Sason Rofman*
>
> Gluster Storage Development
>
> Red Hat Israel <https://www.redhat.com/>
>
> 34 Jerusalem rd. Ra'anana, 43501
>
> bsasonro at redhat.com <mailto:adi at redhat.com> T: _+972-9-7692304_
> M: _+972-52-4326355_
>
> <https://red.ht/sig>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20191122/6bc92281/attachment.html>
More information about the Gluster-devel
mailing list