[Gluster-devel] [Gluster-Maintainers] Modifying gluster's logging mechanism

[Sankarshan Mukhopadhyay]

The original proposal from Barak has merit for planning towards an
alpha form to be available. Has the project moved away from writing
specifications and reviewing those proposals? Earlier, we used to see
those rather than discuss in multi-threaded list emails. While
recorded gains in performance are notable, it would be prudent to
cleanly assess the switch-over cost should the project want to move
over to the new patterns of logging. This seems like a good time to
plan well for a change that has both utility and value.

That said, I'd like to point out some relevant aspects. Logging is
just one (although an important one) part of what is being talked
about as o11y (observability). A log (or, a structured log) is a
record of events. Debugging of distributed systems require
understanding of an unit of work which flowed through a system firing
off events which in turn were recorded. Logs thus are often portions
of events which are part of this unit of work (assume this is a
"transaction" if that helps grasp it better). Or, in other words, logs
are portions of the abstraction ie. events. The key aspect I'd like to
highlight is that (re)designing structured logging in isolation from
o11y principles will not work as we see more customers adopt o11y
patterns, tools within their SRE and other emerging sub-teams.
Focusing just on logs keeps us rooted to the visual display of
information via ELK/EFK models rather than consider the behavior
centric diagnosis of the whole system.



On Fri, Nov 22, 2019 at 3:49 PM Ravishankar N <ravishankar at redhat.com> wrote:
>
>
> On 22/11/19 3:13 pm, Barak Sason Rofman wrote:
> > This is actually one of the main reasons I wanted to bring this up for
> > discussion - will it be fine with the community to run a dedicated
> > tool to reorder the logs offline?
>
> I think it is a bad idea to log without ordering and later relying on an
> external tool to sort it.  This is definitely not something I would want
> to do while doing test and development or debugging field issues.
> Structured logging  is definitely useful for gathering statistics and
> post-processing to make reports and charts and what not,  but from a
> debugging point of view, maintaining causality of messages and working
> with command line text editors and tools on log files is important IMO.
>
> I had a similar concerns when  brick multiplexing feature was developed
> where a single log file was used for logging all multiplexed bricks'
> logs.  So much extra work to weed out messages of 99 processes to read
> the log of the 1 process you are interested in.