[Gluster-devel] [RFC] Enable message IDs in Gluster logs

[Shyamsundar Ranganathan]

Problem statement: Currently there are quite a slew of logs in Gluster that do not lend themselves to trivial analysis by various tools that help collect and monitor logs. 

This FEAT is to make this _tooling_ better by giving logs some form of message IDs so that the tools do not have to do complex log parsing to break it down to problem areas and suggest troubleshooting options.

It is also intended in this RFE, to generate a catalog of such IDs and messages, that can then be elaborated with various troubleshooting and resolution options.

The idea behind the catalog is that, at each release point we have a catalog for that release (updated during further maintenance releases), which can help point to some documentation on common errors and how to recover from them. So the message IDs do not have to be,
- unique across releases 
- possibly needs little reconciliation on an ongoing basis with the catalog, based on string changes
- does not have to be unique for each message, as the documentation can cover collisions to provide clarity

Proposed solution: Add a hash of the message format to the log message, to serve as the message ID.

IOW, the FMT in gf_log(dom, levl, fmt...) (and equivalents) is used to generate a 32bit hash and the log is printed with the hash. As an example,
I [socket.c:3533:socket_init][29004409] 0-testv-client-2: SSL support is NOT enabled

Advantages:
- Minimal code change (only the gf_log_* needs to change to further compute a hash of the message and add it to the log)
- No developers need to remember to generate a message ID before adding any messages etc.
- Cost of hashing during logging as computed using an example proves the addition to be negligible (to almost no impact whatsoever for about 10K messages printed)

Disadvantages:
- Hash collusion, hence 2 different messages having the same ID (but not an issue due to the catalog/documentation as above)
- Hash changes due to changes in messages
  - could be minor string edits lending to better messages, in which case hash may change and we need to retain the same documentation (reconciliation of catalog between releases)
  - could be major string changes or parameter additions, in which case both hash and the corresponding catalog documentation should change (hence not an issue)

Catalogue generation would have to parse the code to extract all used format strings (which is an interesting problem in itself).

This of this ala systemd/journald, without the message ID requirements from the same.

Shyam