[Gluster-devel] Troubleshooting and Diagnostic tools for Gluster

[Shyam]

On 10/23/2015 06:46 AM, Aravinda wrote:
> Hi Gluster developers,
>
> In this mail I am proposing troubleshooting documentation and
> Gluster Tools infrastructure.
>
> Tool to search in documentation
> ===============================
> We recently added message Ids to each error messages in Gluster. Some
> of the error messages are self explanatory. But some error messages
> requires manual intervention to fix the issue. How about identifying
> the error messages which requires more explanation and creating
> documentation for the same. Even though the information about some
> errors available in documentation, it is very difficult to search and
> relate to the error message. It will be very useful if we create a
> tool which looks for documentation and tells us exactly what to do.
>
> For example,(Illustrativepurpose only)
> glusterdoc --explain GEOREP0003
>
>      SSH configuration issue. This error is seen when Pem keys from all
>      master nodes are not distributed properly to Slave
>      nodes. Use Geo-replication create command with force option to
>      redistribute the keys. If issue stillpersists, look for any errors
>      while running hook scripts inGlusterd log file.
>
>
> Note: Inspired from rustc --explain command
> https://twitter.com/jaredforsyth/status/626960244707606528
>
> If we don't know the message id, we can still search from the
> available documentation like,
>
>      glusterdoc --search <SEARCH_KEY_WORD>
>
> These commands can be programmatically consumed, for example
> `--json` will return the output in JSON format. This enables UI
> developers to automatically show help messages when they display
> errors.

The message ID based logging was created for this exact purpose (maybe 
not so elegant a purpose, but leaning towards this :) ). So I am all for it.

(suggestion) The intention of documenting messages with text in DOxygen 
format, was to be able extract this information from the headers, and 
create a catalog, that can then be searched etc. This catalog can be 
processed and shipped as part of the gluster RPMs, which the tool above 
can use.

>
> Gluster Tools infrastructure
> ============================
> Are our Gluster log files sufficient for root causing the issues? Is
> that error caused due to miss configuration? Geo-replication status is
> showing faulty. Where to find the reason for Faulty?
>
> Sac(surs AT redhat.com) mentioned that heis working on gdeploy and many
> developers
> are using their owntools. How about providing common infrastructure(say
> gtool/glustertool) to host all these tools.
>
>  From my toolkit, following tools are available, planning to create
> more such tools for Geo-replication and Gluster.
>
>      volinfo [<VOLNAME>] - Enhanced version of Gluster Volume info
>      command (http://aravindavk.in/blog/glusterfs-tools/ )
>
>      df - df for Gluster Volumes
> (http://aravindavk.in/blog/glusterdf-df-for-gluster-volumes/ )
>
>      georepsetup - A tool to Create Geo-replication session
> easily(http://aravindavk.in/blog/introducing-georepsetup/ )
>
>      gdash - A simple Dashboard for
>      Gluster(http://aravindavk.in/blog/introducing-gdash/ )
>
>      gfid <PATH>   - To get GFID of a file, works both in Mount and
> Backend(https://github.com/aravindavk/gluster_georep_scripts )
>
>      clparser <PATH> - Parse the backend Changelog and print in human
>      readable format(https://github.com/aravindavk/gluster_georep_scripts )
>
>      xtime <PATH>  - To get XTIME xattr from given
> path(https://github.com/aravindavk/gluster_georep_scripts )
>
>      stime <PATH> - To get STIME xattr from given path(Used by
>      Geo-replication https://github.com/aravindavk/gluster_georep_scripts )
>
>      volmark <VOLNAME> - To get Volume Mark of given Volume(Used by
>      Geo-replication https://github.com/aravindavk/gluster_georep_scripts )
>
>
> Geo-replication developers are already using some tools like Changelog
> parser, `arequal-checksum` etc.
>
> Initial idea for Tools Framework:
> ---------------------------------
> A Shell/Python script which looks for the tool in plugins sub directory, if
> exists pass all the arguments and call that script.
>
> `glustertool help` triggers a python Script plugins/help.py which reads
> plugins.yml file to get the list of tools and help messages associated
> with it.
>
> No restrictions on the choice of programming language to create
> tool. It can be bash, Python, Go, Rust, awk, sed etc.
>
> Challenges:
> - Each plugin may have different dependencies, installing all tools
> may install all the dependencies.
> - Multiple programming languages, may be difficult to maintain/build.
> - Maintenance of Third party tools.
> - Creating Plugins registry to discover tools created by other developers.
>
> Tool Ideas:
> -----------
> If you are interested in working on tools for Gluster, I am listing a
> few ideas to start with, feel free to add your ideas to the list.
>
> - A tool to analyze the log file and identify issues. For example,
>    glustertool georep_log_analize <LOG FILE PATH> --after-date <TIMESTAMP>
>
>    Example output: (Illustrative purpose only)
>
>    Number of workers in this node: 2
>    Number of restarts: 5
>    Errors: 10
>    Python Tracebacks: 5
>    Last state: Active
>    Files Skipped: 0
>    Setup issue: No
>
> - Extract skipped GFIDs from Geo-replication logs and re-trigger sync.
>
>    For example,
>    glustertool georep_extract_skipped <LOG_FILE> --after-date<TIMESTAMP>
>
>    This command will
>    1. extract Skipped GFIDs list,
>    2. Mounts Master Volume
>    3. converts GFID to Path
>    4. Set Virtual xattr to re-trigger the sync
>
> - A tool to detect Split brain
>
> - A tool to convert GFID to Path
>
> Created a etherpad to record the available tools and ideas
> https://public.pad.fsfe.org/p/gluster-tools
> Will update once the I make some progress in creating infrastructure.
>
> Comments and Suggestions welcome.
>