[Gluster-devel] Troubleshooting and Diagnostic tools for Gluster

Thu Jan 28 01:34:30 UTC 2016

Nice tool.

Easy to add, clone the glustertool repo, and run

# ./newtool mnt_log_analyze --type=exec --bin=log_analyzer.sh --prog=bash

I used tool name "mnt_log_analyze" just for this example, feel free to 
name your tool.

This will create following dir and files
glustertool/plugins/mnt_log_analyze
glustertool/plugins/mnt_log_analyze/tool.conf
glustertool/plugins/mnt_log_analyze/doc.txt

Update the tool doc in doc.txt and copy your script to 
glustertool/plugins/mnt_log_analyze dir.

Thats all! use your tool using,

# glustertool mnt_log_analyze /var/log/glusterfs/mnt-glusterfs.log

Let me know if something is not working.

regards
Aravinda

On 01/28/2016 04:56 AM, Raghavendra Bhat wrote:
> I have a script written to analyze the log message of gluster process.
>
> It actually scans the log file and identifies the log messages with ERROR
> and WARNING levels.
> It lists the functions (with either ERROR or WARNING logs) and their
> percentage of occcurance.
>
> It also lists the MSGIDs for ERROR and WARNING logs and their percentage of
> occurance.
>
> A sample o/p the script:
>
> [root at hal9000 ~]# ./log_analyzer.sh /var/log/glusterfs/mnt-glusterfs.log
> Number Percentage Function
> 7 0.49 __socket_rwv
> 4 0.28 mgmt_getspec_cbk
> 4 0.28 gf_timer_call_after
> 3 0.21 rpc_clnt_reconfig
> 2 0.14 fuse_thread_proc
> 2 0.14 fini
> 2 0.14 cleanup_and_exit
> 1 0.07 _ios_dump_thread
> 1 0.07 fuse_init
> 1 0.07 fuse_graph_setup
>
> ========= Error Functions ========
>
> 7 0.49 __socket_rwv
> 2 0.14 cleanup_and_exit
>
> Number Percentage MSGID
> 958 67.99 109066
> 424 30.09 109036
> 3 0.21 114057
> 3 0.21 114047
> 3 0.21 114046
> 3 0.21 114035
> 3 0.21 114020
> 3 0.21 114018
> 3 0.21 108031
> 2 0.14 101190
> 1 0.07 7962
> 1 0.07 108006
> 1 0.07 108005
> 1 0.07 108001
> 1 0.07 100030
>
> ========= Error MSGIDs ========
>
> 1 0.07 108006
> 1 0.07 108001
>
> It can be found here.
>
> https://github.com/raghavendrabhat/threaded-io/blob/master/log_analyzer.sh.
>
> Do you think it can be added to the repo?
>
> Regards,
> Raghavendra
>
> On Wed, Jan 27, 2016 at 3:44 AM, Aravinda <avishwan at redhat.com> wrote:
>
>> Hi,
>>
>> I am happy to share the `glustertool` project, which is a
>> infrastructure for adding more tools for Gluster.
>>
>> https://github.com/aravindavk/glustertool
>>
>> Following tools available with the initial release.(`glustertool
>> <TOOLNAME> [ARGS..]`)
>>
>> 1. gfid - To get GFID of given path(Mount or Backend)
>> 2. changelogparser - To parse the Gluster Changelog
>> 3. xtime - To get Xtime from brick backend
>> 4. stime - To get Stime from brick backend
>> 5. volmark - To get Volmark details from Gluster mount
>>
>> rpm/deb packages are not yet available, install this using `sudo
>> python setup.py install`
>>
>> Once installed, run `glustertool list` to see list of tools available.
>> `glustertool doc TOOLNAME` shows documentation about the tool and
>> `glustertool <TOOLNAME> --help` shows the usage of the tool.
>>
>> More tools can be added to this collection easily using `newtool`
>> utility available in this repo.
>>
>>      # ./newtool <TOOLNAME>
>>
>> Read more about adding tools here
>> https://github.com/aravindavk/glustertool/blob/master/CONTRIBUTING.md
>>
>> You can create an issue in github requesting more tools for Gluster
>> https://github.com/aravindavk/glustertool/issues
>>
>> Comments & Suggestions Welcome
>>
>> regards
>> Aravinda
>>
>> On 10/23/2015 11:42 PM, Vijay Bellur wrote:
>>
>>> On Friday 23 October 2015 04:16 PM, Aravinda wrote:
>>>
>>>> Hi Gluster developers,
>>>>
>>>> In this mail I am proposing troubleshooting documentation and
>>>> Gluster Tools infrastructure.
>>>>
>>>> Tool to search in documentation
>>>> ===============================
>>>> We recently added message Ids to each error messages in Gluster. Some
>>>> of the error messages are self explanatory. But some error messages
>>>> requires manual intervention to fix the issue. How about identifying
>>>> the error messages which requires more explanation and creating
>>>> documentation for the same. Even though the information about some
>>>> errors available in documentation, it is very difficult to search and
>>>> relate to the error message. It will be very useful if we create a
>>>> tool which looks for documentation and tells us exactly what to do.
>>>>
>>>> For example,(Illustrativepurpose only)
>>>> glusterdoc --explain GEOREP0003
>>>>
>>>>       SSH configuration issue. This error is seen when Pem keys from all
>>>>       master nodes are not distributed properly to Slave
>>>>       nodes. Use Geo-replication create command with force option to
>>>>       redistribute the keys. If issue stillpersists, look for any errors
>>>>       while running hook scripts inGlusterd log file.
>>>>
>>>>
>>>> Note: Inspired from rustc --explain command
>>>> https://twitter.com/jaredforsyth/status/626960244707606528
>>>>
>>>> If we don't know the message id, we can still search from the
>>>> available documentation like,
>>>>
>>>>       glusterdoc --search <SEARCH_KEY_WORD>
>>>>
>>>> These commands can be programmatically consumed, for example
>>>> `--json` will return the output in JSON format. This enables UI
>>>> developers to automatically show help messages when they display
>>>> errors.
>>>>
>>>> Gluster Tools infrastructure
>>>> ============================
>>>> Are our Gluster log files sufficient for root causing the issues? Is
>>>> that error caused due to miss configuration? Geo-replication status is
>>>> showing faulty. Where to find the reason for Faulty?
>>>>
>>>> Sac(surs AT redhat.com) mentioned that heis working on gdeploy and many
>>>> developers
>>>> are using their owntools. How about providing common infrastructure(say
>>>> gtool/glustertool) to host all these tools.
>>>>
>>>>
>>> Would this be a repository with individual tools being git submodules or
>>> something similar? Is there also a plan to bundle the set of tools into a
>>> binary package?
>>>
>>> Looks like a good idea to me.
>>>
>>> -Vijay
>>>
>>>
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>