[Gluster-devel] Troubleshooting and Diagnostic tools for Gluster

Wed Jan 27 23:26:15 UTC 2016

I have a script written to analyze the log message of gluster process.

It actually scans the log file and identifies the log messages with ERROR
and WARNING levels.
It lists the functions (with either ERROR or WARNING logs) and their
percentage of occcurance.

It also lists the MSGIDs for ERROR and WARNING logs and their percentage of
occurance.

A sample o/p the script:

[root at hal9000 ~]# ./log_analyzer.sh /var/log/glusterfs/mnt-glusterfs.log
Number Percentage Function
7 0.49 __socket_rwv
4 0.28 mgmt_getspec_cbk
4 0.28 gf_timer_call_after
3 0.21 rpc_clnt_reconfig
2 0.14 fuse_thread_proc
2 0.14 fini
2 0.14 cleanup_and_exit
1 0.07 _ios_dump_thread
1 0.07 fuse_init
1 0.07 fuse_graph_setup

========= Error Functions ========

7 0.49 __socket_rwv
2 0.14 cleanup_and_exit

Number Percentage MSGID
958 67.99 109066
424 30.09 109036
3 0.21 114057
3 0.21 114047
3 0.21 114046
3 0.21 114035
3 0.21 114020
3 0.21 114018
3 0.21 108031
2 0.14 101190
1 0.07 7962
1 0.07 108006
1 0.07 108005
1 0.07 108001
1 0.07 100030

========= Error MSGIDs ========

1 0.07 108006
1 0.07 108001

It can be found here.

https://github.com/raghavendrabhat/threaded-io/blob/master/log_analyzer.sh.

Do you think it can be added to the repo?

Regards,
Raghavendra

On Wed, Jan 27, 2016 at 3:44 AM, Aravinda <avishwan at redhat.com> wrote:

> Hi,
>
> I am happy to share the `glustertool` project, which is a
> infrastructure for adding more tools for Gluster.
>
> https://github.com/aravindavk/glustertool
>
> Following tools available with the initial release.(`glustertool
> <TOOLNAME> [ARGS..]`)
>
> 1. gfid - To get GFID of given path(Mount or Backend)
> 2. changelogparser - To parse the Gluster Changelog
> 3. xtime - To get Xtime from brick backend
> 4. stime - To get Stime from brick backend
> 5. volmark - To get Volmark details from Gluster mount
>
> rpm/deb packages are not yet available, install this using `sudo
> python setup.py install`
>
> Once installed, run `glustertool list` to see list of tools available.
> `glustertool doc TOOLNAME` shows documentation about the tool and
> `glustertool <TOOLNAME> --help` shows the usage of the tool.
>
> More tools can be added to this collection easily using `newtool`
> utility available in this repo.
>
>     # ./newtool <TOOLNAME>
>
> Read more about adding tools here
> https://github.com/aravindavk/glustertool/blob/master/CONTRIBUTING.md
>
> You can create an issue in github requesting more tools for Gluster
> https://github.com/aravindavk/glustertool/issues
>
> Comments & Suggestions Welcome
>
> regards
> Aravinda
>
> On 10/23/2015 11:42 PM, Vijay Bellur wrote:
>
>> On Friday 23 October 2015 04:16 PM, Aravinda wrote:
>>
>>> Hi Gluster developers,
>>>
>>> In this mail I am proposing troubleshooting documentation and
>>> Gluster Tools infrastructure.
>>>
>>> Tool to search in documentation
>>> ===============================
>>> We recently added message Ids to each error messages in Gluster. Some
>>> of the error messages are self explanatory. But some error messages
>>> requires manual intervention to fix the issue. How about identifying
>>> the error messages which requires more explanation and creating
>>> documentation for the same. Even though the information about some
>>> errors available in documentation, it is very difficult to search and
>>> relate to the error message. It will be very useful if we create a
>>> tool which looks for documentation and tells us exactly what to do.
>>>
>>> For example,(Illustrativepurpose only)
>>> glusterdoc --explain GEOREP0003
>>>
>>>      SSH configuration issue. This error is seen when Pem keys from all
>>>      master nodes are not distributed properly to Slave
>>>      nodes. Use Geo-replication create command with force option to
>>>      redistribute the keys. If issue stillpersists, look for any errors
>>>      while running hook scripts inGlusterd log file.
>>>
>>>
>>> Note: Inspired from rustc --explain command
>>> https://twitter.com/jaredforsyth/status/626960244707606528
>>>
>>> If we don't know the message id, we can still search from the
>>> available documentation like,
>>>
>>>      glusterdoc --search <SEARCH_KEY_WORD>
>>>
>>> These commands can be programmatically consumed, for example
>>> `--json` will return the output in JSON format. This enables UI
>>> developers to automatically show help messages when they display
>>> errors.
>>>
>>> Gluster Tools infrastructure
>>> ============================
>>> Are our Gluster log files sufficient for root causing the issues? Is
>>> that error caused due to miss configuration? Geo-replication status is
>>> showing faulty. Where to find the reason for Faulty?
>>>
>>> Sac(surs AT redhat.com) mentioned that heis working on gdeploy and many
>>> developers
>>> are using their owntools. How about providing common infrastructure(say
>>> gtool/glustertool) to host all these tools.
>>>
>>>
>> Would this be a repository with individual tools being git submodules or
>> something similar? Is there also a plan to bundle the set of tools into a
>> binary package?
>>
>> Looks like a good idea to me.
>>
>> -Vijay
>>
>>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20160127/cd7d43f2/attachment.html>