[Gluster-devel] Troubleshooting and Diagnostic tools for Gluster

Thu Jan 28 06:53:43 UTC 2016

Also, don't forget to send a pull request to Aravinda, so that others can
use it too.
Maybe add some basic usage note. If it is not already there.

On Thu, Jan 28, 2016 at 7:04 AM, Aravinda <avishwan at redhat.com> wrote:

> Nice tool.
>
> Easy to add, clone the glustertool repo, and run
>
> # ./newtool mnt_log_analyze --type=exec --bin=log_analyzer.sh --prog=bash
>
> I used tool name "mnt_log_analyze" just for this example, feel free to
> name your tool.
>
> This will create following dir and files
> glustertool/plugins/mnt_log_analyze
> glustertool/plugins/mnt_log_analyze/tool.conf
> glustertool/plugins/mnt_log_analyze/doc.txt
>
> Update the tool doc in doc.txt and copy your script to
> glustertool/plugins/mnt_log_analyze dir.
>
> Thats all! use your tool using,
>
> # glustertool mnt_log_analyze /var/log/glusterfs/mnt-glusterfs.log
>
> Let me know if something is not working.
>
> regards
> Aravinda
>
>
> On 01/28/2016 04:56 AM, Raghavendra Bhat wrote:
>
>> I have a script written to analyze the log message of gluster process.
>>
>> It actually scans the log file and identifies the log messages with ERROR
>> and WARNING levels.
>> It lists the functions (with either ERROR or WARNING logs) and their
>> percentage of occcurance.
>>
>> It also lists the MSGIDs for ERROR and WARNING logs and their percentage
>> of
>> occurance.
>>
>> A sample o/p the script:
>>
>> [root at hal9000 ~]# ./log_analyzer.sh /var/log/glusterfs/mnt-glusterfs.log
>> Number Percentage Function
>> 7 0.49 __socket_rwv
>> 4 0.28 mgmt_getspec_cbk
>> 4 0.28 gf_timer_call_after
>> 3 0.21 rpc_clnt_reconfig
>> 2 0.14 fuse_thread_proc
>> 2 0.14 fini
>> 2 0.14 cleanup_and_exit
>> 1 0.07 _ios_dump_thread
>> 1 0.07 fuse_init
>> 1 0.07 fuse_graph_setup
>>
>> ========= Error Functions ========
>>
>> 7 0.49 __socket_rwv
>> 2 0.14 cleanup_and_exit
>>
>> Number Percentage MSGID
>> 958 67.99 109066
>> 424 30.09 109036
>> 3 0.21 114057
>> 3 0.21 114047
>> 3 0.21 114046
>> 3 0.21 114035
>> 3 0.21 114020
>> 3 0.21 114018
>> 3 0.21 108031
>> 2 0.14 101190
>> 1 0.07 7962
>> 1 0.07 108006
>> 1 0.07 108005
>> 1 0.07 108001
>> 1 0.07 100030
>>
>> ========= Error MSGIDs ========
>>
>> 1 0.07 108006
>> 1 0.07 108001
>>
>> It can be found here.
>>
>> https://github.com/raghavendrabhat/threaded-io/blob/master/log_analyzer.sh
>> .
>>
>> Do you think it can be added to the repo?
>>
>> Regards,
>> Raghavendra
>>
>> On Wed, Jan 27, 2016 at 3:44 AM, Aravinda <avishwan at redhat.com> wrote:
>>
>> Hi,
>>>
>>> I am happy to share the `glustertool` project, which is a
>>> infrastructure for adding more tools for Gluster.
>>>
>>> https://github.com/aravindavk/glustertool
>>>
>>> Following tools available with the initial release.(`glustertool
>>> <TOOLNAME> [ARGS..]`)
>>>
>>> 1. gfid - To get GFID of given path(Mount or Backend)
>>> 2. changelogparser - To parse the Gluster Changelog
>>> 3. xtime - To get Xtime from brick backend
>>> 4. stime - To get Stime from brick backend
>>> 5. volmark - To get Volmark details from Gluster mount
>>>
>>> rpm/deb packages are not yet available, install this using `sudo
>>> python setup.py install`
>>>
>>> Once installed, run `glustertool list` to see list of tools available.
>>> `glustertool doc TOOLNAME` shows documentation about the tool and
>>> `glustertool <TOOLNAME> --help` shows the usage of the tool.
>>>
>>> More tools can be added to this collection easily using `newtool`
>>> utility available in this repo.
>>>
>>>      # ./newtool <TOOLNAME>
>>>
>>> Read more about adding tools here
>>> https://github.com/aravindavk/glustertool/blob/master/CONTRIBUTING.md
>>>
>>> You can create an issue in github requesting more tools for Gluster
>>> https://github.com/aravindavk/glustertool/issues
>>>
>>> Comments & Suggestions Welcome
>>>
>>> regards
>>> Aravinda
>>>
>>> On 10/23/2015 11:42 PM, Vijay Bellur wrote:
>>>
>>> On Friday 23 October 2015 04:16 PM, Aravinda wrote:
>>>>
>>>> Hi Gluster developers,
>>>>>
>>>>> In this mail I am proposing troubleshooting documentation and
>>>>> Gluster Tools infrastructure.
>>>>>
>>>>> Tool to search in documentation
>>>>> ===============================
>>>>> We recently added message Ids to each error messages in Gluster. Some
>>>>> of the error messages are self explanatory. But some error messages
>>>>> requires manual intervention to fix the issue. How about identifying
>>>>> the error messages which requires more explanation and creating
>>>>> documentation for the same. Even though the information about some
>>>>> errors available in documentation, it is very difficult to search and
>>>>> relate to the error message. It will be very useful if we create a
>>>>> tool which looks for documentation and tells us exactly what to do.
>>>>>
>>>>> For example,(Illustrativepurpose only)
>>>>> glusterdoc --explain GEOREP0003
>>>>>
>>>>>       SSH configuration issue. This error is seen when Pem keys from
>>>>> all
>>>>>       master nodes are not distributed properly to Slave
>>>>>       nodes. Use Geo-replication create command with force option to
>>>>>       redistribute the keys. If issue stillpersists, look for any
>>>>> errors
>>>>>       while running hook scripts inGlusterd log file.
>>>>>
>>>>>
>>>>> Note: Inspired from rustc --explain command
>>>>> https://twitter.com/jaredforsyth/status/626960244707606528
>>>>>
>>>>> If we don't know the message id, we can still search from the
>>>>> available documentation like,
>>>>>
>>>>>       glusterdoc --search <SEARCH_KEY_WORD>
>>>>>
>>>>> These commands can be programmatically consumed, for example
>>>>> `--json` will return the output in JSON format. This enables UI
>>>>> developers to automatically show help messages when they display
>>>>> errors.
>>>>>
>>>>> Gluster Tools infrastructure
>>>>> ============================
>>>>> Are our Gluster log files sufficient for root causing the issues? Is
>>>>> that error caused due to miss configuration? Geo-replication status is
>>>>> showing faulty. Where to find the reason for Faulty?
>>>>>
>>>>> Sac(surs AT redhat.com) mentioned that heis working on gdeploy and
>>>>> many
>>>>> developers
>>>>> are using their owntools. How about providing common infrastructure(say
>>>>> gtool/glustertool) to host all these tools.
>>>>>
>>>>>
>>>>> Would this be a repository with individual tools being git submodules
>>>> or
>>>> something similar? Is there also a plan to bundle the set of tools into
>>>> a
>>>> binary package?
>>>>
>>>> Looks like a good idea to me.
>>>>
>>>> -Vijay
>>>>
>>>>
>>>> _______________________________________________
>>> Gluster-devel mailing list
>>> Gluster-devel at gluster.org
>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>>
>>>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20160128/fe4dc4d0/attachment.html>