[Gluster-devel] strace like utility for gluster

[Pranith Kumar Karampuri]

On 01/04/2016 11:04 AM, Prashanth Pai wrote:
> Hi,
>
> FYI, ltrace is very helpful for libgfapi clients.
>
> [root at hummingbird gfapi]# ltrace -c -ff -p 28906
>
> ^C% time     seconds  usecs/call     calls      function
> ------ ----------- ----------- --------- --------------------
>   43.27    5.952481       59524       100 glfs_rename
>   21.39    2.942866       29428       100 glfs_fsync
>    7.35    1.011715       10117       100 glfs_fsetxattr
>    7.26    0.999125        9991       100 glfs_creat
>    5.04    0.692812        1154       600 free
>    4.95    0.681092         851       800 __errno_location
>    4.67    0.641790        3208       200 glfs_stat
>    3.96    0.545340         908       600 malloc
>    1.26    0.173293        1732       100 glfs_close
>    0.84    0.115284        1152       100 glfs_write
> ------ ----------- ----------- --------- --------------------
> 100.00   13.755798                  2800 total
Nice, I didn't know about this. What I am looking for with this tool is 
even more granularity. i.e. per xlator information. It shouldn't be so 
difficult to find information like time spent in each xlator. What fop 
from fuse lead to what other fops in each of the xlator etc.

Pranith
>
> Regards,
>   -Prashanth Pai
>
> ----- Original Message -----
>> From: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
>> To: "Gluster Devel" <gluster-devel at gluster.org>
>> Sent: Monday, January 4, 2016 10:49:50 AM
>> Subject: [Gluster-devel] strace like utility for gluster
>>
>> hi,
>>        I have been thinking of coming up with a utility similar to strace
>> for gluster. I am sending this mail to find out if anyone else is also
>> thinking about it and how they are solving this problem to see if we can
>> come up with a concrete solution that we can implement.
>>
>> These are my thoughts about the solution so far:
>>
>> 1) Doing trace for a brick/mount to know just what that process is
>> winding/unwinding (Or whatever else it wants to tell the trace process)
>> seems easier. We can launch a trace process which will open listener
>> unix socket to which the glusterfs process can send whatever it needs to
>> (glfstrace -h <brick-host> -p <brick-port> -s
>> <unix-socket-path-where-the-trace-needs-to-be-sent>). We will have to
>> write gf_trace() infra, which will do the job of sending the information
>> to this trace process only when there is a trace process trying to
>> listen in on what is happening inside the process.
>>
>> 2) Doing end-to-end tracing:
>> Unix socket based listening is not going to work anymore (Unless we send
>> the trace information in xdata or something, which doesn't seem nice to
>> me). We can use network sockets for sending the information from the
>> bricks to the trace process. So at the time of starting a trace on the
>> mount process, mount process will need to send an rpc to the brick
>> process giving the trace process hostname/port information for that
>> client, and bricks can send the trace information to the trace process
>> directly. So we can have multiple trace processes tracing different
>> mounts and bricks will be able to send trace information to different
>> trace processes.
>>
>> We will need to make sure gf_trace() is not going to send the
>> information to trace process in the io-path.
>>
>> 3) Doing glfstrace -p <application-pid>. I have an approach for fuse
>> based mounts. I am hoping nfs/smb folks will respond if we can do
>> something similar in those mounts.
>>
>> glfstrace process now traces the client process with extra information
>> i.e. frame->root->pid information in fuse-bridge which can be used to
>> filter only these fops executed by the application. Rest is similar to 2)
>>
>> 4) Doing glfstrace -c <shell-command>
>>
>> glfstrace process forks, it knows the pid of child, child should wait to
>> hear from parent to start 'exec-of-cmd'. glfstrace process sets things
>> up similar to 3), once it sets up tracing, tells child to exec the cmd.
>>
>> Comments are welcome :-). Happy new year by the way!!
>>
>> Pranith
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>