[Gluster-devel] strace like utility for gluster

[Prashanth Pai]

Hi,

FYI, ltrace is very helpful for libgfapi clients.

[root at hummingbird gfapi]# ltrace -c -ff -p 28906

^C% time     seconds  usecs/call     calls      function
------ ----------- ----------- --------- --------------------
 43.27    5.952481       59524       100 glfs_rename
 21.39    2.942866       29428       100 glfs_fsync
  7.35    1.011715       10117       100 glfs_fsetxattr
  7.26    0.999125        9991       100 glfs_creat
  5.04    0.692812        1154       600 free
  4.95    0.681092         851       800 __errno_location
  4.67    0.641790        3208       200 glfs_stat
  3.96    0.545340         908       600 malloc
  1.26    0.173293        1732       100 glfs_close
  0.84    0.115284        1152       100 glfs_write
------ ----------- ----------- --------- --------------------
100.00   13.755798                  2800 total

Regards,
 -Prashanth Pai

----- Original Message -----
> From: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
> To: "Gluster Devel" <gluster-devel at gluster.org>
> Sent: Monday, January 4, 2016 10:49:50 AM
> Subject: [Gluster-devel] strace like utility for gluster
> 
> hi,
>       I have been thinking of coming up with a utility similar to strace
> for gluster. I am sending this mail to find out if anyone else is also
> thinking about it and how they are solving this problem to see if we can
> come up with a concrete solution that we can implement.
> 
> These are my thoughts about the solution so far:
> 
> 1) Doing trace for a brick/mount to know just what that process is
> winding/unwinding (Or whatever else it wants to tell the trace process)
> seems easier. We can launch a trace process which will open listener
> unix socket to which the glusterfs process can send whatever it needs to
> (glfstrace -h <brick-host> -p <brick-port> -s
> <unix-socket-path-where-the-trace-needs-to-be-sent>). We will have to
> write gf_trace() infra, which will do the job of sending the information
> to this trace process only when there is a trace process trying to
> listen in on what is happening inside the process.
> 
> 2) Doing end-to-end tracing:
> Unix socket based listening is not going to work anymore (Unless we send
> the trace information in xdata or something, which doesn't seem nice to
> me). We can use network sockets for sending the information from the
> bricks to the trace process. So at the time of starting a trace on the
> mount process, mount process will need to send an rpc to the brick
> process giving the trace process hostname/port information for that
> client, and bricks can send the trace information to the trace process
> directly. So we can have multiple trace processes tracing different
> mounts and bricks will be able to send trace information to different
> trace processes.
> 
> We will need to make sure gf_trace() is not going to send the
> information to trace process in the io-path.
> 
> 3) Doing glfstrace -p <application-pid>. I have an approach for fuse
> based mounts. I am hoping nfs/smb folks will respond if we can do
> something similar in those mounts.
> 
> glfstrace process now traces the client process with extra information
> i.e. frame->root->pid information in fuse-bridge which can be used to
> filter only these fops executed by the application. Rest is similar to 2)
> 
> 4) Doing glfstrace -c <shell-command>
> 
> glfstrace process forks, it knows the pid of child, child should wait to
> hear from parent to start 'exec-of-cmd'. glfstrace process sets things
> up similar to 3), once it sets up tracing, tells child to exec the cmd.
> 
> Comments are welcome :-). Happy new year by the way!!
> 
> Pranith
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
>