[Gluster-devel] strace like utility for gluster

[Pranith Kumar Karampuri]

hi,
      I have been thinking of coming up with a utility similar to strace 
for gluster. I am sending this mail to find out if anyone else is also 
thinking about it and how they are solving this problem to see if we can 
come up with a concrete solution that we can implement.

These are my thoughts about the solution so far:

1) Doing trace for a brick/mount to know just what that process is 
winding/unwinding (Or whatever else it wants to tell the trace process) 
seems easier. We can launch a trace process which will open listener 
unix socket to which the glusterfs process can send whatever it needs to 
(glfstrace -h <brick-host> -p <brick-port> -s 
<unix-socket-path-where-the-trace-needs-to-be-sent>). We will have to 
write gf_trace() infra, which will do the job of sending the information 
to this trace process only when there is a trace process trying to 
listen in on what is happening inside the process.

2) Doing end-to-end tracing:
Unix socket based listening is not going to work anymore (Unless we send 
the trace information in xdata or something, which doesn't seem nice to 
me). We can use network sockets for sending the information from the 
bricks to the trace process. So at the time of starting a trace on the 
mount process, mount process will need to send an rpc to the brick 
process giving the trace process hostname/port information for that 
client, and bricks can send the trace information to the trace process 
directly. So we can have multiple trace processes tracing different 
mounts and bricks will be able to send trace information to different 
trace processes.

We will need to make sure gf_trace() is not going to send the 
information to trace process in the io-path.

3) Doing glfstrace -p <application-pid>. I have an approach for fuse 
based mounts. I am hoping nfs/smb folks will respond if we can do 
something similar in those mounts.

glfstrace process now traces the client process with extra information 
i.e. frame->root->pid information in fuse-bridge which can be used to 
filter only these fops executed by the application. Rest is similar to 2)

4) Doing glfstrace -c <shell-command>

glfstrace process forks, it knows the pid of child, child should wait to 
hear from parent to start 'exec-of-cmd'. glfstrace process sets things 
up similar to 3), once it sets up tracing, tells child to exec the cmd.

Comments are welcome :-). Happy new year by the way!!

Pranith