[Gluster-devel] gluster IRC and use systemtap can't output glusterfs backtrace
justglusterfs at gmail.com
justglusterfs at gmail.com
Wed May 14 06:15:32 UTC 2014
to Krishnan Parthasarathi:
I write a systemtap script（probe_glusterfs.stap）： the script content as follows：
probe process("/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so").statement("*")
{
print_ubacktrace();
}
probe process("/usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so").statement("*")
{
print_ubacktrace();
}
probe process("/usr/local/lib/glusterfs/3.5/xlator/protocol/client.so").statement("*")
{
print_ubacktrace();
}
when I do "ls /mnt/glusterfs-volume" test， I execute command: stap probe_glusterfs.stap
systemtap output as follows info:
but the call stack information is not continuous,
Actually, I want to watch glusterfs all function call stacks when I execute "ls /mnt/glusterfs-volume" or "mkdir /mnt/glusterfs-volume/testdir "
(pay attention to: /mnt/glusterfs-volume is my glusterfs volume mount dir)
systemtap output as follows info:
0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
0x7fb46a6980a0 : fuse_resolve_all+0x0/0xb0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a697dde : fuse_resolve+0x24e/0x510 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
0x7fb46a697b90 : fuse_resolve+0x0/0x510 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a69814e : fuse_resolve_all+0xae/0xb0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a697dde : fuse_resolve+0x24e/0x510 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
0x7fb46a6980a0 : fuse_resolve_all+0x0/0xb0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a697e36 : fuse_resolve+0x2a6/0x510 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a69814e : fuse_resolve_all+0xae/0xb0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a697dde : fuse_resolve+0x24e/0x510 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
0x7fb46a698100 : fuse_resolve_all+0x60/0xb0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a697e36 : fuse_resolve+0x2a6/0x510 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a69814e : fuse_resolve_all+0xae/0xb0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a697dde : fuse_resolve+0x24e/0x510 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
0x7fb46a69f590 : fuse_readdir_resume+0x0/0x3f0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a698106 : fuse_resolve_all+0x66/0xb0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a697e36 : fuse_resolve+0x2a6/0x510 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a69814e : fuse_resolve_all+0xae/0xb0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a697dde : fuse_resolve+0x24e/0x510 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
0x7fb46a696660 : get_call_frame_for_req+0x0/0x270 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a69f606 : fuse_readdir_resume+0x76/0x3f0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a698106 : fuse_resolve_all+0x66/0xb0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a697e36 : fuse_resolve+0x2a6/0x510 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a69814e : fuse_resolve_all+0xae/0xb0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a697dde : fuse_resolve+0x24e/0x510 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
0x7fb46a69667f : get_call_frame_for_req+0x1f/0x270 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a69f606 : fuse_readdir_resume+0x76/0x3f0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a698106 : fuse_resolve_all+0x66/0xb0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a697e36 : fuse_resolve+0x2a6/0x510 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a69814e : fuse_resolve_all+0xae/0xb0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a697dde : fuse_resolve+0x24e/0x510 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
0x7fb46a6966d3 : get_call_frame_for_req+0x73/0x270 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a69f606 : fuse_readdir_resume+0x76/0x3f0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a698106 : fuse_resolve_all+0x66/0xb0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a697e36 : fuse_resolve+0x2a6/0x510 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a69814e : fuse_resolve_all+0xae/0xb0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a697dde : fuse_resolve+0x24e/0x510 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
0x7fb46a696724 : get_call_frame_for_req+0xc4/0x270 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a69f606 : fuse_readdir_resume+0x76/0x3f0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a698106 : fuse_resolve_all+0x66/0xb0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a697e36 : fuse_resolve+0x2a6/0x510 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a69814e : fuse_resolve_all+0xae/0xb0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a697dde : fuse_resolve+0x24e/0x510 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
0x7fb46a696769 : get_call_frame_for_req+0x109/0x270 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a69f606 : fuse_readdir_resume+0x76/0x3f0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a698106 : fuse_resolve_all+0x66/0xb0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a697e36 : fuse_resolve+0x2a6/0x510 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a69814e : fuse_resolve_all+0xae/0xb0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a697dde : fuse_resolve+0x24e/0x510 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
0x7fb46a6964a0 : frame_fill_groups+0x0/0x1c0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a69681a : get_call_frame_for_req+0x1ba/0x270 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a69f606 : fuse_readdir_resume+0x76/0x3f0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a698106 : fuse_resolve_all+0x66/0xb0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a697e36 : fuse_resolve+0x2a6/0x510 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a69814e : fuse_resolve_all+0xae/0xb0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6981d1 : fuse_resolve_continue+0x41/0x120 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a697dde : fuse_resolve+0x24e/0x510 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a69812e : fuse_resolve_all+0x8e/0xb0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a698178 : fuse_resolve_and_resume+0x28/0x40 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
0x7fb4675d2c60 : afr_readdirp+0x0/0x20 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4673a14b9 [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]
0x7fb4675d25f0 : afr_do_readdir+0x0/0x670 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d2c73 : afr_readdirp+0x13/0x20 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4673a14b9 [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]
0x7fb46760bea0 : afr_local_init+0x0/0x100 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d2699 : afr_do_readdir+0xa9/0x670 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d2c73 : afr_readdirp+0x13/0x20 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4673a14b9 [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]
0x7fb467609990 : afr_up_children_count+0x0/0x10 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46760bf06 : afr_local_init+0x66/0x100 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d2699 : afr_do_readdir+0xa9/0x670 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d2c73 : afr_readdirp+0x13/0x20 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4673a14b9 [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]
0x7fb467609960 : afr_set_elem_count_get+0x0/0x30 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46760bf06 : afr_local_init+0x66/0x100 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d2699 : afr_do_readdir+0xa9/0x670 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d2c73 : afr_readdirp+0x13/0x20 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4673a14b9 [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]
0x7fb46760b170 : afr_children_create+0x0/0x80 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d26aa : afr_do_readdir+0xba/0x670 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d2c73 : afr_readdirp+0x13/0x20 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4673a14b9 [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]
0x7fb467612f00 : afr_inode_get_read_ctx+0x0/0x40 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d26ca : afr_do_readdir+0xda/0x670 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d2c73 : afr_readdirp+0x13/0x20 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4673a14b9 [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]
0x7fb467612d70 : afr_inode_get_ctx_params+0x0/0x190 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb467612f29 : afr_inode_get_read_ctx+0x29/0x40 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d26ca : afr_do_readdir+0xda/0x670 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d2c73 : afr_readdirp+0x13/0x20 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4673a14b9 [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]
0x7fb467612820 : __afr_inode_ctx_get+0x0/0x110 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb467612dc7 : afr_inode_get_ctx_params+0x57/0x190 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb467612f29 : afr_inode_get_read_ctx+0x29/0x40 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d26ca : afr_do_readdir+0xda/0x670 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d2c73 : afr_readdirp+0x13/0x20 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4673a14b9 [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]
0x7fb46760b550 : afr_get_call_child+0x0/0x260 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d26f1 : afr_do_readdir+0x101/0x670 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d2c73 : afr_readdirp+0x13/0x20 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4673a14b9 [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]
0x7fb4675ec2b0 : afr_fd_ctx_get+0x0/0x60 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d2704 : afr_do_readdir+0x114/0x670 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d2c73 : afr_readdirp+0x13/0x20 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4673a14b9 [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]
0x7fb4675ec1e0 : __afr_fd_ctx_get+0x0/0xd0 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675ec2e0 : afr_fd_ctx_get+0x30/0x60 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d2704 : afr_do_readdir+0x114/0x670 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d2c73 : afr_readdirp+0x13/0x20 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4673a14b9 [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]
0x7fb4678389f0 : client_readdirp+0x0/0x1d0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb4675d2889 : afr_do_readdir+0x299/0x670 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d2c73 : afr_readdirp+0x13/0x20 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4673a14b9 [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]
0x7fb467843420 : client3_3_readdirp+0x0/0x4c0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb467838a79 : client_readdirp+0x89/0x1d0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb4675d2889 : afr_do_readdir+0x299/0x670 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d2c73 : afr_readdirp+0x13/0x20 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4673a14b9 [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]
0x7fb46783e570 : this_fd_get_ctx+0x0/0xd0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb4678435d0 : client3_3_readdirp+0x1b0/0x4c0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb467838a79 : client_readdirp+0x89/0x1d0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb4675d2889 : afr_do_readdir+0x299/0x670 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d2c73 : afr_readdirp+0x13/0x20 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4673a14b9 [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]
0x7fb46783cee0 : client_submit_request+0x0/0x4c0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb467843743 : client3_3_readdirp+0x323/0x4c0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb467838a79 : client_readdirp+0x89/0x1d0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb4675d2889 : afr_do_readdir+0x299/0x670 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d2c73 : afr_readdirp+0x13/0x20 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4673a14b9 [...usr/local/lib/glusterfs/3.5/xlator/cluster/dht.so.0.0.0+0x204b9/0x242000]
0x7fb46a6acf40 : fuse_graph_sync+0x0/0x170 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6ad2e0 : fuse_thread_proc+0x230/0x660 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
0x7fb46784b2b0 : client3_3_readdirp_cbk+0x0/0x3b0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
0x7fb4675d2ca0 : afr_readdirp_cbk+0x0/0x140 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
0x7fb46a6ac8c0 : fuse_readdir_cbk+0x0/0x430 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46673f12e [.../local/lib/glusterfs/3.5/xlator/debug/io-stats.so.0.0.0+0xd12e/0x217000]
0x7fb46a698f70 : send_fuse_data+0x0/0x40 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6acabd : fuse_readdir_cbk+0x1fd/0x430 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46673f12e [.../local/lib/glusterfs/3.5/xlator/debug/io-stats.so.0.0.0+0xd12e/0x217000]
0x7fb46a698da0 : send_fuse_iov+0x0/0x1d0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a698fa6 : send_fuse_data+0x36/0x40 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6acabd : fuse_readdir_cbk+0x1fd/0x430 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46673f12e [.../local/lib/glusterfs/3.5/xlator/debug/io-stats.so.0.0.0+0xd12e/0x217000]
0x7fb46a696a30 : free_fuse_state+0x0/0x150 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6acac5 : fuse_readdir_cbk+0x205/0x430 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46673f12e [.../local/lib/glusterfs/3.5/xlator/debug/io-stats.so.0.0.0+0xd12e/0x217000]
0x7fb46a6969c0 : fuse_resolve_wipe+0x0/0x70 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a696afe : free_fuse_state+0xce/0x150 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6acac5 : fuse_readdir_cbk+0x205/0x430 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46673f12e [.../local/lib/glusterfs/3.5/xlator/debug/io-stats.so.0.0.0+0xd12e/0x217000]
0x7fb46a6969c0 : fuse_resolve_wipe+0x0/0x70 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a696b0a : free_fuse_state+0xda/0x150 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6acac5 : fuse_readdir_cbk+0x205/0x430 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46673f12e [.../local/lib/glusterfs/3.5/xlator/debug/io-stats.so.0.0.0+0xd12e/0x217000]
0x7fb46a6acac9 : fuse_readdir_cbk+0x209/0x430 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46673f12e [.../local/lib/glusterfs/3.5/xlator/debug/io-stats.so.0.0.0+0xd12e/0x217000]
0x7fb46a6acad6 : fuse_readdir_cbk+0x216/0x430 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46673f12e [.../local/lib/glusterfs/3.5/xlator/debug/io-stats.so.0.0.0+0xd12e/0x217000]
0x7fb46760dfc0 : afr_local_cleanup+0x0/0x270 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
0x7fb46a699bc0 : fuse_releasedir+0x0/0x1e0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
0x7fb46760ddd0 : afr_local_sh_cleanup+0x0/0x140 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46760dfda : afr_local_cleanup+0x1a/0x270 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
0x7fb46a6968d0 : get_fuse_state+0x0/0xf0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a699bf4 : fuse_releasedir+0x34/0x1e0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
0x7fb46760d600 : afr_xattr_array_destroy+0x0/0x20 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46760de18 : afr_local_sh_cleanup+0x48/0x140 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46760dfda : afr_local_cleanup+0x1a/0x270 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
0x7fb46a696938 : get_fuse_state+0x68/0xf0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a699bf4 : fuse_releasedir+0x34/0x1e0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
0x7fb46760d5c0 : afr_reset_xattr+0x0/0x40 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46760d609 : afr_xattr_array_destroy+0x9/0x20 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46760de18 : afr_local_sh_cleanup+0x48/0x140 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46760dfda : afr_local_cleanup+0x1a/0x270 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
0x7fb46760adb0 : afr_matrix_cleanup+0x0/0x60 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46760de33 : afr_local_sh_cleanup+0x63/0x140 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46760dfda : afr_local_cleanup+0x1a/0x270 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
0x7fb46760adb0 : afr_matrix_cleanup+0x0/0x60 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46760de42 : afr_local_sh_cleanup+0x72/0x140 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46760dfda : afr_local_cleanup+0x1a/0x270 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
0x7fb46760df10 : afr_local_transaction_cleanup+0x0/0xb0 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46760dfe5 : afr_local_cleanup+0x25/0x270 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
0x7fb46760adb0 : afr_matrix_cleanup+0x0/0x60 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46760df2f : afr_local_transaction_cleanup+0x1f/0xb0 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46760dfe5 : afr_local_cleanup+0x25/0x270 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
0x7fb46760adb0 : afr_matrix_cleanup+0x0/0x60 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46760df3e : afr_local_transaction_cleanup+0x2e/0xb0 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46760dfe5 : afr_local_cleanup+0x25/0x270 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
0x7fb467605120 : afr_entry_lockee_cleanup+0x0/0x70 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46760df6e : afr_local_transaction_cleanup+0x5e/0xb0 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46760dfe5 : afr_local_cleanup+0x25/0x270 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
0x7fb46760adb0 : afr_matrix_cleanup+0x0/0x60 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46760e0ff : afr_local_cleanup+0x13f/0x270 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d2d81 : afr_readdirp_cbk+0xe1/0x140 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46784b44b : client3_3_readdirp_cbk+0x19b/0x3b0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
0x7fb46a699330 : send_fuse_err+0x0/0x140 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a699d0a : fuse_releasedir+0x14a/0x1e0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
0x7fb46a696120 : fuse_ino_to_inode+0x0/0x40 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a69938a : send_fuse_err+0x5a/0x140 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a699d0a : fuse_releasedir+0x14a/0x1e0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
0x7fb46783e510 : client_local_wipe+0x0/0x60 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46784b45f : client3_3_readdirp_cbk+0x1af/0x3b0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
0x7fb46a696138 : fuse_ino_to_inode+0x18/0x40 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a69938a : send_fuse_err+0x5a/0x140 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a699d0a : fuse_releasedir+0x14a/0x1e0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
0x7fb4678377d0 : client_releasedir+0x0/0xe0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
0x7fb467847ab0 : client3_3_releasedir+0x0/0xd0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb467837823 : client_releasedir+0x53/0xe0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
0x7fb46a698da0 : send_fuse_iov+0x0/0x1d0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6993fb : send_fuse_err+0xcb/0x140 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a699d0a : fuse_releasedir+0x14a/0x1e0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
0x7fb46a696a30 : free_fuse_state+0x0/0x150 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a699d12 : fuse_releasedir+0x152/0x1e0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
0x7fb46783e850 : this_fd_del_ctx+0x0/0xd0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb467847b0d : client3_3_releasedir+0x5d/0xd0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb467837823 : client_releasedir+0x53/0xe0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
0x7fb46a6969c0 : fuse_resolve_wipe+0x0/0x70 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a696afe : free_fuse_state+0xce/0x150 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a699d12 : fuse_releasedir+0x152/0x1e0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
0x7fb467847b1c : client3_3_releasedir+0x6c/0xd0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb467837823 : client_releasedir+0x53/0xe0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
0x7fb46a6969c0 : fuse_resolve_wipe+0x0/0x70 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a696b0a : free_fuse_state+0xda/0x150 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a699d12 : fuse_releasedir+0x152/0x1e0 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6ad348 : fuse_thread_proc+0x298/0x660 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
0x7fb467847530 : client_fdctx_destroy+0x0/0x4b0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb467847b51 : client3_3_releasedir+0xa1/0xd0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb467837823 : client_releasedir+0x53/0xe0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
0x7fb46a6acf40 : fuse_graph_sync+0x0/0x170 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x7fb46a6ad2e0 : fuse_thread_proc+0x230/0x660 [/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so.0.0.0]
0x31de6079d1 [/lib64/libpthread-2.12.so+0x79d1/0x219000]
0x7fb46784775f : client_fdctx_destroy+0x22f/0x4b0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb467847b51 : client3_3_releasedir+0xa1/0xd0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb467837823 : client_releasedir+0x53/0xe0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
0x7fb46784779b : client_fdctx_destroy+0x26b/0x4b0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb467847b51 : client3_3_releasedir+0xa1/0xd0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb467837823 : client_releasedir+0x53/0xe0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
0x7fb46783cee0 : client_submit_request+0x0/0x4c0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46784788f : client_fdctx_destroy+0x35f/0x4b0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb467847b51 : client3_3_releasedir+0xa1/0xd0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb467837823 : client_releasedir+0x53/0xe0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
0x7fb4675d24e0 : afr_releasedir+0x0/0x110 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
0x7fb4675d24e2 : afr_releasedir+0x2/0x110 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
0x7fb46760bbb0 : afr_cleanup_fd_ctx+0x0/0xe0 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb4675d259d : afr_releasedir+0xbd/0x110 [...usr/local/lib/glusterfs/3.5/xlator/cluster/afr.so.0.0.0]
0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
0x7fb4678377d0 : client_releasedir+0x0/0xe0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
0x7fb467847ab0 : client3_3_releasedir+0x0/0xd0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb467837823 : client_releasedir+0x53/0xe0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
0x7fb46783e850 : this_fd_del_ctx+0x0/0xd0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb467847b0d : client3_3_releasedir+0x5d/0xd0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb467837823 : client_releasedir+0x53/0xe0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
0x7fb467847b1c : client3_3_releasedir+0x6c/0xd0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb467837823 : client_releasedir+0x53/0xe0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
0x7fb467847530 : client_fdctx_destroy+0x0/0x4b0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb467847b51 : client3_3_releasedir+0xa1/0xd0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb467837823 : client_releasedir+0x53/0xe0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
0x7fb46784775f : client_fdctx_destroy+0x22f/0x4b0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb467847b51 : client3_3_releasedir+0xa1/0xd0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb467837823 : client_releasedir+0x53/0xe0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
0x7fb46784779b : client_fdctx_destroy+0x26b/0x4b0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb467847b51 : client3_3_releasedir+0xa1/0xd0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb467837823 : client_releasedir+0x53/0xe0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
0x7fb46783cee0 : client_submit_request+0x0/0x4c0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46784788f : client_fdctx_destroy+0x35f/0x4b0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb467847b51 : client3_3_releasedir+0xa1/0xd0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb467837823 : client_releasedir+0x53/0xe0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46c0490e1 [/usr/local/lib/libglusterfs.so.0.0.0+0x380e1/0x274000]
0x7fb46783e050 : clnt_readdirp_rsp_cleanup+0x0/0x50 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46784b498 : client3_3_readdirp_cbk+0x1e8/0x3b0 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
0x7fb46784ce70 : client3_3_releasedir_cbk+0x0/0x120 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
0x7fb46784ce7d : client3_3_releasedir_cbk+0xd/0x120 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
0x7fb46784ce8b : client3_3_releasedir_cbk+0x1b/0x120 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
0x7fb46784ce70 : client3_3_releasedir_cbk+0x0/0x120 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
0x7fb46784ce7d : client3_3_releasedir_cbk+0xd/0x120 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
0x7fb46784ce8b : client3_3_releasedir_cbk+0x1b/0x120 [...local/lib/glusterfs/3.5/xlator/protocol/client.so.0.0.0]
0x7fb46be08ba5 [/usr/local/lib/libgfrpc.so.0.0.0+0xdba5/0x215000]
justglusterfs at gmail.com
From: Krishnan Parthasarathi
Date: 2014-05-14 11:26
To: justglusterfs
CC: gluster-devel
Subject: Re: [Gluster-devel] gluster IRC and use systemtap can't output glusterfs backtrace
This systemtap script limits itself to the fuse module and glusterfs binary.
But it is possible to track all the modules that can be possibly (dynamically)
loaded into a glusterfs process using system scripts.
I am not sure what kind of analysis you are looking at, across all function
call stacks.
thanks,
Krish
----- Original Message -----
> to Krishnan Parthasarathi:
> thank you very much for your help,
> As you said, I use stap -d /usr/local/sbin/glusterfs --ldd -e 'probe
> process("/usr/local/lib/glusterfs/3.5/xlator/mount/fuse.so").function("*"){print_ubacktrace()}'
> ,
> systemtap can output the fuse module function of the stack ,
> But now I want to track glusterfs all function'call in order to analyze
> glusterfs system, need output all function call stack ,
> Do you have any better suggestion? or Recommend other tools（similar to
> systemtap）?
> thank you very much!
>
>
>
>
> justglusterfs at gmail.com
>
> From: Krishnan Parthasarathi
> Date: 2014-05-13 13:38
> To: 陈陈
> CC: gluster-devel
> Subject: Re: [Gluster-devel] gluster IRC and use systemtap can't output
> glusterfs backtrace
> To probe functions in specific translators you need to
> specify the path to the corresponding .so file in the process
> provider.
>
> For eg, to probe functions in fuse translator,
>
> stap -d /usr/local/sbin/glusterfs --ldd -e 'probe
> process("/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so").function("*"){print_ubacktrace()}'
> 确实可以输出fuse模块的函数栈
> The following is the output I see, when I write files using dd on
> a GlusterFS native mount,
>
> <snip>
>
> 0x31d7cf5e1d : clone+0x6d/0x90 [/usr/lib64/libc-2.17.so]
> 0x7f7977a1c834 : fuse_resolve+0x74/0x4f0
> [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]
> 0x7f7977a1cd3e : fuse_resolve_all+0x8e/0xb0
> [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]
> 0x7f7977a1cd80 : fuse_resolve_and_resume+0x20/0x30
> [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]
> 0x7f7977a3507e : fuse_thread_proc+0x2ae/0x690
> [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]
> 0x31d8407c53 : start_thread+0xc3/0x310 [/usr/lib64/libpthread-2.17.so]
> 0x31d7cf5e1d : clone+0x6d/0x90 [/usr/lib64/libc-2.17.so]
> 0x7f7977a1e4d0 : fuse_fd_ctx_get+0x0/0x40
> [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]
> 0x7f7977a1c84c : fuse_resolve+0x8c/0x4f0
> [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]
> 0x7f7977a1cd3e : fuse_resolve_all+0x8e/0xb0
> [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]
> 0x7f7977a1cd80 : fuse_resolve_and_resume+0x20/0x30
> [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]
> 0x7f7977a3507e : fuse_thread_proc+0x2ae/0x690
> [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]
> 0x31d8407c53 : start_thread+0xc3/0x310 [/usr/lib64/libpthread-2.17.so]
> 0x31d7cf5e1d : clone+0x6d/0x90 [/usr/lib64/libc-2.17.so]
> 0x7f7977a1c1e0 : fuse_resolve_continue+0x0/0x100
> [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]
> 0x7f7977a1c9f8 : fuse_resolve+0x238/0x4f0
> [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]
> 0x7f7977a1cd3e : fuse_resolve_all+0x8e/0xb0
> [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]
> 0x7f7977a1cd80 : fuse_resolve_and_resume+0x20/0x30
> [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]
> 0x7f7977a3507e : fuse_thread_proc+0x2ae/0x690
> [/usr/local/lib/glusterfs/3.5qa2/xlator/mount/fuse.so]
> 0x31d8407c53 : start_thread+0xc3/0x310 [/usr/lib64/libpthread-2.17.so]
> 0x31d7cf5e1d : clone+0x6d/0x90 [/usr/lib64/libc-2.17.so]
>
> </snip>
>
> Hope that helps,
> Krish
>
>
> ----- Original Message -----
> > See inline for response.
> >
> > ----- Original Message -----
> > > hi everyone!
> > > I meet three question:
> > >
> > > question one:
> > >
> > > I use IRC client(mirc) config server is: irc.freenode.net or
> > > irc.gnu.org
> > > but always can't connect, follwing errors:
> > > Connect retry #1 irc,freenode,net (6667) (dns pool)
> > > unable to connect server (connect timed out)
> > >
> > >
> > >
> > > question two:
> > > I system is centos6.4，glusterfs version is 3.4.
> > > I want to use systemtap watch glusterfs function backtrace,
> > > when I configure glusterfs source code, execute the following operation:
> > > ./configure --enable--systemtap
> > > but checking dtrace happen error，because in my system，no dtrace，my
> > > question is --enable--systemtap why need dtrace? it must have dtrace?
> >
> > I think the systemtap folks have retained certain symbols from dtrace
> > so that code instrumented with systemtap user specified probes are
> > compatible
> > with systems that have dtrace but not systemtap in them.
> >
> > >
> > > question three:
> > > when I do io test in glusterfs client. I execute following command:
> > > stap -d /usr/local/sbin/glusterfs --ldd -e 'probe
> > > process("glusterfs").function("*"){print_ubacktrace()}'
> > >
> > > the result is there is nothing output, and the stap command S state,
> > > no return.
> >
> > For 'process' systemtap provider, you don't need to compile glusterfs with
> > "--enable-systemtap". I am not sure why you are not seeing any of the
> > probes
> > firing in your stap script. Let me try this same and share my results on
> > this
> > thread.
> >
> > Hope that helps,
> > Krish
> >
> > >
> > > thank you answer me!
> > > _______________________________________________
> > > Gluster-devel mailing list
> > > Gluster-devel at gluster.org
> > > http://supercolony.gluster.org/mailman/listinfo/gluster-devel
> > >
> > _______________________________________________
> > Gluster-devel mailing list
> > Gluster-devel at gluster.org
> > http://supercolony.gluster.org/mailman/listinfo/gluster-devel
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20140514/7e318114/attachment-0002.html>
More information about the Gluster-devel
mailing list