[Gluster-devel] 3.6.8 glusterfsd processes not responding
Joe Julian
joe at julianfamily.org
Sat Feb 13 03:16:33 UTC 2016
I've also got several glusterfsd processes that have stopped responding.
A backtrace from a live core, strace, and state dump follow:
Thread 10 (LWP 31587):
#0 0x00007f81d384289c in __lll_lock_wait () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f81d383e065 in _L_lock_858 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007f81d383deba in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007f81ce600ff8 in pl_inodelk_client_cleanup ()
from /usr/lib/x86_64-linux-gnu/glusterfs/3.6.8/xlator/features/locks.so
#4 0x00007f81ce5fe84a in ?? () from /usr/lib/x86_64-linux-gnu/glusterfs/3.6.8/xlator/features/locks.so
#5 0x00007f81d3ef573d in gf_client_disconnect () from /usr/lib/x86_64-linux-gnu/libglusterfs.so.0
#6 0x00007f81cd74e270 in server_connection_cleanup ()
from /usr/lib/x86_64-linux-gnu/glusterfs/3.6.8/xlator/protocol/server.so
#7 0x00007f81cd7486ec in server_rpc_notify ()
from /usr/lib/x86_64-linux-gnu/glusterfs/3.6.8/xlator/protocol/server.so
#8 0x00007f81d3c70f1b in rpcsvc_handle_disconnect () from /usr/lib/x86_64-linux-gnu/libgfrpc.so.0
#9 0x00007f81d3c710b0 in rpcsvc_notify () from /usr/lib/x86_64-linux-gnu/libgfrpc.so.0
#10 0x00007f81d3c74257 in rpc_transport_notify () from /usr/lib/x86_64-linux-gnu/libgfrpc.so.0
#11 0x00007f81cf4d4077 in ?? () from /usr/lib/x86_64-linux-gnu/glusterfs/3.6.8/rpc-transport/socket.so
#12 0x00007f81d3ef793b in ?? () from /usr/lib/x86_64-linux-gnu/libglusterfs.so.0
#13 0x00007f81d4348f71 in main ()
Thread 9 (LWP 3385):
#0 0x00007f81d353408d in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f81d3533f2c in sleep () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f81cee50c38 in ?? () from /usr/lib/x86_64-linux-gnu/glusterfs/3.6.8/xlator/storage/posix.so
#3 0x00007f81d383be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f81d35683fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()
Thread 8 (LWP 20656):
#0 0x00007f81d38400fe in pthread_cond_timedwait@@GLIBC_2.3.2 ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f81ce3ea032 in iot_worker ()
from /usr/lib/x86_64-linux-gnu/glusterfs/3.6.8/xlator/performance/io-threads.so
#2 0x00007f81d383be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007f81d35683fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x0000000000000000 in ?? ()
Thread 7 (LWP 31881):
#0 0x00007f81d383fd84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f81cee50f3b in posix_fsyncer_pick ()
from /usr/lib/x86_64-linux-gnu/glusterfs/3.6.8/xlator/storage/posix.so
#2 0x00007f81cee51155 in posix_fsyncer ()
from /usr/lib/x86_64-linux-gnu/glusterfs/3.6.8/xlator/storage/posix.so
#3 0x00007f81d383be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f81d35683fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()
Thread 6 (LWP 31880):
#0 0x00007f81d38400fe in pthread_cond_timedwait@@GLIBC_2.3.2 ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f81cee4de5a in ?? () from /usr/lib/x86_64-linux-gnu/glusterfs/3.6.8/xlator/storage/posix.so
---Type <return> to continue, or q <return> to quit---
#2 0x00007f81d383be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007f81d35683fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x0000000000000000 in ?? ()
Thread 5 (LWP 31842):
#0 0x00007f81d383fd84 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f81cdfd77fb in index_worker ()
from /usr/lib/x86_64-linux-gnu/glusterfs/3.6.8/xlator/features/index.so
#2 0x00007f81d383be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007f81d35683fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x0000000000000000 in ?? ()
Thread 4 (LWP 31591):
#0 0x00007f81d38400fe in pthread_cond_timedwait@@GLIBC_2.3.2 ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f81d3eddfe3 in syncenv_task () from /usr/lib/x86_64-linux-gnu/libglusterfs.so.0
#2 0x00007f81d3ede440 in syncenv_processor () from /usr/lib/x86_64-linux-gnu/libglusterfs.so.0
#3 0x00007f81d383be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f81d35683fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()
Thread 3 (LWP 31590):
#0 0x00007f81d38400fe in pthread_cond_timedwait@@GLIBC_2.3.2 ()
from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f81d3eddfe3 in syncenv_task () from /usr/lib/x86_64-linux-gnu/libglusterfs.so.0
#2 0x00007f81d3ede440 in syncenv_processor () from /usr/lib/x86_64-linux-gnu/libglusterfs.so.0
#3 0x00007f81d383be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f81d35683fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()
Thread 2 (LWP 31589):
#0 0x00007f81d38439f7 in do_sigwait () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f81d3843a79 in sigwait () from /lib/x86_64-linux-gnu/libpthread.so.0
#2 0x00007f81d434be12 in glusterfs_sigwaiter ()
#3 0x00007f81d383be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007f81d35683fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()
Thread 1 (LWP 31588):
#0 0x00007f81d384352d in nanosleep () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007f81d3ebc4dc in gf_timer_proc () from /usr/lib/x86_64-linux-gnu/libglusterfs.so.0
#2 0x00007f81d383be9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007f81d35683fd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x0000000000000000 in ?? ()
Process 31587 attached with 10 threads - interrupt to quit
[pid 3385] 03:09:19 restart_syscall(<... resuming interrupted call ...> <unfinished ...>
[pid 20656] 03:09:19 restart_syscall(<... resuming interrupted call ...> <unfinished ...>
[pid 31881] 03:09:19 futex(0x7f81d5f538bc, FUTEX_WAIT_PRIVATE, 23707, NULL <unfinished ...>
[pid 31880] 03:09:19 restart_syscall(<... resuming interrupted call ...> <unfinished ...>
[pid 31842] 03:09:19 futex(0x7f81d5f4889c, FUTEX_WAIT_PRIVATE, 41409, NULL <unfinished ...>
[pid 31591] 03:09:19 restart_syscall(<... resuming interrupted call ...> <unfinished ...>
[pid 31590] 03:09:19 restart_syscall(<... resuming interrupted call ...> <unfinished ...>
[pid 31589] 03:09:19 rt_sigtimedwait([HUP INT USR1 USR2 TERM], NULL, NULL, 8 <unfinished ...>
[pid 31588] 03:09:19 restart_syscall(<... resuming interrupted call ...> <unfinished ...>
[pid 31587] 03:09:19 futex(0x7f81900023b0, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 31588] 03:09:19 <... restart_syscall resumed> ) = 0
[pid 31588] 03:09:19 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:20 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:21 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:22 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:23 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:24 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:25 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:26 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:27 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:28 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:29 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:30 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:31 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:32 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:33 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:34 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:35 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:36 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:37 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:38 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:39 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:40 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:41 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:42 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:43 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:44 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:45 nanosleep({1, 0}, <unfinished ...>
[pid 3385] 03:09:46 <... restart_syscall resumed> ) = 0
[pid 3385] 03:09:46 open("/gluster/brick03/cinder-std-01/.glusterfs/health_check", O_RDWR|O_CREAT, 0644) = 27
[pid 3385] 03:09:46 write(27, "2016-02-13 03:09:46", 19) = 19
[pid 3385] 03:09:46 lseek(27, 0, SEEK_SET) = 0
[pid 3385] 03:09:46 read(27, "2016-02-13 03:09:46", 19) = 19
[pid 3385] 03:09:46 close(27) = 0
[pid 3385] 03:09:46 rt_sigprocmask(SIG_BLOCK, [CHLD], ~[ILL ABRT BUS FPE KILL SEGV STOP SYS RTMIN RT_1], 8) = 0
[pid 3385] 03:09:46 nanosleep({30, 0}, <unfinished ...>
[pid 31588] 03:09:46 <... nanosleep resumed> NULL) = 0
[pid 31588] 03:09:46 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:47 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:48 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:49 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:50 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:51 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:52 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:53 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:54 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:55 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:56 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:57 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:58 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:09:59 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:10:00 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:10:01 nanosleep({1, 0}, NULL) = 0
[pid 31588] 03:10:02 nanosleep({1, 0}, NULL) = 0
DUMP-START-TIME: 2016-02-13 02:42:44.670135
[mallinfo]
mallinfo_arena=1187840
mallinfo_ordblks=260
mallinfo_smblks=1
mallinfo_hblks=12
mallinfo_hblkhd=16060416
mallinfo_usmblks=0
mallinfo_fsmblks=112
mallinfo_uordblks=1036864
mallinfo_fordblks=150976
mallinfo_keepcost=97488
[global.glusterfs - Memory usage]
num_types=122
[global.glusterfs - usage-type gf_common_mt_dnscache6 memusage]
size=16
num_allocs=1
max_size=16
max_num_allocs=1
total_allocs=1
[global.glusterfs - usage-type gf_common_mt_event_pool memusage]
size=144
num_allocs=1
max_size=144
max_num_allocs=1
total_allocs=1
[global.glusterfs - usage-type gf_common_mt_reg memusage]
size=393216
num_allocs=1
max_size=393216
max_num_allocs=1
total_allocs=1
[global.glusterfs - usage-type gf_common_mt_epoll_event memusage]
size=3096
num_allocs=1
max_size=3096
max_num_allocs=1
total_allocs=1
[global.glusterfs - usage-type gf_common_mt_fd_ctx memusage]
size=0
num_allocs=0
max_size=192
max_num_allocs=1
total_allocs=2
[global.glusterfs - usage-type gf_common_mt_inode_ctx memusage]
size=0
num_allocs=0
max_size=288
max_num_allocs=1
total_allocs=32
[global.glusterfs - usage-type gf_common_mt_xlator_t memusage]
size=28600
num_allocs=11
max_size=57200
max_num_allocs=22
total_allocs=55
[global.glusterfs - usage-type gf_common_mt_xlator_list_t memusage]
size=320
num_allocs=20
max_size=640
max_num_allocs=40
total_allocs=100
[global.glusterfs - usage-type gf_common_mt_volume_opt_list_t memusage]
size=312
num_allocs=13
max_size=576
max_num_allocs=24
total_allocs=57
[global.glusterfs - usage-type gf_common_mt_gf_timer_t memusage]
size=112
num_allocs=2
max_size=168
max_num_allocs=3
total_allocs=39761
[global.glusterfs - usage-type gf_common_mt_gf_timer_registry_t memusage]
size=168
num_allocs=1
max_size=168
max_num_allocs=1
total_allocs=1
[global.glusterfs - usage-type gf_common_mt_iobuf memusage]
size=109536
num_allocs=8
max_size=109536
max_num_allocs=8
total_allocs=8
[global.glusterfs - usage-type gf_common_mt_iobuf_arena memusage]
size=1872
num_allocs=9
max_size=1872
max_num_allocs=9
total_allocs=9
[global.glusterfs - usage-type gf_common_mt_iobref memusage]
size=0
num_allocs=0
max_size=48
max_num_allocs=2
total_allocs=39
[global.glusterfs - usage-type gf_common_mt_iobuf_pool memusage]
size=1776
num_allocs=1
max_size=1776
max_num_allocs=1
total_allocs=1
[global.glusterfs - usage-type gf_common_mt_memdup memusage]
size=0
num_allocs=0
max_size=118
max_num_allocs=7
total_allocs=7
[global.glusterfs - usage-type gf_common_mt_asprintf memusage]
size=265
num_allocs=12
max_size=680
max_num_allocs=90
total_allocs=524
[global.glusterfs - usage-type gf_common_mt_strdup memusage]
size=3781
num_allocs=172
max_size=4264
max_num_allocs=196
total_allocs=708
[global.glusterfs - usage-type gf_common_mt_socket_private_t memusage]
size=1344
num_allocs=3
max_size=1344
max_num_allocs=3
total_allocs=3
[global.glusterfs - usage-type gf_common_mt_ioq memusage]
size=0
num_allocs=0
max_size=312
max_num_allocs=1
total_allocs=14
[global.glusterfs - usage-type gf_common_mt_char memusage]
size=13504
num_allocs=151
max_size=35426
max_num_allocs=217
total_allocs=698
[global.glusterfs - usage-type gf_common_mt_mem_pool memusage]
size=1200
num_allocs=10
max_size=1200
max_num_allocs=10
total_allocs=10
[global.glusterfs - usage-type gf_common_mt_long memusage]
size=9034400
num_allocs=10
max_size=9034400
max_num_allocs=10
total_allocs=10
[global.glusterfs - usage-type gf_common_mt_rpcsvc_auth_list memusage]
size=288
num_allocs=4
max_size=288
max_num_allocs=4
total_allocs=4
[global.glusterfs - usage-type gf_common_mt_rpcsvc_t memusage]
size=200
num_allocs=1
max_size=200
max_num_allocs=1
total_allocs=1
[global.glusterfs - usage-type gf_common_mt_rpcsvc_program_t memusage]
size=240
num_allocs=2
max_size=240
max_num_allocs=2
total_allocs=2
[global.glusterfs - usage-type gf_common_mt_rpcsvc_listener_t memusage]
size=160
num_allocs=1
max_size=160
max_num_allocs=1
total_allocs=1
[global.glusterfs - usage-type gf_common_mt_rpcsvc_wrapper_t memusage]
size=32
num_allocs=1
max_size=32
max_num_allocs=1
total_allocs=1
[global.glusterfs - usage-type gf_common_mt_rpcclnt_t memusage]
size=296
num_allocs=1
max_size=296
max_num_allocs=1
total_allocs=1
[global.glusterfs - usage-type gf_common_mt_rpcclnt_savedframe_t memusage]
size=200
num_allocs=1
max_size=200
max_num_allocs=1
total_allocs=1
[global.glusterfs - usage-type gf_common_mt_rpc_trans_t memusage]
size=8352
num_allocs=3
max_size=8352
max_num_allocs=3
total_allocs=3
[global.glusterfs - usage-type gf_common_mt_rpc_trans_pollin_t memusage]
size=0
num_allocs=0
max_size=296
max_num_allocs=1
total_allocs=25
[global.glusterfs - usage-type gf_common_mt_rpc_trans_reqinfo_t memusage]
size=0
num_allocs=0
max_size=64
max_num_allocs=1
total_allocs=13
[global.glusterfs - usage-type gf_common_mt_glusterfs_graph_t memusage]
size=192
num_allocs=1
max_size=384
max_num_allocs=2
total_allocs=5
[global.glusterfs - usage-type gf_common_mt_rpcclnt_cb_program_t memusage]
size=88
num_allocs=1
max_size=88
max_num_allocs=1
total_allocs=1
[global.glusterfs - usage-type gf_common_mt_cliententry_t memusage]
size=2048
num_allocs=1
max_size=2048
max_num_allocs=1
total_allocs=1
[global.glusterfs - usage-type gf_common_mt_clienttable_t memusage]
size=32
num_allocs=1
max_size=32
max_num_allocs=1
total_allocs=1
[global.glusterfs - usage-type gf_common_mt_iobrefs memusage]
size=0
num_allocs=0
max_size=256
max_num_allocs=2
total_allocs=39
[global.glusterfs - usage-type gf_sock_mt_lock_array memusage]
size=1640
num_allocs=1
max_size=1640
max_num_allocs=1
total_allocs=1
[global.glusterfs - usage-type gfd_mt_server_cmdline_t memusage]
size=40
num_allocs=1
max_size=40
max_num_allocs=1
total_allocs=1
[global.glusterfs - usage-type gfd_mt_xlator_cmdline_option_t memusage]
size=80
num_allocs=2
max_size=80
max_num_allocs=2
total_allocs=2
[global.glusterfs - usage-type gfd_mt_char memusage]
size=53
num_allocs=4
max_size=53
max_num_allocs=4
total_allocs=4
[global.glusterfs - usage-type gfd_mt_call_pool_t memusage]
size=48
num_allocs=1
max_size=48
max_num_allocs=1
total_allocs=1
[mempool]
-----=-----
pool-name=gv-cinder-server:fd_t
hot-count=2
cold-count=1022
padded_sizeof=108
alloc-count=27465
max-alloc=6
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=gv-cinder-server:dentry_t
hot-count=30
cold-count=16354
padded_sizeof=84
alloc-count=1689
max-alloc=31
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=gv-cinder-server:inode_t
hot-count=32
cold-count=16352
padded_sizeof=156
alloc-count=23516
max-alloc=35
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=gv-cinder-changelog:changelog_local_t
hot-count=0
cold-count=64
padded_sizeof=116
alloc-count=0
max-alloc=0
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=gv-cinder-locks:pl_local_t
hot-count=0
cold-count=32
padded_sizeof=148
alloc-count=119146
max-alloc=5
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=gv-cinder-marker:marker_local_t
hot-count=0
cold-count=128
padded_sizeof=332
alloc-count=25101
max-alloc=2
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=gv-cinder-quota:quota_local_t
hot-count=0
cold-count=64
padded_sizeof=412
alloc-count=0
max-alloc=0
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=gv-cinder-server:rpcsvc_request_t
hot-count=0
cold-count=512
padded_sizeof=2828
alloc-count=134951621
max-alloc=43
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:struct saved_frame
hot-count=0
cold-count=8
padded_sizeof=124
alloc-count=13
max-alloc=3
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:struct rpc_req
hot-count=0
cold-count=8
padded_sizeof=588
alloc-count=13
max-alloc=3
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:rpcsvc_request_t
hot-count=0
cold-count=8
padded_sizeof=2828
alloc-count=1
max-alloc=1
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:log_buf_t
hot-count=0
cold-count=256
padded_sizeof=140
alloc-count=1
max-alloc=1
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:data_t
hot-count=152
cold-count=16232
padded_sizeof=52
alloc-count=88386649
max-alloc=219
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:data_pair_t
hot-count=110
cold-count=16274
padded_sizeof=68
alloc-count=2103720
max-alloc=191
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:dict_t
hot-count=58
cold-count=4038
padded_sizeof=140
alloc-count=85559811
max-alloc=101
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:call_stub_t
hot-count=0
cold-count=1024
padded_sizeof=3756
alloc-count=135291560
max-alloc=27
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:call_stack_t
hot-count=0
cold-count=1024
padded_sizeof=1836
alloc-count=134886246
max-alloc=43
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
-----=-----
pool-name=glusterfs:call_frame_t
hot-count=0
cold-count=4096
padded_sizeof=172
alloc-count=455460023
max-alloc=156
pool-misses=0
cur-stdalloc=0
max-stdalloc=0
[iobuf.global]
iobuf_pool=0x7f81d5ebc980
iobuf_pool.default_page_size=131072
iobuf_pool.arena_size=12976128
iobuf_pool.arena_cnt=8
iobuf_pool.request_misses=0
[purge.1]
purge.1.mem_base=0x7f81d430f000
purge.1.active_cnt=0
purge.1.passive_cnt=1024
purge.1.alloc_cnt=318736820
purge.1.max_active=12
purge.1.page_size=128
[purge.2]
purge.2.mem_base=0x7f81d42cf000
purge.2.active_cnt=0
purge.2.passive_cnt=512
purge.2.alloc_cnt=85775611
purge.2.max_active=8
purge.2.page_size=512
[purge.3]
purge.3.mem_base=0x7f81d41cf000
purge.3.active_cnt=0
purge.3.passive_cnt=512
purge.3.alloc_cnt=25248
purge.3.max_active=1
purge.3.page_size=2048
[purge.4]
purge.4.mem_base=0x7f81d2976000
purge.4.active_cnt=0
purge.4.passive_cnt=128
purge.4.alloc_cnt=1208704
purge.4.max_active=11
purge.4.page_size=8192
[purge.5]
purge.5.mem_base=0x7f81d2776000
purge.5.active_cnt=0
purge.5.passive_cnt=64
purge.5.alloc_cnt=330835
purge.5.max_active=6
purge.5.page_size=32768
[purge.6]
purge.6.mem_base=0x7f81d2376000
purge.6.active_cnt=0
purge.6.passive_cnt=32
purge.6.alloc_cnt=47899288
purge.6.max_active=4
purge.6.page_size=131072
[purge.7]
purge.7.mem_base=0x7f81d2176000
purge.7.active_cnt=0
purge.7.passive_cnt=8
purge.7.alloc_cnt=212
purge.7.max_active=1
purge.7.page_size=262144
[arena.8]
arena.8.mem_base=0x7f81d1f76000
arena.8.active_cnt=0
arena.8.passive_cnt=2
arena.8.alloc_cnt=0
arena.8.max_active=0
arena.8.page_size=1048576
[global.callpool]
callpool_address=0x7f81d5ed8630
callpool.cnt=0
[active graph - 5]
[protocol/server.gv-cinder-server - Memory usage]
num_types=127
[protocol/server.gv-cinder-server - usage-type 0 memusage]
size=0
num_allocs=0
max_size=96
max_num_allocs=3
total_allocs=450
[protocol/server.gv-cinder-server - usage-type gf_common_mt_fdentry_t memusage]
size=51200
num_allocs=25
max_size=81920
max_num_allocs=40
total_allocs=403
[protocol/server.gv-cinder-server - usage-type gf_common_mt_fdtable_t memusage]
size=1536
num_allocs=24
max_size=2496
max_num_allocs=39
total_allocs=150
[protocol/server.gv-cinder-server - usage-type gf_common_mt_fd_ctx memusage]
size=384
num_allocs=2
max_size=1152
max_num_allocs=6
total_allocs=27465
[protocol/server.gv-cinder-server - usage-type gf_common_mt_inode_ctx memusage]
size=1728
num_allocs=6
max_size=2592
max_num_allocs=9
total_allocs=23477
[protocol/server.gv-cinder-server - usage-type gf_common_mt_list_head memusage]
size=1273488
num_allocs=2
max_size=1273488
max_num_allocs=2
total_allocs=2
[protocol/server.gv-cinder-server - usage-type gf_common_mt_inode_table_t memusage]
size=200
num_allocs=1
max_size=200
max_num_allocs=1
total_allocs=1
[protocol/server.gv-cinder-server - usage-type gf_common_mt_volume_opt_list_t memusage]
size=264
num_allocs=11
max_size=264
max_num_allocs=11
total_allocs=11
[protocol/server.gv-cinder-server - usage-type gf_common_mt_auth_handle_t memusage]
size=48
num_allocs=2
max_size=48
max_num_allocs=2
total_allocs=10
[protocol/server.gv-cinder-server - usage-type gf_common_mt_iobref memusage]
size=0
num_allocs=0
max_size=1056
max_num_allocs=44
total_allocs=221069677
[protocol/server.gv-cinder-server - usage-type gf_common_mt_memdup memusage]
size=0
num_allocs=0
max_size=779
max_num_allocs=35
total_allocs=1937699
[protocol/server.gv-cinder-server - usage-type gf_common_mt_asprintf memusage]
size=144
num_allocs=5
max_size=3125
max_num_allocs=10
total_allocs=834957
[protocol/server.gv-cinder-server - usage-type gf_common_mt_strdup memusage]
size=10972
num_allocs=284
max_size=11475
max_num_allocs=303
total_allocs=84802827
[protocol/server.gv-cinder-server - usage-type gf_common_mt_socket_private_t memusage]
size=11200
num_allocs=25
max_size=17920
max_num_allocs=40
total_allocs=151
[protocol/server.gv-cinder-server - usage-type gf_common_mt_ioq memusage]
size=0
num_allocs=0
max_size=936
max_num_allocs=3
total_allocs=134951621
[protocol/server.gv-cinder-server - usage-type gf_common_mt_char memusage]
size=149
num_allocs=7
max_size=3525
max_num_allocs=61
total_allocs=173345106
[protocol/server.gv-cinder-server - usage-type gf_common_mt_mem_pool memusage]
size=480
num_allocs=4
max_size=480
max_num_allocs=4
total_allocs=4
[protocol/server.gv-cinder-server - usage-type gf_common_mt_long memusage]
size=5490688
num_allocs=4
max_size=5490688
max_num_allocs=4
total_allocs=4
[protocol/server.gv-cinder-server - usage-type gf_common_mt_rpcsvc_auth_list memusage]
size=288
num_allocs=4
max_size=288
max_num_allocs=4
total_allocs=4
[protocol/server.gv-cinder-server - usage-type gf_common_mt_rpcsvc_t memusage]
size=200
num_allocs=1
max_size=200
max_num_allocs=1
total_allocs=1
[protocol/server.gv-cinder-server - usage-type gf_common_mt_rpcsvc_program_t memusage]
size=360
num_allocs=3
max_size=360
max_num_allocs=3
total_allocs=3
[protocol/server.gv-cinder-server - usage-type gf_common_mt_rpcsvc_listener_t memusage]
size=160
num_allocs=1
max_size=160
max_num_allocs=1
total_allocs=1
[protocol/server.gv-cinder-server - usage-type gf_common_mt_rpcsvc_wrapper_t memusage]
size=64
num_allocs=2
max_size=64
max_num_allocs=2
total_allocs=128
[protocol/server.gv-cinder-server - usage-type gf_common_mt_rpc_trans_t memusage]
size=69600
num_allocs=25
max_size=111360
max_num_allocs=40
total_allocs=151
[protocol/server.gv-cinder-server - usage-type gf_common_mt_rpc_trans_pollin_t memusage]
size=0
num_allocs=0
max_size=296
max_num_allocs=1
total_allocs=134951621
[protocol/server.gv-cinder-server - usage-type gf_common_mt_fd_lk_ctx_t memusage]
size=48
num_allocs=2
max_size=144
max_num_allocs=6
total_allocs=27465
[protocol/server.gv-cinder-server - usage-type gf_common_mt_client_t memusage]
size=3072
num_allocs=48
max_size=4992
max_num_allocs=78
total_allocs=300
[protocol/server.gv-cinder-server - usage-type gf_common_mt_client_ctx memusage]
size=3072
num_allocs=24
max_size=4992
max_num_allocs=39
total_allocs=150
[protocol/server.gv-cinder-server - usage-type gf_common_mt_iobrefs memusage]
size=0
num_allocs=0
max_size=5632
max_num_allocs=44
total_allocs=221069677
[protocol/server.gv-cinder-server - usage-type gf_server_mt_server_conf_t memusage]
size=50088
num_allocs=25
max_size=50568
max_num_allocs=40
total_allocs=151
[protocol/server.gv-cinder-server - usage-type gf_server_mt_state_t memusage]
size=0
num_allocs=0
max_size=87376
max_num_allocs=43
total_allocs=134886335
[protocol/server.gv-cinder-server - usage-type gf_server_mt_dirent_rsp_t memusage]
size=0
num_allocs=0
max_size=3040
max_num_allocs=19
total_allocs=26251
[protocol/server.gv-cinder-server - usage-type gf_server_mt_rsp_buf_t memusage]
size=0
num_allocs=0
max_size=961
max_num_allocs=19
total_allocs=644
[xlator.protocol.server.priv]
server.total-bytes-read=11424142224
server.total-bytes-write=3014661265649
conn.2.bound_xl./gluster/brick03/cinder-std-01.hashsize=14057
conn.2.bound_xl./gluster/brick03/cinder-std-01.name=/gluster/brick03/cinder-std-01/inode
conn.2.bound_xl./gluster/brick03/cinder-std-01.lru_limit=16384
conn.2.bound_xl./gluster/brick03/cinder-std-01.active_size=3
conn.2.bound_xl./gluster/brick03/cinder-std-01.lru_size=29
conn.2.bound_xl./gluster/brick03/cinder-std-01.purge_size=0
[conn.2.bound_xl./gluster/brick03/cinder-std-01.active.1]
gfid=8f787374-3b1a-4a0e-8ec5-41f41724c22d
nlookup=10372
fd-count=1
ref=4
ia_type=1
[xlator.features.locks.gv-cinder-locks.inode]
path=/vol-820679f7-4d6a-4397-a522-6fff941ab862
mandatory=0
[conn.2.bound_xl./gluster/brick03/cinder-std-01.active.2]
gfid=b8b58762-fd76-4d23-9683-50abf6447d2e
nlookup=11355
fd-count=1
ref=1
ia_type=1
[xlator.features.locks.gv-cinder-locks.inode]
path=/vol-ad3357ba-c264-4e58-a1ac-1d90cb0f960d
mandatory=0
[conn.2.bound_xl./gluster/brick03/cinder-std-01.active.3]
gfid=00000000-0000-0000-0000-000000000001
nlookup=13
fd-count=0
ref=1
ia_type=2
[xlator.features.locks.gv-cinder-locks.inode]
path=/
mandatory=0
[conn.2.bound_xl./gluster/brick03/cinder-std-01.lru.1]
gfid=8bbcfe88-b91e-4643-bb7a-49d2e7a15bcf
nlookup=17
fd-count=0
ref=0
ia_type=1
[conn.2.bound_xl./gluster/brick03/cinder-std-01.lru.2]
gfid=97a98f35-d5c2-4a2c-b900-cacb0b74720e
nlookup=13
fd-count=0
ref=0
ia_type=1
[conn.2.bound_xl./gluster/brick03/cinder-std-01.lru.3]
gfid=4ffec429-231f-4e27-a227-72388d607eed
nlookup=26
fd-count=0
ref=0
ia_type=1
[xlator.features.locks.gv-cinder-locks.inode]
path=/vol-afa9bc2c-5666-422b-88c4-2cb6b1f1fcaa
mandatory=0
[conn.2.bound_xl./gluster/brick03/cinder-std-01.lru.4]
gfid=1e2f6504-5c49-4dda-9a20-5f47f11b7f78
nlookup=13
fd-count=0
ref=0
ia_type=1
[conn.2.bound_xl./gluster/brick03/cinder-std-01.lru.5]
gfid=ddd4bec4-d73b-4649-ab05-63c7219a9c2c
nlookup=13
fd-count=0
ref=0
ia_type=1
[conn.2.bound_xl./gluster/brick03/cinder-std-01.lru.6]
gfid=02b80a30-76d8-4c30-9575-6fbec905d8d3
nlookup=17
fd-count=0
ref=0
ia_type=1
[conn.2.bound_xl./gluster/brick03/cinder-std-01.lru.7]
gfid=43bc1a78-c9b7-471b-9721-4da560745a6e
nlookup=17
fd-count=0
ref=0
ia_type=1
[conn.2.bound_xl./gluster/brick03/cinder-std-01.lru.8]
gfid=396b00de-be60-4700-b617-3d3edfc5aeed
nlookup=17
fd-count=0
ref=0
ia_type=1
[conn.2.bound_xl./gluster/brick03/cinder-std-01.lru.9]
gfid=9f654186-ca48-4e08-8fdd-dd86c9e5dbce
nlookup=17
fd-count=0
ref=0
ia_type=1
[conn.2.bound_xl./gluster/brick03/cinder-std-01.lru.10]
gfid=09d93e82-012b-4e06-abee-34c0d05ca6e8
nlookup=17
fd-count=0
ref=0
ia_type=1
[conn.2.bound_xl./gluster/brick03/cinder-std-01.lru.11]
gfid=ba5986bb-7e18-4d5f-a9e7-04aa08e732c1
nlookup=26
fd-count=0
ref=0
ia_type=1
[xlator.features.locks.gv-cinder-locks.inode]
path=/vol-8a05b90e-820a-4b97-817c-35d9e3a32c20
mandatory=0
[conn.2.bound_xl./gluster/brick03/cinder-std-01.lru.12]
gfid=e298ffe1-49bd-473e-8707-56031118b196
nlookup=30
fd-count=0
ref=0
ia_type=1
[conn.2.bound_xl./gluster/brick03/cinder-std-01.lru.13]
gfid=f83ac264-2cca-4ef6-bbf9-25e0c1c2e307
nlookup=30
fd-count=0
ref=0
ia_type=1
[conn.2.bound_xl./gluster/brick03/cinder-std-01.lru.14]
gfid=22f1caad-537b-4309-b9ca-b23068bc5ca6
nlookup=26
fd-count=0
ref=0
ia_type=1
[conn.2.bound_xl./gluster/brick03/cinder-std-01.lru.15]
gfid=6ae80d7a-76b2-4650-be55-7bee4fb5fd49
nlookup=26
fd-count=0
ref=0
ia_type=1
[conn.2.bound_xl./gluster/brick03/cinder-std-01.lru.16]
gfid=d0e7b11a-ca37-4905-ba87-2b7e5b2e7198
nlookup=30
fd-count=0
ref=0
ia_type=1
[conn.2.bound_xl./gluster/brick03/cinder-std-01.lru.17]
gfid=a8bdac1f-39e3-4d5a-8b96-0da4900dfa91
nlookup=30
fd-count=0
ref=0
ia_type=1
[conn.2.bound_xl./gluster/brick03/cinder-std-01.lru.18]
gfid=edb13fa6-59a3-4374-8645-a17e319559d1
nlookup=17
fd-count=0
ref=0
ia_type=1
[conn.2.bound_xl./gluster/brick03/cinder-std-01.lru.19]
gfid=e13a41d1-2945-486f-9016-8e8ec9e5889d
nlookup=26
fd-count=0
ref=0
ia_type=1
[xlator.features.locks.gv-cinder-locks.inode]
path=/vol-3aee038f-2ba9-4195-883c-62978b0d394d
mandatory=0
[conn.2.bound_xl./gluster/brick03/cinder-std-01.lru.20]
gfid=fc021567-c228-4a7b-ad47-b4b4eaca7fd7
nlookup=26
fd-count=0
ref=0
ia_type=1
[xlator.features.locks.gv-cinder-locks.inode]
path=/vol-3aee038f-2ba9-4195-883c-62978b0d394d.info
mandatory=0
[conn.2.bound_xl./gluster/brick03/cinder-std-01.lru.21]
gfid=337d82da-1104-4f2c-9ad1-900265386532
nlookup=26
fd-count=0
ref=0
ia_type=1
[xlator.features.locks.gv-cinder-locks.inode]
path=/vol-92d4c940-409a-4bfa-ad1f-9c87690dc0f0
mandatory=0
[conn.2.bound_xl./gluster/brick03/cinder-std-01.lru.22]
gfid=989060f7-6bb1-4052-a811-c3136eae0395
nlookup=26
fd-count=0
ref=0
ia_type=1
[xlator.features.locks.gv-cinder-locks.inode]
path=/vol-4d03e51a-da7f-4a53-ae85-6f05e36d4d07
mandatory=0
[conn.2.bound_xl./gluster/brick03/cinder-std-01.lru.23]
gfid=07398a28-6a92-4243-a0e5-d3e03eba4fa5
nlookup=17
fd-count=0
ref=0
ia_type=1
[conn.2.bound_xl./gluster/brick03/cinder-std-01.lru.24]
gfid=50d3c83a-d670-4359-939c-3601abc6ef07
nlookup=17
fd-count=0
ref=0
ia_type=1
[conn.2.bound_xl./gluster/brick03/cinder-std-01.lru.25]
gfid=5b810982-131a-4ab7-8986-034e1f5fec3e
nlookup=17
fd-count=0
ref=0
ia_type=1
[conn.2.bound_xl./gluster/brick03/cinder-std-01.lru.26]
gfid=cfc8019d-69c7-45fb-b95c-1dddbe4e24bb
nlookup=26
fd-count=0
ref=0
ia_type=1
[xlator.features.locks.gv-cinder-locks.inode]
path=/vol-4d46d6d6-3d34-4243-bb86-e4faeae5ba75
mandatory=0
[conn.2.bound_xl./gluster/brick03/cinder-std-01.lru.27]
gfid=99c346af-7edf-4978-b020-af70eba07607
nlookup=7013
fd-count=0
ref=0
ia_type=1
[xlator.features.locks.gv-cinder-locks.inode]
path=/vol-cdf2ad03-e114-4f73-b38d-117c08f2545c
mandatory=0
[conn.2.bound_xl./gluster/brick03/cinder-std-01.lru.28]
gfid=c5915f07-8d1f-45df-8409-f3ea0f7b3da9
nlookup=5768
fd-count=0
ref=0
ia_type=2
[conn.2.bound_xl./gluster/brick03/cinder-std-01.lru.29]
gfid=4a7eb4bb-10a9-43b9-a4dd-d8256697b044
nlookup=6981
fd-count=0
ref=0
ia_type=1
[xlator.features.locks.gv-cinder-locks.inode]
path=/vol-f0e0740f-4e07-4a62-a2b4-9e6ce18496a5
mandatory=0
conn.2.id=controller01-8035-2016/02/11-21:48:33:714247-gv-cinder-client-2-0-0
conn.2.ref=1
conn.2.bound_xl=/gluster/brick03/cinder-std-01
conn.4.id=compute015-24816-2016/02/11-21:56:23:422938-gv-cinder-client-2-0-0
conn.4.ref=1
conn.4.bound_xl=/gluster/brick03/cinder-std-01
conn.6.id=compute013-12886-2016/02/11-22:41:00:857700-gv-cinder-client-2-0-0
conn.6.ref=1
conn.6.bound_xl=/gluster/brick03/cinder-std-01
conn.7.id=compute015-20555-2016/02/11-21:54:15:924117-gv-cinder-client-2-0-0
conn.7.ref=1
conn.7.bound_xl=/gluster/brick03/cinder-std-01
conn.8.id=storage07-29222-2016/02/12-08:16:20:52268-gv-cinder-client-2-0-0
conn.8.ref=1
conn.8.bound_xl=/gluster/brick03/cinder-std-01
conn.9.id=storage07-31653-2016/01/29-06:43:27:2595-gv-cinder-client-2-0-2
conn.9.ref=1
conn.9.bound_xl=/gluster/brick03/cinder-std-01
conn.11.id=compute015-7236-2016/02/11-22:03:52:865440-gv-cinder-client-2-0-0
conn.11.ref=1
conn.11.bound_xl=/gluster/brick03/cinder-std-01
conn.13.id=storage03-7760-2016/02/11-20:45:51:151440-gv-cinder-client-2-0-0
conn.13.ref=1
conn.13.bound_xl=/gluster/brick03/cinder-std-01
conn.14.id=compute021-12040-2016/02/11-22:49:59:497964-gv-cinder-client-2-0-0
conn.14.ref=1
conn.14.bound_xl=/gluster/brick03/cinder-std-01
conn.19.id=compute016-3457-2016/02/11-23:05:18:739493-gv-cinder-client-2-0-0
conn.19.ref=1
conn.19.bound_xl=/gluster/brick03/cinder-std-01
conn.21.id=compute011-17350-2016/02/12-00:04:46:704199-gv-cinder-client-2-0-0
conn.21.ref=1
conn.21.bound_xl=/gluster/brick03/cinder-std-01
conn.22.id=compute014-2359-2016/02/11-22:37:08:48705-gv-cinder-client-2-0-0
conn.22.ref=1
conn.22.bound_xl=/gluster/brick03/cinder-std-01
conn.23.id=storage04-23521-2016/02/11-20:45:50:904106-gv-cinder-client-2-0-0
conn.23.ref=1
conn.23.bound_xl=/gluster/brick03/cinder-std-01
conn.25.id=storage08-20934-2016/02/11-20:45:47:836024-gv-cinder-client-2-0-0
conn.25.ref=1
conn.25.bound_xl=/gluster/brick03/cinder-std-01
conn.26.id=storage04-23562-2016/02/11-20:45:51:830750-gv-cinder-client-2-0-0
conn.26.ref=1
conn.26.bound_xl=/gluster/brick03/cinder-std-01
conn.27.id=compute018-31965-2016/02/11-21:35:27:794540-gv-cinder-client-2-0-0
conn.27.ref=1
conn.27.bound_xl=/gluster/brick03/cinder-std-01
conn.28.id=compute018-16969-2016/02/11-21:43:33:877210-gv-cinder-client-2-0-0
conn.28.ref=1
conn.28.bound_xl=/gluster/brick03/cinder-std-01
conn.29.id=compute017-21189-2016/02/11-22:53:39:559720-gv-cinder-client-2-0-0
conn.29.ref=1
conn.29.bound_xl=/gluster/brick03/cinder-std-01
conn.31.id=compute018-4161-2016/02/11-21:37:35:723582-gv-cinder-client-2-0-0
conn.31.ref=1
conn.31.bound_xl=/gluster/brick03/cinder-std-01
conn.34.id=storage03-1915-2016/02/02-16:51:43:144253-gv-cinder-client-2-0-1
conn.34.ref=1
conn.34.bound_xl=/gluster/brick03/cinder-std-01
conn.35.id=storage08-20965-2016/02/11-20:45:48:848917-gv-cinder-client-2-0-0
conn.35.ref=1
conn.35.bound_xl=/gluster/brick03/cinder-std-01
conn.36.id=compute019-15495-2016/02/11-22:58:56:961354-gv-cinder-client-2-0-0
conn.36.ref=1
conn.36.bound_xl=/gluster/brick03/cinder-std-01
conn.37.id=storage04-29111-2016/02/11-18:26:13:30717-gv-cinder-client-2-0-0
conn.37.ref=1
conn.37.bound_xl=/gluster/brick03/cinder-std-01
conn.38.id=storage03-7806-2016/02/11-20:45:52:54804-gv-cinder-client-2-0-0
conn.38.ref=1
conn.38.bound_xl=/gluster/brick03/cinder-std-01
[debug/io-stats./gluster/brick03/cinder-std-01 - Memory usage]
num_types=121
[debug/io-stats./gluster/brick03/cinder-std-01 - usage-type gf_common_mt_asprintf memusage]
size=0
num_allocs=0
max_size=109
max_num_allocs=1
total_allocs=7
[debug/io-stats./gluster/brick03/cinder-std-01 - usage-type gf_common_mt_strdup memusage]
size=258
num_allocs=6
max_size=406
max_num_allocs=10
total_allocs=10239
[debug/io-stats./gluster/brick03/cinder-std-01 - usage-type gf_common_mt_char memusage]
size=0
num_allocs=0
max_size=154
max_num_allocs=1
total_allocs=7
[debug/io-stats./gluster/brick03/cinder-std-01 - usage-type gf_io_stats_mt_ios_conf memusage]
size=5424
num_allocs=1
max_size=5424
max_num_allocs=1
total_allocs=1
[debug/io-stats./gluster/brick03/cinder-std-01 - usage-type gf_io_stats_mt_ios_fd memusage]
size=1104
num_allocs=2
max_size=2760
max_num_allocs=5
total_allocs=9361
[debug/io-stats./gluster/brick03/cinder-std-01 - usage-type gf_io_stats_mt_ios_stat memusage]
size=928
num_allocs=14
max_size=1080
max_num_allocs=15
total_allocs=888
[debug/io-stats./gluster/brick03/cinder-std-01 - usage-type gf_io_stats_mt_ios_stat_list memusage]
size=512
num_allocs=16
max_size=512
max_num_allocs=16
total_allocs=16
cumulative.data_read=6270431573049
cumulative.data_written=4366462976
incremental.data_read=6270431573049
incremental.data_written=4366462976
/gluster/brick03/cinder-std-01.cumulative.NULL=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.incremental.NULL=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.cumulative.STAT=399758,32878860,21.000,9986.000,82.247
/gluster/brick03/cinder-std-01.incremental.STAT=399758,32878860,21.000,9986.000,82.247
/gluster/brick03/cinder-std-01.cumulative.READLINK=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.incremental.READLINK=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.cumulative.MKNOD=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.incremental.MKNOD=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.cumulative.MKDIR=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.incremental.MKDIR=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.cumulative.UNLINK=1659,210449,32.000,375.000,126.853
/gluster/brick03/cinder-std-01.incremental.UNLINK=1659,210449,32.000,375.000,126.853
/gluster/brick03/cinder-std-01.cumulative.RMDIR=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.incremental.RMDIR=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.cumulative.SYMLINK=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.incremental.SYMLINK=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.cumulative.RENAME=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.incremental.RENAME=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.cumulative.LINK=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.incremental.LINK=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.cumulative.TRUNCATE=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.incremental.TRUNCATE=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.cumulative.OPEN=8487,926077,36.000,260.000,109.117
/gluster/brick03/cinder-std-01.incremental.OPEN=8487,926077,36.000,260.000,109.117
/gluster/brick03/cinder-std-01.cumulative.READ=48833565,12708669647,14.000,1602328.000,260.245
/gluster/brick03/cinder-std-01.incremental.READ=48833565,12708669647,14.000,1602328.000,260.245
/gluster/brick03/cinder-std-01.cumulative.WRITE=303078,62499624,17.000,172831.000,206.216
/gluster/brick03/cinder-std-01.incremental.WRITE=303078,62499624,17.000,172831.000,206.216
/gluster/brick03/cinder-std-01.cumulative.STATFS=11584,1266007,21.000,55380.000,109.289
/gluster/brick03/cinder-std-01.incremental.STATFS=11584,1266007,21.000,55380.000,109.289
/gluster/brick03/cinder-std-01.cumulative.FLUSH=882,54202,16.000,221.000,61.454
/gluster/brick03/cinder-std-01.incremental.FLUSH=882,54202,16.000,221.000,61.454
/gluster/brick03/cinder-std-01.cumulative.FSYNC=63714,6442491979,50.000,906161.000,101115.798
/gluster/brick03/cinder-std-01.incremental.FSYNC=63714,6442491979,50.000,906161.000,101115.798
/gluster/brick03/cinder-std-01.cumulative.SETXATTR=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.incremental.SETXATTR=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.cumulative.GETXATTR=25101,2796339,19.000,59253.000,111.403
/gluster/brick03/cinder-std-01.incremental.GETXATTR=25101,2796339,19.000,59253.000,111.403
/gluster/brick03/cinder-std-01.cumulative.REMOVEXATTR=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.incremental.REMOVEXATTR=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.cumulative.OPENDIR=15586,1103872,1.000,275.000,70.825
/gluster/brick03/cinder-std-01.incremental.OPENDIR=15586,1103872,1.000,275.000,70.825
/gluster/brick03/cinder-std-01.cumulative.FSYNCDIR=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.incremental.FSYNCDIR=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.cumulative.ACCESS=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.incremental.ACCESS=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.cumulative.CREATE=874,50472530,232.000,207796.000,57748.890
/gluster/brick03/cinder-std-01.incremental.CREATE=874,50472530,232.000,207796.000,57748.890
/gluster/brick03/cinder-std-01.cumulative.FTRUNCATE=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.incremental.FTRUNCATE=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.cumulative.FSTAT=17,2177,79.000,171.000,128.059
/gluster/brick03/cinder-std-01.incremental.FSTAT=17,2177,79.000,171.000,128.059
/gluster/brick03/cinder-std-01.cumulative.LK=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.incremental.LK=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.cumulative.LOOKUP=125661,33305278,11.000,85401.000,265.041
/gluster/brick03/cinder-std-01.incremental.LOOKUP=125661,33305278,11.000,85401.000,265.041
/gluster/brick03/cinder-std-01.cumulative.READDIR=9962,733765,11.000,22711.000,73.656
/gluster/brick03/cinder-std-01.incremental.READDIR=9962,733765,11.000,22711.000,73.656
/gluster/brick03/cinder-std-01.cumulative.INODELK=84548282,8422980210,10.000,3541049.000,99.623
/gluster/brick03/cinder-std-01.incremental.INODELK=84548282,8422980210,10.000,3541049.000,99.623
/gluster/brick03/cinder-std-01.cumulative.FINODELK=160805,38053301587,10.000,160453695.000,236642.527
/gluster/brick03/cinder-std-01.incremental.FINODELK=160805,38053301587,10.000,160453695.000,236642.527
/gluster/brick03/cinder-std-01.cumulative.ENTRYLK=13410,926496,12.000,202.000,69.090
/gluster/brick03/cinder-std-01.incremental.ENTRYLK=13410,926496,12.000,202.000,69.090
/gluster/brick03/cinder-std-01.cumulative.FENTRYLK=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.incremental.FENTRYLK=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.cumulative.XATTROP=3719,655734,42.000,17867.000,176.320
/gluster/brick03/cinder-std-01.incremental.XATTROP=3719,655734,42.000,17867.000,176.320
/gluster/brick03/cinder-std-01.cumulative.FXATTROP=361596,56177842,27.000,142536.000,155.361
/gluster/brick03/cinder-std-01.incremental.FXATTROP=361596,56177842,27.000,142536.000,155.361
/gluster/brick03/cinder-std-01.cumulative.FGETXATTR=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.incremental.FGETXATTR=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.cumulative.FSETXATTR=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.incremental.FSETXATTR=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.cumulative.RCHECKSUM=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.incremental.RCHECKSUM=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.cumulative.SETATTR=23,3450,93.000,178.000,150.000
/gluster/brick03/cinder-std-01.incremental.SETATTR=23,3450,93.000,178.000,150.000
/gluster/brick03/cinder-std-01.cumulative.FSETATTR=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.incremental.FSETATTR=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.cumulative.READDIRP=39,421236,40.000,97679.000,10800.923
/gluster/brick03/cinder-std-01.incremental.READDIRP=39,421236,40.000,97679.000,10800.923
/gluster/brick03/cinder-std-01.cumulative.FORGET=874,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.incremental.FORGET=874,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.cumulative.RELEASE=9359,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.incremental.RELEASE=9359,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.cumulative.RELEASEDIR=15586,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.incremental.RELEASEDIR=15586,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.cumulative.GETSPEC=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.incremental.GETSPEC=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.cumulative.FREMOVEXATTR=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.incremental.FREMOVEXATTR=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.cumulative.FALLOCATE=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.incremental.FALLOCATE=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.cumulative.DISCARD=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.incremental.DISCARD=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.cumulative.ZEROFILL=0,0,0.000,0.000,0.000
/gluster/brick03/cinder-std-01.incremental.ZEROFILL=0,0,0.000,0.000,0.000
[features/quota.gv-cinder-quota - Memory usage]
num_types=128
[features/quota.gv-cinder-quota - usage-type gf_common_mt_asprintf memusage]
size=30
num_allocs=1
max_size=124
max_num_allocs=2
total_allocs=6
[features/quota.gv-cinder-quota - usage-type gf_common_mt_char memusage]
size=0
num_allocs=0
max_size=133
max_num_allocs=1
total_allocs=5
[features/quota.gv-cinder-quota - usage-type gf_common_mt_mem_pool memusage]
size=120
num_allocs=1
max_size=120
max_num_allocs=1
total_allocs=1
[features/quota.gv-cinder-quota - usage-type gf_common_mt_long memusage]
size=26368
num_allocs=1
max_size=26368
max_num_allocs=1
total_allocs=1
[features/quota.gv-cinder-quota - usage-type gf_quota_mt_quota_priv_t memusage]
size=96
num_allocs=1
max_size=96
max_num_allocs=1
total_allocs=1
[xlators.features.quota.priv]
soft-timeout=60
hard-timeout=5
alert-time=86400
quota-on=0
statfs=0
volume-uuid=gv-cinder
validation-count=0
[features/marker.gv-cinder-marker - Memory usage]
num_types=124
[features/marker.gv-cinder-marker - usage-type gf_common_mt_asprintf memusage]
size=32
num_allocs=1
max_size=115
max_num_allocs=2
total_allocs=1745
[features/marker.gv-cinder-marker - usage-type gf_common_mt_strdup memusage]
size=0
num_allocs=0
max_size=42
max_num_allocs=2
total_allocs=25101
[features/marker.gv-cinder-marker - usage-type gf_common_mt_char memusage]
size=0
num_allocs=0
max_size=99
max_num_allocs=1
total_allocs=1744
[features/marker.gv-cinder-marker - usage-type gf_common_mt_mem_pool memusage]
size=120
num_allocs=1
max_size=120
max_num_allocs=1
total_allocs=1
[features/marker.gv-cinder-marker - usage-type gf_common_mt_long memusage]
size=42496
num_allocs=1
max_size=42496
max_num_allocs=1
total_allocs=1
[features/marker.gv-cinder-marker - usage-type gf_marker_mt_marker_conf_t memusage]
size=80
num_allocs=1
max_size=80
max_num_allocs=1
total_allocs=1
[features/index.gv-cinder-index - Memory usage]
num_types=120
[features/index.gv-cinder-index - usage-type gf_common_mt_gf_dirent_t memusage]
size=0
num_allocs=0
max_size=3826
max_num_allocs=19
total_allocs=25757
[features/index.gv-cinder-index - usage-type gf_common_mt_strdup memusage]
size=0
num_allocs=0
max_size=162
max_num_allocs=3
total_allocs=21020
[features/index.gv-cinder-index - usage-type gf_common_mt_char memusage]
size=0
num_allocs=0
max_size=29
max_num_allocs=1
total_allocs=4981
[features/index.gv-cinder-index - usage-type gf_index_mt_priv_t memusage]
size=152
num_allocs=1
max_size=152
max_num_allocs=1
total_allocs=1
[features/index.gv-cinder-index - usage-type gf_index_inode_ctx_t memusage]
size=160
num_allocs=5
max_size=192
max_num_allocs=6
total_allocs=791
[features/index.gv-cinder-index - usage-type gf_index_fd_ctx_t memusage]
size=0
num_allocs=0
max_size=32
max_num_allocs=2
total_allocs=4981
[features/barrier.gv-cinder-barrier - Memory usage]
num_types=118
[features/barrier.gv-cinder-barrier - usage-type gf_common_mt_asprintf memusage]
size=0
num_allocs=0
max_size=96
max_num_allocs=1
total_allocs=4
[features/barrier.gv-cinder-barrier - usage-type gf_common_mt_char memusage]
size=0
num_allocs=0
max_size=138
max_num_allocs=1
total_allocs=4
[features/barrier.gv-cinder-barrier - usage-type gf_barrier_mt_priv_t memusage]
size=56
num_allocs=1
max_size=56
max_num_allocs=1
total_allocs=1
[xlator.features.barrier.priv]
barrier.enabled=0
barrier.timeout=120
[performance/io-threads.gv-cinder-io-threads - Memory usage]
num_types=118
[performance/io-threads.gv-cinder-io-threads - usage-type gf_common_mt_iovec memusage]
size=0
num_allocs=0
max_size=208
max_num_allocs=13
total_allocs=303078
[performance/io-threads.gv-cinder-io-threads - usage-type gf_common_mt_asprintf memusage]
size=0
num_allocs=0
max_size=176
max_num_allocs=2
total_allocs=410217
[performance/io-threads.gv-cinder-io-threads - usage-type gf_common_mt_strdup memusage]
size=0
num_allocs=0
max_size=212
max_num_allocs=8
total_allocs=169882954
[performance/io-threads.gv-cinder-io-threads - usage-type gf_common_mt_char memusage]
size=0
num_allocs=0
max_size=238
max_num_allocs=2
total_allocs=410217
[performance/io-threads.gv-cinder-io-threads - usage-type gf_iot_mt_iot_conf_t memusage]
size=376
num_allocs=1
max_size=376
max_num_allocs=1
total_allocs=1
[performance/io-threads.gv-cinder-io-threads]
maximum_threads_count=16
current_threads_count=1
sleep_count=1
idle_time=120
stack_size=1048576
high_priority_threads=16
normal_priority_threads=16
low_priority_threads=16
least_priority_threads=1
cached least rate=24
least rate limit=0
[features/locks.gv-cinder-locks - Memory usage]
num_types=126
[features/locks.gv-cinder-locks - usage-type gf_common_mt_asprintf memusage]
size=27
num_allocs=1
max_size=145
max_num_allocs=7
total_allocs=271430
[features/locks.gv-cinder-locks - usage-type gf_common_mt_strdup memusage]
size=286
num_allocs=10
max_size=4023
max_num_allocs=66
total_allocs=84845125
[features/locks.gv-cinder-locks - usage-type gf_common_mt_char memusage]
size=0
num_allocs=0
max_size=201
max_num_allocs=6
total_allocs=251416
[features/locks.gv-cinder-locks - usage-type gf_common_mt_mem_pool memusage]
size=120
num_allocs=1
max_size=120
max_num_allocs=1
total_allocs=1
[features/locks.gv-cinder-locks - usage-type gf_common_mt_long memusage]
size=4736
num_allocs=1
max_size=4736
max_num_allocs=1
total_allocs=1
[features/locks.gv-cinder-locks - usage-type gf_locks_mt_pl_dom_list_t memusage]
size=880
num_allocs=10
max_size=880
max_num_allocs=10
total_allocs=10
[features/locks.gv-cinder-locks - usage-type gf_locks_mt_pl_inode_t memusage]
size=2112
num_allocs=12
max_size=2288
max_num_allocs=13
total_allocs=886
[features/locks.gv-cinder-locks - usage-type gf_locks_mt_posix_lock_t memusage]
size=504
num_allocs=7
max_size=576
max_num_allocs=8
total_allocs=25
[features/locks.gv-cinder-locks - usage-type gf_locks_mt_pl_entry_lock_t memusage]
size=0
num_allocs=0
max_size=3552
max_num_allocs=3
total_allocs=13410
[features/locks.gv-cinder-locks - usage-type gf_locks_mt_pl_inode_lock_t memusage]
size=0
num_allocs=0
max_size=125440
max_num_allocs=56
total_allocs=84709087
[features/locks.gv-cinder-locks - usage-type gf_locks_mt_posix_locks_private_t memusage]
size=16
num_allocs=1
max_size=16
max_num_allocs=1
total_allocs=1
[features/locks.gv-cinder-locks - usage-type gf_locks_mt_pl_fdctx_t memusage]
size=48
num_allocs=3
max_size=96
max_num_allocs=6
total_allocs=19966
[features/access-control.gv-cinder-access-control - Memory usage]
num_types=121
[features/access-control.gv-cinder-access-control - usage-type gf_common_mt_asprintf memusage]
size=0
num_allocs=0
max_size=20
max_num_allocs=10
total_allocs=238292
[features/access-control.gv-cinder-access-control - usage-type gf_common_mt_char memusage]
size=0
num_allocs=0
max_size=245
max_num_allocs=10
total_allocs=237822
[features/access-control.gv-cinder-access-control - usage-type gf_posix_acl_mt_ctx_t memusage]
size=992
num_allocs=31
max_size=1024
max_num_allocs=32
total_allocs=921
[features/access-control.gv-cinder-access-control - usage-type gf_posix_acl_mt_posix_ace_t memusage]
size=32
num_allocs=1
max_size=32
max_num_allocs=1
total_allocs=1
[features/access-control.gv-cinder-access-control - usage-type gf_posix_acl_mt_conf_t memusage]
size=16
num_allocs=1
max_size=16
max_num_allocs=1
total_allocs=1
[features/changelog.gv-cinder-changelog - Memory usage]
num_types=127
[features/changelog.gv-cinder-changelog - usage-type gf_common_mt_asprintf memusage]
size=38
num_allocs=1
max_size=136
max_num_allocs=2
total_allocs=7
[features/changelog.gv-cinder-changelog - usage-type gf_common_mt_strdup memusage]
size=84
num_allocs=2
max_size=84
max_num_allocs=2
total_allocs=6
[features/changelog.gv-cinder-changelog - usage-type gf_common_mt_char memusage]
size=0
num_allocs=0
max_size=186
max_num_allocs=1
total_allocs=6
[features/changelog.gv-cinder-changelog - usage-type gf_common_mt_mem_pool memusage]
size=120
num_allocs=1
max_size=120
max_num_allocs=1
total_allocs=1
[features/changelog.gv-cinder-changelog - usage-type gf_common_mt_long memusage]
size=7424
num_allocs=1
max_size=7424
max_num_allocs=1
total_allocs=1
[features/changelog.gv-cinder-changelog - usage-type gf_changelog_mt_priv_t memusage]
size=760
num_allocs=1
max_size=760
max_num_allocs=1
total_allocs=1
[features/changelog.gv-cinder-changelog - usage-type gf_changelog_mt_rt_t memusage]
size=4
num_allocs=1
max_size=4
max_num_allocs=1
total_allocs=1
[storage/posix.gv-cinder-posix - Memory usage]
num_types=125
[storage/posix.gv-cinder-posix - usage-type gf_common_mt_gf_dirent_t memusage]
size=0
num_allocs=0
max_size=4055
max_num_allocs=19
total_allocs=494
[storage/posix.gv-cinder-posix - usage-type gf_common_mt_inode_ctx memusage]
size=7488
num_allocs=26
max_size=7776
max_num_allocs=27
total_allocs=39
[storage/posix.gv-cinder-posix - usage-type gf_common_mt_iobref memusage]
size=0
num_allocs=0
max_size=144
max_num_allocs=6
total_allocs=48833565
[storage/posix.gv-cinder-posix - usage-type gf_common_mt_asprintf memusage]
size=0
num_allocs=0
max_size=95
max_num_allocs=7
total_allocs=201906
[storage/posix.gv-cinder-posix - usage-type gf_common_mt_strdup memusage]
size=31
num_allocs=1
max_size=166
max_num_allocs=2
total_allocs=5110
[storage/posix.gv-cinder-posix - usage-type gf_common_mt_char memusage]
size=256
num_allocs=1
max_size=691
max_num_allocs=16
total_allocs=522677
[storage/posix.gv-cinder-posix - usage-type gf_common_mt_iobrefs memusage]
size=0
num_allocs=0
max_size=768
max_num_allocs=6
total_allocs=48833565
[storage/posix.gv-cinder-posix - usage-type gf_posix_mt_posix_fd memusage]
size=96
num_allocs=2
max_size=240
max_num_allocs=5
total_allocs=22484
[storage/posix.gv-cinder-posix - usage-type gf_posix_mt_char memusage]
size=0
num_allocs=0
max_size=345
max_num_allocs=15
total_allocs=1011303
[storage/posix.gv-cinder-posix - usage-type gf_posix_mt_posix_private memusage]
size=608
num_allocs=1
max_size=608
max_num_allocs=1
total_allocs=1
[storage/posix.gv-cinder-posix - usage-type gf_posix_mt_trash_path memusage]
size=51
num_allocs=1
max_size=51
max_num_allocs=1
total_allocs=1
[storage/posix.gv-cinder-posix]
base_path=/gluster/brick03/cinder-std-01
base_path_length=30
max_read=-220679111
max_write=71495680
nr_files=-2516
DUMP-END-TIME: 2016-02-13 02:42:44.674718
More information about the Gluster-devel
mailing list