[Gluster-devel] Profiling GlusterFS FUSE client with Valgrind's Massif tool

[Oleksandr Natalenko]

More info here.

Massif puts the following warning on volume unmount:

===
valgrind: m_mallocfree.c:304 (get_bszB_as_is): Assertion 'bszB_lo == 
bszB_hi' failed.
valgrind: Heap block lo/hi size mismatch: lo = 1, hi = 0.
This is probably caused by your program erroneously writing past the
end of a heap block and corrupting heap metadata.  If you fix any
invalid writes reported by Memcheck, this assertion failure will
probably go away.  Please try that before reporting this as a bug.
...
Thread 1: status = VgTs_Runnable
==30590==    at 0x4C29037: free (in 
/usr/lib64/valgrind/vgpreload_massif-amd64-linux.so)
==30590==    by 0x67CE63B: __libc_freeres (in /usr/lib64/libc-2.17.so)
==30590==    by 0x4A246B4: _vgnU_freeres (in 
/usr/lib64/valgrind/vgpreload_core-amd64-linux.so)
==30590==    by 0x66A2E2A: __run_exit_handlers (in 
/usr/lib64/libc-2.17.so)
==30590==    by 0x66A2EB4: exit (in /usr/lib64/libc-2.17.so)
==30590==    by 0x1117E9: cleanup_and_exit (glusterfsd.c:1308)
==30590==    by 0x669F66F: ??? (in /usr/lib64/libc-2.17.so)
==30590==    by 0x606EEF4: pthread_join (in 
/usr/lib64/libpthread-2.17.so)
==30590==    by 0x4EC2687: event_dispatch_epoll (event-epoll.c:762)
==30590==    by 0x10E876: main (glusterfsd.c:2370)
...
===

I rechecked mount/ls/unmount with memcheck tool as suggested and got the 
following:

===
...
==30315== Thread 8:
==30315== Syscall param writev(vector[...]) points to uninitialised 
byte(s)
==30315==    at 0x675FEA0: writev (in /usr/lib64/libc-2.17.so)
==30315==    by 0xE664795: send_fuse_iov (fuse-bridge.c:158)
==30315==    by 0xE6649B9: send_fuse_data (fuse-bridge.c:197)
==30315==    by 0xE666F7A: fuse_attr_cbk (fuse-bridge.c:753)
==30315==    by 0xE6671A6: fuse_root_lookup_cbk (fuse-bridge.c:783)
==30315==    by 0x14519937: io_stats_lookup_cbk (io-stats.c:1512)
==30315==    by 0x14300B3E: mdc_lookup_cbk (md-cache.c:867)
==30315==    by 0x13EE9226: qr_lookup_cbk (quick-read.c:446)
==30315==    by 0x13CD8B66: ioc_lookup_cbk (io-cache.c:260)
==30315==    by 0x1346405D: dht_revalidate_cbk (dht-common.c:985)
==30315==    by 0x1320EC60: afr_discover_done (afr-common.c:2316)
==30315==    by 0x1320EC60: afr_discover_cbk (afr-common.c:2361)
==30315==    by 0x12F9EE91: client3_3_lookup_cbk 
(client-rpc-fops.c:2981)
==30315==  Address 0x170b238c is on thread 8's stack
==30315==  in frame #3, created by fuse_attr_cbk (fuse-bridge.c:723)
...
==30315== Warning: invalid file descriptor -1 in syscall close()
==30315== Thread 1:
==30315== Invalid free() / delete / delete[] / realloc()
==30315==    at 0x4C2AD17: free (in 
/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==30315==    by 0x67D663B: __libc_freeres (in /usr/lib64/libc-2.17.so)
==30315==    by 0x4A246B4: _vgnU_freeres (in 
/usr/lib64/valgrind/vgpreload_core-amd64-linux.so)
==30315==    by 0x66AAE2A: __run_exit_handlers (in 
/usr/lib64/libc-2.17.so)
==30315==    by 0x66AAEB4: exit (in /usr/lib64/libc-2.17.so)
==30315==    by 0x1117E9: cleanup_and_exit (glusterfsd.c:1308)
==30315==    by 0x66A766F: ??? (in /usr/lib64/libc-2.17.so)
==30315==    by 0x6076EF4: pthread_join (in 
/usr/lib64/libpthread-2.17.so)
==30315==    by 0x4ECA687: event_dispatch_epoll (event-epoll.c:762)
==30315==    by 0x10E876: main (glusterfsd.c:2370)
==30315==  Address 0x6a2d3d0 is 0 bytes inside data symbol 
"noai6ai_cached"
===

It seems Massif crashes (?) because of invalid memory access in 
glusterfs process cleanup stage.

Pranith? Nithya?

29.08.2016 13:14, Oleksandr Natalenko wrote:
> ===
> valgrind --tool=massif --trace-children=yes /usr/sbin/glusterfs -N
> --volfile-server=server.example.com --volfile-id=test
> /mnt/net/glusterfs/test
> ===