[Gluster-users] [Gluster-devel] Profiling GlusterFS FUSE client with Valgrind's Massif tool

[Oleksandr Natalenko]

Created BZ for it [1].

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1373630

On вівторок, 6 вересня 2016 р. 23:32:51 EEST Pranith Kumar Karampuri wrote:
> I included you on a thread on users, let us see if he can help you out.
> 
> On Mon, Aug 29, 2016 at 4:02 PM, Oleksandr Natalenko <
> 
> oleksandr at natalenko.name> wrote:
> > More info here.
> > 
> > Massif puts the following warning on volume unmount:
> > 
> > ===
> > valgrind: m_mallocfree.c:304 (get_bszB_as_is): Assertion 'bszB_lo ==
> > bszB_hi' failed.
> > valgrind: Heap block lo/hi size mismatch: lo = 1, hi = 0.
> > This is probably caused by your program erroneously writing past the
> > end of a heap block and corrupting heap metadata.  If you fix any
> > invalid writes reported by Memcheck, this assertion failure will
> > probably go away.  Please try that before reporting this as a bug.
> > ...
> > Thread 1: status = VgTs_Runnable
> > ==30590==    at 0x4C29037: free (in /usr/lib64/valgrind/vgpreload_
> > massif-amd64-linux.so)
> > ==30590==    by 0x67CE63B: __libc_freeres (in /usr/lib64/libc-2.17.so)
> > ==30590==    by 0x4A246B4: _vgnU_freeres (in
> > /usr/lib64/valgrind/vgpreload_
> > core-amd64-linux.so)
> > ==30590==    by 0x66A2E2A: __run_exit_handlers (in /usr/lib64/libc-2.17.so
> > )
> > ==30590==    by 0x66A2EB4: exit (in /usr/lib64/libc-2.17.so)
> > ==30590==    by 0x1117E9: cleanup_and_exit (glusterfsd.c:1308)
> > ==30590==    by 0x669F66F: ??? (in /usr/lib64/libc-2.17.so)
> > ==30590==    by 0x606EEF4: pthread_join (in /usr/lib64/libpthread-2.17.so)
> > ==30590==    by 0x4EC2687: event_dispatch_epoll (event-epoll.c:762)
> > ==30590==    by 0x10E876: main (glusterfsd.c:2370)
> > ...
> > ===
> > 
> > I rechecked mount/ls/unmount with memcheck tool as suggested and got the
> > following:
> > 
> > ===
> > ...
> > ==30315== Thread 8:
> > ==30315== Syscall param writev(vector[...]) points to uninitialised
> > byte(s)
> > ==30315==    at 0x675FEA0: writev (in /usr/lib64/libc-2.17.so)
> > ==30315==    by 0xE664795: send_fuse_iov (fuse-bridge.c:158)
> > ==30315==    by 0xE6649B9: send_fuse_data (fuse-bridge.c:197)
> > ==30315==    by 0xE666F7A: fuse_attr_cbk (fuse-bridge.c:753)
> > ==30315==    by 0xE6671A6: fuse_root_lookup_cbk (fuse-bridge.c:783)
> > ==30315==    by 0x14519937: io_stats_lookup_cbk (io-stats.c:1512)
> > ==30315==    by 0x14300B3E: mdc_lookup_cbk (md-cache.c:867)
> > ==30315==    by 0x13EE9226: qr_lookup_cbk (quick-read.c:446)
> > ==30315==    by 0x13CD8B66: ioc_lookup_cbk (io-cache.c:260)
> > ==30315==    by 0x1346405D: dht_revalidate_cbk (dht-common.c:985)
> > ==30315==    by 0x1320EC60: afr_discover_done (afr-common.c:2316)
> > ==30315==    by 0x1320EC60: afr_discover_cbk (afr-common.c:2361)
> > ==30315==    by 0x12F9EE91: client3_3_lookup_cbk (client-rpc-fops.c:2981)
> > ==30315==  Address 0x170b238c is on thread 8's stack
> > ==30315==  in frame #3, created by fuse_attr_cbk (fuse-bridge.c:723)
> > ...
> > ==30315== Warning: invalid file descriptor -1 in syscall close()
> > ==30315== Thread 1:
> > ==30315== Invalid free() / delete / delete[] / realloc()
> > ==30315==    at 0x4C2AD17: free (in /usr/lib64/valgrind/vgpreload_
> > memcheck-amd64-linux.so)
> > ==30315==    by 0x67D663B: __libc_freeres (in /usr/lib64/libc-2.17.so)
> > ==30315==    by 0x4A246B4: _vgnU_freeres (in
> > /usr/lib64/valgrind/vgpreload_
> > core-amd64-linux.so)
> > ==30315==    by 0x66AAE2A: __run_exit_handlers (in /usr/lib64/libc-2.17.so
> > )
> > ==30315==    by 0x66AAEB4: exit (in /usr/lib64/libc-2.17.so)
> > ==30315==    by 0x1117E9: cleanup_and_exit (glusterfsd.c:1308)
> > ==30315==    by 0x66A766F: ??? (in /usr/lib64/libc-2.17.so)
> > ==30315==    by 0x6076EF4: pthread_join (in /usr/lib64/libpthread-2.17.so)
> > ==30315==    by 0x4ECA687: event_dispatch_epoll (event-epoll.c:762)
> > ==30315==    by 0x10E876: main (glusterfsd.c:2370)
> > ==30315==  Address 0x6a2d3d0 is 0 bytes inside data symbol
> > "noai6ai_cached"
> > ===
> > 
> > It seems Massif crashes (?) because of invalid memory access in glusterfs
> > process cleanup stage.
> > 
> > Pranith? Nithya?
> > 
> > 29.08.2016 13:14, Oleksandr Natalenko wrote:
> >> ===
> >> valgrind --tool=massif --trace-children=yes /usr/sbin/glusterfs -N
> >> --volfile-server=server.example.com --volfile-id=test
> >> /mnt/net/glusterfs/test
> >> ===
> > 
> > _______________________________________________
> > Gluster-devel mailing list
> > Gluster-devel at gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-devel