[Gluster-devel] Profiling GlusterFS FUSE client with Valgrind's Massif tool

[Pranith Kumar Karampuri]

I included you on a thread on users, let us see if he can help you out.

On Mon, Aug 29, 2016 at 4:02 PM, Oleksandr Natalenko <
oleksandr at natalenko.name> wrote:

> More info here.
>
> Massif puts the following warning on volume unmount:
>
> ===
> valgrind: m_mallocfree.c:304 (get_bszB_as_is): Assertion 'bszB_lo ==
> bszB_hi' failed.
> valgrind: Heap block lo/hi size mismatch: lo = 1, hi = 0.
> This is probably caused by your program erroneously writing past the
> end of a heap block and corrupting heap metadata.  If you fix any
> invalid writes reported by Memcheck, this assertion failure will
> probably go away.  Please try that before reporting this as a bug.
> ...
> Thread 1: status = VgTs_Runnable
> ==30590==    at 0x4C29037: free (in /usr/lib64/valgrind/vgpreload_
> massif-amd64-linux.so)
> ==30590==    by 0x67CE63B: __libc_freeres (in /usr/lib64/libc-2.17.so)
> ==30590==    by 0x4A246B4: _vgnU_freeres (in /usr/lib64/valgrind/vgpreload_
> core-amd64-linux.so)
> ==30590==    by 0x66A2E2A: __run_exit_handlers (in /usr/lib64/libc-2.17.so
> )
> ==30590==    by 0x66A2EB4: exit (in /usr/lib64/libc-2.17.so)
> ==30590==    by 0x1117E9: cleanup_and_exit (glusterfsd.c:1308)
> ==30590==    by 0x669F66F: ??? (in /usr/lib64/libc-2.17.so)
> ==30590==    by 0x606EEF4: pthread_join (in /usr/lib64/libpthread-2.17.so)
> ==30590==    by 0x4EC2687: event_dispatch_epoll (event-epoll.c:762)
> ==30590==    by 0x10E876: main (glusterfsd.c:2370)
> ...
> ===
>
> I rechecked mount/ls/unmount with memcheck tool as suggested and got the
> following:
>
> ===
> ...
> ==30315== Thread 8:
> ==30315== Syscall param writev(vector[...]) points to uninitialised byte(s)
> ==30315==    at 0x675FEA0: writev (in /usr/lib64/libc-2.17.so)
> ==30315==    by 0xE664795: send_fuse_iov (fuse-bridge.c:158)
> ==30315==    by 0xE6649B9: send_fuse_data (fuse-bridge.c:197)
> ==30315==    by 0xE666F7A: fuse_attr_cbk (fuse-bridge.c:753)
> ==30315==    by 0xE6671A6: fuse_root_lookup_cbk (fuse-bridge.c:783)
> ==30315==    by 0x14519937: io_stats_lookup_cbk (io-stats.c:1512)
> ==30315==    by 0x14300B3E: mdc_lookup_cbk (md-cache.c:867)
> ==30315==    by 0x13EE9226: qr_lookup_cbk (quick-read.c:446)
> ==30315==    by 0x13CD8B66: ioc_lookup_cbk (io-cache.c:260)
> ==30315==    by 0x1346405D: dht_revalidate_cbk (dht-common.c:985)
> ==30315==    by 0x1320EC60: afr_discover_done (afr-common.c:2316)
> ==30315==    by 0x1320EC60: afr_discover_cbk (afr-common.c:2361)
> ==30315==    by 0x12F9EE91: client3_3_lookup_cbk (client-rpc-fops.c:2981)
> ==30315==  Address 0x170b238c is on thread 8's stack
> ==30315==  in frame #3, created by fuse_attr_cbk (fuse-bridge.c:723)
> ...
> ==30315== Warning: invalid file descriptor -1 in syscall close()
> ==30315== Thread 1:
> ==30315== Invalid free() / delete / delete[] / realloc()
> ==30315==    at 0x4C2AD17: free (in /usr/lib64/valgrind/vgpreload_
> memcheck-amd64-linux.so)
> ==30315==    by 0x67D663B: __libc_freeres (in /usr/lib64/libc-2.17.so)
> ==30315==    by 0x4A246B4: _vgnU_freeres (in /usr/lib64/valgrind/vgpreload_
> core-amd64-linux.so)
> ==30315==    by 0x66AAE2A: __run_exit_handlers (in /usr/lib64/libc-2.17.so
> )
> ==30315==    by 0x66AAEB4: exit (in /usr/lib64/libc-2.17.so)
> ==30315==    by 0x1117E9: cleanup_and_exit (glusterfsd.c:1308)
> ==30315==    by 0x66A766F: ??? (in /usr/lib64/libc-2.17.so)
> ==30315==    by 0x6076EF4: pthread_join (in /usr/lib64/libpthread-2.17.so)
> ==30315==    by 0x4ECA687: event_dispatch_epoll (event-epoll.c:762)
> ==30315==    by 0x10E876: main (glusterfsd.c:2370)
> ==30315==  Address 0x6a2d3d0 is 0 bytes inside data symbol "noai6ai_cached"
> ===
>
> It seems Massif crashes (?) because of invalid memory access in glusterfs
> process cleanup stage.
>
> Pranith? Nithya?
>
> 29.08.2016 13:14, Oleksandr Natalenko wrote:
>
>> ===
>> valgrind --tool=massif --trace-children=yes /usr/sbin/glusterfs -N
>> --volfile-server=server.example.com --volfile-id=test
>> /mnt/net/glusterfs/test
>> ===
>>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
>



-- 
Pranith
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20160906/7abd0381/attachment.html>