[Bugs] [Bug 1233136] Libgfapi client program crashes during glfs_fini() because io-cache xlator is leaking iobufs

bugzilla at redhat.com bugzilla at redhat.com
Tue Sep 29 09:44:29 UTC 2015 

https://bugzilla.redhat.com/show_bug.cgi?id=1233136

Soumya Koduri <skoduri at redhat.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |mchangir at redhat.com



--- Comment #2 from Soumya Koduri <skoduri at redhat.com> ---
Milind has seen similar crash while trying to take backup from gluster volumes
using baroeas. And we have observed that there is no issue when io-cache xlator
is disabled.


After adding few debug statements in iobref_ref & iobref_unref (attached the
log file),
just before the crash -->

[2015-09-29 08:45:47.623190] E [MSGID: 101039] [iobuf.c:536:__iobuf_ref]
0-sun-client-0: $$$$$$$$$$$$$$$$$ in iobuf_ref iobuf(0x7fbb9402bae0), ref(1)
/usr/local/lib/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xf2)[0x7fbb9d4f0c52]
/usr/local/lib/libglusterfs.so.0(__iobuf_ref+0x8b)[0x7fbb9d53527b]
/usr/local/lib/libglusterfs.so.0(iobuf_ref+0x87)[0x7fbb9d535e54]
/usr/local/lib/libglusterfs.so.0(iobuf_get2+0x12b)[0x7fbb9d53577e]
/usr/local/lib/libgfrpc.so.0(rpc_clnt_record_build_record+0x1f1)[0x7fbb9d9d9422]
/usr/local/lib/libgfrpc.so.0(rpc_clnt_record+0x2b3)[0x7fbb9d9d97ba]
/usr/local/lib/libgfrpc.so.0(rpc_clnt_submit+0x1e5)[0x7fbb9d9d9c68]
/usr/local/lib/glusterfs/3.8dev/xlator/protocol/client.so(client_submit_request+0x540)[0x7fbb86dc0ef3]
/usr/local/lib/glusterfs/3.8dev/xlator/protocol/client.so(client3_3_getxattr+0x53a)[0x7fbb86de4e10]
/usr/local/lib/glusterfs/3.8dev/xlator/protocol/client.so(client_getxattr+0xee)[0x7fbb86dc5d1f]
/usr/local/lib/glusterfs/3.8dev/xlator/cluster/distribute.so(dht_getxattr+0x165f)[0x7fbb86b6ee09]
/usr/local/lib/libglusterfs.so.0(default_getxattr+0xa5)[0x7fbb9d5089d0]
/usr/local/lib/libglusterfs.so.0(default_getxattr+0xa5)[0x7fbb9d5089d0]
/usr/local/lib/libglusterfs.so.0(default_getxattr+0xa5)[0x7fbb9d5089d0]
/usr/local/lib/libglusterfs.so.0(default_getxattr+0xa5)[0x7fbb9d5089d0]
/usr/local/lib/libglusterfs.so.0(default_getxattr+0xa5)[0x7fbb9d5089d0]
/usr/local/lib/libglusterfs.so.0(default_getxattr+0xa5)[0x7fbb9d5089d0]
/usr/local/lib/glusterfs/3.8dev/xlator/performance/md-cache.so(mdc_getxattr+0x462)[0x7fbb85cd7956]
/usr/local/lib/glusterfs/3.8dev/xlator/debug/io-stats.so(io_stats_getxattr+0x23a)[0x7fbb85ac01cc]
/usr/local/lib/libglusterfs.so.0(default_getxattr+0xa5)[0x7fbb9d5089d0]
/usr/local/lib/libglusterfs.so.0(syncop_getxattr+0x2fb)[0x7fbb9d54aab7]
/usr/local/lib/libgfapi.so.0(+0x12de7)[0x7fbb9dbf8de7]
/usr/local/lib/libgfapi.so.0(glfs_llistxattr+0x36)[0x7fbb9dbf8f0d]
/usr/lib64/bareos/plugins/gfapi-fd.so(+0x6a13)[0x7fbb9de0ea13]
/usr/sbin/bareos-fd[0x417828]
/usr/sbin/bareos-fd[0x40d1d8]
/usr/sbin/bareos-fd[0x415f6f]
/usr/lib64/libbareosfind-15.3.0.so(_Z10find_filesP3JCRP6FF_PKTPFiS0_S2_bES4_+0x385)[0x7fbba8478905]
/usr/sbin/bareos-fd[0x40b435]
/usr/sbin/bareos-fd[0x41200f]
/usr/sbin/bareos-fd[0x413665]
/usr/sbin/bareos-fd[0x41cefc]
/usr/lib64/libbareos-15.3.0.so(workq_server+0x1f5)[0x7fbba7e35ec5]
/usr/lib64/libbareos-15.3.0.so(lmgr_thread_launcher+0x5f)[0x7fbba7e1f73f]
/usr/lib64/libpthread.so.0(+0x7555)[0x7fbba7369555]
/usr/lib64/libc.so.6(clone+0x6d)[0x7fbba65ffb9d]
[2015-09-29 08:45:47.623944] T [rpc-clnt.c:1256:rpc_clnt_record_build_header]
0-rpc-clnt: Request fraglen 96, payload: 28, rpc hdr: 68
[2015-09-29 08:45:47.623964] E [MSGID: 101039] [iobuf.c:536:__iobuf_ref]
0-sun-client-0: $$$$$$$$$$$$$$$$$ in iobuf_ref iobuf(0x7fbb9402bae0), ref(2)
/usr/local/lib/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xf2)[0x7fbb9d4f0c52]
/usr/local/lib/libglusterfs.so.0(__iobuf_ref+0x8b)[0x7fbb9d53527b]
/usr/local/lib/libglusterfs.so.0(iobuf_ref+0x87)[0x7fbb9d535e54]
/usr/local/lib/libglusterfs.so.0(__iobref_add+0x164)[0x7fbb9d53645d]
/usr/local/lib/libglusterfs.so.0(iobref_add+0xf7)[0x7fbb9d53658a]
/usr/local/lib/libgfrpc.so.0(rpc_clnt_submit+0x244)[0x7fbb9d9d9cc7]
/usr/local/lib/glusterfs/3.8dev/xlator/protocol/client.so(client_submit_request+0x540)[0x7fbb86dc0ef3]
/usr/local/lib/glusterfs/3.8dev/xlator/protocol/client.so(client3_3_getxattr+0x53a)[0x7fbb86de4e10]
/usr/local/lib/glusterfs/3.8dev/xlator/protocol/client.so(client_getxattr+0xee)[0x7fbb86dc5d1f]
/usr/local/lib/glusterfs/3.8dev/xlator/cluster/distribute.so(dht_getxattr+0x165f)[0x7fbb86b6ee09]
/usr/local/lib/libglusterfs.so.0(default_getxattr+0xa5)[0x7fbb9d5089d0]
/usr/local/lib/libglusterfs.so.0(default_getxattr+0xa5)[0x7fbb9d5089d0]
/usr/local/lib/libglusterfs.so.0(default_getxattr+0xa5)[0x7fbb9d5089d0]
/usr/local/lib/libglusterfs.so.0(default_getxattr+0xa5)[0x7fbb9d5089d0]
/usr/local/lib/libglusterfs.so.0(default_getxattr+0xa5)[0x7fbb9d5089d0]
/usr/local/lib/libglusterfs.so.0(default_getxattr+0xa5)[0x7fbb9d5089d0]
/usr/local/lib/glusterfs/3.8dev/xlator/performance/md-cache.so(mdc_getxattr+0x462)[0x7fbb85cd7956]
/usr/local/lib/glusterfs/3.8dev/xlator/debug/io-stats.so(io_stats_getxattr+0x23a)[0x7fbb85ac01cc]
/usr/local/lib/libglusterfs.so.0(default_getxattr+0xa5)[0x7fbb9d5089d0]
/usr/local/lib/libglusterfs.so.0(syncop_getxattr+0x2fb)[0x7fbb9d54aab7]
/usr/local/lib/libgfapi.so.0(+0x12de7)[0x7fbb9dbf8de7]
/usr/local/lib/libgfapi.so.0(glfs_llistxattr+0x36)[0x7fbb9dbf8f0d]
/usr/lib64/bareos/plugins/gfapi-fd.so(+0x6a13)[0x7fbb9de0ea13]
/usr/sbin/bareos-fd[0x417828]
/usr/sbin/bareos-fd[0x40d1d8]
/usr/sbin/bareos-fd[0x415f6f]
/usr/lib64/libbareosfind-15.3.0.so(_Z10find_filesP3JCRP6FF_PKTPFiS0_S2_bES4_+0x385)[0x7fbba8478905]
/usr/sbin/bareos-fd[0x40b435]
/usr/sbin/bareos-fd[0x41200f]
/usr/sbin/bareos-fd[0x413665]
/usr/sbin/bareos-fd[0x41cefc]
/usr/lib64/libbareos-15.3.0.so(workq_server+0x1f5)[0x7fbba7e35ec5]
/usr/lib64/libbareos-15.3.0.so(lmgr_thread_launcher+0x5f)[0x7fbba7e1f73f]
/usr/lib64/libpthread.so.0(+0x7555)[0x7fbba7369555]
/usr/lib64/libc.so.6(clone+0x6d)[0x7fbba65ffb9d]
[2015-09-29 08:45:47.624264] T [rpc-clnt.c:1593:rpc_clnt_submit] 0-rpc-clnt:
submitted request (XID: 0x1e Program: GlusterFS 3.3, ProgVers: 330, Proc: 18)
to rpc-transport (sun-client-0)
[2015-09-29 08:45:47.624271] E [MSGID: 101039] [iobuf.c:549:__iobuf_unref]
0-sun-client-0: $$$$$$$$$$$$$$$$$ in iobuf_unref iobuf(0x7fbb9402bae0), ref(1)
/usr/local/lib/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xf2)[0x7fbb9d4f0c52]
/usr/local/lib/libglusterfs.so.0(__iobuf_unref+0x8b)[0x7fbb9d535310]
/usr/local/lib/libglusterfs.so.0(iobuf_unref+0x8e)[0x7fbb9d535d9d]
/usr/local/lib/libgfrpc.so.0(rpc_clnt_submit+0x493)[0x7fbb9d9d9f16]
/usr/local/lib/glusterfs/3.8dev/xlator/protocol/client.so(client_submit_request+0x540)[0x7fbb86dc0ef3]
/usr/local/lib/glusterfs/3.8dev/xlator/protocol/client.so(client3_3_getxattr+0x53a)[0x7fbb86de4e10]
/usr/local/lib/glusterfs/3.8dev/xlator/protocol/client.so(client_getxattr+0xee)[0x7fbb86dc5d1f]
/usr/local/lib/glusterfs/3.8dev/xlator/cluster/distribute.so(dht_getxattr+0x165f)[0x7fbb86b6ee09]
/usr/local/lib/libglusterfs.so.0(default_getxattr+0xa5)[0x7fbb9d5089d0]
/usr/local/lib/libglusterfs.so.0(default_getxattr+0xa5)[0x7fbb9d5089d0]
/usr/local/lib/libglusterfs.so.0(default_getxattr+0xa5)[0x7fbb9d5089d0]
/usr/local/lib/libglusterfs.so.0(default_getxattr+0xa5)[0x7fbb9d5089d0]
/usr/local/lib/libglusterfs.so.0(default_getxattr+0xa5)[0x7fbb9d5089d0]
/usr/local/lib/libglusterfs.so.0(default_getxattr+0xa5)[0x7fbb9d5089d0]
/usr/local/lib/glusterfs/3.8dev/xlator/performance/md-cache.so(mdc_getxattr+0x462)[0x7fbb85cd7956]
/usr/local/lib/glusterfs/3.8dev/xlator/debug/io-stats.so(io_stats_getxattr+0x23a)[0x7fbb85ac01cc]
/usr/local/lib/libglusterfs.so.0(default_getxattr+0xa5)[0x7fbb9d5089d0]
/usr/local/lib/libglusterfs.so.0(syncop_getxattr+0x2fb)[0x7fbb9d54aab7]
/usr/local/lib/libgfapi.so.0(+0x12de7)[0x7fbb9dbf8de7]
/usr/local/lib/libgfapi.so.0(glfs_llistxattr+0x36)[0x7fbb9dbf8f0d]
/usr/lib64/bareos/plugins/gfapi-fd.so(+0x6a13)[0x7fbb9de0ea13]
/usr/sbin/bareos-fd[0x417828]
/usr/sbin/bareos-fd[0x40d1d8]
/usr/sbin/bareos-fd[0x415f6f]
/usr/lib64/libbareosfind-15.3.0.so(_Z10find_filesP3JCRP6FF_PKTPFiS0_S2_bES4_+0x385)[0x7fbba8478905]
/usr/sbin/bareos-fd[0x40b435]
/usr/sbin/bareos-fd[0x41200f]
/usr/sbin/bareos-fd[0x413665]
/usr/sbin/bareos-fd[0x41cefc]
/usr/lib64/libbareos-15.3.0.so(workq_server+0x1f5)[0x7fbba7e35ec5]
/usr/lib64/libbareos-15.3.0.so(lmgr_thread_launcher+0x5f)[0x7fbba7e1f73f]
/usr/lib64/libpthread.so.0(+0x7555)[0x7fbba7369555]
/usr/lib64/libc.so.6(clone+0x6d)[0x7fbba65ffb9d]


There is a ref taken on iobuf(0x7fbb9402bae0) twice in rpc_clnt_submit. But it
has got unref'ed only once which led to the crash.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=mrmiKNtRfj&a=cc_unsubscribe

More information about the Bugs mailing list