[Bugs] [Bug 1233136] New: Libgfapi client program crashes during glfs_fini() because io-cache xlator is leaking iobufs

Thu Jun 18 09:55:08 UTC 2015

https://bugzilla.redhat.com/show_bug.cgi?id=1233136

            Bug ID: 1233136
           Summary: Libgfapi client program crashes during glfs_fini()
                    because io-cache xlator is leaking iobufs
           Product: GlusterFS
           Version: mainline
         Component: io-cache
          Severity: high
          Assignee: bugs at gluster.org
          Reporter: ppai at redhat.com
                CC: bugs at gluster.org, gluster-bugs at redhat.com

Created attachment 1040398
  --> https://bugzilla.redhat.com/attachment.cgi?id=1040398&action=edit
Sample program to reproduce the crash

Description of problem:
I have a binary file (a PNG picture). I read the first four bytes which works
as expected. During glfs_fini() which frees resources, the program crashes.

Workaround:
This crash is not observed when io-cache xlator is turned off
# gluster volume set test io-cache off

Always reproducible with attached program and binary file given below.

Setup:

[root at fed22 glusterfs]# uname -a
Linux fed22 4.0.4-303.fc22.x86_64 #1 SMP Thu May 28 12:37:06 UTC 2015 x86_64
x86_64 x86_64 GNU/Linux

[root at fed22 glusterfs]# gcc --version
gcc (GCC) 5.1.1 20150422 (Red Hat 5.1.1-1)
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[root at fed22 glusterfs]# gluster volume info

Volume Name: test
Type: Distributed-Replicate
Volume ID: e52a4cfe-9af8-40cf-ad20-71522bfc26cb
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: fed22:/export/brick1/b
Brick2: fed22:/export/brick2/b
Brick3: fed22:/export/brick3/b
Brick4: fed22:/export/brick4/b
Options Reconfigured:
performance.quick-read: off
performance.read-ahead: off
performance.write-behind: off
performance.open-behind: off
performance.io-cache: on
performance.readdir-ahead: off
server.allow-insecure: on
[root at fed22 glusterfs]# 

Steps to Reproduce:

# mount -t glusterfs localhost:test /mnt
# cd /mnt
# wget https://upload.wikimedia.org/wikipedia/en/2/24/Lenna.png
# umount /mnt

# gcc crash.c -I /usr/local/include/glusterfs/ -lgfapi -O0 -ggdb -o ./crash
# ./crash

Actual results:
[root at fed22 ~]# ./crash
read returned: 4
crash: iobuf.c:128: __iobuf_arena_destroy_iobufs: Assertion `iobuf->ref == 0'
failed.
Aborted (core dumped)

Expected results: The client program should not crash.
[root at fed22 ~]# ./crash
read returned: 4
glfs_fini returned: 0

Debug core:
[root at fed22 ~]# gdb crash
GNU gdb (GDB) Fedora 7.9.1-13.fc22
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from crash...done.
(gdb) r
Starting program: /root/crash 
Missing separate debuginfos, use: dnf debuginfo-install
glibc-2.21-5.fc22.x86_64
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7fffedbe2700 (LWP 26296)]
[New Thread 0x7ffff4a9d700 (LWP 26295)]
[New Thread 0x7ffff529e700 (LWP 26294)]
[New Thread 0x7fffeb12a700 (LWP 26301)]
read returned: 4
[Thread 0x7ffff529e700 (LWP 26294) exited]
[Thread 0x7ffff4a9d700 (LWP 26295) exited]
[Thread 0x7fffeb12a700 (LWP 26301) exited]
[Thread 0x7fffedbe2700 (LWP 26296) exited]
crash: iobuf.c:128: __iobuf_arena_destroy_iobufs: Assertion `iobuf->ref == 0'
failed.

Program received signal SIGABRT, Aborted.
0x00007ffff782fa98 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install
keyutils-libs-1.5.9-4.fc22.x86_64 krb5-libs-1.13.1-3.fc22.x86_64
libacl-2.2.52-7.fc22.x86_64 libattr-2.4.47-9.fc22.x86_64
libcom_err-1.42.12-4.fc22.x86_64 libselinux-2.3-10.fc22.x86_64
openssl-libs-1.0.1k-8.fc22.x86_64 pcre-8.37-1.fc22.x86_64
zlib-1.2.8-7.fc22.x86_64
(gdb) bt
#0  0x00007ffff782fa98 in raise () from /lib64/libc.so.6
#1  0x00007ffff783172a in abort () from /lib64/libc.so.6
#2  0x00007ffff7828247 in __assert_fail_base () from /lib64/libc.so.6
#3  0x00007ffff78282f2 in __assert_fail () from /lib64/libc.so.6
#4  0x00007ffff6f3ce3f in __iobuf_arena_destroy_iobufs (iobuf_arena=0x6201b0)
at iobuf.c:128
#5  0x00007ffff6f3cf2f in __iobuf_arena_destroy (iobuf_pool=0x605060,
iobuf_arena=0x6201b0) at iobuf.c:152
#6  0x00007ffff6f3d660 in iobuf_pool_destroy (iobuf_pool=0x605060) at
iobuf.c:321
#7  0x00007ffff7bc33a0 in glusterfs_ctx_destroy (ctx=0x602190) at glfs.c:973
#8  0x00007ffff7bc38e9 in pub_glfs_fini (fs=0x602010) at glfs.c:1150
#9  0x0000000000400a89 in main () at crash.c:27
(gdb) f 4
#4  0x00007ffff6f3ce3f in __iobuf_arena_destroy_iobufs (iobuf_arena=0x6201b0)
at iobuf.c:128
128                    GF_ASSERT (iobuf->ref == 0);
(gdb) p *iobuf
$1 = {{list = {next = 0x620208, prev = 0x620208}, {next = 0x620208, prev =
0x620208}}, iobuf_arena = 0x6201b0, lock = 1, ref = 1, ptr = 0x7ffff5a5f000,
free_ptr = 0x0}
(gdb) 

Possible reason:
io-cache xlator is taking a ref somewhere and does not unref.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.