[Gluster-devel] Recent dict changes affecting QEMU-GlusterFS patches

[Jeff Darcy]

On Mon, 2012-06-18 at 09:33 +0530, Bharata B Rao wrote:
> Hi,
> 
> I recently posted patches to integrate GlusterFS with QEMU.
> (http://lists.nongnu.org/archive/html/qemu-devel/2012-06/msg01745.html).
> While updating those patches to latest gluster git, I am seeing a
> problem and I tracked that down to this commit:
> 
> e8eb0a9cb6539a7607d4c134daf331400a93d136 (Optimize for small dicts,
> and avoid an overrun).
> 
> With this commit, I see an invalid memory reference in _dict_lookup().
> Some details from gdb are shown below:

I've seen something like this before, when commonly used structures
(like dict_t) change.  It seems like somehow not all dependencies are
getting updated properly, resulting in a mix of code that uses the old
srtucture and code that uses the new one.  I don't know how such a
problem can survive the rpmbuild process, which I always use even during
development, but I have seen the symptoms disappear when I've carefully
nuked all GlusterFS source and binaries from my system to guarantee that
I'm starting fresh.

In any case, I'll look into this a bit further and see if it might be
something else.  The dict_t structure did change with that commit, as
did the usage of some fields, so if your code relies somehow on old
behavior then it's possible that an update is needed.

> [root at bharata qemu]# gdb ./x86_64-softmmu/qemu-system-x86_64
> (gdb) set args --enable-kvm --nographic -m 1024 -smp 4 -drive
> file=gluster:/home/bharata/c-qemu-rpcbypass.vol:/dir1/F16,format=gluster,cache=none
> -net nic,model=virtio -net user -redir tcp:2000::22
> (gdb) r
> Starting program: x86_64-softmmu/qemu-system-x86_64 --enable-kvm
> --nographic -m 1024 -smp 4 -drive
> file=gluster:/home/bharata/c-qemu-rpcbypass.vol:/dir1/F16,format=gluster,cache=none
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib64/libthread_db.so.1".
> 
> Program received signal SIGSEGV, Segmentation fault.
> 0x00007ffff66e8ff6 in __strcmp_sse42 () from /lib64/libc.so.6
> Missing separate debuginfos, use: debuginfo-install
> glib2-2.30.3-1.fc16.x86_64 glibc-2.14.90-24.fc16.7.x86_64
> libuuid-2.20.1-2.3.fc16.x86_64 openssl-1.0.0j-1.fc16.x86_64
> zlib-1.2.5-6.fc16.x86_64
> (gdb) bt
> #0  0x00007ffff66e8ff6 in __strcmp_sse42 () from /lib64/libc.so.6
> #1  0x00007ffff7241ab1 in _dict_lookup (key=0x5555564e11b0 "directory",
>     this=<optimized out>) at dict.c:204
> #2  _dict_lookup (this=<optimized out>, key=0x5555564e11b0
> "directory") at dict.c:192
> #3  0x00007ffff72427ae in _dict_set (value=0x7ffff534302c, key=
>     0x5555564e11b0 "directory", this=0x5555564c6c6c) at dict.c:254
> #4  dict_set (value=0x7ffff534302c, key=<optimized out>, this=0x5555564c6c6c)
>     at dict.c:327
> #5  dict_set (this=0x5555564c6c6c, key=<optimized out>, value=0x7ffff534302c)
>     at dict.c:313
> #6  0x00007ffff728c2a8 in volume_option (value=0x5555564e2470 "/vm", key=
>     0x5555564e11b0 "directory") at ./graph.y:249
> #7  yyparse () at ./graph.y:76
> #8  0x00007ffff728cbbc in glusterfs_graph_construct
> (fp=0x5555564dcbe0) at ./graph.y:597
> <snipped>
> 
> (gdb) up
> #1  0x00007ffff7241ab1 in _dict_lookup (key=0x5555564e11b0 "directory",
>     this=<optimized out>) at dict.c:204
> 204                     if (pair->key && !strcmp (pair->key, key))
> (gdb) p *pair
> $1 = {hash_next = 0x5555564c6ca4, prev = 0x5555564dbbfc, next =
> 0x3ff00000001, value =
>     0x1, key = 0x54 <Address 0x54 out of bounds>}
> 
> You can see that pair->key has invalid address.
> 
> I am using QEMU in RPC-bypass  mode and the volume file looks like this:
> # cat c-qemu-rpcbypass.vol
> volume vm
>   type storage/posix
>   option directory /vm
> end-volume
> 
> I am not familiar with this part of the code and hence will need time
> to debug this. Meanwhile if anyone else familiar with this part of the
> code could give some pointers, it will be useful.
> 
> Regards,
> Bharata.