[Gluster-devel] Possible bug in the communications layer ?

Raghavendra Gowdappa rgowdapp at redhat.com
Fri Apr 29 03:56:30 UTC 2016


Thanks xavi.

@Prasanna/Rafi/Rtalur,
We had earlier seen "decoding failed" errors in quotad in another user setup. Possibly these are related. Can you please take a look?

regards,
Raghavendra

----- Original Message -----
> From: "Xavier Hernandez" <xhernandez at datalab.es>
> To: "Gluster Devel" <gluster-devel at gluster.org>
> Sent: Thursday, April 28, 2016 6:13:01 PM
> Subject: [Gluster-devel] Possible bug in the communications layer ?
> 
> Hi,
> 
> I've seen what seems a bug in the communications layer. The first sign
> is an "XDR decoding failed" error in the logs.
> 
> This happens with Gluster 3.7.11 accessed through Ganesha and gfapi. The
> volume is a distributed-disperse 4*(4+2).
> 
> I'm able to reproduce the problem easily doing the following test:
> 
> iozone -t2 -s10g -r1024k -i0 -w -F <nfs mount>/iozone{1..2}.dat
> echo 3 >/proc/sys/vm/drop_caches
> iozone -t2 -s10g -r1024k -i1 -w -F <nfs mount>/iozone{1..2}.dat
> 
> The error happens soon after starting the read test.
> 
> As can be seen in the data below, client3_3_readv_cbk() is processing an
> iovec of 116 bytes, however it should be of 154 bytes (the buffer in
> memory really seems to contain 154 bytes). The data on the network seems
> ok (at least I haven't been able to identify any problem), so this must
> be a processing error on the client side.
> 
> The last field in cut buffer of the sequentialized data corresponds to
> the length of the xdata field: 0x26. So at least 38 more byte should be
> present.
> 
> My guess is that some corner case is hit reading fragmented network
> packets due to a high load.
> 
> Debug information:
> 
> Breakpoint 1, client3_3_readv_cbk (req=0x7f540e64106c,
> iov=0x7f540e6410ac, count=<optimized out>, myframe=0x7f54259a4d54) at
> client-rpc-fops.c:3021
> 3021                    gf_msg (this->name, GF_LOG_ERROR, EINVAL,
> (gdb) print *iov
> $1 = {iov_base = 0x7f53e994e018, iov_len = 116}
> (gdb) x/116xb 0x7f53e994e018
> 0x7f53e994e018: 0x00    0x00    0x80    0x00    0x00    0x00    0x00    0x00
> 0x7f53e994e020: 0xa8    0xbf    0xa3    0xe0    0x5f    0x48    0x4c    0x1e
> 0x7f53e994e028: 0x80    0xa3    0x8a    0xd8    0x9d    0xa1    0x1c    0x75
> 0x7f53e994e030: 0x80    0xa3    0x8a    0xd8    0x9d    0xa1    0x1c    0x75
> 0x7f53e994e038: 0x00    0x00    0x00    0x00    0x00    0x00    0x08    0x00
> 0x7f53e994e040: 0x00    0x00    0x81    0xa0    0x00    0x00    0x00    0x01
> 0x7f53e994e048: 0x00    0x00    0x00    0x00    0x00    0x00    0x00    0x00
> 0x7f53e994e050: 0x00    0x00    0x00    0x00    0x00    0x00    0x00    0x00
> 0x7f53e994e058: 0x00    0x00    0x00    0x00    0xa0    0x00    0x00    0x00
> 0x7f53e994e060: 0x00    0x00    0x10    0x00    0x00    0x00    0x00    0x00
> 0x7f53e994e068: 0x00    0x50    0x00    0x00    0x57    0x22    0x04    0x1f
> 0x7f53e994e070: 0x25    0x38    0x92    0x91    0x57    0x22    0x04    0xb3
> 0x7f53e994e078: 0x03    0x53    0x1b    0x13    0x57    0x22    0x04    0xb3
> 0x7f53e994e080: 0x06    0xf5    0xe1    0x99    0x00    0x00    0x80    0x00
> 0x7f53e994e088: 0x00    0x00    0x00    0x26
> (gdb) x/154xb 0x7f53e994e018
> 0x7f53e994e018: 0x00    0x00    0x80    0x00    0x00    0x00    0x00    0x00
> 0x7f53e994e020: 0xa8    0xbf    0xa3    0xe0    0x5f    0x48    0x4c    0x1e
> 0x7f53e994e028: 0x80    0xa3    0x8a    0xd8    0x9d    0xa1    0x1c    0x75
> 0x7f53e994e030: 0x80    0xa3    0x8a    0xd8    0x9d    0xa1    0x1c    0x75
> 0x7f53e994e038: 0x00    0x00    0x00    0x00    0x00    0x00    0x08    0x00
> 0x7f53e994e040: 0x00    0x00    0x81    0xa0    0x00    0x00    0x00    0x01
> 0x7f53e994e048: 0x00    0x00    0x00    0x00    0x00    0x00    0x00    0x00
> 0x7f53e994e050: 0x00    0x00    0x00    0x00    0x00    0x00    0x00    0x00
> 0x7f53e994e058: 0x00    0x00    0x00    0x00    0xa0    0x00    0x00    0x00
> 0x7f53e994e060: 0x00    0x00    0x10    0x00    0x00    0x00    0x00    0x00
> 0x7f53e994e068: 0x00    0x50    0x00    0x00    0x57    0x22    0x04    0x1f
> 0x7f53e994e070: 0x25    0x38    0x92    0x91    0x57    0x22    0x04    0xb3
> 0x7f53e994e078: 0x03    0x53    0x1b    0x13    0x57    0x22    0x04    0xb3
> 0x7f53e994e080: 0x06    0xf5    0xe1    0x99    0x00    0x00    0x80    0x00
> 0x7f53e994e088: 0x00    0x00    0x00    0x26    0x00    0x00    0x00    0x01
> 0x7f53e994e090: 0x00    0x00    0x00    0x17    0x00    0x00    0x00    0x02
> 0x7f53e994e098: 0x67    0x6c    0x75    0x73    0x74    0x65    0x72    0x66
> 0x7f53e994e0a0: 0x73    0x2e    0x69    0x6e    0x6f    0x64    0x65    0x6c
> 0x7f53e994e0a8: 0x6b    0x2d    0x63    0x6f    0x75    0x6e    0x74    0x00
> 0x7f53e994e0b0: 0x31    0x00
> (gdb) bt
> #0  client3_3_readv_cbk (req=0x7f540e64106c, iov=0x7f540e6410ac,
> count=<optimized out>, myframe=0x7f54259a4d54) at client-rpc-fops.c:3021
> #1  0x00007f542a677ab0 in rpc_clnt_handle_reply
> (clnt=clnt at entry=0x7f54101cdef0, pollin=pollin at entry=0x7f54000091f0) at
> rpc-clnt.c:764
> #2  0x00007f542a677d6f in rpc_clnt_notify (trans=<optimized out>,
> mydata=0x7f54101cdf20, event=<optimized out>, data=0x7f54000091f0) at
> rpc-clnt.c:925
> #3  0x00007f542a673853 in rpc_transport_notify
> (this=this at entry=0x7f54101ddb70,
> event=event at entry=RPC_TRANSPORT_MSG_RECEIVED,
> data=data at entry=0x7f54000091f0) at rpc-transport.c:546
> #4  0x00007f541d881666 in socket_event_poll_in
> (this=this at entry=0x7f54101ddb70) at socket.c:2237
> #5  0x00007f541d8842c4 in socket_event_handler (fd=fd at entry=30,
> idx=idx at entry=20, data=0x7f54101ddb70, poll_in=1, poll_out=0,
> poll_err=0) at socket.c:2350
> #6  0x00007f542a90aa4a in event_dispatch_epoll_handler
> (event=0x7f540effc540, event_pool=0xd3d9a0) at event-epoll.c:575
> #7  event_dispatch_epoll_worker (data=0x7f54100d0890) at event-epoll.c:678
> #8  0x00007f542d0fbdc5 in start_thread () from /lib64/libpthread.so.0
> #9  0x00007f542c7bc28d in clone () from /lib64/libc.so.6
> 
> Xavi
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
> 


More information about the Gluster-devel mailing list