[Gluster-devel] Possible bug in the communications layer ?

Xavier Hernandez xhernandez at datalab.es
Mon May 9 11:07:04 UTC 2016


I've uploaded a patch for this problem:

http://review.gluster.org/14270

Any review will be very appreciated :)

Thanks,

Xavi

On 09/05/16 12:35, Raghavendra Gowdappa wrote:
>
>
> ----- Original Message -----
>> From: "Xavier Hernandez" <xhernandez at datalab.es>
>> To: "Raghavendra Gowdappa" <rgowdapp at redhat.com>
>> Cc: "Gluster Devel" <gluster-devel at gluster.org>
>> Sent: Monday, May 9, 2016 3:07:16 PM
>> Subject: Re: [Gluster-devel] Possible bug in the communications layer ?
>>
>> Hi Raghavendra,
>>
>> I've finally found the bug. It was obvious but I didn't see it.
>
> Same here :).
>
>>
>>   1561         case SP_STATE_ACCEPTED_SUCCESS_REPLY_INIT:
>>   1562                 default_read_size = xdr_sizeof ((xdrproc_t)
>> xdr_gfs3_read_rsp,
>>   1563                                                 &read_rsp);
>>   1564
>>   1565                 proghdr_buf = frag->fragcurrent;
>>   1566
>>   1567                 __socket_proto_init_pending (priv,
>> default_read_size);
>>   1568
>>   1569                 frag->call_body.reply.accepted_success_state
>>   1570                         = SP_STATE_READING_PROC_HEADER;
>>   1571
>>   1572                 /* fall through */
>>   1573
>>   1574         case SP_STATE_READING_PROC_HEADER:
>>   1575                 __socket_proto_read (priv, ret);
>>   1576
>>   1577                 gf_trace_add("xdrmem_create", default_read_size,
>> (uintptr_t)proghdr_buf);
>>   1578                 /* there can be 'xdata' in read response, figure
>> it out */
>>   1579                 xdrmem_create (&xdr, proghdr_buf, default_read_size,
>>   1580                                XDR_DECODE);
>>   1581
>>   1582                 /* This will fail if there is xdata sent from
>> server, if not,
>>   1583                    well and good, we don't need to worry about  */
>>   1584                 xdr_gfs3_read_rsp (&xdr, &read_rsp);
>>   1585
>>   1586                 free (read_rsp.xdata.xdata_val);
>>   1587
>>   1588                 /* need to round off to proper roof (%4), as XDR
>> packing pads
>>   1589                    the end of opaque object with '0' */
>>   1590                 size = roof (read_rsp.xdata.xdata_len, 4);
>>   1591
>>   1592                 if (!size) {
>>   1593                         frag->call_body.reply.accepted_success_state
>>   1594                                 = SP_STATE_READ_PROC_OPAQUE;
>>   1595                         goto read_proc_opaque;
>>   1596                 }
>>   1597
>>   1598                 __socket_proto_init_pending (priv, size);
>>   1599
>>   1600                 frag->call_body.reply.accepted_success_state
>>   1601                         = SP_STATE_READING_PROC_OPAQUE;
>>
>> The main problem here is that we are using two local variables
>> (proghdr_buf and default_read_size) in two distinct states that might be
>> called at different times.
>>
>> The particular case that is failing is the following:
>>
>> 1. In state SP_STATE_ACCEPTED_SUCCESS_REPLY_INIT, everything is prepared
>> to read 116 bytes. default_read_size is set to 116 and proghdr_buf
>> points to the buffer where data will be written.
>>
>> 2. In state SP_STATE_READING_PROC_HEADER, a partial read of 88 bytes is
>> done. At this point the function returns and proghdr_buf and
>> default_read_size are lost.
>>
>> 3. When more data is available, this function is called again and it
>> starts executing at state SP_STATE_READING_PROC_HEADER.
>>
>> 4. The remaining 28 bytes are read.
>>
>> 5. When it checks the buffer and tries to decode it to see if there's
>> xdata present, it uses the default values of proghdr_buf and
>> default_read_size, that are 0. This causes the decode to leave
>> read_rsp.xdata.xdata_len set to 0.
>>
>> 6. The program interprets that xdata_len being 0 means that there's no
>> xdata, so it continues reading the remaining of the RPC packet into the
>> payload buffer.
>>
>> If you want, I can send a patch for this.
>
> Yes. That would be helpful. The analysis is correct and moving initialization of prog_hdrbuf to line 1578 will fix the issue. If you are too busy, please let me know and I can patch it up too :).
>
> Thanks for debugging the issue :).
>
> regards,
> Raghavendra.
>
>>
>> Xavi
>>
>> On 05/05/16 10:21, Xavier Hernandez wrote:
>>> I've undone all changes and now I'm unable to reproduce the problem, so
>>> the modification I did is probably incorrect and not the root cause, as
>>> you described.
>>>
>>> I'll continue investigating...
>>>
>>> Xavi
>>>
>>> On 04/05/16 15:01, Xavier Hernandez wrote:
>>>> On 04/05/16 14:47, Raghavendra Gowdappa wrote:
>>>>>
>>>>>
>>>>> ----- Original Message -----
>>>>>> From: "Xavier Hernandez" <xhernandez at datalab.es>
>>>>>> To: "Raghavendra Gowdappa" <rgowdapp at redhat.com>
>>>>>> Cc: "Gluster Devel" <gluster-devel at gluster.org>
>>>>>> Sent: Wednesday, May 4, 2016 5:37:56 PM
>>>>>> Subject: Re: [Gluster-devel] Possible bug in the communications layer ?
>>>>>>
>>>>>> I think I've found the problem.
>>>>>>
>>>>>> 1567         case SP_STATE_READING_PROC_HEADER:
>>>>>>   1568                 __socket_proto_read (priv, ret);
>>>>>>   1569
>>>>>>   1570                 /* there can be 'xdata' in read response, figure
>>>>>> it out */
>>>>>>   1571                 xdrmem_create (&xdr, proghdr_buf,
>>>>>> default_read_size,
>>>>>>   1572                                XDR_DECODE);
>>>>>>   1573
>>>>>>   1574                 /* This will fail if there is xdata sent from
>>>>>> server, if not,
>>>>>>   1575                    well and good, we don't need to worry
>>>>>> about  */
>>>>>>   1576                 xdr_gfs3_read_rsp (&xdr, &read_rsp);
>>>>>>   1577
>>>>>>   1578                 free (read_rsp.xdata.xdata_val);
>>>>>>   1579
>>>>>>   1580                 /* need to round off to proper roof (%4), as XDR
>>>>>> packing pads
>>>>>>   1581                    the end of opaque object with '0' */
>>>>>>   1582                 size = roof (read_rsp.xdata.xdata_len, 4);
>>>>>>   1583
>>>>>>   1584                 if (!size) {
>>>>>>   1585
>>>>>> frag->call_body.reply.accepted_success_state
>>>>>>   1586                                 = SP_STATE_READ_PROC_OPAQUE;
>>>>>>   1587                         goto read_proc_opaque;
>>>>>>   1588                 }
>>>>>>   1589
>>>>>>   1590                 __socket_proto_init_pending (priv, size);
>>>>>>   1591
>>>>>>   1592                 frag->call_body.reply.accepted_success_state
>>>>>>   1593                         = SP_STATE_READING_PROC_OPAQUE;
>>>>>>   1594
>>>>>>   1595         case SP_STATE_READING_PROC_OPAQUE:
>>>>>>   1596                 __socket_proto_read (priv, ret);
>>>>>>   1597
>>>>>>   1598                 frag->call_body.reply.accepted_success_state
>>>>>>   1599                         = SP_STATE_READ_PROC_OPAQUE;
>>>>>>
>>>>>> On line 1568 we read, at most, 116 bytes because we calculate the size
>>>>>> of a read response without xdata. Then we detect that we really need
>>>>>> more data for xdata (BTW, read_rsp.xdata.xdata_val will be always
>>>>>> allocated even if xdr_gfs3_read_rsp() fails ?)
>>>>>
>>>>> No. It need not be. Its guaranteed that only on a successful
>>>>> completion it is allocated. However, _if_ decoding fails only because
>>>>> xdr stream doesn't include xdata bits, but xdata_len is zero (by
>>>>> initializing it to default_read_size), then xdr library would've
>>>>> filled read_rsp.xdata.xdata_len (read_rsp.xdata.xdata_val can still be
>>>>> NULL).
>>>>
>>>> The question is: is it guaranteed that after an unsuccessful completion
>>>> xdata_val will be NULL (i.e. not touched by the function, even if
>>>> xadata_len is != 0) ? otherwise the free() could corrupt memory.
>>>>
>>>>>
>>>>>>
>>>>>> So we get into line 1596 with the pending info initialized to read the
>>>>>> remaining data. This is the __socket_proto_read macro:
>>>>>>
>>>>>>    166 /* This will be used in a switch case and breaks from the switch
>>>>>> case if all
>>>>>>    167  * the pending data is not read.
>>>>>>    168  */
>>>>>>    169 #define __socket_proto_read(priv, ret)
>>>>>>        \
>>>>>>    170                 {
>>>>>>        \
>>>>>>    171                 size_t bytes_read = 0;
>>>>>>        \
>>>>>>    172                 struct gf_sock_incoming *in;
>>>>>>        \
>>>>>>    173                 in = &priv->incoming;
>>>>>>        \
>>>>>>    174
>>>>>>        \
>>>>>>    175                 __socket_proto_update_pending (priv);
>>>>>>        \
>>>>>>    176
>>>>>>        \
>>>>>>    177                 ret = __socket_readv (this,
>>>>>>        \
>>>>>>    178                                       in->pending_vector, 1,
>>>>>>        \
>>>>>>    179                                       &in->pending_vector,
>>>>>>        \
>>>>>>    180                                       &in->pending_count,
>>>>>>        \
>>>>>>    181                                       &bytes_read);
>>>>>>        \
>>>>>>    182                 if (ret == -1)
>>>>>>        \
>>>>>>    183                         break;
>>>>>>        \
>>>>>>    184                 __socket_proto_update_priv_after_read (priv,
>>>>>> ret,
>>>>>> bytes_read); \
>>>>>>    185         }
>>>>>>
>>>>>> We read from the socket using __socket_readv(). It it fails, we quit.
>>>>>> However if the socket doesn't have more data to read, this function
>>>>>> does
>>>>>> not return -1:
>>>>>>
>>>>>>    555                         ret = __socket_cached_read (this,
>>>>>> opvector, opcount);
>>>>>>    556
>>>>>>    557                         if (ret == 0) {
>>>>>>    558
>>>>>> gf_log(this->name,GF_LOG_DEBUG,"EOF on socket");
>>>>>>    559                                 errno = ENODATA;
>>>>>>    560                                 ret = -1;
>>>>>>    561                         }
>>>>>>    562                         if (ret == -1 && errno == EAGAIN) {
>>>>>>    563                                 /* done for now */
>>>>>>    564                                 break;
>>>>>>    565                         }
>>>>>>    566                         this->total_bytes_read += ret;
>>>>>>
>>>>>> If __socket_cached_read() fails with errno == EAGAIN, we break and
>>>>>> return opcount, which is >= 0. Causing the process to continue instead
>>>>>> of waiting for more data.
>>>>>
>>>>> No. If you observe, there is a call to another macro
>>>>> __socket_proto_update_priv_after_read at line 184. As can be seen in
>>>>> the definition below, if (ret > 0) - which is the case in case of
>>>>> short-read - we do break. I hope I am not missing anything here :).
>>>>>
>>>>> #define __socket_proto_update_priv_after_read(priv, ret,
>>>>> bytes_read)    \
>>>>>
>>>>> {                                                               \
>>>>>         struct gf_sock_incoming_frag *frag;                     \
>>>>>                 frag =
>>>>> &priv->incoming.frag;                            \
>>>>>
>>>>> \
>>>>>                 frag->fragcurrent +=
>>>>> bytes_read;                        \
>>>>>                 frag->bytes_read +=
>>>>> bytes_read;                         \
>>>>>
>>>>> \
>>>>>                 if ((ret > 0) || (frag->remaining_size != 0))
>>>>> {         \
>>>>>             if (frag->remaining_size != 0 && ret == 0) {    \
>>>>>                 __socket_proto_reset_pending (priv);    \
>>>>>
>>>>> }                                               \
>>>>>
>>>>> \
>>>>>                         gf_log (this->name,
>>>>> GF_LOG_TRACE,               \
>>>>>                                 "partial read on non-blocking
>>>>> socket"); \
>>>>>
>>>>> \
>>>>>
>>>>> break;                                          \
>>>>>
>>>>> }                                                       \
>>>>>         }
>>>>>
>>>>
>>>> I didn't see this. However the fact is that doing the change I've
>>>> described, it seems to work.
>>>>
>>>> I'll analyze it a bit more.
>>>>
>>>> Xavi
>>>>
>>>>>>
>>>>>> As a side note, there's another problem here: if errno is not EAGAIN,
>>>>>> we'll update this->total_bytes_read substracting one. This shouldn't be
>>>>>> done when ret < 0.
>>>>>>
>>>>>> There are other places where ret is set to -1, but opcount is returned.
>>>>>> I guess that we should also set opcount = -1 on these places, but I
>>>>>> don't have a deep knowledge about this implementation.
>>>>>>
>>>>>> I've done a quick test checking for (ret != 0) instead of (ret ==
>>>>>> -1) in
>>>>>> __socket_proto_read() and it seemed to work.
>>>>>>
>>>>>> Could anyone with more knowledge about the communications layer verify
>>>>>> this and explain what would be the best solution ?
>>>>>>
>>>>>> Xavi
>>>>>>
>>>>>> On 29/04/16 14:52, Xavier Hernandez wrote:
>>>>>>> With your patch applied, it seems that the bug is not hit.
>>>>>>>
>>>>>>> I guess it's a timing issue that the new logging hides. Maybe no more
>>>>>>> data available after reading the partial readv header ? (it will
>>>>>>> arrive
>>>>>>> later).
>>>>>>>
>>>>>>> I'll continue testing...
>>>>>>>
>>>>>>> Xavi
>>>>>>>
>>>>>>> On 29/04/16 13:48, Raghavendra Gowdappa wrote:
>>>>>>>> Attaching the patch.
>>>>>>>>
>>>>>>>> ----- Original Message -----
>>>>>>>>> From: "Raghavendra Gowdappa" <rgowdapp at redhat.com>
>>>>>>>>> To: "Xavier Hernandez" <xhernandez at datalab.es>
>>>>>>>>> Cc: "Gluster Devel" <gluster-devel at gluster.org>
>>>>>>>>> Sent: Friday, April 29, 2016 5:14:02 PM
>>>>>>>>> Subject: Re: [Gluster-devel] Possible bug in the communications
>>>>>>>>> layer ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ----- Original Message -----
>>>>>>>>>> From: "Xavier Hernandez" <xhernandez at datalab.es>
>>>>>>>>>> To: "Raghavendra Gowdappa" <rgowdapp at redhat.com>
>>>>>>>>>> Cc: "Gluster Devel" <gluster-devel at gluster.org>
>>>>>>>>>> Sent: Friday, April 29, 2016 1:21:57 PM
>>>>>>>>>> Subject: Re: [Gluster-devel] Possible bug in the communications
>>>>>>>>>> layer ?
>>>>>>>>>>
>>>>>>>>>> Hi Raghavendra,
>>>>>>>>>>
>>>>>>>>>> yes, the readv response contains xdata. The dict length is 38
>>>>>>>>>> (0x26)
>>>>>>>>>> and, at the moment of failure, rsp.xdata.xdata_len already contains
>>>>>>>>>> 0x26.
>>>>>>>>>
>>>>>>>>> rsp.xdata.xdata_len having 0x26 even when decoding failed indicates
>>>>>>>>> that the
>>>>>>>>> approach used in socket.c to get the length of xdata is correct.
>>>>>>>>> However, I
>>>>>>>>> cannot find any other way of xdata going into payload vector other
>>>>>>>>> than
>>>>>>>>> xdata_len being zero. Just to be double sure, I've a patch
>>>>>>>>> containing
>>>>>>>>> debug
>>>>>>>>> message printing xdata_len when decoding fails in socket.c. Can you
>>>>>>>>> please
>>>>>>>>> apply the patch, run the tests and revert back with results?
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Xavi
>>>>>>>>>>
>>>>>>>>>> On 29/04/16 09:10, Raghavendra Gowdappa wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>> From: "Raghavendra Gowdappa" <rgowdapp at redhat.com>
>>>>>>>>>>>> To: "Xavier Hernandez" <xhernandez at datalab.es>
>>>>>>>>>>>> Cc: "Gluster Devel" <gluster-devel at gluster.org>
>>>>>>>>>>>> Sent: Friday, April 29, 2016 12:36:43 PM
>>>>>>>>>>>> Subject: Re: [Gluster-devel] Possible bug in the communications
>>>>>>>>>>>> layer ?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>>> From: "Raghavendra Gowdappa" <rgowdapp at redhat.com>
>>>>>>>>>>>>> To: "Xavier Hernandez" <xhernandez at datalab.es>
>>>>>>>>>>>>> Cc: "Jeff Darcy" <jdarcy at redhat.com>, "Gluster Devel"
>>>>>>>>>>>>> <gluster-devel at gluster.org>
>>>>>>>>>>>>> Sent: Friday, April 29, 2016 12:07:59 PM
>>>>>>>>>>>>> Subject: Re: [Gluster-devel] Possible bug in the communications
>>>>>>>>>>>>> layer ?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>>>> From: "Xavier Hernandez" <xhernandez at datalab.es>
>>>>>>>>>>>>>> To: "Jeff Darcy" <jdarcy at redhat.com>
>>>>>>>>>>>>>> Cc: "Gluster Devel" <gluster-devel at gluster.org>
>>>>>>>>>>>>>> Sent: Thursday, April 28, 2016 8:15:36 PM
>>>>>>>>>>>>>> Subject: Re: [Gluster-devel] Possible bug in the communications
>>>>>>>>>>>>>> layer
>>>>>>>>>>>>>> ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Jeff,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 28.04.2016 15:20, Jeff Darcy wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This happens with Gluster 3.7.11 accessed through Ganesha and
>>>>>>>>>>>>>> gfapi.
>>>>>>>>>>>>>> The
>>>>>>>>>>>>>> volume is a distributed-disperse 4*(4+2). I'm able to
>>>>>>>>>>>>>> reproduce the
>>>>>>>>>>>>>> problem
>>>>>>>>>>>>>> easily doing the following test: iozone -t2 -s10g -r1024k
>>>>>>>>>>>>>> -i0 -w
>>>>>>>>>>>>>> -F/iozone{1..2}.dat echo 3 >/proc/sys/vm/drop_caches iozone -t2
>>>>>>>>>>>>>> -s10g
>>>>>>>>>>>>>> -r1024k -i1 -w -F/iozone{1..2}.dat The error happens soon after
>>>>>>>>>>>>>> starting
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> read test. As can be seen in the data below,
>>>>>>>>>>>>>> client3_3_readv_cbk() is
>>>>>>>>>>>>>> processing an iovec of 116 bytes, however it should be of 154
>>>>>>>>>>>>>> bytes
>>>>>>>>>>>>>> (the
>>>>>>>>>>>>>> buffer in memory really seems to contain 154 bytes). The
>>>>>>>>>>>>>> data on
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> network
>>>>>>>>>>>>>> seems ok (at least I haven't been able to identify any
>>>>>>>>>>>>>> problem), so
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>> must be a processing error on the client side. The last
>>>>>>>>>>>>>> field in
>>>>>>>>>>>>>> cut
>>>>>>>>>>>>>> buffer
>>>>>>>>>>>>>> of the sequentialized data corresponds to the length of the
>>>>>>>>>>>>>> xdata
>>>>>>>>>>>>>> field:
>>>>>>>>>>>>>> 0x26. So at least 38 more byte should be present.
>>>>>>>>>>>>>> Nice detective work, Xavi.  It would be *very* interesting to
>>>>>>>>>>>>>> see what
>>>>>>>>>>>>>> the value of the "count" parameter is (it's unfortunately
>>>>>>>>>>>>>> optimized
>>>>>>>>>>>>>> out).
>>>>>>>>>>>>>> I'll bet it's two, and iov[1].iov_len is 38.  I have a weak
>>>>>>>>>>>>>> memory of
>>>>>>>>>>>>>> some problems with how this iov is put together, a couple of
>>>>>>>>>>>>>> years
>>>>>>>>>>>>>> ago,
>>>>>>>>>>>>>> and it looks like you might have tripped over one more.
>>>>>>>>>>>>>> It seems you are right. The count is 2 and the first 38
>>>>>>>>>>>>>> bytes of
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> second
>>>>>>>>>>>>>> vector contains the remaining data of xdata field.
>>>>>>>>>>>>>
>>>>>>>>>>>>> This is the bug. client3_3_readv_cbk (and for that matter all
>>>>>>>>>>>>> the
>>>>>>>>>>>>> actors/cbks) expects response in utmost two vectors:
>>>>>>>>>>>>> 1. Program header containing request or response. This is
>>>>>>>>>>>>> subjected to
>>>>>>>>>>>>> decoding/encoding. This vector should point to a buffer that
>>>>>>>>>>>>> contains
>>>>>>>>>>>>> the
>>>>>>>>>>>>> entire program header/response contiguously.
>>>>>>>>>>>>> 2. If the procedure returns payload (like readv response or a
>>>>>>>>>>>>> write
>>>>>>>>>>>>> request),
>>>>>>>>>>>>> second vector contains the buffer pointing to the entire
>>>>>>>>>>>>> (contiguous)
>>>>>>>>>>>>> payload. Note that this payload is raw and is not subjected to
>>>>>>>>>>>>> encoding/decoding.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In your case, this _clean_ separation is broken with part of
>>>>>>>>>>>>> program
>>>>>>>>>>>>> header
>>>>>>>>>>>>> slipping into 2nd vector supposed to contain read data (may be
>>>>>>>>>>>>> because
>>>>>>>>>>>>> of
>>>>>>>>>>>>> rpc fragmentation). I think this is a bug in socket layer. I'll
>>>>>>>>>>>>> update
>>>>>>>>>>>>> more
>>>>>>>>>>>>> on this.
>>>>>>>>>>>>
>>>>>>>>>>>> Does your read response include xdata too? I think the code
>>>>>>>>>>>> related to
>>>>>>>>>>>> reading xdata in readv response is a bit murky.
>>>>>>>>>>>>
>>>>>>>>>>>> <socket.c/__socket_read_accepted_successful_reply>
>>>>>>>>>>>>
>>>>>>>>>>>>         case SP_STATE_ACCEPTED_SUCCESS_REPLY_INIT:
>>>>>>>>>>>>                 default_read_size = xdr_sizeof ((xdrproc_t)
>>>>>>>>>>>>                 xdr_gfs3_read_rsp,
>>>>>>>>>>>>                                                 &read_rsp);
>>>>>>>>>>>>
>>>>>>>>>>>>                 proghdr_buf = frag->fragcurrent;
>>>>>>>>>>>>
>>>>>>>>>>>>                 __socket_proto_init_pending (priv,
>>>>>>>>>>>> default_read_size);
>>>>>>>>>>>>
>>>>>>>>>>>>                 frag->call_body.reply.accepted_success_state
>>>>>>>>>>>>                         = SP_STATE_READING_PROC_HEADER;
>>>>>>>>>>>>
>>>>>>>>>>>>                 /* fall through */
>>>>>>>>>>>>
>>>>>>>>>>>>         case SP_STATE_READING_PROC_HEADER:
>>>>>>>>>>>>                 __socket_proto_read (priv, ret);
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> By this time we've read read response _minus_ the xdata
>>>>>>>>>>>
>>>>>>>>>>> I meant we have read "readv response header"
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>                 /* there can be 'xdata' in read response, figure
>>>>>>>>>>>> it out
>>>>>>>>>>>>                 */
>>>>>>>>>>>>                 xdrmem_create (&xdr, proghdr_buf,
>>>>>>>>>>>> default_read_size,
>>>>>>>>>>>>                                XDR_DECODE);
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> We created xdr stream above with "default_read_size" (this
>>>>>>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>>>>> include xdata)
>>>>>>>>>>>>
>>>>>>>>>>>>                 /* This will fail if there is xdata sent from
>>>>>>>>>>>> server, if
>>>>>>>>>>>>                 not,
>>>>>>>>>>>>                    well and good, we don't need to worry
>>>>>>>>>>>> about  */
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> what if xdata is present and decoding failed (as length
>>>>>>>>>>>>>>>>>> of xdr
>>>>>>>>>>>>>>>>>> stream
>>>>>>>>>>>>>>>>>> above - default_read_size - doesn't include xdata)?
>>>>>>>>>>>>>>>>>> would we
>>>>>>>>>>>>>>>>>> have
>>>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>>> valid value in read_rsp.xdata.xdata_len? This is the part
>>>>>>>>>>>>>>>>>> I am
>>>>>>>>>>>>>>>>>> confused about. If read_rsp.xdata.xdata_len is not
>>>>>>>>>>>>>>>>>> correct then
>>>>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>>>>> is a possibility that xdata might not be entirely
>>>>>>>>>>>>>>>>>> present in
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> vector socket passes to higher layers as progheader (with
>>>>>>>>>>>>>>>>>> part or
>>>>>>>>>>>>>>>>>> entire xdata spilling over to payload vector).
>>>>>>>>>>>>
>>>>>>>>>>>>                 xdr_gfs3_read_rsp (&xdr, &read_rsp);
>>>>>>>>>>>>
>>>>>>>>>>>>                 free (read_rsp.xdata.xdata_val);
>>>>>>>>>>>>
>>>>>>>>>>>>                 /* need to round off to proper roof (%4), as XDR
>>>>>>>>>>>> packing
>>>>>>>>>>>>                 pads
>>>>>>>>>>>>                    the end of opaque object with '0' */
>>>>>>>>>>>>                 size = roof (read_rsp.xdata.xdata_len, 4);
>>>>>>>>>>>>
>>>>>>>>>>>>                 if (!size) {
>>>>>>>>>>>>
>>>>>>>>>>>> frag->call_body.reply.accepted_success_state
>>>>>>>>>>>>                                 = SP_STATE_READ_PROC_OPAQUE;
>>>>>>>>>>>>                         goto read_proc_opaque;
>>>>>>>>>>>>                 }
>>>>>>>>>>>>
>>>>>>>>>>>> </socket.c>
>>>>>>>>>>>>
>>>>>>>>>>>> Can you please confirm whether there was an xdata in the readv
>>>>>>>>>>>> response
>>>>>>>>>>>> (may
>>>>>>>>>>>> be by looking in bricks) whose decoding failed?
>>>>>>>>>>>>
>>>>>>>>>>>> regards,
>>>>>>>>>>>> Raghavendra
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Gluster-devel mailing list
>>>>>>>>>>>> Gluster-devel at gluster.org
>>>>>>>>>>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Gluster-devel mailing list
>>>>>>> Gluster-devel at gluster.org
>>>>>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>>>>>
>>>> _______________________________________________
>>>> Gluster-devel mailing list
>>>> Gluster-devel at gluster.org
>>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>> _______________________________________________
>>> Gluster-devel mailing list
>>> Gluster-devel at gluster.org
>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>


More information about the Gluster-devel mailing list