[Gluster-devel] Usage of xdata to send anything on wire

[Niels de Vos]

On Fri, May 13, 2016 at 06:44:58AM -0400, Raghavendra Gowdappa wrote:
> 
> 
> ----- Original Message -----
> > From: "Poornima Gurusiddaiah" <pgurusid at redhat.com>
> > To: "Gluster Devel" <gluster-devel at gluster.org>, "Niels de Vos" <ndevos at redhat.com>, "Jeff Darcy"
> > <jdarcy at redhat.com>, "Raghavendra Talur" <rtalur at redhat.com>, "Rajesh Joseph" <rjoseph at redhat.com>, "Pranith Kumar
> > Karampuri" <pkarampu at redhat.com>, "Soumya Koduri" <skoduri at redhat.com>, "Raghavendra Gowdappa"
> > <rgowdapp at redhat.com>, "Anoop Chirayath Manjiyil Sajan" <achiraya at redhat.com>
> > Sent: Friday, May 13, 2016 3:18:51 PM
> > Subject: Usage of xdata to send anything on wire
> > 
> > Hi,
> > 
> > In the recent past, have seen multiple instances where we needed to send some
> > data along with the fop on wire. And we have been using the xdata for the
> > same. Eg:
> > 1. Add lease-id, transaction id to every fop.
> > 2. Send the xattrs to invalidate in xdata.
> > 3. Send the share flags during open.
> > 
> > There were concerns raised around this for the below reasons:
> > 1. word-size and endianness
> > 2. security issues
> > 3. Backward compatibility issues i.e. old/new client and server combination.
> 
> Can you please elaborate on the above 3 points? A bit more verbose
> explanation of what and how the above factors cause problems will
> help.

The "word-size and endianness" point relates to XDR not being able to
parse the values in a dictionary. The key is always a string, but the
value can be something binary. There are a few occurences where the
value contains a GFID as binary structure. When sending this over the
network, the GFID on the client can be different than on the server if
the architectures do not have the same memory layout. We definitely need
to prevent this for anything that gets sent over the network. A real
structure defined in a .x file helps with that. A dict is not such a
structure (at the moment).

I dont know what the "security issues" are.

The issue with "old/new client and server combination" is explained
further below.

> > Initiating this mail to arrive at a conclusion, whether we can use xdata or
> > we need to find a different solution, if so what is the solution.
> > Your thoughts comments are appreciated.
> > 
> > Solution 1:
> > To get rid of sending xdata on wire, one of the solution could be to have
> > protocol versioning in Gluster. With this we can modify xdr structures for
> > each release and get rid of xdata. But this will be huge work.

I am not a big fan of this. It is not very flexible and we may need to
maintain different versions of certain functions. I think it is likely
that we would forget to correct some of the functions for older protocol
versions when we fix new versions. I expect this to be a relatively high
maintenance burden.

That said, it is surely a safe way to prevent surprising
incompatibilities.

> > Solution 2:
> > - Change dict, to not be an opaque structure , but an array of data elements
> > which is a union of (int, string etc.).

This is what I have been thinking about for a while too already.
Anything that gets put in a dict should also set a 'type'. Probably
default to 'string', but for anything else we need to make it possible
for generated XDR functions to (de)serialize the values.

> > - Backward compatibility issues is when the newer server/client adds data to
> > dict but the old client/server fails to read the dict. This is the
> > responsibility of the programmer to make sure, thta this case doesn't fail
> > silently, op version can be used if it is done as a part of adding new
> > feature/volume set. Another approach would be, if client has a list of
> > capabilities(features) the server supports it can accordingly tune itself to
> > access the xdata.

The solution you describe is something that each single feature can
implement themselves. Depending on the op-version is IMHO not always the
right thing to do, it forces an all-or-nothing upgrade plan for users. I
prefer to prevent that.

Thanks,
Niels
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20160513/dcccbf20/attachment.sig>