[Gluster-devel] Usage of xdata to send anything on wire

[Jeff Darcy]

> In the recent past, have seen multiple instances where we needed to
> send some data along with the fop on wire. And we have been using the
> xdata for the same. Eg:
>
> 1. Add lease-id, transaction id to every fop. 

The original purpose of xdata was to convey extra information without
having to bump a protocol version - e.g. hints that could safely be
ignored, or new fields that are subject to experimentation during the
prototype phase of a new translator.  For something that applies to
every fop, and particularly that we expect to be a permanent part of how
our protocol operates, I believe we should bump the protocol version.

> 2. Send the xattrs to invalidate in xdata.
> 3. Send the share flags during open.
>
> There were concerns raised around this for the below reasons:
> 1. word-size and endianness

This should only be an issue if people are using dict_set_bin.  In most 
cases I've seen, xdata values are ints that are represented internally
as strings, which should avoid any endian-ness issues.

> 2. security issues

I don't see how this is an issue for xdata any more than it is for
separate fields.  All of our security is either at the connection level
(TLS) or the request level (uid/gid).  Whatever's in xdata now would be
exactly as in/secure if it were in a new field associated with a new
protocol version.

> 3. Backward compatibility issues i.e. old/new client and server
> combination.

This is the biggest issue, though I'll add one more:
performance/complexity.  Creating dicts and adding keys to them involves
more memory allocations than we really want in our main I/O paths.
Because allocation errors have to be checked, this also makes code more 
complex and cumbersome than it would be with normal fields.

> Initiating this mail to arrive at a conclusion, whether we can use
> xdata or we need to find a different solution, if so what is the
> solution.  Your thoughts comments are appreciated.
>
> Solution 1:
> To get rid of sending xdata on wire, one of the solution could be to
> have protocol versioning in Gluster. With this we can modify xdr
> structures for each release and get rid of xdata. But this will be
> huge work.

We should have protocol versioning anyway, but that's no reason to get
rid of xdata.  It's still a useful feature, even if we change things
that are currently using it to use separate fields instead.  We can
consider these on a case-by-case basis.

> Solution 2:
> - Change dict, to not be an opaque structure , but an array of data 
> elements which is a union of (int, string etc.).

I don't see how this would be an improvement.  Each key in xdata is
already such a union, just represented a bit inefficiently, and
addressing them by name instead of by index is more portable/extensible.
We'd gain some in efficiency, but at the cost of having to coordinate
use of indices across all translators (including third-party).  That
would also increase the risk of misinterpreting a field set for one
purpose as meaning something totally different, which could be
disastrous.  With named fields the risk of such collisions is
negligible.

> - Backward compatibility issues is when the newer server/client adds
> data to dict but the old client/server fails to read the dict. This is
> the responsibility of the programmer to make sure, thta this case
> doesn't fail silently, op version can be used if it is done as a part
> of adding new feature/volume set. Another approach would be, if client
> has a list of capabilities(features) the server supports it can
> accordingly tune itself to access the xdata.

I'm a big fan of negotiating specific capabilities during the initial
connection handshake, instead of tying everything to a protocol version.
Believe it or not, several of the DECnet protocols had a nice way of
doing this using extensible bitfields.  I first encountered this in '89;
for me it remains the gold standard against which other
protocol-versioning or capability-negotiation strategies tend to fall
short.  We could easily do such negotiation for "global" protocol
features (e.g.  lease/transaction ID) as part of our connection
handshake, and individual translators could do something similar via
GF_FOP_IPC.