[Gluster-devel] Transparent encryption in GlusterFS

[Jeff Darcy]

On 05/05/2011 04:23 PM, Edward Shishkin wrote:
> The straightforward solution is to serialize read-modify-writes.
> I wonder if GlusterFS has any per-file serialization means,
> that would allow to resolve this problem. Or maybe there are
> possibilities to create such means. Any hints would be highly
> appreciated.

At a first approximation, you could just wrap the read-modify-write in
POSIX locks. That would conflict with other uses of POSIX locks, though,
and might not address the issue of "self-conflict" induced e.g. by some
of the performance translators issuing parallel writes to the same fd.
There is an "oplock" translator in CloudFS which was co-developed with
the encryption translator you're working on and which attempts to
provide the necessary conflict detection without scalability-destroying
serialization. The code does need some improvement, though, as has been
discussed on the cloudfs-devel thread you started at
https://fedorahosted.org/pipermail/cloudfs-devel/2011-May/000038.html.
In particular, we need to address not just race conditions but also e.g.
forward-progress guarantees, and (as I said in that thread) I think
judicious use of server-side request queuing is the way to do that.

This kind of synchronization is needed for other things besides
encryption, by the way. For example, I've considered adding a
data-integrity translator using checksums or hashes. That would run into
exactly the same need for atomic read-modify-write sequences, requiring
exactly the same kind of coordination, so as we design this we should
try to account for the fact that there might be multiple concurrent
users (translators) at different levels.