[Gluster-devel] Change in glusterfs[master]: Transparent data encryption and metadata authentication in t...
Anand Avati
avati at gluster.org
Thu Oct 24 20:59:43 UTC 2013
On Thu, Oct 24, 2013 at 1:18 PM, Edward Shishkin <edward at redhat.com> wrote:
> Hi all,
>
> So, here is the "all-in-one-translator" version represented by the
> Patch Set #2 at review.gluster.org/4667
>
> Everything has been addressed except encryption in NFS mounts (see
> next mail for details). That is:
>
> . New design of EOF (end-of-file) handling;
> . No oplock translator on the server side;
> . All locks are acquired/released by the crypt translator;
> . Now we can encrypt srtiped and(or) replicated volumes.
>
> Common comments.
>
> In the new design all files on the server are "padded", whereas the
> real file size is stored as xattr. So we introduce a special layer
> in the crypt translator, which performs file size translations: every
> time when any callback returns struct iatt, we update its ia_size
> with the real (non-padded) value.
>
> The most unpleasant thing in this new design is FOP->readdirp_cbk():
> in this case we need N translations, i.e. N calls to the server (N is
> number of directory entries).
>
> To perform translations we spawn N children. We need a valid list of
> dirents after returning from FOP->readdirp_cbk() of previous
> translator, but we don't want to create a copy of this list (which
> can be large enough). For this reason we introduce a reference counter
> in struct gf_dirent_t and allocate dynamic structures gf_dirent_t
> (instead of on-stack ones), see respective changes in
>
> ./libglusterfs/src/gf-dirent.c
> ./libglusterfs/src/gf-dirent.h
> ./xlators/cluster/dht/src/dht-common.c
> ./xlators/protocol/client/src/client-rpc-fops.c
>
[pasting from internal email reply]
I had a look at the way you are handling readdirplus. I think it is overly
complex. FOP->readdirplus() already has a parameter @xdata in which you can
request per-entry xattr replies.
So in crypt_readdirp() you need to: dict_set(xdata, FSIZE_XATTR_PREFIX, 0);
Once you do that, in crypt_readdirp_cbk, you can expect each gf_dirent_t to
have its dirent->dict set with FSIZE_XATTR_PREFIX.
So you just need to iterate over replies in crypt_readdirp_cbk, update each
dirent->d_stat.ia_size with value from dict_get_uint64(dirent->xdata,
FSIZE_XATTR_PREFIX)
Please look at how posix-acl does something very similar (loading per-entry
ACLs into respective inodes via xattrs returned in readdirplus)
Avati
>
> Thanks,
> Edward.
>
>
> On Mon, 14 Oct 2013 14:27:01 -0700
> Anand Avati <avati at redhat.com> wrote:
>
> > Edward,
> > It looks like this patch requires a higher version of openssl (I
> > recall you have mentioned before that that dependency was on version
> > 1.0.1c? I checked yum update on the build server and the latest
> > available version is 1.0.0-27. Is there a "clean" way to get the
> > right version of openssl to a RHEL/CENTOS-6.x server?
> >
> > Also note that the previous submission of the patch was at
> > http://review.gluster.org/4667. The recent on
> > (http://review.gluster.org/6086) has a different Change-Id: in the
> > commit log. It will be good if you can re-submit the patch with the
> > old Change-Id (and abandon #6086) so that we can maintain the history
> > of resubmission and the old work on records.
> >
> > Thanks!
> > Avati
> >
> > On 10/14/2013 07:26 AM, Edward Shishkin (Code Review) wrote:
> > > Edward Shishkin has uploaded a new change for review.
> > >
> > > http://review.gluster.org/6086
> > >
> > >
> > > Change subject: Transparent data encryption and metadata
> > > authentication in the systems with non-trusted server (take
> > > II)
> ......................................................................
> > >
> > > Transparent data encryption and metadata authentication
> > > in the systems with non-trusted server (take II)
> > >
> > > This new functionality can be useful in various cloud technologies.
> > > It is implemented via a special encryption/crypt translator, which
> > > works on the client side and performs encryption and authentication;
> > >
> > > 1. Class of supported algorithms
> > >
> > > The crypt translator can support any atomic symmetric block cipher
> > > algorithms (which require to pad plain/cipher text before performing
> > > encryption/decryption transform (see glossary in atom.c for
> > > definitions). In particular, it can support algorithms with the EOF
> > > issue (which require to pad the end of file by extra-data).
> > >
> > > Crypt translator performs translations
> > > user -> (offset, size) -> (aligned-offset, padded-size) ->server
> > > (and backward), and resolves individual FOPs (write(), truncate(),
> > > etc) to read-modify-write sequences.
> > >
> > > A volume can contain files encrypted by different algorithms of the
> > > mentioned class. To change some option value just reconfigure the
> > > volume.
> > >
> > > Currently only one algorithm is supported: AES_XTS.
> > >
> > > Example of algorithms, which can not be supported by the crypt
> > > translator:
> > >
> > > 1. Asymmetric block cipher algorithms, which inflate data, e.g. RSA;
> > > 2. Symmetric block cipher algorithms with inline MACs for data
> > > authentication.
> > >
> > > 2. Implementation notes.
> > >
> > > a) Atomic algorithms
> > >
> > > Since any process in a stackable file system manipulates with local
> > > data (which can be obsoleted by local data of another process), any
> > > atomic cipher algorithm without proper support can lead to non-POSIX
> > > behavior. To resolve the "collisions" we introduce locks: before
> > > performing FOP->read(), FOP->write(), etc. the process should first
> > > lock the file.
> > >
> > > b) Algorithms with EOF issue
> > >
> > > Such algorithms require to pad the end of file with some extra-data.
> > > Without proper support this will result in losing information about
> > > real file size. Keeping a track of real file size is a
> > > responsibility of the crypt translator. A special extended
> > > attribute with the name "trusted.glusterfs.crypt.att.size" is used
> > > for this purpose. All files contained in bricks of encrypted volume
> > > do have "padded" sizes.
> > >
> > > 3. Non-trusted servers and
> > > Metadata authentication
> > >
> > > We assume that server, where user's data is stored on is
> > > non-trusted. It means that the server can be subjected to various
> > > attacks directed to reveal user's encrypted personal data. We
> > > provide protection against such attacks.
> > >
> > > Every encrypted file has specific private attributes (cipher
> > > algorithm id, atom size, etc), which are packed to a string
> > > (so-called "format string") and stored as a special extended
> > > attribute with the name "trusted.glusterfs.crypt.att.cfmt". We
> > > protect the string from tampering. This protection is mandatory,
> > > hardcoded and is always on. Without such protection various attacks
> > > (based on extending the scope of per-file secret keys) are possible.
> > >
> > > Our authentication method has been developed in tight collaboration
> > > with Red Hat security team and is implemented as "metadata loader of
> > > version 1" (see file metadata.c). This method is NIST-compliant and
> > > is based on checking 8-byte per-hardlink MACs created(updated) by
> > > FOP->create(), FOP->link(), FOP->unlink(), FOP->rename() by the
> > > following unique entities:
> > >
> > > . file (hardlink) name;
> > > . verified file's object id (gfid).
> > >
> > > Every time, before manipulating with a file, we check it's MACs at
> > > FOP->open() time. Some FOPs don't require a file to be opened (e.g.
> > > FOP->truncate()). In such cases the crypt translator opens the file
> > > mandatory.
> > >
> > > 4. Generating keys
> > >
> > > Unique per-file keys are derived by NIST-compliant methods from the
> > >
> > > a) parent key;
> > > b) unique verified object-id of the file (gfid);
> > > Per-volume master key, provided by user at mount time is in the root
> > > of this "tree of keys".
> > >
> > > Those keys are used to:
> > >
> > > 1) encrypt/decrypt file data;
> > > 2) encrypt/decrypt file metadata;
> > > 3) create per-file and per-link MACs for metadata authentication.
> > >
> > > 5. Instructions
> > > Getting started with crypt translator
> > >
> > > Example:
> > >
> > > 1) Create a volume "myvol" by specifying the option "encrypt":
> > >
> > > # gluster volume create myvol encrypt pepelac:/vols/xvol
> > >
> > > 2) Set location (absolute pathname) of your master key:
> > >
> > > # gluster volume set myvol encryption.master-key /home/me/mykey
> > >
> > > 3) Set other options to override default options, if needed.
> > >
> > > 4) On the client side make sure that the file /home/me/mykey exists
> > > and contains proper per-volume master key (that is 256-bit AES
> > > key). This key has to be in hex form, i.e. should be represented
> > > by 64 symbols from the set {'0', ..., '9', 'a', ..., 'f'}.
> > > The key should start at the beginning of the file. All symbols
> > > at offsets >= 64 are ignored.
> > >
> > > 5) Mount the volume "myvol" on the client side:
> > >
> > > # glusterfs --volfile-server=pepelac --volfile-id=myvol /mnt
> > >
> > > After successful mount the file which contains master key may be
> > > removed. NOTE: Keeping the master key between mount sessions is
> > > in user's competence.
> > >
> > > **********************************************************************
> > >
> > > WARNING! Losing the master key will make content of all regular
> > > files inaccessible. Mount with improper master key allows to access
> > > content of directories: file names are not encrypted.
> > >
> > > **********************************************************************
> > >
> > > 6. Options of crypt translator
> > >
> > > 1) "master-key": specifies location (absolute pathname) of the file
> > > which contains per-volume master key. There is no default
> > > location for master key.
> > >
> > > 2) "data-key-size": specifies size of per-file key for data
> > > encryption Possible values:
> > > . "256" default value
> > > . "512"
> > >
> > > 3) "block-size": specifies atom size. Possible values:
> > > . "512"
> > > . "1024"
> > > . "2048"
> > > . "4096" default value;
> > >
> > > 7. Test cases
> > >
> > > Any workload, which involves the following file operations:
> > >
> > > ->create();
> > > ->open();
> > > ->readv();
> > > ->writev();
> > > ->truncate();
> > > ->ftruncate();
> > > ->link();
> > > ->unlink();
> > > ->rename();
> > > ->readdirp().
> > >
> > > 8. TODOs:
> > >
> > > 1) Currently size of IOs issued by crypt translator is restricted
> > > by block_size (4K by default). We can use larger IOs to improve
> > > performance.
> > >
> > > Change-Id: Idc3523a8752888d3dd10f4fe71aa38dfc53d9ded
> > > Signed-off-by: Edward Shishkin <edward at redhat.com>
> > > ---
> > > M cli/src/cli-cmd-parser.c
> > > M configure.ac
> > > M doc/gluster.8
> > > M libglusterfs/src/gf-dirent.c
> > > M libglusterfs/src/gf-dirent.h
> > > M xlators/cluster/dht/src/dht-common.c
> > > M xlators/encryption/Makefile.am
> > > A xlators/encryption/crypt/Makefile.am
> > > A xlators/encryption/crypt/src/Makefile.am
> > > A xlators/encryption/crypt/src/atom.c
> > > A xlators/encryption/crypt/src/crypt-common.h
> > > A xlators/encryption/crypt/src/crypt-mem-types.h
> > > A xlators/encryption/crypt/src/crypt.c
> > > A xlators/encryption/crypt/src/crypt.h
> > > A xlators/encryption/crypt/src/data.c
> > > A xlators/encryption/crypt/src/keys.c
> > > A xlators/encryption/crypt/src/metadata.c
> > > A xlators/encryption/crypt/src/metadata.h
> > > M xlators/mgmt/glusterd/src/glusterd-store.c
> > > M xlators/mgmt/glusterd/src/glusterd-store.h
> > > M xlators/mgmt/glusterd/src/glusterd-volgen.c
> > > M xlators/mgmt/glusterd/src/glusterd-volume-ops.c
> > > M xlators/mgmt/glusterd/src/glusterd-volume-set.c
> > > M xlators/mgmt/glusterd/src/glusterd.h
> > > M xlators/protocol/client/src/client-rpc-fops.c
> > > 25 files changed, 8,733 insertions(+), 28 deletions(-)
> > >
> > >
> > > git pull ssh://git.gluster.org/glusterfs refs/changes/86/6086/1
> > >
> >
> >
> > _______________________________________________
> > Gluster-devel mailing list
> > Gluster-devel at nongnu.org
> > https://lists.nongnu.org/mailman/listinfo/gluster-devel
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> https://lists.nongnu.org/mailman/listinfo/gluster-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20131024/9380d7f9/attachment-0001.html>
More information about the Gluster-devel
mailing list