[Gluster-devel] Preventing lookups from serving metadata.

[Frank Filz]

On 11/08/2016 06:30 AM, Shyam wrote:
> On 11/08/2016 08:10 AM, Ravishankar N wrote:
>> So there is a class of bugs* in exposed in replicate volumes where if
>> the only good copy of the file is down, we still end up serving stale
>> data to the application because of caching
>> in various layers outside gluster. In fuse, this can be mitigated by
>> setting attribute and entry-timeout to zero so that the actual FOP
>> (stat, read, write, etc) reaches AFR which will
>> then fail it with EIO. But this does not work for NFS based clients.
>>
>> 1) Is there a way by which we can make the 'lookup' FOP in gluster do
>> just that- i.e. tell whether the entry exists or not, and *not serve*
>> any other (stat) information except the gfid?
>
> To my reading of NFSv3 this seems possible and here is what I have
> gathered,
>
> LOOKUP RPC returns post_op_attr for the object being looked up, and
> this can contain no attribute information. See [3] and [4] below.
> LOOKUP also returns 'nfs_fh3 object' which is where the GFID related
> information sits.
>
> Gluster NFS code, seems to adhere to this as seen in [5]
>
> NFSv3 RFC does not encourage this (reading [4] "This appears to make
> returning attributes optional. However,.." states this), but is not
> prevented.
>
> So overall with NFSv3 this seems to be doable. We possibly need data
> from Ganesha implementation and NFSv4 related reading on this.
>
In NFS v4, the operations are performed using COMPOUND. There is a
GETATTR compound op which can fail separately from the LOOKUP op.
READDIR can also return attributes for each entry and is able to fail to
return attributes for some entries (as long as ATTR4_RDATTR_ERR is one
of the requested attributes). I know all of this works at least for
directory entries and the client handles it.

You can see what happens if attributes are not available by having an
export that requires a security flavor (such as krb5) that is not
available to the mount. If you mount the pseudofs, and the ls -l the
pseudofs directory the exports are in, you will see some listings with ?
for the attributes, for example:

[root at localhost src]# mount /mnt4
[root at localhost src]# ls -l /mnt4
ls: cannot access /mnt4/test4: Operation not permitted
ls: cannot access /mnt4/aaa: Operation not permitted
total 28
???????????  ? ?    ?       ?            ? aaa
drwxr-xr-x.  3 root root 4096 Jun  3  2015 default
drwxr-xr-x.  4 root root 4096 Apr 13  2016 exp1
drwxr-xr-x.  3 root root 4096 Apr 13  2016 exp4
dr-xr-xr-x.  2 root root 4096 Sep 16  2015 none
drwxr-xr-x.  2 root root 4096 Sep 16  2015 sys
drwxrwxrwx. 11 root root 4096 Nov  7 10:15 test1
drwxr-xr-x.  3 root root 4096 Sep 14 13:43 test2
???????????  ? ?    ?       ?            ? test4
drwxr-xr-x.  3 root root   29 Apr 17  2014 xfs1


aaa and test4 are exports that require krb5 (my client is not set up with krb5).

The only issue is that FSAL readdir callback API MAY not actually have a way to indicate failure of attribute fetch...

Ganesha does presume on initial caching of an object that attribute fetch was successful since it needs to know a few attributes to even instantiate an inode.

Frank


>>
>> 2) If that is not possible, it is okay for AFR to fail lookups with EIO
>> if when client-quorum is met and there is no source available? The
>> downside is that if we fail lookups with EIO, even unlink cannot be
>> served.
>> (Think of  a user who doesn't want to resolve a file in split-brain, but
>> rather delete it).
>>
>> Thanks,
>> Ravi
>>
>> *bugs:
>> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1356974
>> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1224709
> [3] NFS v3 RFC definition of LOOKUP:
> https://tools.ietf.org/html/rfc1813#section-3.3.3
> [4] NFS v3 definition of post_op_attr
> https://tools.ietf.org/html/rfc1813#section-2.6 (search for
> post_op_attr after reaching this section)
> [5] Gluster NFS code pointer:
> https://github.com/gluster/glusterfs/blob/master/xlators/nfs/server/src/nfs3-helpers.c#L365
> and more specifically,
> https://github.com/gluster/glusterfs/blob/master/xlators/nfs/server/src/nfs3-helpers.c#L376
>
>>
>>
>>
>>
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-devel
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel