[Gluster-devel] 3.5.1qa4 performances

[Pranith Kumar Karampuri]

Emmanuel,
     When files are created and deleted even before self-heal completes, stale index files will be left in .glusterfs/indices/xattrop directory. Self-heal-daemon is supposed to delete these stale indices. Self-heal-daemon figures out that these indices are stale by checking if the op_errno on the lookup is ENOENT. But now it could return ESTALE with the following patch. So the stale index file remains and lookups keep on happening forever.

This is the patch which introduced the change in behavior:

commit d1879d04e39258ea25a49eed3244b395d4af2c1d
Author: Anand Avati <avati at redhat.com>
Date:   Thu Nov 21 06:48:17 2013 -0800

    core: fix errno for non-existent GFID
    
    When clients refer to a GFID which does not exist, the errno to
    be returned in ESTALE (and not ENOENT). Even though ENOENT might
    look "proper" most of the time, as the application eventually expects
    ENOENT even if a parent directory does not exist, not returning
    ESTALE results in resolvers (FUSE and GFAPI) to not retry resolution
    in uncached mode. This can result in spurious ENOENTs during
    concurrent path modification operations.

Pranith
----- Original Message -----
> From: "Emmanuel Dreyfus" <manu at netbsd.org>
> To: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
> Cc: gluster-devel at nongnu.org
> Sent: Thursday, December 26, 2013 11:22:45 AM
> Subject: Re: [Gluster-devel] 3.5.1qa4 performances
> 
> Emmanuel Dreyfus <manu at netbsd.org> wrote:
> 
> > I search .glusterfs/cf/1b/cf1bdf4f-b71c-4fda-963d-b7e4547e1b7c on each
> > bricks: it does not exist anywhere. I tried with other "Stale NFS file
> > handle" messages, and the file never exists in glusterfs index tree.
> 
> I suspect it has something to do with that warning on client log, which
> happens
> once:
> 
> [2013-12-25 06:04:00.111064] W [client-rpc-fops.c:1744:client3_3_xattrop_cbk]
> 0-gfs351-client-0: remote operation failed: Undefined error: 0. Path:
> /manu/usr/src/games/backgammon/common_source/obj/one.o
> (cf1bdf4f-b71c-4fda-963d-b7e4547e1b7c)
> 
> Althought the index does not exist, the file does:
> silo:/export/wd2a//manu/usr/src/games/backgammon/common_source/obj/one.o
> hangar:/export/wd1a//manu/usr/src/games/backgammon/common_source/obj/one.o
> 
> Both have same extended attributes:
> 
> trusted.afr.gfs351-client-0
>    000   00 00 00 00 00 00 00 00 00 00 00 00                ............
> trusted.afr.gfs351-client-1
>    000   00 00 00 00 00 00 00 00 00 00 00 00                ............
> trusted.afr.gfs351-gfid
>    000   da 17 69 27 49 8f 4d e6 99 2b 53 25 be 9d 11 a3    ..i'I.M..+S%....
> 
> For parent directory, trusted.gfid is the same on all bricks.
> trusted.glusterfs.dht it:
> silo:/export/wd2a//manu/usr/src/games/backgammon/common_source/obj
>    000   00 00 00 01 00 00 00 00 00 00 00 00 7f ff ff fe    ...............
> hangar:/export/wd1a//manu/usr/src/games/backgammon/common_source/obj
>    000   00 00 00 01 00 00 00 00 00 00 00 00 7f ff ff fe    ...............
> hangar:/export/wd3a//manu/usr/src/games/backgammon/common_source/obj
>    000   00 00 00 01 00 00 00 00 7f ff ff ff ff ff ff ff    ...............
> debacle:/export/wd1a//manu/usr/src/games/backgammon/common_source/obj
>    000   00 00 00 01 00 00 00 00 7f ff ff ff ff ff ff ff    ...............
> 
> --
> Emmanuel Dreyfus
> http://hcpnet.free.fr/pubz
> manu at netbsd.org
>