[Gluster-devel] 3.5.1qa4 performances

[Pranith Kumar Karampuri]

Emmanuel,
    For some reason self-heal on those files must be failing. Wonder why? Could you find what is the file on the bricks for each of those files (using find -inum) and give me the getfattr output of those files and their parent directories please.

Pranith

----- Original Message -----
> From: "Emmanuel Dreyfus" <manu at netbsd.org>
> To: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
> Cc: gluster-devel at nongnu.org
> Sent: Wednesday, December 25, 2013 11:03:06 AM
> Subject: Re: [Gluster-devel] 3.5.1qa4 performances
> 
> Pranith Kumar Karampuri <pkarampu at redhat.com> wrote:
> 
> >      The only reason I can think of Lookups taking that long is if they
> > are stuck in io-threads least priority queue. You can probably add debugs
> > in io-threads xlator to do it temporarily. But I will add some code there
> > to get such stats in future.
> 
> I collected stats again, and here is the only "pig" remaining:
>  %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
>  Fop
>  ---------   -----------   -----------   -----------   ------------
>  ----
> (...)
>      94.53    5694.96 us      86.00 us 6964122.00 us          20862
>      LOOKUP
> 
> The log of that brick is flood with NFS stale handle messages:
> 
> [2013-12-25 05:26:41.506742] I [server-rpc-fops.c:154:server_lookup_cbk]
> 0-gfs351-server: 3352413: LOOKUP (null)
> (4e2b7640-9303-4edb-b84d-7e68ee95e786)
> ==> (Stale NFS file handle)
> [2013-12-25 05:26:41.507985] I [server-rpc-fops.c:154:server_lookup_cbk]
> 0-gfs351-server: 3352414: LOOKUP (null)
> (d93d337b-3d61-4a80-9516-3ba379ed2ab7)
> ==> (Stale NFS file handle)
> [2013-12-25 05:26:41.508997] I [server-rpc-fops.c:154:server_lookup_cbk]
> 0-gfs351-server: 3352415: LOOKUP (null)
> (0438b8f0-2268-4f51-b014-13b70cdfd555)
> ==> (Stale NFS file handle)
> [2013-12-25 05:26:41.510713] I [server-rpc-fops.c:154:server_lookup_cbk]
> 0-gfs351-server: 3352416: LOOKUP (null)
> (bf230ee9-6942-4eb0-a7b4-c9707ce31c19)
> ==> (Stale NFS file handle)
> [2013-12-25 05:26:41.512476] I [server-rpc-fops.c:154:server_lookup_cbk]
> 0-gfs351-server: 3352417: LOOKUP (null)
> (80726ba2-1b6e-426c-9a7d-45ab477e25f7)
> ==> (Stale NFS file handle)
> 
> It loops with around 60 messages per second for each of the folllowing files:
>   0438b8f0-2268-4f51-b014-13b70cdfd555
>   3835947b-823e-4940-80ad-36861cbe87a2
>   4e2b7640-9303-4edb-b84d-7e68ee95e786
>   5210739b-4a20-4356-821f-645214309e31
>   70654ae9-9311-428e-8aeb-a11bdfe999ea
>   80726ba2-1b6e-426c-9a7d-45ab477e25f7
>   8c1907fa-2dac-42f5-94a1-301acbab47d3
>   a3d6d354-74db-49f7-95ef-986b0df87bbc
>   ae4fe8de-bc9f-42cb-9419-5aa5657bd082
>   bf230ee9-6942-4eb0-a7b4-c9707ce31c19
>   d93d337b-3d61-4a80-9516-3ba379ed2ab7
>   dc7e344e-f75b-45fd-884c-933a10de8833
> 
> What does it means?
> 
> --
> Emmanuel Dreyfus
> http://hcpnet.free.fr/pubz
> manu at netbsd.org
>