[Gluster-devel] 3.5.1qa4 performances

[Emmanuel Dreyfus]

Pranith Kumar Karampuri <pkarampu at redhat.com> wrote:

>      The only reason I can think of Lookups taking that long is if they
> are stuck in io-threads least priority queue. You can probably add debugs
> in io-threads xlator to do it temporarily. But I will add some code there
> to get such stats in future.

I collected stats again, and here is the only "pig" remaining:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
(...)
     94.53    5694.96 us      86.00 us 6964122.00 us          20862      LOOKUP

The log of that brick is flood with NFS stale handle messages:

[2013-12-25 05:26:41.506742] I [server-rpc-fops.c:154:server_lookup_cbk]
0-gfs351-server: 3352413: LOOKUP (null) (4e2b7640-9303-4edb-b84d-7e68ee95e786)
==> (Stale NFS file handle)
[2013-12-25 05:26:41.507985] I [server-rpc-fops.c:154:server_lookup_cbk]
0-gfs351-server: 3352414: LOOKUP (null) (d93d337b-3d61-4a80-9516-3ba379ed2ab7)
==> (Stale NFS file handle)
[2013-12-25 05:26:41.508997] I [server-rpc-fops.c:154:server_lookup_cbk]
0-gfs351-server: 3352415: LOOKUP (null) (0438b8f0-2268-4f51-b014-13b70cdfd555)
==> (Stale NFS file handle)
[2013-12-25 05:26:41.510713] I [server-rpc-fops.c:154:server_lookup_cbk]
0-gfs351-server: 3352416: LOOKUP (null) (bf230ee9-6942-4eb0-a7b4-c9707ce31c19)
==> (Stale NFS file handle)
[2013-12-25 05:26:41.512476] I [server-rpc-fops.c:154:server_lookup_cbk]
0-gfs351-server: 3352417: LOOKUP (null) (80726ba2-1b6e-426c-9a7d-45ab477e25f7)
==> (Stale NFS file handle)

It loops with around 60 messages per second for each of the folllowing files:
  0438b8f0-2268-4f51-b014-13b70cdfd555
  3835947b-823e-4940-80ad-36861cbe87a2
  4e2b7640-9303-4edb-b84d-7e68ee95e786
  5210739b-4a20-4356-821f-645214309e31
  70654ae9-9311-428e-8aeb-a11bdfe999ea
  80726ba2-1b6e-426c-9a7d-45ab477e25f7
  8c1907fa-2dac-42f5-94a1-301acbab47d3
  a3d6d354-74db-49f7-95ef-986b0df87bbc
  ae4fe8de-bc9f-42cb-9419-5aa5657bd082
  bf230ee9-6942-4eb0-a7b4-c9707ce31c19
  d93d337b-3d61-4a80-9516-3ba379ed2ab7
  dc7e344e-f75b-45fd-884c-933a10de8833

What does it means?

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu at netbsd.org