[Gluster-devel] 3.5.1qa4 performances

Emmanuel Dreyfus manu at netbsd.org
Tue Dec 24 05:18:57 UTC 2013


Hi 

My tests on 3.5.1qa4 are doing well, but performance seems weak. Here is gluster
volume profile information at the end of this message.

If I understand correctly, I have INODELK and LOOKUP operations that take more
than one second, which seems huge. However, the average is still reasonable,
therefore I wonder if it is really significant. How can I investigate this
further?

It may be related: on bricks that have slow LOOKUP, glustershd.log gets filled
with entries like this:
[2013-12-24 05:10:18.562322] W [client-rpc-fops.c:1103:client3_3_getxattr_cbk]
0-gfs351-client-2: remote operation failed: Stale NFS file handle. Path:
<gfid:fe59ae94-f6a4-4b28-a279-4b5b8d75c936>
(fe59ae94-f6a4-4b28-a279-4b5b8d75c936). Key: glusterfs.gfid2path
[2013-12-24 05:10:18.564198] W [client-rpc-fops.c:1103:client3_3_getxattr_cbk]
0-gfs351-client-2: remote operation failed: Stale NFS file handle. Path:
<gfid:0b70f051-df96-4a53-94a8-bc2c340a77af>
(0b70f051-df96-4a53-94a8-bc2c340a77af). Key: glusterfs.gfid2path
[2013-12-24 05:10:18.566070] W [client-rpc-fops.c:1103:client3_3_getxattr_cbk]
0-gfs351-client-2: remote operation failed: Stale NFS file handle. Path:
<gfid:f822879c-29d4-425a-ad8b-023bb5e51f6c>
(f822879c-29d4-425a-ad8b-023bb5e51f6c). Key: glusterfs.gfid2path

And the brick log loops on the same kind of errors:
[2013-12-24 05:15:21.477211] I [server-rpc-fops.c:776:server_getxattr_cbk]
0-gfs351-server: 18770957: GETXATTR (null)
(d93d337b-3d61-4a80-9516-3ba379ed2ab7) (glusterfs.gfid2path) ==> (Stale NFS file
handle)
[2013-12-24 05:15:21.477663] I [server-rpc-fops.c:154:server_lookup_cbk]
0-gfs351-server: 18770958: LOOKUP (null) (0438b8f0-2268-4f51-b014-13b70cdfd555)
==> (Stale NFS file handle)
[2013-12-24 05:15:21.478267] I [server-rpc-fops.c:154:server_lookup_cbk]
0-gfs351-server: 9014209: LOOKUP (null) (5379c125-b0dd-4636-8b7f-880247a6368f)
==> (Stale NFS file handle)
[2013-12-24 05:15:21.479118] I [server-rpc-fops.c:776:server_getxattr_cbk]
0-gfs351-server: 18770959: GETXATTR (null)
(0438b8f0-2268-4f51-b014-13b70cdfd555) (glusterfs.gfid2path) ==> (Stale NFS file
handle)
[2013-12-24 05:15:21.479546] I [server-rpc-fops.c:154:server_lookup_cbk]
0-gfs351-server: 18770960: LOOKUP (null) (bf230ee9-6942-4eb0-a7b4-c9707ce31c19)
==> (Stale NFS file handle)
[2013-12-24 05:15:21.480171] I [server-rpc-fops.c:154:server_lookup_cbk]
0-gfs351-server: 9014210: LOOKUP (null) (9dcb2827-21c4-4dab-b55e-a0102efe4456)
==> (Stale NFS file handle)

What does it means?

gluster volume profile information:

Brick: silo:/export/wd2a
------------------------
Cumulative Stats:
   Block Size:                  1b+                   2b+                   4b+
 No. of Reads:                    0                     0                     0
No. of Writes:                    3                     7                    26
 
   Block Size:                  8b+                  16b+                  32b+
 No. of Reads:                    1                     0                     0
No. of Writes:                  517                   226                   396
 
   Block Size:                 64b+                 128b+                 256b+
 No. of Reads:                  166                   391                   292
No. of Writes:                  523                   861                  1486
 
   Block Size:                512b+                1024b+                2048b+
 No. of Reads:                  137                   172                   161
No. of Writes:                 7074                  8974                 11592
 
   Block Size:               4096b+                8192b+               16384b+
 No. of Reads:                 1303                  1331                  1380
No. of Writes:                 6597                  4380                  3152
 
   Block Size:              32768b+               65536b+              131072b+
 No. of Reads:                 1198                 12077               2574709
No. of Writes:                 3846                   619                   940
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          28171      FORGET
      0.00       0.00 us       0.00 us       0.00 us          67177     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          36654  RELEASEDIR
      0.00      30.00 us      30.00 us      30.00 us              2        STAT
      0.00      69.00 us      69.00 us      69.00 us              1       FSTAT
      0.00     111.00 us     111.00 us     111.00 us              1    GETXATTR
      0.00      43.00 us      40.00 us      45.00 us              3    READLINK
      0.01      38.75 us      23.00 us      68.00 us              4     ENTRYLK
      0.01      47.50 us      33.00 us      76.00 us              4      STATFS
      0.01      65.25 us      36.00 us      98.00 us              4   FTRUNCATE
      0.03      43.56 us      23.00 us      97.00 us             16    FINODELK
      0.06     413.25 us     335.00 us     493.00 us              4       FSYNC
      0.06     432.50 us      39.00 us    1589.00 us              4       WRITE
      0.47      40.28 us      21.00 us     322.00 us            314     INODELK
      0.62    8402.00 us    1877.00 us   14927.00 us              2      CREATE
      0.73     148.71 us      28.00 us     381.00 us            132        READ
      3.84    6478.19 us    2798.00 us   23951.00 us             16    FXATTROP
     34.45     221.39 us      79.00 us  296105.00 us           4195      LOOKUP
     59.70   10253.01 us      59.00 us  392512.00 us            157     SETATTR
 
    Duration: 402686 seconds
   Data Read: 338773217672 bytes
Data Written: 502853550 bytes
 
Interval 0 Stats:
   Block Size:                  1b+                   2b+                   4b+
 No. of Reads:                    0                     0                     0
No. of Writes:                    3                     7                    26
 
   Block Size:                  8b+                  16b+                  32b+
 No. of Reads:                    1                     0                     0
No. of Writes:                  517                   226                   396
 
   Block Size:                 64b+                 128b+                 256b+
 No. of Reads:                  166                   391                   292
No. of Writes:                  523                   861                  1486
 
   Block Size:                512b+                1024b+                2048b+
 No. of Reads:                  137                   172                   161
No. of Writes:                 7074                  8974                 11592
 
   Block Size:               4096b+                8192b+               16384b+
 No. of Reads:                 1303                  1331                  1380
No. of Writes:                 6597                  4380                  3152
 
   Block Size:              32768b+               65536b+              131072b+
 No. of Reads:                 1198                 12077               2574709
No. of Writes:                 3846                   619                   940
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          28171      FORGET
      0.00       0.00 us       0.00 us       0.00 us          67177     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          36654  RELEASEDIR
      0.00      30.00 us      30.00 us      30.00 us              2        STAT
      0.00      69.00 us      69.00 us      69.00 us              1       FSTAT
      0.00     111.00 us     111.00 us     111.00 us              1    GETXATTR
      0.00      43.00 us      40.00 us      45.00 us              3    READLINK
      0.01      38.75 us      23.00 us      68.00 us              4     ENTRYLK
      0.01      47.50 us      33.00 us      76.00 us              4      STATFS
      0.01      65.25 us      36.00 us      98.00 us              4   FTRUNCATE
      0.03      43.56 us      23.00 us      97.00 us             16    FINODELK
      0.06     413.25 us     335.00 us     493.00 us              4       FSYNC
      0.06     432.50 us      39.00 us    1589.00 us              4       WRITE
      0.47      40.28 us      21.00 us     322.00 us            314     INODELK
      0.62    8402.00 us    1877.00 us   14927.00 us              2      CREATE
      0.73     148.71 us      28.00 us     381.00 us            132        READ
      3.84    6478.19 us    2798.00 us   23951.00 us             16    FXATTROP
     34.45     221.39 us      79.00 us  296105.00 us           4195      LOOKUP
     59.70   10253.01 us      59.00 us  392512.00 us            157     SETATTR
 
    Duration: 402686 seconds
   Data Read: 338773217672 bytes
Data Written: 502853550 bytes
 
Brick: debacle:/export/wd1a
---------------------------
Cumulative Stats:
   Block Size:                  1b+                   2b+                   4b+
 No. of Reads:                    1                     0                     0
No. of Writes:                    3                    12                    29
 
   Block Size:                  8b+                  16b+                  32b+
 No. of Reads:                    5                     3                     3
No. of Writes:                  549                   388                   543
 
   Block Size:                 64b+                 128b+                 256b+
 No. of Reads:                  188                   478                   270
No. of Writes:                  686                  1264                  2242
 
   Block Size:                512b+                1024b+                2048b+
 No. of Reads:                  141                   237                   158
No. of Writes:                 8500                 11940                 17533
 
   Block Size:               4096b+                8192b+               16384b+
 No. of Reads:                 1357                  1331                  3030
No. of Writes:                 6858                  4259                  3119
 
   Block Size:              32768b+               65536b+              131072b+
 No. of Reads:                 1652                  5581                 44543
No. of Writes:                 3610                   629                  1132
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          28094      FORGET
      0.00       0.00 us       0.00 us       0.00 us          61813     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          36654  RELEASEDIR
      0.00      32.00 us      32.00 us      32.00 us              1    GETXATTR
      0.00      22.50 us      21.00 us      24.00 us              2        STAT
      0.00      39.00 us      37.00 us      41.00 us              2       WRITE
      0.00      20.75 us      17.00 us      24.00 us              4     ENTRYLK
      0.00      17.75 us      13.00 us      21.00 us              8    FINODELK
      0.00      73.00 us      41.00 us     105.00 us              2   FTRUNCATE
      0.00      54.75 us      20.00 us     155.00 us              4      STATFS
      0.00     169.50 us     164.00 us     175.00 us              2       FSYNC
      0.00      50.38 us      44.00 us      58.00 us              8       FSTAT
      0.01     101.32 us      91.00 us     123.00 us             44     READDIR
      0.06    3362.38 us    1069.00 us   17592.00 us              8    FXATTROP
      0.19     328.27 us      45.00 us   62554.00 us            249     SETATTR
      0.67  140898.50 us   96307.00 us  185490.00 us              2      CREATE
      8.15    6867.40 us      11.00 us 1675137.00 us            499     INODELK
     90.91    6765.62 us      57.00 us 3022228.00 us           5649      LOOKUP
 
    Duration: 402683 seconds
   Data Read: 6536179250 bytes
Data Written: 545243668 bytes
 
Interval 0 Stats:
   Block Size:                  1b+                   2b+                   4b+
 No. of Reads:                    1                     0                     0
No. of Writes:                    3                    12                    29
 
   Block Size:                  8b+                  16b+                  32b+
 No. of Reads:                    5                     3                     3
No. of Writes:                  549                   388                   543
 
   Block Size:                 64b+                 128b+                 256b+
 No. of Reads:                  188                   478                   270
No. of Writes:                  686                  1264                  2242
 
   Block Size:                512b+                1024b+                2048b+
 No. of Reads:                  141                   237                   158
No. of Writes:                 8500                 11940                 17533
 
   Block Size:               4096b+                8192b+               16384b+
 No. of Reads:                 1357                  1331                  3030
No. of Writes:                 6858                  4259                  3119
 
   Block Size:              32768b+               65536b+              131072b+
 No. of Reads:                 1652                  5581                 44543
No. of Writes:                 3610                   629                  1132
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          28094      FORGET
      0.00       0.00 us       0.00 us       0.00 us          61813     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          36654  RELEASEDIR
      0.00      32.00 us      32.00 us      32.00 us              1    GETXATTR
      0.00      22.50 us      21.00 us      24.00 us              2        STAT
      0.00      39.00 us      37.00 us      41.00 us              2       WRITE
      0.00      20.75 us      17.00 us      24.00 us              4     ENTRYLK
      0.00      17.75 us      13.00 us      21.00 us              8    FINODELK
      0.00      73.00 us      41.00 us     105.00 us              2   FTRUNCATE
      0.00      54.75 us      20.00 us     155.00 us              4      STATFS
      0.00     169.50 us     164.00 us     175.00 us              2       FSYNC
      0.00      50.38 us      44.00 us      58.00 us              8       FSTAT
      0.01     101.32 us      91.00 us     123.00 us             44     READDIR
      0.06    3362.38 us    1069.00 us   17592.00 us              8    FXATTROP
      0.19     328.27 us      45.00 us   62554.00 us            249     SETATTR
      0.67  140898.50 us   96307.00 us  185490.00 us              2      CREATE
      8.15    6867.40 us      11.00 us 1675137.00 us            499     INODELK
     90.91    6765.62 us      57.00 us 3022228.00 us           5649      LOOKUP
 
    Duration: 402683 seconds
   Data Read: 6536179250 bytes
Data Written: 545243668 bytes
 
Brick: hangar:/export/wd1a
--------------------------
Cumulative Stats:
   Block Size:                  1b+                   2b+                   4b+
 No. of Reads:                    0                     0                     0
No. of Writes:                    3                     7                    26
 
   Block Size:                  8b+                  16b+                  32b+
 No. of Reads:                    1                     2                     0
No. of Writes:                  517                   226                   396
 
   Block Size:                 64b+                 128b+                 256b+
 No. of Reads:                  205                   432                   335
No. of Writes:                  523                   861                  1486
 
   Block Size:                512b+                1024b+                2048b+
 No. of Reads:                  120                   124                   175
No. of Writes:                 7074                  8974                 11592
 
   Block Size:               4096b+                8192b+               16384b+
 No. of Reads:                 1376                  5933                  1325
No. of Writes:                 6597                  4380                  3152
 
   Block Size:              32768b+               65536b+              131072b+
 No. of Reads:                 1116                  6665               2201096
No. of Writes:                 3846                   619                   940
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          28311      FORGET
      0.00       0.00 us       0.00 us       0.00 us          67175     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          36654  RELEASEDIR
      0.01      47.00 us      47.00 us      47.00 us              1    GETXATTR
      0.02      38.25 us      25.00 us      51.00 us              4     ENTRYLK
      0.02      47.25 us      34.00 us      76.00 us              4      STATFS
      0.03      58.00 us      46.00 us      75.00 us              4       WRITE
      0.04      74.50 us      40.00 us     115.00 us              4   FTRUNCATE
      0.06      31.56 us      20.00 us      61.00 us             16    FINODELK
      0.22     435.00 us     368.00 us     516.00 us              4       FSYNC
      0.32      39.58 us      32.00 us      50.00 us             65    READLINK
      1.46      37.34 us      20.00 us     197.00 us            314     INODELK
      4.23   17009.00 us   12995.00 us   21023.00 us              2      CREATE
      7.76    3901.38 us    2761.00 us   10117.00 us             16    FXATTROP
      8.62     441.71 us      53.00 us   44487.00 us            157     SETATTR
     77.23     149.34 us      82.00 us   10172.00 us           4162      LOOKUP
 
    Duration: 402683 seconds
   Data Read: 289280402340 bytes
Data Written: 502853550 bytes
 
Interval 0 Stats:
   Block Size:                  1b+                   2b+                   4b+
 No. of Reads:                    0                     0                     0
No. of Writes:                    3                     7                    26
 
   Block Size:                  8b+                  16b+                  32b+
 No. of Reads:                    1                     2                     0
No. of Writes:                  517                   226                   396
 
   Block Size:                 64b+                 128b+                 256b+
 No. of Reads:                  205                   432                   335
No. of Writes:                  523                   861                  1486
 
   Block Size:                512b+                1024b+                2048b+
 No. of Reads:                  120                   124                   175
No. of Writes:                 7074                  8974                 11592
 
   Block Size:               4096b+                8192b+               16384b+
 No. of Reads:                 1376                  5933                  1325
No. of Writes:                 6597                  4380                  3152
 
   Block Size:              32768b+               65536b+              131072b+
 No. of Reads:                 1116                  6665               2201096
No. of Writes:                 3846                   619                   940
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          28311      FORGET
      0.00       0.00 us       0.00 us       0.00 us          67175     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          36654  RELEASEDIR
      0.01      47.00 us      47.00 us      47.00 us              1    GETXATTR
      0.02      38.25 us      25.00 us      51.00 us              4     ENTRYLK
      0.02      47.25 us      34.00 us      76.00 us              4      STATFS
      0.03      58.00 us      46.00 us      75.00 us              4       WRITE
      0.04      74.50 us      40.00 us     115.00 us              4   FTRUNCATE
      0.06      31.56 us      20.00 us      61.00 us             16    FINODELK
      0.22     435.00 us     368.00 us     516.00 us              4       FSYNC
      0.32      39.58 us      32.00 us      50.00 us             65    READLINK
      1.46      37.34 us      20.00 us     197.00 us            314     INODELK
      4.23   17009.00 us   12995.00 us   21023.00 us              2      CREATE
      7.76    3901.38 us    2761.00 us   10117.00 us             16    FXATTROP
      8.62     441.71 us      53.00 us   44487.00 us            157     SETATTR
     77.23     149.34 us      82.00 us   10172.00 us           4162      LOOKUP
 
    Duration: 402683 seconds
   Data Read: 289280402340 bytes
Data Written: 502853550 bytes
 
Brick: hangar:/export/wd3a
--------------------------
Cumulative Stats:
   Block Size:                  1b+                   2b+                   4b+
 No. of Reads:                    0                     0                     0
No. of Writes:                    3                    12                    29
 
   Block Size:                  8b+                  16b+                  32b+
 No. of Reads:                    5                     4                     2
No. of Writes:                  549                   388                   543
 
   Block Size:                 64b+                 128b+                 256b+
 No. of Reads:                   89                   383                   302
No. of Writes:                  686                  1264                  2241
 
   Block Size:                512b+                1024b+                2048b+
 No. of Reads:                  179                   170                   146
No. of Writes:                 8499                 11938                 17532
 
   Block Size:               4096b+                8192b+               16384b+
 No. of Reads:                 1367                  1366                  1816
No. of Writes:                 6858                  4258                  3119
 
   Block Size:              32768b+               65536b+              131072b+
 No. of Reads:                 1962                 13661                 92348
No. of Writes:                 3610                   629                  1132
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          27647      FORGET
      0.00       0.00 us       0.00 us       0.00 us          61843     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          36654  RELEASEDIR
      0.00      34.25 us      21.00 us      50.00 us              4     ENTRYLK
      0.00      71.50 us      68.00 us      75.00 us              2       WRITE
      0.00      72.50 us      72.00 us      73.00 us              2   FTRUNCATE
      0.00      68.50 us      39.00 us     141.00 us              4      STATFS
      0.00     186.50 us      88.00 us     285.00 us              2    GETXATTR
      0.00      64.25 us      26.00 us     224.00 us              8    FINODELK
      0.00     288.50 us     266.00 us     311.00 us              2       FSYNC
      0.01     168.28 us     115.00 us     249.00 us             32     READDIR
      0.08    4721.12 us    2756.00 us    9548.00 us              8    FXATTROP
      0.12   28292.50 us   12698.00 us   43887.00 us              2      CREATE
      1.10    1024.51 us      23.00 us  208491.00 us            500     INODELK
      3.59    6658.77 us      59.00 us  333387.00 us            250     SETATTR
     95.09    7812.38 us      92.00 us 2591816.00 us           5651      LOOKUP
 
    Duration: 402683 seconds
   Data Read: 13735588693 bytes
Data Written: 545224769 bytes
 
Interval 0 Stats:
   Block Size:                  1b+                   2b+                   4b+
 No. of Reads:                    0                     0                     0
No. of Writes:                    3                    12                    29
 
   Block Size:                  8b+                  16b+                  32b+
 No. of Reads:                    5                     4                     2
No. of Writes:                  549                   388                   543
 
   Block Size:                 64b+                 128b+                 256b+
 No. of Reads:                   89                   383                   302
No. of Writes:                  686                  1264                  2241
 
   Block Size:                512b+                1024b+                2048b+
 No. of Reads:                  179                   170                   146
No. of Writes:                 8499                 11938                 17532
 
   Block Size:               4096b+                8192b+               16384b+
 No. of Reads:                 1367                  1366                  1816
No. of Writes:                 6858                  4258                  3119
 
   Block Size:              32768b+               65536b+              131072b+
 No. of Reads:                 1962                 13661                 92348
No. of Writes:                 3610                   629                  1132
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          27647      FORGET
      0.00       0.00 us       0.00 us       0.00 us          61843     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          36654  RELEASEDIR
      0.00      34.25 us      21.00 us      50.00 us              4     ENTRYLK
      0.00      71.50 us      68.00 us      75.00 us              2       WRITE
      0.00      72.50 us      72.00 us      73.00 us              2   FTRUNCATE
      0.00      68.50 us      39.00 us     141.00 us              4      STATFS
      0.00     186.50 us      88.00 us     285.00 us              2    GETXATTR
      0.00      64.25 us      26.00 us     224.00 us              8    FINODELK
      0.00     288.50 us     266.00 us     311.00 us              2       FSYNC
      0.01     168.28 us     115.00 us     249.00 us             32     READDIR
      0.08    4721.12 us    2756.00 us    9548.00 us              8    FXATTROP
      0.12   28292.50 us   12698.00 us   43887.00 us              2      CREATE
      1.10    1024.51 us      23.00 us  208491.00 us            500     INODELK
      3.59    6658.77 us      59.00 us  333387.00 us            250     SETATTR
     95.09    7812.38 us      92.00 us 2591816.00 us           5651      LOOKUP
 
    Duration: 402683 seconds
   Data Read: 13735588693 bytes
Data Written: 545224769 bytes
 

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu at netbsd.org




More information about the Gluster-devel mailing list