[Gluster-users] RDMA high cpu usage and poor performance

[Bryan Whitehead]

I see both glusterfsd and glusterfs eat a good 70-100% of CPU while dd
runs (see below)

[root at lab0 ~]# gluster volume info

Volume Name: testrdma
Type: Replicate
Volume ID: bf7b42aa-5680-4f5c-8027-d0a56cc5e65d
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: rdma
Bricks:
Brick1: 10.1.0.10:/glust
Brick2: 10.1.0.11:/glust
Options Reconfigured:
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
performance.io-thread-count: 64
[root at lab0 ~]#

output from dd bs=1M if=/dev/zero of=/test/test.file :
29992+0 records in
29992+0 records out
31448891392 bytes (31 GB) copied, 319.076 s, 98.6 MB/s



[root at lab0 ~]# gluster volume profile testrdma info
Brick: 10.1.0.10:/glust
-----------------------
Cumulative Stats:
   Block Size:             131072b+
 No. of Reads:                    0
No. of Writes:               245984
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us              1      FORGET
      0.00       0.00 us       0.00 us       0.00 us              5     RELEASE
      0.00       0.00 us       0.00 us       0.00 us              1  RELEASEDIR
      0.00      10.50 us       9.00 us      12.00 us              2       FLUSH
      0.00      13.00 us      11.00 us      15.00 us              2     ENTRYLK
      0.00      28.00 us      28.00 us      28.00 us              1    GETXATTR
      0.00      59.50 us      18.00 us     101.00 us              2     READDIR
      0.00     201.00 us     201.00 us     201.00 us              1      CREATE
      0.00      56.00 us      18.00 us      93.00 us              5      LOOKUP
     17.89     123.50 us      47.00 us   29371.00 us         239936       WRITE
     23.06      79.60 us       2.00 us   41325.00 us         479877    FINODELK
     59.06     227.42 us      18.00 us   52835.00 us         430226    FXATTROP

    Duration: 3790 seconds
   Data Read: 0 bytes
Data Written: 32241614848 bytes

Interval 3 Stats:
   Block Size:             131072b+
 No. of Reads:                    0
No. of Writes:                82837
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us              1     RELEASE
      0.00       9.00 us       9.00 us       9.00 us              1       FLUSH
      0.00      18.00 us      18.00 us      18.00 us              1      LOOKUP
      0.00      28.00 us      28.00 us      28.00 us              1    GETXATTR
      0.00      59.50 us      18.00 us     101.00 us              2     READDIR
     16.61     129.30 us      47.00 us   23023.00 us          82837       WRITE
     19.79      77.02 us       3.00 us   30051.00 us         165668    FINODELK
     63.61     282.00 us      37.00 us   48149.00 us         145458    FXATTROP

    Duration: 472 seconds
   Data Read: 0 bytes
Data Written: 10857611264 bytes

Brick: 10.1.0.11:/glust
-----------------------
Cumulative Stats:
   Block Size:             131072b+
 No. of Reads:                    0
No. of Writes:               245984
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us              1      FORGET
      0.00       0.00 us       0.00 us       0.00 us              5     RELEASE
      0.00       0.00 us       0.00 us       0.00 us              1  RELEASEDIR
      0.00      13.00 us      12.00 us      14.00 us              2       FLUSH
      0.00      28.00 us      28.00 us      28.00 us              1    GETXATTR
      0.00      32.00 us      25.00 us      39.00 us              2     ENTRYLK
      0.00      63.50 us      32.00 us      95.00 us              2     READDIR
      0.00     179.00 us     179.00 us     179.00 us              1      CREATE
      0.00      57.40 us      27.00 us      87.00 us              5      LOOKUP
      7.76     177.75 us      40.00 us   54211.00 us         239936       WRITE
     44.67     512.94 us      17.00 us   75783.00 us         478774    FXATTROP
     47.57     544.98 us       2.00 us   99430.00 us         479877    FINODELK

    Duration: 3790 seconds
   Data Read: 0 bytes
Data Written: 32241614848 bytes

Interval 3 Stats:
   Block Size:             131072b+
 No. of Reads:                    0
No. of Writes:                82837
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us              1     RELEASE
      0.00      14.00 us      14.00 us      14.00 us              1       FLUSH
      0.00      27.00 us      27.00 us      27.00 us              1      LOOKUP
      0.00      28.00 us      28.00 us      28.00 us              1    GETXATTR
      0.00      63.50 us      32.00 us      95.00 us              2     READDIR
      7.43     218.70 us      40.00 us   37871.00 us          82837       WRITE
     45.45     670.49 us      37.00 us   75783.00 us         165308    FXATTROP
     47.12     693.66 us       2.00 us   99430.00 us         165678    FINODELK

    Duration: 472 seconds
   Data Read: 0 bytes
Data Written: 10857611264 bytes


I'll open a bug once redhat gets out of maintenance mode.

-Bryan