[Gluster-users] improving performance question

denmat tu2bgone at gmail.com
Tue Jul 12 03:54:44 UTC 2016


Hi all,

I'm running a 3 node gluster cluster in AWS (m4.xlarge, 4 x st1 500G disks)
and just starting out.
glusterfs 3.7.11 built on Apr 18 2016 12:57:30 --servers
glusterfs 3.7.13 built on Jul  8 2016 15:46:50 -- fuse clients

Volume Name: marketplace_nfs
Type: Distributed-Replicate
Volume ID: 261f1e1e-49c1-4e93-b7a5-082590fd606f
Status: Started
Number of Bricks: 6 x 2 = 12
Transport-type: tcp
Bricks:
Brick1: 10.90.3.4:/data/data1/marketplace_nfs
Brick2: 10.90.4.82:/data/data1/marketplace_nfs
Brick3: 10.90.5.26:/data/data1/marketplace_nfs
Brick4: 10.90.3.4:/data/data0/marketplace_nfs
Brick5: 10.90.4.82:/data/data0/marketplace_nfs
Brick6: 10.90.5.26:/data/data0/marketplace_nfs
Brick7: 10.90.3.4:/data/data3/marketplace_nfs
Brick8: 10.90.4.82:/data/data3/marketplace_nfs
Brick9: 10.90.5.26:/data/data3/marketplace_nfs
Brick10: 10.90.3.4:/data/data2/marketplace_nfs
Brick11: 10.90.4.82:/data/data2/marketplace_nfs
Brick12: 10.90.5.26:/data/data2/marketplace_nfs
Options Reconfigured:
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
network.ping-timeout: 15
cluster.quorum-type: none
performance.readdir-ahead: on


The idea is that at least 2 nodes have a copy of the data in the volume.

I mount this on the client
10.90.3.4:/marketplace_nfs on /data type fuse.glusterfs
(rw,default_permissions,allow_other,max_read=131072)
(10.90.3.4:/marketplace_nfs /data glusterfs
backupvolfile-server=nfs.market.customer.envato.internal 0 0)

When I execute on a fuse mounted client:

$ time find ./* -maxdepth 1 -type d  |wc -l
6219
real 10m42.753s
user 0m0.730s
sys 0m0.467s

Locally on one of the gluster nodes (over a larger number of files).
time find /data/data*/marketplace_nfs/ -maxdepth 1 -type d |wc -l
24908
real 0m0.202s
user 0m0.058s
sys 0m0.126s

I am not sure of the performance I can reasonably expect. The question I
have is is this reasonable and if not, how can I improve performance?
Should I reduce the number of bricks per node? Should I increase the
replica count? Is there a fundamental problem in my configuration?

Any help appreciated,
Den

further information:

Each gluster node has similar top:

top - 03:19:57 up 7 days, 22:19,  1 user,  load average: 12.84, 9.13, 6.56
Tasks: 142 total,   2 running, 140 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.4 us,  0.3 sy,  0.0 ni, 99.2 id,  0.0 wa,  0.0 hi,  0.0 si,
 0.0 st
KiB Mem : 16431672 total,   239224 free,   644316 used, 15548132 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 15365572 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 3782 root      20   0 2002532  83264   7736 S 206.2  0.5 101:34.25
/usr/sbin/glusterfsd -s 10.90.3.4 --volfile-id
marketplace_nfs.10.90.3.4.data-data1-marketplace_nfs -p
/var/lib/glusterd/vols/marketplace_nfs/run/10.90.3.4-data-data1-marketplace_nfs.pid
-S /var/run/gluster/aef8f955e6f50664936da6eee0158803.socket --brick-name
/data/data1/marketplace_nfs -l /va+
 3820 root      20   0 2068064  83124   8120 S 175.0  0.5  99:59.40
/usr/sbin/glusterfsd -s 10.90.3.4 --volfile-id
marketplace_nfs.10.90.3.4.data-data3-marketplace_nfs -p
/var/lib/glusterd/vols/marketplace_nfs/run/10.90.3.4-data-data3-marketplace_nfs.pid
-S /var/run/gluster/a4ab633dc313d2e6fd8e53a37c77c30b.socket --brick-name
/data/data3/marketplace_nfs -l /va+
 3801 root      20   0 1335868  72172   8128 S   6.2  0.4   9:12.61
/usr/sbin/glusterfsd -s 10.90.3.4 --volfile-id
marketplace_nfs.10.90.3.4.data-data0-marketplace_nfs -p
/var/lib/glusterd/vols/marketplace_nfs/run/10.90.3.4-data-data0-marketplace_nfs.pid
-S /var/run/gluster/ff9b43a6f3ea1b1ec0c04b9aee936dc8.socket --brick-name
/data/data0/marketplace_nfs -l /va+
    1 root      20   0  128556   8576   5736 S   0.0  0.1   0:06.74
/usr/lib/systemd/systemd --switched-root --system --deserialize 21
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.10
[kthreadd]


Each glusterfs server is showing this behaviour, massive network sends and
little disk usage:

---total-cpu-usage---- --dsk/xvdg----dsk/xvdh----dsk/xvdi----dsk/xvdj-
-net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ: read  writ: read  writ: read  writ|
recv  send|  in   out | int   csw
 24  74   2   0   0   0|   0     0 :   0  1024B:   0     0 :   0     0 |
129k   14M|   0     0 |6533  6308
 25  74   1   0   0   0|   0     0 :   0     0 :   0     0 :   0     0 |
160k   14M|   0     0 |6973  6669
 24  74   2   0   0   0|   0     0 :   0     0 :   0     0 :   0     0 |
161k   15M|   0     0 |7300  7562
 25  74   2   0   0   0|   0     0 :   0     0 :   0     0 :   0     0 |
149k   14M|   0     0 |6642  6528
 31  68   1   0   0   0|   0     0 :   0     0 :   0     0 :   0     0 |
124k   14M|   0     0 |6683  6343
 24  64  12   0   0   0|   0     0 :   0     0 :   0     0 :   0  1024B|
142k   13M|   0     0 |6800  6939
 25  73   1   0   0   1|   0     0 :   0     0 :   0     0 :   0     0 |
154k   14M|   0     0 |7054  6484
 26  70   4   0   0   0|   0     0 :   0     0 :   0     0 :   0     0 |
132k   14M|   0     0 |6803  6539
 25  69   5   0   0   1|   0     0 :   0     0 :   0     0 :   0     0 |
160k   14M|   0     0 |7323  9557
 24  70   5   0   0   0|   0     0 :   0     0 :   0     0 :   0     0 |
143k   14M|   0     0 |6769  6935
 25  73   2   0   0   0|   0     0 :   0     0 :   0     0 :   0     0 |
137k   14M|   0     0 |6976  6257
 29  70   0   0   0   0|   0  1024B:   0     0 :   0     0 :   0     0 |
171k   14M|   0     0 |6836  5853
 27  70   3   0   0   0|   0     0 :   0     0 :   0     0 :   0     0 |
179k   14M|   0     0 |7081  7359
 22  62  16   0   0   0|   0     0 :   0     0 :   0     0 :   0     0 |
133k   13M|   0     0 |6499  6478
 15  36  48   0   0   0|   0     0 :   0     0 :   0     0 :   0     0 |
 91k 8798k|   0     0 |4686  3832
 15  44  41   0   0   0|   0     0 :   0     0 :   0     0 :   0     0 |
132k   10M|   0     0 |5708  5834
 19  53  28   0   0   0|   0     0 :   0     0 :   0     0 :   0     0 |
140k   12M|   0     0 |6471  5599
 27  70   3   0   0   1|   0     0 :   0     0 :   0     0 :   0     0 |
151k   13M|   0     0 |7134  7197
 24  74   2   0   0   0|   0     0 :   0     0 :   0     0 :   0     0 |
159k   14M|   0     0 |7682  8206
 28  69   3   0   0   0|   0     0 :   0     0 :   0     0 :   0     0 |
134k   13M|   0     0 |7194  8798
 29  70   1   0   0   0|   0     0 :   0     0 :   0     0 :   0     0 |
 72k   14M|   0     0 |6365  6426
 15  44  42   0   0   0|   0     0 :   0     0 :   0     0 :   0     0 |
 61k   11M|   0     0 |4929  6895
  0   0 100   0   0   0|   0     0 :   0     0 :   0  1024B:   0     0 |
 25k   15k|   0     0 |1121  1259
  0   0 100   0   0   0|   0     0 :   0     0 :   0     0 :   0     0
|4268B 3186B|   0     0 | 458   562


The profile info is:

$ cat /tmp/dataout1
Brick: 10.90.3.4:/data/data2/marketplace_nfs
--------------------------------------------
Cumulative Stats:
   Block Size:                  1b+                  64b+
128b+
 No. of Reads:                    0                     0
  0
No. of Writes:                   15                     1
  2

   Block Size:                256b+                 512b+
 1024b+
 No. of Reads:                    0                     0
  0
No. of Writes:                    4                     5
 20

   Block Size:               2048b+                4096b+
 8192b+
 No. of Reads:                    0                     0
  0
No. of Writes:                   50                    67
114

   Block Size:              16384b+               32768b+
65536b+
 No. of Reads:                    0                     1
  0
No. of Writes:                  273                   391
831

   Block Size:             131072b+
 No. of Reads:                14301
No. of Writes:               397404
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us           8687
 FORGET
      0.00       0.00 us       0.00 us       0.00 us           1800
RELEASE
      0.00       0.00 us       0.00 us       0.00 us         199314
 RELEASEDIR
      0.00      30.00 us      26.00 us      34.00 us              3
 STATFS
      0.01     121.00 us      94.00 us     155.00 us              9
 UNLINK
      0.01      31.45 us      14.00 us      51.00 us             44
READDIR
      0.03     106.95 us      13.00 us     618.00 us             60
 GETXATTR
      0.08      32.99 us      16.00 us    1321.00 us            470
 STAT
      3.68      40.79 us      19.00 us     588.00 us          16548
INODELK
      5.16     114.49 us      60.00 us    8674.00 us           8274
RMDIR
      7.20      39.89 us       8.00 us    1114.00 us          33114
ENTRYLK
     17.10      49.37 us       1.00 us    9456.00 us          63574
OPENDIR
     31.39     115.54 us      15.00 us    4769.00 us          49862
 LOOKUP
     35.33     123.04 us      23.00 us    7915.00 us          52693
 READDIRP

    Duration: 684652 seconds
   Data Read: 1874518431 bytes
Data Written: 52198234474 bytes

Interval 9 Stats:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us            926
 RELEASEDIR
      0.06      23.40 us      23.00 us      24.00 us              5
 STAT
     23.14      45.95 us      29.00 us    3089.00 us            929
OPENDIR
     34.89     115.37 us      25.00 us     301.00 us            558
 READDIRP
     41.91     116.09 us      75.00 us    1106.00 us            666
 LOOKUP

    Duration: 58 seconds
   Data Read: 0 bytes
Data Written: 0 bytes

Brick: 10.90.3.4:/data/data0/marketplace_nfs
--------------------------------------------
Cumulative Stats:
   Block Size:                  1b+                  64b+
128b+
 No. of Reads:                    0                     0
  0
No. of Writes:                   19                     1
  1

   Block Size:                256b+                 512b+
 1024b+
 No. of Reads:                    0                     0
  0
No. of Writes:                    3                     8
 15

   Block Size:               2048b+                4096b+
 8192b+
 No. of Reads:                    0                     0
  0
No. of Writes:                   55                    71
 92

   Block Size:              16384b+               32768b+
65536b+
 No. of Reads:                    0                     1
  1
No. of Writes:                  274                   413
838

   Block Size:             131072b+
 No. of Reads:                    3
No. of Writes:               246923
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us           8688
 FORGET
      0.00       0.00 us       0.00 us       0.00 us           1792
RELEASE
      0.00       0.00 us       0.00 us       0.00 us         199385
 RELEASEDIR
      0.00      35.67 us      32.00 us      38.00 us              3
 STATFS
      0.01      31.00 us      17.00 us      56.00 us             44
READDIR
      0.01     178.92 us      94.00 us     632.00 us             13
 UNLINK
      0.03     119.72 us      13.00 us     597.00 us             60
 GETXATTR
      0.08      37.79 us      18.00 us    1467.00 us            470
 STAT
      3.29      42.40 us      19.00 us     624.00 us          16548
INODELK
      5.06     130.39 us      58.00 us   12961.00 us           8274
RMDIR
      7.53      48.42 us       9.00 us    2416.00 us          33122
ENTRYLK
     19.89      66.66 us       1.00 us    8525.00 us          63574
OPENDIR
     31.02     125.13 us      24.00 us    6020.00 us          52812
 READDIRP
     33.08     140.87 us      16.00 us    8952.00 us          50033
 LOOKUP

    Duration: 684652 seconds
   Data Read: 558591 bytes
Data Written: 32474329315 bytes

Interval 9 Stats:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us            926
 RELEASEDIR
      0.06      27.60 us      21.00 us      37.00 us              5
 STAT
     30.85      76.03 us      29.00 us    4056.00 us            929
OPENDIR
     31.14     130.61 us      29.00 us    3018.00 us            546
 READDIRP
     37.95     130.47 us      52.00 us    1154.00 us            666
 LOOKUP

    Duration: 58 seconds
   Data Read: 0 bytes
Data Written: 0 bytes

Brick: 10.90.3.4:/data/data1/marketplace_nfs
--------------------------------------------
Cumulative Stats:
   Block Size:                  1b+                 128b+
256b+
 No. of Reads:                    0                     0
  0
No. of Writes:                   11                     3
  2

   Block Size:                512b+                1024b+
 2048b+
 No. of Reads:                    0                     0
  0
No. of Writes:                    5                     6
 47

   Block Size:               4096b+                8192b+
16384b+
 No. of Reads:                    0                     0
  0
No. of Writes:                   62                   100
263

   Block Size:              32768b+               65536b+
 131072b+
 No. of Reads:                    0                     3
168
No. of Writes:                  412                   838
 334487

 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us           8657
 FORGET
      0.00       0.00 us       0.00 us       0.00 us           1779
RELEASE
      0.00       0.00 us       0.00 us       0.00 us         199385
 RELEASEDIR
      0.00      54.33 us      42.00 us      66.00 us              3
 STATFS
      0.00      29.43 us      15.00 us      44.00 us             44
READDIR
      0.00    5174.50 us      81.00 us   33257.00 us              8
 UNLINK
      0.01      44.76 us      19.00 us   26107.00 us          16548
INODELK
      0.01     131.59 us      59.00 us   10776.00 us           8274
RMDIR
      0.01      48.54 us       9.00 us   28337.00 us          33112
ENTRYLK
      0.04    4452.26 us      18.00 us  132302.00 us           1263
 STAT
      0.24  548915.85 us      13.00 us 5224650.00 us             60
 GETXATTR
      0.93    2414.17 us      16.00 us  203291.00 us          51699
 LOOKUP
      1.48    3129.09 us       1.00 us  228575.00 us          63574
OPENDIR
     97.28   68862.50 us      17.00 us  460340.00 us         189998
 READDIRP

    Duration: 684653 seconds
   Data Read: 22303103 bytes
Data Written: 43952561838 bytes

Interval 9 Stats:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us            926
 RELEASEDIR
      0.02    5927.05 us      25.00 us   75030.00 us             20
 STAT
      0.73    5543.27 us      78.00 us  141158.00 us            668
 LOOKUP
      1.29    7060.32 us      27.00 us  144956.00 us            930
OPENDIR
     97.96  105543.88 us      25.00 us  384946.00 us           4722
 READDIRP

    Duration: 58 seconds
   Data Read: 0 bytes
Data Written: 0 bytes

Brick: 10.90.3.4:/data/data3/marketplace_nfs
--------------------------------------------
Cumulative Stats:
   Block Size:                  1b+                 128b+
256b+
 No. of Reads:                    0                     0
  0
No. of Writes:                    8                     1
  3

   Block Size:                512b+                1024b+
 2048b+
 No. of Reads:                    0                     0
  0
No. of Writes:                    7                    21
 44

   Block Size:               4096b+                8192b+
16384b+
 No. of Reads:                    0                     0
  0
No. of Writes:                   62                   104
241

   Block Size:              32768b+               65536b+
 131072b+
 No. of Reads:                    2                     1
  6
No. of Writes:                  464                   791
 285704

 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us           8659
 FORGET
      0.00       0.00 us       0.00 us       0.00 us           1762
RELEASE
      0.00       0.00 us       0.00 us       0.00 us         199385
 RELEASEDIR
      0.00      38.33 us      30.00 us      52.00 us              3
 STATFS
      0.00      31.25 us      15.00 us     111.00 us             44
READDIR
      0.00    1703.00 us      97.00 us   15797.00 us             14
 UNLINK
      0.00      42.25 us      19.00 us   13149.00 us          16548
INODELK
      0.01     121.61 us      61.00 us    4492.00 us           8274
RMDIR
      0.01      40.81 us       9.00 us    5622.00 us          33124
ENTRYLK
      0.05  123021.10 us      13.00 us 2265444.00 us             60
 GETXATTR
      0.12   13702.60 us      17.00 us  296082.00 us           1263
 STAT
      1.16    3452.46 us      15.00 us  272058.00 us          50064
 LOOKUP
      1.45    3379.65 us       1.00 us  325140.00 us          63574
OPENDIR
     97.20   75905.73 us      21.00 us  533385.00 us         190235
 READDIRP

    Duration: 684652 seconds
   Data Read: 974227 bytes
Data Written: 37555843577 bytes

Interval 9 Stats:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us            926
 RELEASEDIR
      0.08   20941.70 us      20.00 us  179031.00 us             20
 STAT
      0.98    5891.36 us      27.00 us  180270.00 us            929
OPENDIR
      1.15    9582.95 us      81.00 us  194474.00 us            666
 LOOKUP
     97.79  113708.24 us      22.00 us  424985.00 us           4784
 READDIRP

    Duration: 58 seconds
   Data Read: 0 bytes
Data Written: 0 bytes

Brick: 10.90.4.82:/data/data1/marketplace_nfs
---------------------------------------------
Cumulative Stats:
   Block Size:                  1b+                 128b+
256b+
 No. of Reads:                    0                     0
  0
No. of Writes:                   11                     3
  2

   Block Size:                512b+                1024b+
 2048b+
 No. of Reads:                    0                     0
  0
No. of Writes:                    5                     6
 47

   Block Size:               4096b+                8192b+
16384b+
 No. of Reads:                    0                     0
  0
No. of Writes:                   62                   100
263

   Block Size:              32768b+               65536b+
 131072b+
 No. of Reads:                    0                     2
 43
No. of Writes:                  412                   838
 334487

 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us           8662
 FORGET
      0.00       0.00 us       0.00 us       0.00 us           1779
RELEASE
      0.00       0.00 us       0.00 us       0.00 us         199383
 RELEASEDIR
      0.00      56.33 us      35.00 us      67.00 us              3
 STATFS
      0.01     155.88 us     112.00 us     235.00 us              8
 UNLINK
      0.01      30.86 us      16.00 us      61.00 us             44
READDIR
      0.03     120.55 us      13.00 us     343.00 us             60
 GETXATTR
      0.10      45.39 us      22.00 us    2903.00 us            470
 STAT
      3.27      41.38 us      20.00 us    1282.00 us          16548
INODELK
      4.04     102.23 us      60.00 us    6847.00 us           8274
RMDIR
      5.91      37.39 us       9.00 us    1601.00 us          33112
ENTRYLK
     21.07      69.38 us      37.00 us    9144.00 us          63574
OPENDIR
     30.59     121.72 us      36.00 us    7740.00 us          52616
 READDIRP
     34.97     141.63 us      87.00 us    5463.00 us          51699
 LOOKUP

    Duration: 684653 seconds
   Data Read: 5784405 bytes
Data Written: 43952561838 bytes

Interval 9 Stats:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us            926
 RELEASEDIR
      0.04      22.60 us      22.00 us      43.00 us              5
 STAT
     27.35     131.09 us      36.00 us    2312.00 us            554
 READDIRP
     31.73      90.78 us      37.00 us    9144.00 us            928
OPENDIR
     40.88     162.48 us      87.00 us    4127.00 us            668
 LOOKUP

    Duration: 57 seconds
   Data Read: 0 bytes
Data Written: 0 bytes

Brick: 10.90.4.82:/data/data0/marketplace_nfs
---------------------------------------------
Cumulative Stats:
   Block Size:                  1b+                  64b+
128b+
 No. of Reads:                    0                     0
  0
No. of Writes:                   11                     1
  1

   Block Size:                256b+                 512b+
 1024b+
 No. of Reads:                    0                     0
  0
No. of Writes:                    7                    10
 15

   Block Size:               2048b+                4096b+
 8192b+
 No. of Reads:                    0                     0
  2
No. of Writes:                   46                    63
120

   Block Size:              16384b+               32768b+
65536b+
 No. of Reads:                    0                     0
  0
No. of Writes:                  248                   438
866

   Block Size:             131072b+
 No. of Reads:                  171
No. of Writes:               350568
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us           8702
 FORGET
      0.00       0.00 us       0.00 us       0.00 us           1862
RELEASE
      0.00       0.00 us       0.00 us       0.00 us         199383
 RELEASEDIR
      0.00      33.45 us      15.00 us      66.00 us             44
READDIR
      0.00    1804.00 us      43.00 us    5262.00 us              3
 STATFS
      0.00     678.89 us      90.00 us    5115.00 us              9
 UNLINK
      0.00     465.45 us      13.00 us   11085.00 us             60
 GETXATTR
      0.01      45.88 us      19.00 us   11892.00 us          16548
INODELK
      0.01     105.35 us      63.00 us    2869.00 us           8274
RMDIR
      0.02      38.91 us       9.00 us   18766.00 us          33114
ENTRYLK
      0.10    6311.11 us      30.00 us  230926.00 us           1264
 STAT
      0.96    1596.82 us     101.00 us  304352.00 us          50059
 LOOKUP
      1.11    1456.50 us      30.00 us  309713.00 us          63574
OPENDIR
     97.81   43329.13 us      27.00 us 1075779.00 us         188746
 READDIRP

    Duration: 684653 seconds
   Data Read: 22434084 bytes
Data Written: 46064215951 bytes

Interval 9 Stats:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us            926
 RELEASEDIR
      0.03    2744.90 us      30.00 us   49925.00 us             20
 STAT
      0.66    2076.87 us     101.00 us  149921.00 us            666
 LOOKUP
      0.83    1897.86 us      30.00 us  204514.00 us            928
OPENDIR
     98.48   44524.85 us      27.00 us 1075779.00 us           4669
 READDIRP

    Duration: 57 seconds
   Data Read: 0 bytes
Data Written: 0 bytes

Brick: 10.90.4.82:/data/data3/marketplace_nfs
---------------------------------------------
Cumulative Stats:
   Block Size:                  1b+                 128b+
256b+
 No. of Reads:                    0                     0
  0
No. of Writes:                    8                     1
  3

   Block Size:                512b+                1024b+
 2048b+
 No. of Reads:                    0                     0
  0
No. of Writes:                    7                    21
 44

   Block Size:               4096b+                8192b+
16384b+
 No. of Reads:                    0                     1
  0
No. of Writes:                   62                   104
241

   Block Size:              32768b+               65536b+
 131072b+
 No. of Reads:                    0                     0
 2099
No. of Writes:                  464                   791
 285704

 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us           8665
 FORGET
      0.00       0.00 us       0.00 us       0.00 us           1762
RELEASE
      0.00       0.00 us       0.00 us       0.00 us         199383
 RELEASEDIR
      0.00      28.00 us      24.00 us      31.00 us              3
 STATFS
      0.01      32.00 us      16.00 us      61.00 us             44
READDIR
      0.01     110.00 us      84.00 us     138.00 us             14
 UNLINK
      0.04     117.45 us      13.00 us     636.00 us             60
 GETXATTR
      0.09      37.84 us      58.00 us    2402.00 us            470
 STAT
      3.58      40.91 us      19.00 us    3205.00 us          16548
INODELK
      4.53     103.60 us      59.00 us    4561.00 us           8274
RMDIR
      6.06      34.60 us       9.00 us    1404.00 us          33124
ENTRYLK
     18.68      55.57 us      33.00 us    5689.00 us          63575
OPENDIR
     32.75     117.30 us      30.00 us    6893.00 us          52807
 READDIRP
     34.25     129.40 us     103.00 us    5439.00 us          50064
 LOOKUP

    Duration: 684653 seconds
   Data Read: 275135247 bytes
Data Written: 37555843577 bytes

Interval 9 Stats:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us            926
 RELEASEDIR
      0.06      27.80 us      58.00 us      58.00 us              5
 STAT
     28.17      70.89 us      33.00 us    4767.00 us            928
OPENDIR
     30.52     126.15 us      30.00 us    3003.00 us            565
 READDIRP
     41.25     144.62 us     103.00 us    2822.00 us            666
 LOOKUP

    Duration: 57 seconds
   Data Read: 0 bytes
Data Written: 0 bytes

Brick: 10.90.4.82:/data/data2/marketplace_nfs
---------------------------------------------
Cumulative Stats:
   Block Size:                  1b+                  16b+
 64b+
 No. of Reads:                    0                     0
  0
No. of Writes:                   12                     1
  4

   Block Size:                128b+                 256b+
512b+
 No. of Reads:                    0                     0
  0
No. of Writes:                    1                     3
  7

   Block Size:               1024b+                2048b+
 4096b+
 No. of Reads:                    0                     0
  0
No. of Writes:                   14                    43
 68

   Block Size:               8192b+               16384b+
32768b+
 No. of Reads:                    0                     1
  0
No. of Writes:                  114                   263
419

   Block Size:              65536b+              131072b+
 No. of Reads:                    2                   352
No. of Writes:                  824                322197
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us           8705
 FORGET
      0.00       0.00 us       0.00 us       0.00 us           1807
RELEASE
      0.00       0.00 us       0.00 us       0.00 us         199308
 RELEASEDIR
      0.00      29.33 us      22.00 us      36.00 us              3
 STATFS
      0.00      68.50 us      47.00 us      90.00 us              2
 READLINK
      0.00      28.57 us      14.00 us      46.00 us             44
READDIR
      0.00     344.00 us      10.00 us    4695.00 us             60
 GETXATTR
      0.00    1442.17 us      93.00 us    8467.00 us             18
 UNLINK
      0.01     100.33 us      60.00 us    8658.00 us           8274
RMDIR
      0.01      70.57 us      20.00 us  485119.00 us          16548
INODELK
      0.03      86.06 us       9.00 us  889356.00 us          33132
ENTRYLK
      0.12    9973.16 us      24.00 us  403909.00 us           1264
 STAT
      1.06    2120.37 us      89.00 us  411175.00 us          50520
 LOOKUP
      1.20    1900.07 us      30.00 us  374073.00 us          63575
OPENDIR
     97.57   52219.09 us      28.00 us 1051061.00 us         188703
 READDIRP

    Duration: 684653 seconds
   Data Read: 46382302 bytes
Data Written: 42340686672 bytes

Interval 9 Stats:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us            926
 RELEASEDIR
      0.06    6935.55 us      24.00 us  135183.00 us             20
 STAT
      0.37    1290.03 us      89.00 us  168901.00 us            668
 LOOKUP
      0.60    1503.80 us      30.00 us  185597.00 us            929
OPENDIR
     98.97   49757.74 us      28.00 us 1051061.00 us           4625
 READDIRP

    Duration: 57 seconds
   Data Read: 0 bytes
Data Written: 0 bytes

Brick: 10.90.5.26:/data/data0/marketplace_nfs
---------------------------------------------
Cumulative Stats:
   Block Size:                  1b+                  64b+
128b+
 No. of Reads:                    0                     0
  0
No. of Writes:                   11                     1
  1

   Block Size:                256b+                 512b+
 1024b+
 No. of Reads:                    0                     0
  0
No. of Writes:                    7                    10
 15

   Block Size:               2048b+                4096b+
 8192b+
 No. of Reads:                    0                     1
  0
No. of Writes:                   46                    63
120

   Block Size:              16384b+               32768b+
65536b+
 No. of Reads:                    2                     0
  1
No. of Writes:                  248                   438
866

   Block Size:             131072b+
 No. of Reads:                  905
No. of Writes:               350568
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us           8710
 FORGET
      0.00       0.00 us       0.00 us       0.00 us           1862
RELEASE
      0.00       0.00 us       0.00 us       0.00 us         199383
 RELEASEDIR
      0.00     109.67 us      32.00 us     238.00 us              3
 STATFS
      0.01      37.95 us      20.00 us      82.00 us             44
READDIR
      0.02     407.44 us      95.00 us    2742.00 us              9
 UNLINK
      0.03     114.35 us      17.00 us     417.00 us             60
 GETXATTR
      0.10      45.22 us      18.00 us    1975.00 us            470
 STAT
      3.23      39.57 us      17.00 us     512.00 us          16548
INODELK
      4.09     100.09 us      61.00 us    5656.00 us           8274
RMDIR
      6.11      37.33 us       9.00 us    1217.00 us          33114
ENTRYLK
     20.72      65.99 us       1.00 us    6995.00 us          63574
OPENDIR
     31.05     119.10 us      24.00 us    7180.00 us          52780
 READDIRP
     34.64     140.09 us      22.00 us    7989.00 us          50059
 LOOKUP

    Duration: 684616 seconds
   Data Read: 118761609 bytes
Data Written: 46064215951 bytes

Interval 9 Stats:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us            926
 RELEASEDIR
      0.45     229.20 us      22.00 us    1051.00 us              5
 STAT
     29.27     133.60 us      25.00 us    5005.00 us            558
 READDIRP
     29.65      81.38 us      30.00 us    3405.00 us            928
OPENDIR
     40.63     155.39 us      67.00 us    2723.00 us            666
 LOOKUP

    Duration: 57 seconds
   Data Read: 0 bytes
Data Written: 0 bytes

Brick: 10.90.5.26:/data/data2/marketplace_nfs
---------------------------------------------
Cumulative Stats:
   Block Size:                  1b+                  16b+
 64b+
 No. of Reads:                    0                     0
  0
No. of Writes:                   12                     1
  4

   Block Size:                128b+                 256b+
512b+
 No. of Reads:                    0                     0
  0
No. of Writes:                    1                     3
  7

   Block Size:               1024b+                2048b+
 4096b+
 No. of Reads:                    0                     0
  1
No. of Writes:                   14                    43
 68

   Block Size:               8192b+               16384b+
32768b+
 No. of Reads:                    0                     0
  1
No. of Writes:                  114                   263
419

   Block Size:              65536b+              131072b+
 No. of Reads:                    1                 10072
No. of Writes:                  824                322197
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us           8714
 FORGET
      0.00       0.00 us       0.00 us       0.00 us           1807
RELEASE
      0.00       0.00 us       0.00 us       0.00 us         199308
 RELEASEDIR
      0.00      44.33 us      25.00 us      73.00 us              3
 STATFS
      0.01      35.36 us      17.00 us      64.00 us             44
READDIR
      0.07     204.87 us      14.00 us    3290.00 us             60
 GETXATTR
      0.07     706.28 us      98.00 us   10710.00 us             18
 UNLINK
      0.09      34.82 us      18.00 us    2003.00 us            470
 STAT
      2.80      53.16 us      24.00 us    5770.00 us           9488
 READLINK
      3.50      38.14 us      19.00 us    2399.00 us          16548
INODELK
      4.40      95.87 us      54.00 us   10825.00 us           8274
RMDIR
      5.82      31.65 us       8.00 us    2535.00 us          33132
ENTRYLK
     17.28      49.00 us       1.00 us    4676.00 us          63574
OPENDIR
     32.61     116.38 us      21.00 us    8548.00 us          50519
 LOOKUP
     33.36     114.35 us      25.00 us    8772.00 us          52588
 READDIRP

    Duration: 684615 seconds
   Data Read: 1320327387 bytes
Data Written: 42340686672 bytes

Interval 9 Stats:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us            926
 RELEASEDIR
      0.05      45.50 us      45.00 us      46.00 us              2
 READLINK
      0.07      24.20 us      21.00 us      28.00 us              5
 STAT
     21.85      43.76 us      28.00 us     829.00 us            928
OPENDIR
     35.71     121.53 us      25.00 us    2901.00 us            546
 READDIRP
     42.33     117.76 us      55.00 us     296.00 us            668
 LOOKUP

    Duration: 57 seconds
   Data Read: 0 bytes
Data Written: 0 bytes

Brick: 10.90.5.26:/data/data1/marketplace_nfs
---------------------------------------------
Cumulative Stats:
   Block Size:                  1b+                  64b+
128b+
 No. of Reads:                    0                     0
  0
No. of Writes:                   19                     1
  1

   Block Size:                256b+                 512b+
 1024b+
 No. of Reads:                    0                     0
  0
No. of Writes:                    3                     8
 15

   Block Size:               2048b+                4096b+
 8192b+
 No. of Reads:                    0                     0
  0
No. of Writes:                   55                    71
 92

   Block Size:              16384b+               32768b+
65536b+
 No. of Reads:                    2                     0
  4
No. of Writes:                  274                   413
838

   Block Size:             131072b+
 No. of Reads:                15212
No. of Writes:               246923
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us           8683
 FORGET
      0.00       0.00 us       0.00 us       0.00 us           1792
RELEASE
      0.00       0.00 us       0.00 us       0.00 us         199383
 RELEASEDIR
      0.00      73.67 us      24.00 us     152.00 us              3
 STATFS
      0.00      28.59 us      14.00 us      55.00 us             44
READDIR
      0.00     179.22 us      11.00 us    2137.00 us             60
 GETXATTR
      0.00    1016.15 us      93.00 us    8833.00 us             13
 UNLINK
      0.01      43.64 us      20.00 us    8567.00 us          16548
INODELK
      0.01     105.54 us      60.00 us   18316.00 us           8274
RMDIR
      0.02      37.47 us      10.00 us    2952.00 us          33122
ENTRYLK
      0.07    3915.24 us      17.00 us  210047.00 us           1263
 STAT
      0.76    1109.99 us      19.00 us  206017.00 us          50033
 LOOKUP
      0.99    1141.17 us       1.00 us  240750.00 us          63574
OPENDIR
     98.15   40529.31 us      23.00 us 1009517.00 us         177968
 READDIRP

    Duration: 684616 seconds
   Data Read: 1994272237 bytes
Data Written: 32474329315 bytes

Interval 9 Stats:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us            926
 RELEASEDIR
      0.18   16199.70 us      21.00 us  169000.00 us             20
 STAT
      0.45    1231.90 us      57.00 us   80090.00 us            666
 LOOKUP
      0.51     990.99 us      28.00 us  117563.00 us            928
OPENDIR
     98.86   42375.20 us      27.00 us  425098.00 us           4223
 READDIRP

    Duration: 57 seconds
   Data Read: 0 bytes
Data Written: 0 bytes

Brick: 10.90.5.26:/data/data3/marketplace_nfs
---------------------------------------------
Cumulative Stats:
   Block Size:                  1b+                  64b+
128b+
 No. of Reads:                    0                     0
  0
No. of Writes:                   15                     1
  2

   Block Size:                256b+                 512b+
 1024b+
 No. of Reads:                    0                     0
  0
No. of Writes:                    4                     5
 20

   Block Size:               2048b+                4096b+
 8192b+
 No. of Reads:                    0                     0
  0
No. of Writes:                   50                    67
114

   Block Size:              16384b+               32768b+
65536b+
 No. of Reads:                    2                     1
  1
No. of Writes:                  273                   391
831

   Block Size:             131072b+
 No. of Reads:                14324
No. of Writes:               397404
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us           8669
 FORGET
      0.00       0.00 us       0.00 us       0.00 us           1800
RELEASE
      0.00       0.00 us       0.00 us       0.00 us         199389
 RELEASEDIR
      0.00      47.00 us      30.00 us      79.00 us              3
 STATFS
      0.00     171.32 us      15.00 us    6207.00 us             44
READDIR
      0.00     463.12 us      13.00 us    7289.00 us             60
 GETXATTR
      0.00    3234.22 us      99.00 us   24983.00 us              9
 UNLINK
      0.01      45.34 us      19.00 us   21238.00 us          16548
INODELK
      0.01     104.78 us      54.00 us   15752.00 us           8274
RMDIR
      0.01      38.32 us       9.00 us   22041.00 us          33114
ENTRYLK
      0.14   11662.03 us      15.00 us  313625.00 us           1263
 STAT
      1.07    2314.18 us      29.00 us  288612.00 us          49863
 LOOKUP
      1.30    2209.79 us       1.00 us  363556.00 us          63573
OPENDIR
     97.46   55158.65 us      25.00 us  602484.00 us         190328
 READDIRP

    Duration: 684616 seconds
   Data Read: 1877692288 bytes
Data Written: 52198234474 bytes

Interval 9 Stats:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls
Fop
 ---------   -----------   -----------   -----------   ------------
 ----
      0.00       0.00 us       0.00 us       0.00 us            926
 RELEASEDIR
      0.16   18884.20 us      21.00 us  172352.00 us             20
 STAT
      0.61    1582.79 us      27.00 us  232040.00 us            927
OPENDIR
      0.75    2693.21 us      58.00 us  212646.00 us            666
 LOOKUP
     98.49   50363.67 us      26.00 us  438706.00 us           4708
 READDIRP

    Duration: 57 seconds
   Data Read: 0 bytes
Data Written: 0 bytes
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160712/fdb9042a/attachment.html>


More information about the Gluster-users mailing list