[Gluster-users] glusterfsd/glusterfs process taking CPU load higher than usual

ABHISHEK PALIWAL abhishpaliwal at gmail.com
Tue Nov 29 13:26:01 UTC 2016


Hi Team,

I have two board setup and on which we have one volume with two brick on
each board.

When I was checking the cpu load I found glusterfsd/glusterfs process
taking higher CPU load then usual like below:

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+
COMMAND
 3117 root      20   0 1429m  48m 3636 R   98  0.2 212:12.35
fpipt_main_thre
12299 root      20   0 1139m  52m 4192 R   73  0.2 120:41.69
glusterfsd
 4517 root      20   0 1139m  52m 4192 S   72  0.2 121:01.54
glusterfsd
 1915 root      20   0 1139m  52m 4192 R   62  0.2 121:16.22
glusterfsd
14633 root      20   0 1139m  52m 4192 S   62  0.2 120:37.13
glusterfsd
 1992 root      20   0  634m 154m 4340 S   57  0.7  68:11.18
glusterfs
17886 root      20   0 1139m  52m 4192 R   55  0.2 120:28.57
glusterfsd
 2664 root      20   0  783m  31m 4708 S   52  0.1 100:13.12
Scc_SctpHost_pr
 1914 root      20   0 1139m  52m 4192 S   50  0.2 121:20.19
glusterfsd
12556 root      20   0 1139m  52m 4192 S   50  0.2 120:31.38
glusterfsd
 1583 root      20   0 1139m  52m 4192 R   48  0.2 121:16.83
glusterfsd
12112 root      20   0 1139m  52m 4192 R   43  0.2 120:58.73
glusterfsd

Is there any way to identify the way or to reduce this high load.

I have also collected the volume profile logs but don't know how to
understand or analyze those logs.

I am attaching those logs here.
-- 
Regards
Abhishek Paliwal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20161129/9b0583d6/attachment.html>
-------------- next part --------------
Log start: 161129-130518 - 10.67.29.150 - moshell 16.0y - /home/emamiko/EVO8300/Issues/CPU_highLoad_C1MP/Gluster_vol_profile.txt

EVOA_8300-1> 

EVOA_8300-1> gluster volume profile c_glusterfs start

161129-13:05:26 10.67.29.150 16.0y CPP_MOM-CPP-LSV203-gen2_gen2_COMPLETE stopfile=/tmp/15640
$ gluster volume profile c_glusterfs start
Starting volume profile on c_glusterfs has been successful 
$ 

EVOA_8300-1> 

EVOA_8300-1> # wait for a minute or two

EVOA_8300-1> wait 120 

Waiting from [2016-11-29 13:05:31] to [2016-11-29 13:07:31]...Done.

EVOA_8300-1> gluster volume profile c_glusterfs info 

161129-13:07:32 10.67.29.150 16.0y CPP_MOM-CPP-LSV203-gen2_gen2_COMPLETE stopfile=/tmp/15640
$ gluster volume profile c_glusterfs info
Brick: 10.32.0.48:/opt/lvmdir/c2/brick
--------------------------------------
Cumulative Stats:
   Block Size:                  1b+                   4b+                   8b+ 
 No. of Reads:                    0                     1                     0 
No. of Writes:                    6                     2                     9 
 
   Block Size:                 16b+                  32b+                  64b+ 
 No. of Reads:                    0                     6                     6 
No. of Writes:                    2                     9                     6 
 
   Block Size:                128b+                 256b+                 512b+ 
 No. of Reads:                    1                     2                     7 
No. of Writes:                   22                    26                    67 
 
   Block Size:               1024b+                2048b+                4096b+ 
 No. of Reads:                   14                     3                     5 
No. of Writes:                   79                   129                    35 
 
   Block Size:               8192b+               16384b+               32768b+ 
 No. of Reads:                    3                    14                    12 
No. of Writes:                   13                     0                     1 
 
   Block Size:              65536b+              131072b+ 
 No. of Reads:                   16                   224 
No. of Writes:                   20                    16 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us              2      FORGET
      0.00       0.00 us       0.00 us       0.00 us            248     RELEASE
      0.00       0.00 us       0.00 us       0.00 us         189505  RELEASEDIR
      0.00      70.33 us      63.00 us      74.00 us              3      STATFS
      0.00     172.50 us      90.00 us     255.00 us              2     READDIR
      0.00      92.67 us      46.00 us     128.00 us              6    GETXATTR
      6.29     192.50 us      83.00 us    4065.00 us           4443    SETXATTR
      7.19      88.90 us       3.00 us    1724.00 us          10996     OPENDIR
     14.39     210.96 us      33.00 us   45438.00 us           9282     INODELK
     72.13     278.64 us      44.00 us    1452.00 us          35214      LOOKUP
 
    Duration: 7146 seconds
   Data Read: 32013955 bytes
Data Written: 4931237 bytes
 
Interval 2 Stats:
   Block Size:                  8b+                 512b+                1024b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                    1                     5                     5 
 
   Block Size:               2048b+                4096b+                8192b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                    5                     6                     5 
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us              1     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          15632  RELEASEDIR
      0.01     172.50 us      90.00 us     255.00 us              2     READDIR
      0.01      97.00 us      46.00 us     128.00 us              4    GETXATTR
      6.38     192.98 us      85.00 us     460.00 us           1879    SETXATTR
      7.46      88.90 us       3.00 us     629.00 us           4767     OPENDIR
      7.82     114.99 us      33.00 us   16289.00 us           3865     INODELK
     78.32     278.35 us      44.00 us    1452.00 us          15988      LOOKUP
 
    Duration: 541 seconds
   Data Read: 0 bytes
Data Written: 103460 bytes
 
Brick: 10.32.1.144:/opt/lvmdir/c2/brick
---------------------------------------
Cumulative Stats:
   Block Size:                  1b+                   4b+                   8b+ 
 No. of Reads:                    0                     2                     1 
No. of Writes:                   12                     2                     9 
 
   Block Size:                 16b+                  32b+                  64b+ 
 No. of Reads:                    0                    10                    10 
No. of Writes:                    4                    18                    34 
 
   Block Size:                128b+                 256b+                 512b+ 
 No. of Reads:                    1                     2                     5 
No. of Writes:                   47                    66                    90 
 
   Block Size:               1024b+                2048b+                4096b+ 
 No. of Reads:                   13                     5                    15 
No. of Writes:                   93                   131                    36 
 
   Block Size:               8192b+               16384b+               32768b+ 
 No. of Reads:                   28                    47                   100 
No. of Writes:                   15                     1                     1 
 
   Block Size:              65536b+              131072b+ 
 No. of Reads:                  431                   881 
No. of Writes:                    2                     7 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us              8      FORGET
      0.00       0.00 us       0.00 us       0.00 us            766     RELEASE
      0.00       0.00 us       0.00 us       0.00 us         249890  RELEASEDIR
      0.00      80.67 us      59.00 us      99.00 us              3      STATFS
      0.00     153.50 us      95.00 us     235.00 us              4    GETXATTR
      0.12     608.38 us     124.00 us   82081.00 us           4446    SETXATTR
      0.20     459.28 us      42.00 us  102079.00 us           9334     INODELK
      0.85    1692.49 us      51.00 us   91491.00 us          10996     OPENDIR
      0.95     571.52 us      24.00 us   84614.00 us          36224        STAT
      1.32     815.53 us      87.00 us   87059.00 us          35231      LOOKUP
     96.57   58347.51 us      62.00 us  246355.00 us          36156    READDIRP
 
    Duration: 8695 seconds
   Data Read: 162964890 bytes
Data Written: 2025544 bytes
 
Interval 2 Stats:
   Block Size:                  8b+                 512b+                1024b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                    1                     5                     5 
 
   Block Size:               2048b+                4096b+                8192b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                    5                     6                     5 
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us              1     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          15632  RELEASEDIR
      0.00     200.00 us     165.00 us     235.00 us              2    GETXATTR
      0.12     659.87 us     124.00 us   82081.00 us           1882    SETXATTR
      0.19     501.62 us      48.00 us   80022.00 us           3885     INODELK
      0.82    1769.25 us      52.00 us   64336.00 us           4769     OPENDIR
      1.19     583.49 us      26.00 us   84614.00 us          20858        STAT
      1.31     839.80 us      87.00 us   87059.00 us          16005      LOOKUP
     96.37   61324.40 us      74.00 us  241437.00 us          16125    READDIRP
 
    Duration: 541 seconds
   Data Read: 0 bytes
Data Written: 103460 bytes
 
$ 

EVOA_8300-1> 

EVOA_8300-1> # wait for a minute

EVOA_8300-1> wait 60

Waiting from [2016-11-29 13:07:33] to [2016-11-29 13:08:33]...Done.

EVOA_8300-1> gluster volume profile c_glusterfs info 

161129-13:08:35 10.67.29.150 16.0y CPP_MOM-CPP-LSV203-gen2_gen2_COMPLETE stopfile=/tmp/15640
$ gluster volume profile c_glusterfs info
Brick: 10.32.0.48:/opt/lvmdir/c2/brick
--------------------------------------
Cumulative Stats:
   Block Size:                  1b+                   4b+                   8b+ 
 No. of Reads:                    0                     1                     0 
No. of Writes:                    6                     2                     9 
 
   Block Size:                 16b+                  32b+                  64b+ 
 No. of Reads:                    0                     6                     6 
No. of Writes:                    2                     9                     6 
 
   Block Size:                128b+                 256b+                 512b+ 
 No. of Reads:                    1                     2                     7 
No. of Writes:                   22                    26                    67 
 
   Block Size:               1024b+                2048b+                4096b+ 
 No. of Reads:                   14                     3                     5 
No. of Writes:                   79                   129                    35 
 
   Block Size:               8192b+               16384b+               32768b+ 
 No. of Reads:                    3                    14                    12 
No. of Writes:                   13                     0                     1 
 
   Block Size:              65536b+              131072b+ 
 No. of Reads:                   16                   224 
No. of Writes:                   20                    16 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us              2      FORGET
      0.00       0.00 us       0.00 us       0.00 us            248     RELEASE
      0.00       0.00 us       0.00 us       0.00 us         191343  RELEASEDIR
      0.00      70.33 us      63.00 us      74.00 us              3      STATFS
      0.00     172.50 us      90.00 us     255.00 us              2     READDIR
      0.00      92.67 us      46.00 us     128.00 us              6    GETXATTR
      6.04     192.80 us      83.00 us    4065.00 us           4703    SETXATTR
      7.60      88.92 us       3.00 us    1724.00 us          12830     OPENDIR
     14.10     215.83 us      33.00 us   48369.00 us           9809     INODELK
     72.26     279.27 us      44.00 us    1452.00 us          38848      LOOKUP
 
    Duration: 7208 seconds
   Data Read: 32013955 bytes
Data Written: 4931237 bytes
 
Interval 3 Stats:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us           1838  RELEASEDIR
      3.65     198.02 us      84.00 us     466.00 us            260    SETXATTR
     11.27     301.67 us      46.00 us   48369.00 us            527     INODELK
     11.57      89.04 us      49.00 us     660.00 us           1834     OPENDIR
     73.51     285.37 us     200.00 us     640.00 us           3634      LOOKUP
 
    Duration: 62 seconds
   Data Read: 0 bytes
Data Written: 0 bytes
 
Brick: 10.32.1.144:/opt/lvmdir/c2/brick
---------------------------------------
Cumulative Stats:
   Block Size:                  1b+                   4b+                   8b+ 
 No. of Reads:                    0                     2                     1 
No. of Writes:                   12                     2                     9 
 
   Block Size:                 16b+                  32b+                  64b+ 
 No. of Reads:                    0                    10                    10 
No. of Writes:                    4                    18                    34 
 
   Block Size:                128b+                 256b+                 512b+ 
 No. of Reads:                    1                     2                     5 
No. of Writes:                   47                    66                    90 
 
   Block Size:               1024b+                2048b+                4096b+ 
 No. of Reads:                   13                     5                    15 
No. of Writes:                   93                   131                    36 
 
   Block Size:               8192b+               16384b+               32768b+ 
 No. of Reads:                   28                    47                   100 
No. of Writes:                   15                     1                     1 
 
   Block Size:              65536b+              131072b+ 
 No. of Reads:                  431                   881 
No. of Writes:                    2                     7 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us              8      FORGET
      0.00       0.00 us       0.00 us       0.00 us            766     RELEASE
      0.00       0.00 us       0.00 us       0.00 us         251728  RELEASEDIR
      0.00      80.67 us      59.00 us      99.00 us              3      STATFS
      0.00     153.50 us      95.00 us     235.00 us              4    GETXATTR
      0.12     628.59 us     112.00 us   82081.00 us           4706    SETXATTR
      0.18     469.51 us      42.00 us  102079.00 us           9864     INODELK
      0.84    1667.88 us      49.00 us   91491.00 us          12830     OPENDIR
      0.89     580.14 us      24.00 us   84614.00 us          39055        STAT
      1.25     823.75 us      87.00 us   87059.00 us          38866      LOOKUP
     96.72   58491.54 us      62.00 us  248778.00 us          42190    READDIRP
 
    Duration: 8757 seconds
   Data Read: 162964890 bytes
Data Written: 2025544 bytes
 
Interval 3 Stats:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us           1838  RELEASEDIR
      0.07     974.16 us     112.00 us   81951.00 us            260    SETXATTR
      0.09     649.60 us      53.00 us   59116.00 us            530     INODELK
      0.53     690.48 us      30.00 us   77783.00 us           2831        STAT
      0.76    1520.32 us      49.00 us   60490.00 us           1834     OPENDIR
      0.90     903.37 us     220.00 us   52687.00 us           3635      LOOKUP
     97.65   59354.60 us     162.00 us  248778.00 us           6034    READDIRP
 
    Duration: 62 seconds
   Data Read: 0 bytes
Data Written: 0 bytes
 
$ 

EVOA_8300-1> 

EVOA_8300-1> gluster volume profile c_glusterfs stop

161129-13:08:36 10.67.29.150 16.0y CPP_MOM-CPP-LSV203-gen2_gen2_COMPLETE stopfile=/tmp/15640
$ gluster volume profile c_glusterfs stop
Stopping volume profile on c_glusterfs has been successful 
$ 

EVOA_8300-1> 

EVOA_8300-1> gluster volume c_glusterfs top

161129-13:08:37 10.67.29.150 16.0y CPP_MOM-CPP-LSV203-gen2_gen2_COMPLETE stopfile=/tmp/15640
$ gluster volume c_glusterfs top
unrecognized word: c_glusterfs (position 1)
$ 

EVOA_8300-1> gluster volume top c_glusterfs read

161129-13:10:14 10.67.29.150 16.0y CPP_MOM-CPP-LSV203-gen2_gen2_COMPLETE stopfile=/tmp/15640
$ gluster volume top c_glusterfs read
Brick: 10.32.0.48:/opt/lvmdir/c2/brick
Count		filename
=======================
42		/logfiles/availability/CELLO_AVAILABILITY2_LOG.xml
8		/logfiles/alarm_event/ALARM_LOG.xml
5		/logfiles/alarm_event/EVENT_LOG.xml
4		/configuration/oamrdncy.cfg
3		/security/usrmgmt_data/shadow
3		/security/usrmgmt_data/gshadow
3		/security/usrmgmt_data/group
3		/security/usrmgmt_data/passwd
2		/loadmodules_norepl/CXC1724447%25_R13Z/cello/emas/CPPClientConfig.xml
2		/logfiles/security/CELLO_SECURITYEVENT_LOG.xml
2		/logfiles/audit_trail/CORBA_AUDITTRAIL_LOG.xml
2		/systemfiles/cello/ss7/segments
2		/license/parameters_01.lic
1		/loadmodules_norepl/CXC1724447%25_R13Z/cello/emas/CPPClientConfig.xml.tmp
1		/systemfiles/cello/osa/oei
1		/license/licenseKeyInfo.lic
Brick: 10.32.1.144:/opt/lvmdir/c2/brick
Count		filename
=======================
127		<gfid:11924eca-0be7-4998-a784-f5e8c3efae3e>
91		<gfid:f807fd10-816c-471f-aba4-af4334ca4a04>
43		<gfid:8bdb94b6-aaf2-4cc0-974a-74124a5f9f97>
5		<gfid:8629e4b6-3291-4ef7-b731-c79ede4d261c>
4		/logfiles/systemlog/syslog
2		<gfid:7a108660-4fb7-47bf-b182-8361bcf1b544>
2		<gfid:23806fe8-1c01-4ec7-837a-55af95998fe3>
2		<gfid:5ea755d2-8c2a-4bb6-9ef2-89d254ebfa22>
2		<gfid:cf5acaaf-d3e4-442d-99d3-c790b167f762>
2		<gfid:0ef8d31f-5b73-4e1c-a4a2-de0f4213e9af>
2		<gfid:65384e0c-9ea6-4ed2-a51d-64f11ef22d3a>
2		<gfid:a148ee86-e321-4afd-a00c-70407917358f>
2		<gfid:0c03ea31-a961-4c2d-b471-7209d91f1cf9>
2		<gfid:d9a0cba5-e555-4533-bce5-4752838053b1>
2		<gfid:d5eb913b-4b5c-4b34-acd6-6e294cd14b82>
2		<gfid:d7e0add1-a01a-4710-a863-2e1b26e56e67>
2		<gfid:547cc4a9-399b-4b56-9c7c-fce47a430854>
2		<gfid:17293ca0-0779-4b72-94fb-5de85d127bed>
2		<gfid:adb643a5-8c95-4cb6-bb2e-0bfee8c74d8d>
2		<gfid:f29481ca-7ba7-4b7c-ad1d-61cbd52c36b9>
2		<gfid:bc6e0010-c9b2-45fa-bbc0-b82cdad56139>
2		<gfid:e8bf7ca1-888f-4b2e-88c8-b0467fc46e4b>
2		<gfid:22eee50d-2aee-40c7-a764-9c4e29bed522>
2		<gfid:691fc8dc-5945-4f96-a044-16addbe6e924>
2		<gfid:a87ab5e1-6f59-42f3-96bf-641fec842c7b>
2		/loadmodules_norepl/CXC1724447%25_R13Z/cello/emas/CPPClientConfig.xml.tmp
2		<gfid:70422951-de13-4111-8563-62f0af3a24fd>
2		/systemfiles/cello/ss7/segments
2		<gfid:2f2ab403-5497-45c7-b377-d4b093c758bd>
2		/loadmodules_norepl/CXC1723372_R94B01
2		/loadmodules_norepl/CXC1720772_R86A01
2		/configuration/oamrdncy.cfg
1		/pmd/96/002500/pmd-ospi_sccadm-ppc-2166-20161129-100851.tgz
1		<gfid:0f456e93-14c5-4ab9-9b4c-3125d28425fc>
1		<gfid:6599b7e8-adeb-45f2-9dc4-0d373598fe95>
1		<gfid:ec9dab77-e44f-4785-b878-bea8bf8c53c8>
1		/pmd/pmd.data
1		<gfid:9e814c7c-6ac4-40c0-a9de-5e65042f0d78>
1		<gfid:848ee4b0-d84f-4fba-99a3-49cc0bd90dc3>
1		/loadmodules_norepl/CXC1720773_R86A01
1		/loadmodules_norepl/CXC1723373_R93A01
1		<gfid:cf442ff0-fdfc-48b1-87ef-d71364d10835>
1		<gfid:7ae1e5da-90b2-4d70-9e80-6115c7f91d6a>
1		<gfid:30151495-2dbd-445e-b1cb-c7dd66a9cdf2>
1		<gfid:4a5105b8-82f5-464c-9737-5bae7b052492>
1		<gfid:a7d3a075-e064-40b4-8ecb-49e0aae43614>
1		<gfid:b67d631b-36a2-46e5-a587-462c891f0177>
$ 

EVOA_8300-1> 

EVOA_8300-1> gluster volume info

161129-13:10:55 10.67.29.150 16.0y CPP_MOM-CPP-LSV203-gen2_gen2_COMPLETE stopfile=/tmp/15640
$ gluster volume info
 
Volume Name: c_glusterfs
Type: Replicate
Volume ID: 560ca2c3-6d79-45ce-bca1-50ec9a2f25ac
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 10.32.0.48:/opt/lvmdir/c2/brick
Brick2: 10.32.1.144:/opt/lvmdir/c2/brick
Options Reconfigured:
nfs.disable: on
network.ping-timeout: 4
performance.readdir-ahead: on
$ 

EVOA_8300-1> 

Log close: 161129-131102 - /home/emamiko/EVO8300/Issues/CPU_highLoad_C1MP/Gluster_vol_profile.txt


More information about the Gluster-users mailing list