[root@master ~]# oc logs heketi-storage-6-8dcdn Setting up heketi database File: /var/lib/heketi/heketi.db Size: 262144 Blocks: 512 IO Block: 131072 regular file Device: 200054h/2097236d Inode: 13717556549865401166 Links: 1 Access: (0600/-rw-------) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2019-01-21 14:17:53.065291018 +0000 Modify: 2019-02-14 06:20:40.547041647 +0000 Change: 2019-02-14 06:20:40.558041722 +0000 Birth: - Heketi v8.0.0-1-g082b556-release-8 [heketi] INFO 2019/02/14 07:12:08 Loaded kubernetes executor [heketi] ERROR 2019/02/14 07:12:08 /src/github.com/heketi/heketi/apps/glusterfs/app.go:177: Heketi was terminated while performing one or more operations. Server may refuse to start as long as pending operations are present in the db. [heketi] INFO 2019/02/14 07:12:08 Please refer to the Heketi troubleshooting documentation for more information on how to resolve this issue. [heketi] WARNING 2019/02/14 07:12:08 Ignoring stale pending operations.Server will be running with incomplete/inconsistent state in DB. [heketi] INFO 2019/02/14 07:12:08 Block: Auto Create Block Hosting Volume set to true [heketi] INFO 2019/02/14 07:12:08 Block: New Block Hosting Volume size 100 GB [heketi] INFO 2019/02/14 07:12:08 GlusterFS Application Loaded [heketi] INFO 2019/02/14 07:12:08 Started Node Health Cache Monitor Authorization loaded Listening on port 8080 [heketi] INFO 2019/02/14 07:12:18 Starting Node Health Status refresh [cmdexec] INFO 2019/02/14 07:12:18 Check Glusterd service status in node storage.matrix.nokia.com [kubeexec] DEBUG 2019/02/14 07:12:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: storage.matrix.nokia.com Pod: glusterfs-storage-9h5cf Command: systemctl status glusterd Result: ? glusterd.service - GlusterFS, a clustered file-system server Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) Active: active (running) since Fri 2019-01-18 13:27:57 UTC; 3 weeks 5 days ago Process: 23951 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) Main PID: 23952 (glusterd) CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b11242_cd44_11e8_a361_5254007b74fe.slice/docker-ce7cfe74053c991b94bda7e05b33316da1af1ce519ce170748f1ece89f266143.scope/system.slice/glusterd.service +- 558 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id heketidbstorage.192.168.89.220.var-lib-heketi-mounts-vg_018e77166eb3b38a0c250eb8faede2f7-brick_2b6ed29607d007a75f7a7c1bf0da7bae-brick -p /var/run/gluster/vols/heketidbstorage/192.168.89.220-var-lib-heketi-mounts-vg_018e77166eb3b38a0c250eb8faede2f7-brick_2b6ed29607d007a75f7a7c1bf0da7bae-brick.pid -S /var/run/gluster/88c0da28ee877a32.socket --brick-name /var/lib/heketi/mounts/vg_018e77166eb3b38a0c250eb8faede2f7/brick_2b6ed29607d007a75f7a7c1bf0da7bae/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_018e77166eb3b38a0c250eb8faede2f7-brick_2b6ed29607d007a75f7a7c1bf0da7bae-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49154 --xlator-option heketidbstorage-server.listen-port=49154 +- 901 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id glusterfs-registry-volume.192.168.89.220.var-lib-heketi-mounts-vg_5054cf122b4f54021cb9d67752d23c1d-brick_b22c3ab4d7bbc420e05340e8e187bcf5-brick -p /var/run/gluster/vols/glusterfs-registry-volume/192.168.89.220-var-lib-heketi-mounts-vg_5054cf122b4f54021cb9d67752d23c1d-brick_b22c3ab4d7bbc420e05340e8e187bcf5-brick.pid -S /var/run/gluster/b64892014146a0d5.socket --brick-name /var/lib/heketi/mounts/vg_5054cf122b4f54021cb9d67752d23c1d/brick_b22c3ab4d7bbc420e05340e8e187bcf5/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5054cf122b4f54021cb9d67752d23c1d-brick_b22c3ab4d7bbc420e05340e8e187bcf5-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49155 --xlator-option glusterfs-registry-volume-server.listen-port=49155 +- 2130 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id vol_2e2e6b9cf174901b370ea79a266c651b.192.168.89.220.var-lib-heketi-mounts-vg_25eb42cb007aab028afd0ee42b72bae9-brick_62c4432fc7a6e4aeb11573e22eaf53d8-brick -p /var/run/gluster/vols/vol_2e2e6b9cf174901b370ea79a266c651b/192.168.89.220-var-lib-heketi-mounts-vg_25eb42cb007aab028afd0ee42b72bae9-brick_62c4432fc7a6e4aeb11573e22eaf53d8-brick.pid -S /var/run/gluster/5ab438223cc6750c.socket --brick-name /var/lib/heketi/mounts/vg_25eb42cb007aab028afd0ee42b72bae9/brick_62c4432fc7a6e4aeb11573e22eaf53d8/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_25eb42cb007aab028afd0ee42b72bae9-brick_62c4432fc7a6e4aeb11573e22eaf53d8-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49161 --xlator-option vol_2e2e6b9cf174901b370ea79a266c651b-server.listen-port=49161 +- 2713 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id vol_c5fa076446479cf414397591c0af1c7f.192.168.89.220.var-lib-heketi-mounts-vg_25eb42cb007aab028afd0ee42b72bae9-brick_ac7b22198b6424785edd32d6f0a078d2-brick -p /var/run/gluster/vols/vol_c5fa076446479cf414397591c0af1c7f/192.168.89.220-var-lib-heketi-mounts-vg_25eb42cb007aab028afd0ee42b72bae9-brick_ac7b22198b6424785edd32d6f0a078d2-brick.pid -S /var/run/gluster/b586da88dc5b2118.socket --brick-name /var/lib/heketi/mounts/vg_25eb42cb007aab028afd0ee42b72bae9/brick_ac7b22198b6424785edd32d6f0a078d2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_25eb42cb007aab028afd0ee42b72bae9-brick_ac7b22198b6424785edd32d6f0a078d2-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49162 --xlator-option vol_c5fa076446479cf414397591c0af1c7f-server.listen-port=49162 +- 9064 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id vol_bc5012fd39c5b3ab958b9da4b0256d3a.192.168.89.220.var-lib-heketi-mounts-vg_5054cf122b4f54021cb9d67752d23c1d-brick_fded2411a02e03c7e0219eb9a2b08c9a-brick -p /var/run/gluster/vols/vol_bc5012fd39c5b3ab958b9da4b0256d3a/192.168.89.220-var-lib-heketi-mounts-vg_5054cf122b4f54021cb9d67752d23c1d-brick_fded2411a02e03c7e0219eb9a2b08c9a-brick.pid -S /var/run/gluster/8b6fba20e4ec888e.socket --brick-name /var/lib/heketi/mounts/vg_5054cf122b4f54021cb9d67752d23c1d/brick_fded2411a02e03c7e0219eb9a2b08c9a/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5054cf122b4f54021cb9d67752d23c1d-brick_fded2411a02e03c7e0219eb9a2b08c9a-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49166 --xlator-option vol_bc5012fd39c5b3ab958b9da4b0256d3a-server.listen-port=49166 +- 23952 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO +- 32058 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/06a8fe3d13500a44.socket --xlator-option *replicate*.node-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name glustershd +- 71770 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id vol_88fbf36bf87106e549dca765f171cf69.192.168.89.220.var-lib-heketi-mounts-vg_9c007bfbe66a2522ad5dd851bb18d03a-brick_22982c432dd089512ab2bde1e0663f9a-brick -p /var/run/gluster/vols/vol_88fbf36bf87106e549dca765f171cf69/192.168.89.220-var-lib-heketi-mounts-vg_9c007bfbe66a2522ad5dd851bb18d03a-brick_22982c432dd089512ab2bde1e0663f9a-brick.pid -S /var/run/gluster/adbe241d384df1c2.socket --brick-name /var/lib/heketi/mounts/vg_9c007bfbe66a2522ad5dd851bb18d03a/brick_22982c432dd089512ab2bde1e0663f9a/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_9c007bfbe66a2522ad5dd851bb18d03a-brick_22982c432dd089512ab2bde1e0663f9a-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49167 --xlator-option vol_88fbf36bf87106e549dca765f171cf69-server.listen-port=49167 +- 72776 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id vol_47c2af6849ac9ad7d7fbc897ae3ae80c.192.168.89.220.var-lib-heketi-mounts-vg_5054cf122b4f54021cb9d67752d23c1d-brick_025b1bf4bbabea839ef139c4453f9bcd-brick -p /var/run/gluster/vols/vol_47c2af6849ac9ad7d7fbc897ae3ae80c/192.168.89.220-var-lib-heketi-mounts-vg_5054cf122b4f54021cb9d67752d23c1d-brick_025b1bf4bbabea839ef139c4453f9bcd-brick.pid -S /var/run/gluster/e6f521dfc39ee69b.socket --brick-name /var/lib/heketi/mounts/vg_5054cf122b4f54021cb9d67752d23c1d/brick_025b1bf4bbabea839ef139c4453f9bcd/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5054cf122b4f54021cb9d67752d23c1d-brick_025b1bf4bbabea839ef139c4453f9bcd-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49158 --xlator-option vol_47c2af6849ac9ad7d7fbc897ae3ae80c-server.listen-port=49158 +- 82920 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id vol_63428c902b5d577a7739eb66a050a420.192.168.89.220.var-lib-heketi-mounts-vg_25eb42cb007aab028afd0ee42b72bae9-brick_85c1670a17f956abd0f3dd99cad53a7e-brick -p /var/run/gluster/vols/vol_63428c902b5d577a7739eb66a050a420/192.168.89.220-var-lib-heketi-mounts-vg_25eb42cb007aab028afd0ee42b72bae9-brick_85c1670a17f956abd0f3dd99cad53a7e-brick.pid -S /var/run/gluster/c82b614596a19c6c.socket --brick-name /var/lib/heketi/mounts/vg_25eb42cb007aab028afd0ee42b72bae9/brick_85c1670a17f956abd0f3dd99cad53a7e/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_25eb42cb007aab028afd0ee42b72bae9-brick_85c1670a17f956abd0f3dd99cad53a7e-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49171 --xlator-option vol_63428c902b5d577a7739eb66a050a420-server.listen-port=49171 +- 86318 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id vol_937aa673e0d60126082bbd8c1589e383.192.168.89.220.var-lib-heketi-mounts-vg_018e77166eb3b38a0c250eb8faede2f7-brick_275d68b56cfc42c8b46aa90eb227eb25-brick -p /var/run/gluster/vols/vol_937aa673e0d60126082bbd8c1589e383/192.168.89.220-var-lib-heketi-mounts-vg_018e77166eb3b38a0c250eb8faede2f7-brick_275d68b56cfc42c8b46aa90eb227eb25-brick.pid -S /var/run/gluster/5ae55f9a21c8e488.socket --brick-name /var/lib/heketi/mounts/vg_018e77166eb3b38a0c250eb8faede2f7/brick_275d68b56cfc42c8b46aa90eb227eb25/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_018e77166eb3b38a0c250eb8faede2f7-brick_275d68b56cfc42c8b46aa90eb227eb25-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49172 --xlator-option vol_937aa673e0d60126082bbd8c1589e383-server.listen-port=49172 +- 87342 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id vol_62905ab0e959e7663f4576501c0c9b69.192.168.89.220.var-lib-heketi-mounts-vg_5054cf122b4f54021cb9d67752d23c1d-brick_9684d9e56109e011585af33ecd8039cc-brick -p /var/run/gluster/vols/vol_62905ab0e959e7663f4576501c0c9b69/192.168.89.220-var-lib-heketi-mounts-vg_5054cf122b4f54021cb9d67752d23c1d-brick_9684d9e56109e011585af33ecd8039cc-brick.pid -S /var/run/gluster/e63ba38f2720a00f.socket --brick-name /var/lib/heketi/mounts/vg_5054cf122b4f54021cb9d67752d23c1d/brick_9684d9e56109e011585af33ecd8039cc/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5054cf122b4f54021cb9d67752d23c1d-brick_9684d9e56109e011585af33ecd8039cc-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49157 --xlator-option vol_62905ab0e959e7663f4576501c0c9b69-server.listen-port=49157 +- 97945 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id vol_dcee4f7f020d47e58fb48612bbba19d1.192.168.89.220.var-lib-heketi-mounts-vg_018e77166eb3b38a0c250eb8faede2f7-brick_16de636f8337f47a87996fca50313260-brick -p /var/run/gluster/vols/vol_dcee4f7f020d47e58fb48612bbba19d1/192.168.89.220-var-lib-heketi-mounts-vg_018e77166eb3b38a0c250eb8faede2f7-brick_16de636f8337f47a87996fca50313260-brick.pid -S /var/run/gluster/5bbb028f813f880f.socket --brick-name /var/lib/heketi/mounts/vg_018e77166eb3b38a0c250eb8faede2f7/brick_16de636f8337f47a87996fca50313260/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_018e77166eb3b38a0c250eb8faede2f7-brick_16de636f8337f47a87996fca50313260-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49168 --xlator-option vol_dcee4f7f020d47e58fb48612bbba19d1-server.listen-port=49168 +-103617 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id vol_5101331e9d33a5d04adab92837b9d5ad.192.168.89.220.var-lib-heketi-mounts-vg_9c007bfbe66a2522ad5dd851bb18d03a-brick_079acb5b4a0b0f3c45ed87f9c7def14b-brick -p /var/run/gluster/vols/vol_5101331e9d33a5d04adab92837b9d5ad/192.168.89.220-var-lib-heketi-mounts-vg_9c007bfbe66a2522ad5dd851bb18d03a-brick_079acb5b4a0b0f3c45ed87f9c7def14b-brick.pid -S /var/run/gluster/20aa83a13cec2563.socket --brick-name /var/lib/heketi/mounts/vg_9c007bfbe66a2522ad5dd851bb18d03a/brick_079acb5b4a0b0f3c45ed87f9c7def14b/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_9c007bfbe66a2522ad5dd851bb18d03a-brick_079acb5b4a0b0f3c45ed87f9c7def14b-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49159 --xlator-option vol_5101331e9d33a5d04adab92837b9d5ad-server.listen-port=49159 +-103808 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id vol_2e6ad6aa03f7fe219807cb135ca1c766.192.168.89.220.var-lib-heketi-mounts-vg_9c007bfbe66a2522ad5dd851bb18d03a-brick_2fd555bc1cbd1b49ef04e7b88d34297b-brick -p /var/run/gluster/vols/vol_2e6ad6aa03f7fe219807cb135ca1c766/192.168.89.220-var-lib-heketi-mounts-vg_9c007bfbe66a2522ad5dd851bb18d03a-brick_2fd555bc1cbd1b49ef04e7b88d34297b-brick.pid -S /var/run/gluster/a375f4b3a8cba7c0.socket --brick-name /var/lib/heketi/mounts/vg_9c007bfbe66a2522ad5dd851bb18d03a/brick_2fd555bc1cbd1b49ef04e7b88d34297b/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_9c007bfbe66a2522ad5dd851bb18d03a-brick_2fd555bc1cbd1b49ef04e7b88d34297b-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49163 --xlator-option vol_2e6ad6aa03f7fe219807cb135ca1c766-server.listen-port=49163 +-103990 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id vol_78d426889b7bd661f3d6c6f7814b6d4f.192.168.89.220.var-lib-heketi-mounts-vg_9c007bfbe66a2522ad5dd851bb18d03a-brick_8dba95329e797beb06eca079b9ae6a4b-brick -p /var/run/gluster/vols/vol_78d426889b7bd661f3d6c6f7814b6d4f/192.168.89.220-var-lib-heketi-mounts-vg_9c007bfbe66a2522ad5dd851bb18d03a-brick_8dba95329e797beb06eca079b9ae6a4b-brick.pid -S /var/run/gluster/8d3f3f80d9e62092.socket --brick-name /var/lib/heketi/mounts/vg_9c007bfbe66a2522ad5dd851bb18d03a/brick_8dba95329e797beb06eca079b9ae6a4b/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_9c007bfbe66a2522ad5dd851bb18d03a-brick_8dba95329e797beb06eca079b9ae6a4b-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49164 --xlator-option vol_78d426889b7bd661f3d6c6f7814b6d4f-server.listen-port=49164 +-105270 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id vol_9f04d82be19e2ea8ee80deb9098cd390.192.168.89.220.var-lib-heketi-mounts-vg_018e77166eb3b38a0c250eb8faede2f7-brick_0bd819f4969e7adbba32e2c77afc202c-brick -p /var/run/gluster/vols/vol_9f04d82be19e2ea8ee80deb9098cd390/192.168.89.220-var-lib-heketi-mounts-vg_018e77166eb3b38a0c250eb8faede2f7-brick_0bd819f4969e7adbba32e2c77afc202c-brick.pid -S /var/run/gluster/0882e30a8d5b1272.socket --brick-name /var/lib/heketi/mounts/vg_018e77166eb3b38a0c250eb8faede2f7/brick_0bd819f4969e7adbba32e2c77afc202c/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_018e77166eb3b38a0c250eb8faede2f7-brick_0bd819f4969e7adbba32e2c77afc202c-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49165 --xlator-option vol_9f04d82be19e2ea8ee80deb9098cd390-server.listen-port=49165 +-121771 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id vol_7e73769a754d7f9f6213ef3b4551af0e.192.168.89.220.var-lib-heketi-mounts-vg_018e77166eb3b38a0c250eb8faede2f7-brick_c2fe3a63636192e85d8adeefff18c92b-brick -p /var/run/gluster/vols/vol_7e73769a754d7f9f6213ef3b4551af0e/192.168.89.220-var-lib-heketi-mounts-vg_018e77166eb3b38a0c250eb8faede2f7-brick_c2fe3a63636192e85d8adeefff18c92b-brick.pid -S /var/run/gluster/130cc033b68aedd6.socket --brick-name /var/lib/heketi/mounts/vg_018e77166eb3b38a0c250eb8faede2f7/brick_c2fe3a63636192e85d8adeefff18c92b/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_018e77166eb3b38a0c250eb8faede2f7-brick_c2fe3a63636192e85d8adeefff18c92b-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49156 --xlator-option vol_7e73769a754d7f9f6213ef3b4551af0e-server.listen-port=49156 +-130924 /usr/sbin/glusterfsd -s 192.168.89.220 --volfile-id vol_dbc601f7162df326784aea34c4ebe8f2.192.168.89.220.var-lib-heketi-mounts-vg_5054cf122b4f54021cb9d67752d23c1d-brick_4fa8a73223d69de183d8e58618722084-brick -p /var/run/gluster/vols/vol_dbc601f7162df326784aea34c4ebe8f2/192.168.89.220-var-lib-heketi-mounts-vg_5054cf122b4f54021cb9d67752d23c1d-brick_4fa8a73223d69de183d8e58618722084-brick.pid -S /var/run/gluster/88109cbb53cf4280.socket --brick-name /var/lib/heketi/mounts/vg_5054cf122b4f54021cb9d67752d23c1d/brick_4fa8a73223d69de183d8e58618722084/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_5054cf122b4f54021cb9d67752d23c1d-brick_4fa8a73223d69de183d8e58618722084-brick.log --xlator-option *-posix.glusterd-uuid=25b5f142-4890-4315-a352-cf947fdf649c --process-name brick --brick-port 49160 --xlator-option vol_dbc601f7162df326784aea34c4ebe8f2-server.listen-port=49160 Jan 18 13:27:56 storage.matrix.nokia.com systemd[1]: Starting GlusterFS, a clustered file-system server... Jan 18 13:27:57 storage.matrix.nokia.com systemd[1]: Started GlusterFS, a clustered file-system server. [heketi] INFO 2019/02/14 07:12:18 Periodic health check status: node 5f090e0aa757345d22ebdb15897bd68a up=true [cmdexec] INFO 2019/02/14 07:12:18 Check Glusterd service status in node app1.matrix.nokia.com [kubeexec] DEBUG 2019/02/14 07:12:18 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: app1.matrix.nokia.com Pod: glusterfs-storage-31fjz Command: systemctl status glusterd Result: ? glusterd.service - GlusterFS, a clustered file-system server Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled) Active: active (running) since Sun 2019-02-03 10:05:10 UTC; 1 weeks 3 days ago Process: 198 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) Main PID: 199 (glusterd) CGroup: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b10041_cd44_11e8_a361_5254007b74fe.slice/docker-b96fad2fc728c53d50ae7c4bc5a83b155d21566fd3b3877d4bd364513804907d.scope/system.slice/glusterd.service +-199 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO +-238 /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/46eb4bfe4652baa1.socket --xlator-option *replicate*.node-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name glustershd +-282 /usr/sbin/glusterfsd -s 192.168.89.218 --volfile-id glusterfs-registry-volume.192.168.89.218.var-lib-heketi-mounts-vg_cea64ccf8a96a13dacc1cb3ac3ea13de-brick_7ef53adf2bce0e240229941ac7f43661-brick -p /var/run/gluster/vols/glusterfs-registry-volume/192.168.89.218-var-lib-heketi-mounts-vg_cea64ccf8a96a13dacc1cb3ac3ea13de-brick_7ef53adf2bce0e240229941ac7f43661-brick.pid -S /var/run/gluster/f223a3f7986d97b5.socket --brick-name /var/lib/heketi/mounts/vg_cea64ccf8a96a13dacc1cb3ac3ea13de/brick_7ef53adf2bce0e240229941ac7f43661/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_cea64ccf8a96a13dacc1cb3ac3ea13de-brick_7ef53adf2bce0e240229941ac7f43661-brick.log --xlator-option *-posix.glusterd-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name brick --brick-port 49152 --xlator-option glusterfs-registry-volume-server.listen-port=49152 +-291 /usr/sbin/glusterfsd -s 192.168.89.218 --volfile-id heketidbstorage.192.168.89.218.var-lib-heketi-mounts-vg_127a3d697690febddfa2a41b3a820bde-brick_79e211c7bee7d61bfdabb0add8e42b33-brick -p /var/run/gluster/vols/heketidbstorage/192.168.89.218-var-lib-heketi-mounts-vg_127a3d697690febddfa2a41b3a820bde-brick_79e211c7bee7d61bfdabb0add8e42b33-brick.pid -S /var/run/gluster/f905b930c13d9816.socket --brick-name /var/lib/heketi/mounts/vg_127a3d697690febddfa2a41b3a820bde/brick_79e211c7bee7d61bfdabb0add8e42b33/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_127a3d697690febddfa2a41b3a820bde-brick_79e211c7bee7d61bfdabb0add8e42b33-brick.log --xlator-option *-posix.glusterd-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name brick --brick-port 49153 --xlator-option heketidbstorage-server.listen-port=49153 +-306 /usr/sbin/glusterfsd -s 192.168.89.218 --volfile-id vol_2e2e6b9cf174901b370ea79a266c651b.192.168.89.218.var-lib-heketi-mounts-vg_88c314d3366471a0dd89ca4d5a3c8ef8-brick_6f738c7cc82f66dbda8bc0b6d74883c0-brick -p /var/run/gluster/vols/vol_2e2e6b9cf174901b370ea79a266c651b/192.168.89.218-var-lib-heketi-mounts-vg_88c314d3366471a0dd89ca4d5a3c8ef8-brick_6f738c7cc82f66dbda8bc0b6d74883c0-brick.pid -S /var/run/gluster/ac938e354448789e.socket --brick-name /var/lib/heketi/mounts/vg_88c314d3366471a0dd89ca4d5a3c8ef8/brick_6f738c7cc82f66dbda8bc0b6d74883c0/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_88c314d3366471a0dd89ca4d5a3c8ef8-brick_6f738c7cc82f66dbda8bc0b6d74883c0-brick.log --xlator-option *-posix.glusterd-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name brick --brick-port 49154 --xlator-option vol_2e2e6b9cf174901b370ea79a266c651b-server.listen-port=49154 +-321 /usr/sbin/glusterfsd -s 192.168.89.218 --volfile-id vol_2e6ad6aa03f7fe219807cb135ca1c766.192.168.89.218.var-lib-heketi-mounts-vg_e7e08adf3c8cd3dee08dd006b1596074-brick_980bd302213bbdd2d9aa09dbe8abf62b-brick -p /var/run/gluster/vols/vol_2e6ad6aa03f7fe219807cb135ca1c766/192.168.89.218-var-lib-heketi-mounts-vg_e7e08adf3c8cd3dee08dd006b1596074-brick_980bd302213bbdd2d9aa09dbe8abf62b-brick.pid -S /var/run/gluster/db1300311bc7697d.socket --brick-name /var/lib/heketi/mounts/vg_e7e08adf3c8cd3dee08dd006b1596074/brick_980bd302213bbdd2d9aa09dbe8abf62b/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e7e08adf3c8cd3dee08dd006b1596074-brick_980bd302213bbdd2d9aa09dbe8abf62b-brick.log --xlator-option *-posix.glusterd-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name brick --brick-port 49155 --xlator-option vol_2e6ad6aa03f7fe219807cb135ca1c766-server.listen-port=49155 +-330 /usr/sbin/glusterfsd -s 192.168.89.218 --volfile-id vol_47c2af6849ac9ad7d7fbc897ae3ae80c.192.168.89.218.var-lib-heketi-mounts-vg_cea64ccf8a96a13dacc1cb3ac3ea13de-brick_1f52101af46329a1d5e758156c8267e0-brick -p /var/run/gluster/vols/vol_47c2af6849ac9ad7d7fbc897ae3ae80c/192.168.89.218-var-lib-heketi-mounts-vg_cea64ccf8a96a13dacc1cb3ac3ea13de-brick_1f52101af46329a1d5e758156c8267e0-brick.pid -S /var/run/gluster/17890cacc5ba5074.socket --brick-name /var/lib/heketi/mounts/vg_cea64ccf8a96a13dacc1cb3ac3ea13de/brick_1f52101af46329a1d5e758156c8267e0/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_cea64ccf8a96a13dacc1cb3ac3ea13de-brick_1f52101af46329a1d5e758156c8267e0-brick.log --xlator-option *-posix.glusterd-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name brick --brick-port 49156 --xlator-option vol_47c2af6849ac9ad7d7fbc897ae3ae80c-server.listen-port=49156 +-339 /usr/sbin/glusterfsd -s 192.168.89.218 --volfile-id vol_5101331e9d33a5d04adab92837b9d5ad.192.168.89.218.var-lib-heketi-mounts-vg_127a3d697690febddfa2a41b3a820bde-brick_17f8162940314babf2b7d75976890130-brick -p /var/run/gluster/vols/vol_5101331e9d33a5d04adab92837b9d5ad/192.168.89.218-var-lib-heketi-mounts-vg_127a3d697690febddfa2a41b3a820bde-brick_17f8162940314babf2b7d75976890130-brick.pid -S /var/run/gluster/6d420a71217c335e.socket --brick-name /var/lib/heketi/mounts/vg_127a3d697690febddfa2a41b3a820bde/brick_17f8162940314babf2b7d75976890130/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_127a3d697690febddfa2a41b3a820bde-brick_17f8162940314babf2b7d75976890130-brick.log --xlator-option *-posix.glusterd-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name brick --brick-port 49157 --xlator-option vol_5101331e9d33a5d04adab92837b9d5ad-server.listen-port=49157 +-348 /usr/sbin/glusterfsd -s 192.168.89.218 --volfile-id vol_62905ab0e959e7663f4576501c0c9b69.192.168.89.218.var-lib-heketi-mounts-vg_cea64ccf8a96a13dacc1cb3ac3ea13de-brick_31ed93bf445cf1fd00d34660a80e1078-brick -p /var/run/gluster/vols/vol_62905ab0e959e7663f4576501c0c9b69/192.168.89.218-var-lib-heketi-mounts-vg_cea64ccf8a96a13dacc1cb3ac3ea13de-brick_31ed93bf445cf1fd00d34660a80e1078-brick.pid -S /var/run/gluster/73b2ef5a8bf5baa1.socket --brick-name /var/lib/heketi/mounts/vg_cea64ccf8a96a13dacc1cb3ac3ea13de/brick_31ed93bf445cf1fd00d34660a80e1078/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_cea64ccf8a96a13dacc1cb3ac3ea13de-brick_31ed93bf445cf1fd00d34660a80e1078-brick.log --xlator-option *-posix.glusterd-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name brick --brick-port 49158 --xlator-option vol_62905ab0e959e7663f4576501c0c9b69-server.listen-port=49158 +-357 /usr/sbin/glusterfsd -s 192.168.89.218 --volfile-id vol_63428c902b5d577a7739eb66a050a420.192.168.89.218.var-lib-heketi-mounts-vg_88c314d3366471a0dd89ca4d5a3c8ef8-brick_586c05c5ce5bd7474cfb9a9cd190225c-brick -p /var/run/gluster/vols/vol_63428c902b5d577a7739eb66a050a420/192.168.89.218-var-lib-heketi-mounts-vg_88c314d3366471a0dd89ca4d5a3c8ef8-brick_586c05c5ce5bd7474cfb9a9cd190225c-brick.pid -S /var/run/gluster/27400e8d3608aa83.socket --brick-name /var/lib/heketi/mounts/vg_88c314d3366471a0dd89ca4d5a3c8ef8/brick_586c05c5ce5bd7474cfb9a9cd190225c/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_88c314d3366471a0dd89ca4d5a3c8ef8-brick_586c05c5ce5bd7474cfb9a9cd190225c-brick.log --xlator-option *-posix.glusterd-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name brick --brick-port 49159 --xlator-option vol_63428c902b5d577a7739eb66a050a420-server.listen-port=49159 +-366 /usr/sbin/glusterfsd -s 192.168.89.218 --volfile-id vol_78d426889b7bd661f3d6c6f7814b6d4f.192.168.89.218.var-lib-heketi-mounts-vg_127a3d697690febddfa2a41b3a820bde-brick_d3230ee5de8fe8300f47eb4bcea6f2e2-brick -p /var/run/gluster/vols/vol_78d426889b7bd661f3d6c6f7814b6d4f/192.168.89.218-var-lib-heketi-mounts-vg_127a3d697690febddfa2a41b3a820bde-brick_d3230ee5de8fe8300f47eb4bcea6f2e2-brick.pid -S /var/run/gluster/87c81a7cb1ad328b.socket --brick-name /var/lib/heketi/mounts/vg_127a3d697690febddfa2a41b3a820bde/brick_d3230ee5de8fe8300f47eb4bcea6f2e2/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_127a3d697690febddfa2a41b3a820bde-brick_d3230ee5de8fe8300f47eb4bcea6f2e2-brick.log --xlator-option *-posix.glusterd-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name brick --brick-port 49160 --xlator-option vol_78d426889b7bd661f3d6c6f7814b6d4f-server.listen-port=49160 +-375 /usr/sbin/glusterfsd -s 192.168.89.218 --volfile-id vol_7e73769a754d7f9f6213ef3b4551af0e.192.168.89.218.var-lib-heketi-mounts-vg_88c314d3366471a0dd89ca4d5a3c8ef8-brick_84e395f5464cd25dce66b8e60f395982-brick -p /var/run/gluster/vols/vol_7e73769a754d7f9f6213ef3b4551af0e/192.168.89.218-var-lib-heketi-mounts-vg_88c314d3366471a0dd89ca4d5a3c8ef8-brick_84e395f5464cd25dce66b8e60f395982-brick.pid -S /var/run/gluster/c7917e048093731a.socket --brick-name /var/lib/heketi/mounts/vg_88c314d3366471a0dd89ca4d5a3c8ef8/brick_84e395f5464cd25dce66b8e60f395982/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_88c314d3366471a0dd89ca4d5a3c8ef8-brick_84e395f5464cd25dce66b8e60f395982-brick.log --xlator-option *-posix.glusterd-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name brick --brick-port 49161 --xlator-option vol_7e73769a754d7f9f6213ef3b4551af0e-server.listen-port=49161 +-384 /usr/sbin/glusterfsd -s 192.168.89.218 --volfile-id vol_937aa673e0d60126082bbd8c1589e383.192.168.89.218.var-lib-heketi-mounts-vg_127a3d697690febddfa2a41b3a820bde-brick_5f22e9906d3566116ec474bb618213e5-brick -p /var/run/gluster/vols/vol_937aa673e0d60126082bbd8c1589e383/192.168.89.218-var-lib-heketi-mounts-vg_127a3d697690febddfa2a41b3a820bde-brick_5f22e9906d3566116ec474bb618213e5-brick.pid -S /var/run/gluster/926dadf68659a463.socket --brick-name /var/lib/heketi/mounts/vg_127a3d697690febddfa2a41b3a820bde/brick_5f22e9906d3566116ec474bb618213e5/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_127a3d697690febddfa2a41b3a820bde-brick_5f22e9906d3566116ec474bb618213e5-brick.log --xlator-option *-posix.glusterd-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name brick --brick-port 49162 --xlator-option vol_937aa673e0d60126082bbd8c1589e383-server.listen-port=49162 +-393 /usr/sbin/glusterfsd -s 192.168.89.218 --volfile-id vol_9f04d82be19e2ea8ee80deb9098cd390.192.168.89.218.var-lib-heketi-mounts-vg_88c314d3366471a0dd89ca4d5a3c8ef8-brick_281606b3024ac98416fedee94528e3c7-brick -p /var/run/gluster/vols/vol_9f04d82be19e2ea8ee80deb9098cd390/192.168.89.218-var-lib-heketi-mounts-vg_88c314d3366471a0dd89ca4d5a3c8ef8-brick_281606b3024ac98416fedee94528e3c7-brick.pid -S /var/run/gluster/a3fd18961e1f360b.socket --brick-name /var/lib/heketi/mounts/vg_88c314d3366471a0dd89ca4d5a3c8ef8/brick_281606b3024ac98416fedee94528e3c7/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_88c314d3366471a0dd89ca4d5a3c8ef8-brick_281606b3024ac98416fedee94528e3c7-brick.log --xlator-option *-posix.glusterd-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name brick --brick-port 49163 --xlator-option vol_9f04d82be19e2ea8ee80deb9098cd390-server.listen-port=49163 +-402 /usr/sbin/glusterfsd -s 192.168.89.218 --volfile-id vol_bc5012fd39c5b3ab958b9da4b0256d3a.192.168.89.218.var-lib-heketi-mounts-vg_cea64ccf8a96a13dacc1cb3ac3ea13de-brick_08599e162e105063a8e2efec868d2a76-brick -p /var/run/gluster/vols/vol_bc5012fd39c5b3ab958b9da4b0256d3a/192.168.89.218-var-lib-heketi-mounts-vg_cea64ccf8a96a13dacc1cb3ac3ea13de-brick_08599e162e105063a8e2efec868d2a76-brick.pid -S /var/run/gluster/42ae4fefd1cb6554.socket --brick-name /var/lib/heketi/mounts/vg_cea64ccf8a96a13dacc1cb3ac3ea13de/brick_08599e162e105063a8e2efec868d2a76/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_cea64ccf8a96a13dacc1cb3ac3ea13de-brick_08599e162e105063a8e2efec868d2a76-brick.log --xlator-option *-posix.glusterd-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name brick --brick-port 49164 --xlator-option vol_bc5012fd39c5b3ab958b9da4b0256d3a-server.listen-port=49164 +-411 /usr/sbin/glusterfsd -s 192.168.89.218 --volfile-id vol_c5fa076446479cf414397591c0af1c7f.192.168.89.218.var-lib-heketi-mounts-vg_cea64ccf8a96a13dacc1cb3ac3ea13de-brick_ae2fa0410160c9fb489af8e5ad063ac3-brick -p /var/run/gluster/vols/vol_c5fa076446479cf414397591c0af1c7f/192.168.89.218-var-lib-heketi-mounts-vg_cea64ccf8a96a13dacc1cb3ac3ea13de-brick_ae2fa0410160c9fb489af8e5ad063ac3-brick.pid -S /var/run/gluster/8bb099144ac69c55.socket --brick-name /var/lib/heketi/mounts/vg_cea64ccf8a96a13dacc1cb3ac3ea13de/brick_ae2fa0410160c9fb489af8e5ad063ac3/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_cea64ccf8a96a13dacc1cb3ac3ea13de-brick_ae2fa0410160c9fb489af8e5ad063ac3-brick.log --xlator-option *-posix.glusterd-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name brick --brick-port 49165 --xlator-option vol_c5fa076446479cf414397591c0af1c7f-server.listen-port=49165 +-420 /usr/sbin/glusterfsd -s 192.168.89.218 --volfile-id vol_dbc601f7162df326784aea34c4ebe8f2.192.168.89.218.var-lib-heketi-mounts-vg_e7e08adf3c8cd3dee08dd006b1596074-brick_ff62a0d3e0fe5c53c3725235d2832893-brick -p /var/run/gluster/vols/vol_dbc601f7162df326784aea34c4ebe8f2/192.168.89.218-var-lib-heketi-mounts-vg_e7e08adf3c8cd3dee08dd006b1596074-brick_ff62a0d3e0fe5c53c3725235d2832893-brick.pid -S /var/run/gluster/53bb40a037c87c29.socket --brick-name /var/lib/heketi/mounts/vg_e7e08adf3c8cd3dee08dd006b1596074/brick_ff62a0d3e0fe5c53c3725235d2832893/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e7e08adf3c8cd3dee08dd006b1596074-brick_ff62a0d3e0fe5c53c3725235d2832893-brick.log --xlator-option *-posix.glusterd-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name brick --brick-port 49166 --xlator-option vol_dbc601f7162df326784aea34c4ebe8f2-server.listen-port=49166 +-429 /usr/sbin/glusterfsd -s 192.168.89.218 --volfile-id vol_dcee4f7f020d47e58fb48612bbba19d1.192.168.89.218.var-lib-heketi-mounts-vg_e7e08adf3c8cd3dee08dd006b1596074-brick_800e2a99110af93ca69bd950c9ea7ef8-brick -p /var/run/gluster/vols/vol_dcee4f7f020d47e58fb48612bbba19d1/192.168.89.218-var-lib-heketi-mounts-vg_e7e08adf3c8cd3dee08dd006b1596074-brick_800e2a99110af93ca69bd950c9ea7ef8-brick.pid -S /var/run/gluster/fbf6e334c35a83bb.socket --brick-name /var/lib/heketi/mounts/vg_e7e08adf3c8cd3dee08dd006b1596074/brick_800e2a99110af93ca69bd950c9ea7ef8/brick -l /var/log/glusterfs/bricks/var-lib-heketi-mounts-vg_e7e08adf3c8cd3dee08dd006b1596074-brick_800e2a99110af93ca69bd950c9ea7ef8-brick.log --xlator-option *-posix.glusterd-uuid=0574c4d2-6900-4447-a752-2bf7477b443e --process-name brick --brick-port 49167 --xlator-option vol_dcee4f7f020d47e58fb48612bbba19d1-server.listen-port=49167 Feb 03 10:05:08 app1.matrix.nokia.com systemd[1]: Starting GlusterFS, a clustered file-system server... Feb 03 10:05:10 app1.matrix.nokia.com systemd[1]: Started GlusterFS, a clustered file-system server. [heketi] INFO 2019/02/14 07:12:18 Periodic health check status: node b72660bc8e28725f612d33bd675f8341 up=true [cmdexec] INFO 2019/02/14 07:12:18 Check Glusterd service status in node app2.matrix.nokia.com [negroni] Started POST /volumes [heketi] INFO 2019/02/14 07:12:40 Allocating brick set #0 [negroni] Started POST /volumes [negroni] Started GET /clusters [negroni] Completed 401 Unauthorized in 119.867µs [negroni] Started POST /volumes [negroni] Started GET /operations [negroni] Completed 200 OK in 449.406µs [negroni] Started GET /operations [negroni] Completed 200 OK in 322.589µs [negroni] Started POST /volumes [negroni] Started POST /volumes [negroni] Started POST /volumes [negroni] Started GET /db/dump [negroni] Completed 200 OK in 7.607085ms [negroni] Started POST /volumes [negroni] Started POST /volumes [negroni] Started POST /volumes [heketi] WARNING 2019/02/14 07:17:41 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 250.4µs [negroni] Started POST /volumes [heketi] WARNING 2019/02/14 07:17:56 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 191.165µs [negroni] Started POST /volumes [heketi] WARNING 2019/02/14 07:18:11 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 339.575µs [negroni] Started POST /volumes [heketi] WARNING 2019/02/14 07:18:26 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 200.437µs [negroni] Started POST /volumes [heketi] WARNING 2019/02/14 07:18:41 operations in-flight (8) exceeds limit (8) [negroni] Completed 429 Too Many Requests in 456.234µs