[Gluster-users] GlusterFS Scale
Lindolfo Meira
meira at cesup.ufrgs.br
Wed Feb 20 14:56:24 UTC 2019
Hi Amar. Thanks for taking the time.
Yes, I know stripping has been deprecated, but we have a specific interest
in testing striped volumes at the moment. We're trying to stablish a
performance comparison (stripe vs shard).
Each brick contributes a single hardware controlled RAID-6 volume with 16
disks. Running the benchmark with exactly the same parameters directly on
these volumes we manage to get around 13GB/s (way above the 3.36GB/s the
network adapters are able to achieve). We're benchmarking this 6-brick
gluster system using the MPIIO API of IOR.
The output of the profiling follows attached. It refers to an IOR write
test, iterated 8 times. Again, max I got was 11GB/s. IOR parameters used
were -B -E -F -q -w -k -z -m -i=8 -t=2m -b=1g -a=MPIIO.
Cheers,
Lindolfo Meira, MSc
Diretor Geral, Centro Nacional de Supercomputação
Universidade Federal do Rio Grande do Sul
+55 (51) 3308-3139
On Wed, 20 Feb 2019, Amar Tumballi Suryanarayan wrote:
> On Mon, Feb 18, 2019 at 11:23 PM Lindolfo Meira <meira at cesup.ufrgs.br>
> wrote:
>
> > We're running some benchmarks on a striped glusterfs volume.
> >
> >
> Hi Lindolfo,
>
> We are not supporting Stripe anymore, and planning to remove it from build
> too by glusterfs-6.0 (ie, next release). See if you can use 'Shard' for the
> usecase.
>
>
> > We have 6 identical servers acting as bricks. Measured link speed between
> > these servers is 3.36GB/s. Link speed between clients of the parallel file
> > system and its servers is also 3.36GB/s. So we're expecting this system to
> > have a write performance of around 20.16GB/s (6 times 3.36GB/s) minus some
> > write overhead.
> >
> > If we write to the system from a single client, we manage to write at
> > around 3.36GB/s. That's okay, because we're limited by the max throughput
> > of that client's network adapter. But when we account for that and write
> > from 6 or more clients, we can never get past 11GB/s. Is that right? Is
> > this really the overhead to be expected? We'd appreciate any inputs.
> >
> >
> Lame question: Are we getting more than 11GB/s from disks ?
>
> Please collect `gluster volume profile gfs0 info`, that can give more
> information.
>
>
> > Output of gluster volume info:
> >
> > Volume Name: gfs0
> > Type: Stripe
> > Volume ID: 2ca3dd45-6209-43ff-a164-7f2694097c64
> > Status: Started
> > Snapshot Count: 0
> > Number of Bricks: 1 x 6 = 6
> > Transport-type: tcp
> > Bricks:
> > Brick1: pfs01-ib:/mnt/data
> > Brick2: pfs02-ib:/mnt/data
> > Brick3: pfs03-ib:/mnt/data
> > Brick4: pfs04-ib:/mnt/data
> > Brick5: pfs05-ib:/mnt/data
> > Brick6: pfs06-ib:/mnt/data
> > Options Reconfigured:
> > cluster.stripe-block-size: 128KB
> > performance.cache-size: 32MB
> > performance.write-behind-window-size: 1MB
> > performance.strict-write-ordering: off
> > performance.strict-o-direct: off
> > performance.stat-prefetch: off
> > server.event-threads: 4
> > client.event-threads: 2
> > performance.io-thread-count: 16
> > transport.address-family: inet
> > nfs.disable: on
> > cluster.localtime-logging: enable
> >
> >
> >
>
-------------- next part --------------
Brick: pfs01-ib:/mnt/data
-------------------------
Cumulative Stats:
Block Size: 131072b+
No. of Reads: 0
No. of Writes: 131136
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 192 FORGET
0.00 0.00 us 0.00 us 0.00 us 576 RELEASE
0.05 51.18 us 20.63 us 256.30 us 192 LK
0.07 93.88 us 50.88 us 362.80 us 163 SETATTR
0.09 107.73 us 43.98 us 240.06 us 163 SETXATTR
0.11 40.00 us 12.66 us 217.19 us 576 FLUSH
0.14 41.94 us 15.70 us 184.47 us 669 STAT
0.14 98.87 us 50.81 us 309.65 us 288 OPEN
0.15 164.14 us 81.69 us 370.35 us 192 UNLINK
0.20 47.53 us 19.10 us 272.89 us 867 STATFS
0.40 286.72 us 197.48 us 521.78 us 288 CREATE
0.61 99.54 us 39.43 us 436.23 us 1252 LOOKUP
7.17 4521.94 us 23.03 us 28429.53 us 326 INODELK
90.87 142.49 us 67.03 us 13285.58 us 131136 WRITE
Duration: 375 seconds
Data Read: 0 bytes
Data Written: 17188257792 bytes
Interval 0 Stats:
Block Size: 131072b+
No. of Reads: 0
No. of Writes: 131136
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 192 FORGET
0.00 0.00 us 0.00 us 0.00 us 576 RELEASE
0.05 51.18 us 20.63 us 256.30 us 192 LK
0.07 93.88 us 50.88 us 362.80 us 163 SETATTR
0.09 107.73 us 43.98 us 240.06 us 163 SETXATTR
0.11 40.00 us 12.66 us 217.19 us 576 FLUSH
0.14 41.94 us 15.70 us 184.47 us 669 STAT
0.14 98.87 us 50.81 us 309.65 us 288 OPEN
0.15 164.14 us 81.69 us 370.35 us 192 UNLINK
0.20 47.53 us 19.10 us 272.89 us 867 STATFS
0.40 286.72 us 197.48 us 521.78 us 288 CREATE
0.61 99.54 us 39.43 us 436.23 us 1252 LOOKUP
7.17 4521.94 us 23.03 us 28429.53 us 326 INODELK
90.87 142.49 us 67.03 us 13285.58 us 131136 WRITE
Duration: 375 seconds
Data Read: 0 bytes
Data Written: 17188257792 bytes
Brick: pfs04-ib:/mnt/data
-------------------------
Cumulative Stats:
Block Size: 131072b+
No. of Reads: 0
No. of Writes: 131040
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 192 FORGET
0.00 0.00 us 0.00 us 0.00 us 576 RELEASE
0.05 57.99 us 19.80 us 224.35 us 192 LK
0.08 114.89 us 46.33 us 378.40 us 163 SETATTR
0.10 43.42 us 11.99 us 1391.00 us 576 FLUSH
0.12 44.41 us 16.43 us 249.75 us 669 STAT
0.13 104.06 us 54.86 us 1862.77 us 288 OPEN
0.16 45.24 us 20.35 us 246.34 us 867 STATFS
0.64 122.02 us 42.30 us 25803.78 us 1252 LOOKUP
1.05 1309.44 us 85.68 us 44130.35 us 192 UNLINK
3.46 2868.13 us 199.93 us 212895.24 us 288 CREATE
94.20 171.45 us 67.48 us 21658.19 us 131040 WRITE
Duration: 375 seconds
Data Read: 0 bytes
Data Written: 17175674880 bytes
Interval 0 Stats:
Block Size: 131072b+
No. of Reads: 0
No. of Writes: 131040
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 192 FORGET
0.00 0.00 us 0.00 us 0.00 us 576 RELEASE
0.05 57.99 us 19.80 us 224.35 us 192 LK
0.08 114.89 us 46.33 us 378.40 us 163 SETATTR
0.10 43.42 us 11.99 us 1391.00 us 576 FLUSH
0.12 44.41 us 16.43 us 249.75 us 669 STAT
0.13 104.06 us 54.86 us 1862.77 us 288 OPEN
0.16 45.24 us 20.35 us 246.34 us 867 STATFS
0.64 122.09 us 42.30 us 25803.78 us 1252 LOOKUP
1.05 1309.44 us 85.68 us 44130.35 us 192 UNLINK
3.46 2868.13 us 199.93 us 212895.24 us 288 CREATE
94.20 171.41 us 67.48 us 21658.19 us 131040 WRITE
Duration: 375 seconds
Data Read: 0 bytes
Data Written: 17175674880 bytes
Brick: pfs05-ib:/mnt/data
-------------------------
Cumulative Stats:
Block Size: 131072b+
No. of Reads: 0
No. of Writes: 131040
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 192 FORGET
0.00 0.00 us 0.00 us 0.00 us 576 RELEASE
0.04 55.49 us 15.39 us 301.38 us 192 LK
0.07 106.98 us 48.60 us 398.94 us 163 SETATTR
0.09 37.23 us 11.51 us 284.55 us 576 FLUSH
0.11 94.69 us 55.42 us 260.94 us 288 OPEN
0.11 41.13 us 16.07 us 203.31 us 669 STAT
0.16 45.45 us 19.89 us 270.21 us 867 STATFS
0.50 98.73 us 43.35 us 360.68 us 1252 LOOKUP
3.78 4820.73 us 85.53 us 343928.29 us 192 UNLINK
8.03 6823.51 us 204.49 us 344265.24 us 288 CREATE
87.10 162.75 us 67.78 us 20562.18 us 131040 WRITE
Duration: 375 seconds
Data Read: 0 bytes
Data Written: 17175674880 bytes
Interval 0 Stats:
Block Size: 131072b+
No. of Reads: 0
No. of Writes: 131040
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 192 FORGET
0.00 0.00 us 0.00 us 0.00 us 576 RELEASE
0.04 55.49 us 15.39 us 301.38 us 192 LK
0.07 106.98 us 48.60 us 398.94 us 163 SETATTR
0.09 37.23 us 11.51 us 284.55 us 576 FLUSH
0.11 94.69 us 55.42 us 260.94 us 288 OPEN
0.11 41.13 us 16.07 us 203.31 us 669 STAT
0.16 45.45 us 19.89 us 270.21 us 867 STATFS
0.50 98.73 us 43.35 us 360.68 us 1252 LOOKUP
3.78 4820.73 us 85.53 us 343928.29 us 192 UNLINK
8.03 6823.51 us 204.49 us 344265.24 us 288 CREATE
87.10 162.73 us 67.78 us 20562.18 us 131040 WRITE
Duration: 375 seconds
Data Read: 0 bytes
Data Written: 17175674880 bytes
Brick: pfs02-ib:/mnt/data
-------------------------
Cumulative Stats:
Block Size: 131072b+
No. of Reads: 0
No. of Writes: 131136
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 192 FORGET
0.00 0.00 us 0.00 us 0.00 us 576 RELEASE
0.06 65.96 us 17.65 us 2289.39 us 192 LK
0.08 100.75 us 50.54 us 323.66 us 163 SETATTR
0.10 36.51 us 12.29 us 250.04 us 576 FLUSH
0.13 90.32 us 50.30 us 226.21 us 288 OPEN
0.16 47.30 us 15.82 us 2281.57 us 669 STAT
0.20 46.76 us 18.38 us 2912.04 us 867 STATFS
0.61 97.93 us 40.99 us 780.38 us 1252 LOOKUP
1.56 1639.43 us 82.56 us 196463.69 us 192 UNLINK
5.95 4164.89 us 187.42 us 195441.47 us 288 CREATE
91.14 140.09 us 66.06 us 16004.89 us 131136 WRITE
Duration: 375 seconds
Data Read: 0 bytes
Data Written: 17188257792 bytes
Interval 0 Stats:
Block Size: 131072b+
No. of Reads: 0
No. of Writes: 131136
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 192 FORGET
0.00 0.00 us 0.00 us 0.00 us 576 RELEASE
0.06 65.96 us 17.65 us 2289.39 us 192 LK
0.08 100.75 us 50.54 us 323.66 us 163 SETATTR
0.10 36.51 us 12.29 us 250.04 us 576 FLUSH
0.13 90.32 us 50.30 us 226.21 us 288 OPEN
0.16 47.30 us 15.82 us 2281.57 us 669 STAT
0.20 46.76 us 18.38 us 2912.04 us 867 STATFS
0.61 97.93 us 40.99 us 780.38 us 1252 LOOKUP
1.56 1639.43 us 82.56 us 196463.69 us 192 UNLINK
5.95 4164.89 us 187.42 us 195441.47 us 288 CREATE
91.14 140.07 us 66.06 us 16004.89 us 131136 WRITE
Duration: 375 seconds
Data Read: 0 bytes
Data Written: 17188257792 bytes
Brick: pfs06-ib:/mnt/data
-------------------------
Cumulative Stats:
Block Size: 131072b+
No. of Reads: 0
No. of Writes: 131040
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 192 FORGET
0.00 0.00 us 0.00 us 0.00 us 576 RELEASE
0.06 72.23 us 19.21 us 3624.32 us 192 LK
0.08 107.83 us 52.61 us 314.04 us 163 SETATTR
0.09 35.56 us 12.99 us 282.80 us 576 FLUSH
0.12 91.34 us 50.59 us 234.47 us 288 OPEN
0.13 44.96 us 16.94 us 3535.79 us 669 STAT
0.18 45.68 us 19.13 us 321.56 us 867 STATFS
0.61 110.06 us 41.37 us 15742.64 us 1252 LOOKUP
1.40 1649.38 us 87.82 us 64877.34 us 192 UNLINK
3.19 2498.30 us 195.74 us 111136.27 us 288 CREATE
94.14 162.01 us 68.96 us 28557.12 us 131040 WRITE
Duration: 375 seconds
Data Read: 0 bytes
Data Written: 17175674880 bytes
Interval 0 Stats:
Block Size: 131072b+
No. of Reads: 0
No. of Writes: 131040
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 192 FORGET
0.00 0.00 us 0.00 us 0.00 us 576 RELEASE
0.06 72.54 us 19.21 us 3624.32 us 192 LK
0.08 107.83 us 52.61 us 314.04 us 163 SETATTR
0.09 35.56 us 12.99 us 282.80 us 576 FLUSH
0.12 91.34 us 50.59 us 234.47 us 288 OPEN
0.13 44.96 us 16.94 us 3535.79 us 669 STAT
0.18 45.68 us 19.13 us 321.56 us 867 STATFS
0.61 110.07 us 41.37 us 15742.64 us 1252 LOOKUP
1.40 1649.38 us 87.82 us 64877.34 us 192 UNLINK
3.19 2498.30 us 195.74 us 111136.27 us 288 CREATE
94.14 161.96 us 68.96 us 28557.12 us 131040 WRITE
Duration: 375 seconds
Data Read: 0 bytes
Data Written: 17175674880 bytes
Brick: pfs03-ib:/mnt/data
-------------------------
Cumulative Stats:
Block Size: 131072b+
No. of Reads: 0
No. of Writes: 131040
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 192 FORGET
0.00 0.00 us 0.00 us 0.00 us 576 RELEASE
0.05 57.70 us 19.13 us 345.61 us 192 LK
0.07 104.82 us 49.85 us 500.99 us 163 SETATTR
0.09 36.39 us 10.25 us 406.40 us 576 FLUSH
0.12 41.01 us 13.58 us 177.65 us 669 STAT
0.12 98.04 us 48.32 us 312.39 us 288 OPEN
0.17 46.52 us 18.59 us 186.27 us 867 STATFS
0.53 99.96 us 39.17 us 393.71 us 1252 LOOKUP
2.09 2566.82 us 84.69 us 136604.30 us 192 UNLINK
7.48 6129.64 us 202.67 us 198818.41 us 288 CREATE
89.28 160.74 us 67.20 us 47777.54 us 131040 WRITE
Duration: 375 seconds
Data Read: 0 bytes
Data Written: 17175674880 bytes
Interval 0 Stats:
Block Size: 131072b+
No. of Reads: 0
No. of Writes: 131040
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 192 FORGET
0.00 0.00 us 0.00 us 0.00 us 576 RELEASE
0.05 57.70 us 19.13 us 345.61 us 192 LK
0.07 104.82 us 49.85 us 500.99 us 163 SETATTR
0.09 36.39 us 10.25 us 406.40 us 576 FLUSH
0.12 41.01 us 13.58 us 177.65 us 669 STAT
0.12 98.04 us 48.32 us 312.39 us 288 OPEN
0.17 46.52 us 18.59 us 186.27 us 867 STATFS
0.53 99.96 us 39.17 us 393.71 us 1252 LOOKUP
2.09 2566.82 us 84.69 us 136604.30 us 192 UNLINK
7.48 6129.64 us 202.67 us 198818.41 us 288 CREATE
89.28 160.72 us 67.20 us 47777.54 us 131040 WRITE
Duration: 375 seconds
Data Read: 0 bytes
Data Written: 17175674880 bytes
More information about the Gluster-users
mailing list