[Gluster-users] Gluster performance issues - need advise

Amar Tumballi Suryanarayan atumball at redhat.com
Wed Jan 23 15:49:21 UTC 2019


I didn't understand the issue properly. Mostly I missed something.

Are you concerned the performance is 49MB/s with and without perf options?
or are you expecting it to be 123MB/s as over the n/w you get that speed?

If it is the first problem, then you are actually having
'performance.write-behind on' in both options, and it is the only perf
xlator which comes into action during the test you ran.

If it is the second, then please be informed that gluster does client side
replication, which means, n/w would be split in half for write operations
(like write(), creat() etc), so the number you are getting is almost the
maximum with 1GbE.

Regards,
Amar

On Wed, Jan 23, 2019 at 8:38 PM Strahil Nikolov <hunter86_bg at yahoo.com>
wrote:

> Hello Community,
>
> recently I have built a new lab based on oVirt and CentOS 7.
> During deployment I had some hicups, but now the engine is up and running
> - but gluster is causing me trouble.
>
> Symptoms: Slow VM install from DVD, poor write performance. The latter has
> been tested via:
> dd if=/dev/zero
> of=/rhev/data-center/mnt/glusterSD/ovirt1.localdomain:_data bs=1M
> count=1000 status=progress
>
> The reported speed is 60MB/s which is way too low for my setup.
>
> My lab design:
>
> https://drive.google.com/file/d/1SiW21ASPXHRAEuE_jZ50R3FoO-NcnFqT/view?usp=sharing
> Gluster version is 3.12.15
>
> So far I have done:
>
> 1. Added 'server.allow-insecure on' (with 'option rpc-auth-allow-insecure
> on' in glusterd.vol)
> Volume info after that change:
>
> Volume Name: data
> Type: Replicate
> Volume ID: 9b06a1e9-8102-4cd7-bc56-84960a1efaa2
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x (2 + 1) = 3
> Transport-type: tcp
> Bricks:
> Brick1: ovirt1.localdomain:/gluster_bricks/data/data
> Brick2: ovirt2.localdomain:/gluster_bricks/data/data
> Brick3: ovirt3.localdomain:/gluster_bricks/data/data (arbiter)
> Options Reconfigured:
> performance.client-io-threads: off
> nfs.disable: on
> transport.address-family: inet
> performance.quick-read: off
> performance.read-ahead: off
> performance.io-cache: off
> performance.low-prio-threads: 32
> network.remote-dio: off
> cluster.eager-lock: enable
> cluster.quorum-type: auto
> cluster.server-quorum-type: server
> cluster.data-self-heal-algorithm: full
> cluster.locking-scheme: granular
> cluster.shd-max-threads: 8
> cluster.shd-wait-qlength: 10000
> features.shard: on
> user.cifs: off
> storage.owner-uid: 36
> storage.owner-gid: 36
> network.ping-timeout: 30
> performance.strict-o-direct: on
> cluster.granular-entry-heal: enable
> server.allow-insecure: on
>
> Seems no positive or negative effect so far.
>
> 2. Tested with tmpfs  on all bricks -> ovirt1 mounted gluster volume ->
> max 60MB/s (bs=1M without 'oflag=direct')
>
>
> [root at ovirt1 data]# dd if=/dev/zero of=large_io bs=1M  count=4000
> status=progress
> 4177526784 bytes (4.2 GB) copied, 70.843409 s, 59.0 MB/s
> 4000+0 records in
> 4000+0 records out
> 4194304000 bytes (4.2 GB) copied, 71.1407 s, 59.0 MB/s
> [root at ovirt1 data]# rm -f large_io
> [root at ovirt1 data]# gluster volume profile data info
> Brick: ovirt1.localdomain:/gluster_bricks/data/data
> ---------------------------------------------------
> Cumulative Stats:
> Block Size:            131072b+
> No. of Reads:                    8
> No. of Writes:                44968
> %-latency  Avg-latency  Min-Latency  Max-Latency  No. of calls        Fop
> ---------  -----------  -----------  -----------  ------------        ----
> 0.00      0.00 us      0.00 us      0.00 us              3      FORGET
> 0.00      0.00 us      0.00 us      0.00 us            35    RELEASE
> 0.00      0.00 us      0.00 us      0.00 us            28  RELEASEDIR
> 0.00      78.00 us      78.00 us      78.00 us              1      FSTAT
> 0.00      35.67 us      26.00 us      73.00 us              6      FLUSH
> 0.00    324.00 us    324.00 us    324.00 us              1    XATTROP
> 0.00      45.80 us      38.00 us      54.00 us            10        STAT
> 0.00    227.67 us    216.00 us    242.00 us              3      CREATE
> 0.00    113.38 us      68.00 us    381.00 us              8        READ
> 0.00      39.82 us      1.00 us    148.00 us            28    OPENDIR
> 0.00      67.54 us      10.00 us    283.00 us            24    GETXATTR
> 0.00      59.97 us      45.00 us    113.00 us            32        OPEN
> 0.00      24.41 us      13.00 us      89.00 us            161    INODELK
> 0.00      43.43 us      28.00 us    214.00 us            93      STATFS
> 0.00    246.35 us      11.00 us    1155.00 us            20    READDIR
> 0.00    283.00 us    233.00 us    353.00 us            18    READDIRP
> 0.00    153.23 us    122.00 us    259.00 us            87      MKNOD
> 0.01      99.77 us      10.00 us    258.00 us            442      LOOKUP
> 0.31      49.22 us      27.00 us    540.00 us          45620    FXATTROP
> 0.77    124.24 us      87.00 us    604.00 us          44968      WRITE
> 0.93  15767.71 us      15.00 us  305833.00 us            431    ENTRYLK
> 1.99  160711.39 us    3332.00 us  406037.00 us            90      UNLINK
> 96.00    5167.82 us      18.00 us  55972.00 us        135349    FINODELK
>
> Duration: 380 seconds
> Data Read: 1048576 bytes
> Data Written: 5894045696 bytes
>
> Interval 0 Stats:
> Block Size:            131072b+
> No. of Reads:                    8
> No. of Writes:                44968
> %-latency  Avg-latency  Min-Latency  Max-Latency  No. of calls        Fop
> ---------  -----------  -----------  -----------  ------------        ----
> 0.00      0.00 us      0.00 us      0.00 us              3      FORGET
> 0.00      0.00 us      0.00 us      0.00 us            35    RELEASE
> 0.00      0.00 us      0.00 us      0.00 us            28  RELEASEDIR
> 0.00      78.00 us      78.00 us      78.00 us              1      FSTAT
> 0.00      35.67 us      26.00 us      73.00 us              6      FLUSH
> 0.00    324.00 us    324.00 us    324.00 us              1    XATTROP
> 0.00      45.80 us      38.00 us      54.00 us            10        STAT
> 0.00    227.67 us    216.00 us    242.00 us              3      CREATE
> 0.00    113.38 us      68.00 us    381.00 us              8        READ
> 0.00      39.82 us      1.00 us    148.00 us            28    OPENDIR
> 0.00      67.54 us      10.00 us    283.00 us            24    GETXATTR
> 0.00      59.97 us      45.00 us    113.00 us            32        OPEN
> 0.00      24.41 us      13.00 us      89.00 us            161    INODELK
> 0.00      43.43 us      28.00 us    214.00 us            93      STATFS
> 0.00    246.35 us      11.00 us    1155.00 us            20    READDIR
> 0.00    283.00 us    233.00 us    353.00 us            18    READDIRP
> 0.00    153.23 us    122.00 us    259.00 us            87      MKNOD
> 0.01      99.77 us      10.00 us    258.00 us            442      LOOKUP
> 0.31      49.22 us      27.00 us    540.00 us          45620    FXATTROP
> 0.77    124.24 us      87.00 us    604.00 us          44968      WRITE
> 0.93  15767.71 us      15.00 us  305833.00 us            431    ENTRYLK
> 1.99  160711.39 us    3332.00 us  406037.00 us            90      UNLINK
> 96.00    5167.82 us      18.00 us  55972.00 us        135349    FINODELK
>
> Duration: 380 seconds
> Data Read: 1048576 bytes
> Data Written: 5894045696 bytes
>
> Brick: ovirt3.localdomain:/gluster_bricks/data/data
> ---------------------------------------------------
> Cumulative Stats:
> Block Size:                  1b+
> No. of Reads:                    0
> No. of Writes:                39328
> %-latency  Avg-latency  Min-Latency  Max-Latency  No. of calls        Fop
> ---------  -----------  -----------  -----------  ------------        ----
> 0.00      0.00 us      0.00 us      0.00 us              2      FORGET
> 0.00      0.00 us      0.00 us      0.00 us            12    RELEASE
> 0.00      0.00 us      0.00 us      0.00 us            17  RELEASEDIR
> 0.00    101.00 us    101.00 us    101.00 us              1      FSTAT
> 0.00      51.50 us      20.00 us      81.00 us              4      FLUSH
> 0.01    219.50 us    188.00 us    251.00 us              2      CREATE
> 0.01      43.45 us      11.00 us      90.00 us            11    GETXATTR
> 0.01      62.30 us      38.00 us    119.00 us            10        OPEN
> 0.01      50.59 us      1.00 us    102.00 us            17    OPENDIR
> 0.01      24.60 us      12.00 us      64.00 us            40    INODELK
> 0.02    176.30 us      10.00 us    765.00 us            10    READDIR
> 0.07      63.08 us      39.00 us    133.00 us            78      UNLINK
> 0.13      27.35 us      10.00 us      91.00 us            333    ENTRYLK
> 0.13    126.89 us      99.00 us    179.00 us            76      MKNOD
> 0.42    116.70 us      8.00 us    8661.00 us            261      LOOKUP
> 28.73      51.79 us      22.00 us    2574.00 us          39822    FXATTROP
> 29.52      53.87 us      16.00 us    3290.00 us          39328      WRITE
> 40.92      24.71 us      10.00 us    3224.00 us        118864    FINODELK
>
> Duration: 189 seconds
> Data Read: 0 bytes
> Data Written: 39328 bytes
>
> Interval 0 Stats:
> Block Size:                  1b+
> No. of Reads:                    0
> No. of Writes:                39328
> %-latency  Avg-latency  Min-Latency  Max-Latency  No. of calls        Fop
> ---------  -----------  -----------  -----------  ------------        ----
> 0.00      0.00 us      0.00 us      0.00 us              2      FORGET
> 0.00      0.00 us      0.00 us      0.00 us            12    RELEASE
> 0.00      0.00 us      0.00 us      0.00 us            17  RELEASEDIR
> 0.00    101.00 us    101.00 us    101.00 us              1      FSTAT
> 0.00      51.50 us      20.00 us      81.00 us              4      FLUSH
> 0.01    219.50 us    188.00 us    251.00 us              2      CREATE
> 0.01      43.45 us      11.00 us      90.00 us            11    GETXATTR
> 0.01      62.30 us      38.00 us    119.00 us            10        OPEN
> 0.01      50.59 us      1.00 us    102.00 us            17    OPENDIR
> 0.01      24.60 us      12.00 us      64.00 us            40    INODELK
> 0.02    176.30 us      10.00 us    765.00 us            10    READDIR
> 0.07      63.08 us      39.00 us    133.00 us            78      UNLINK
> 0.13      27.35 us      10.00 us      91.00 us            333    ENTRYLK
> 0.13    126.89 us      99.00 us    179.00 us            76      MKNOD
> 0.42    116.70 us      8.00 us    8661.00 us            261      LOOKUP
> 28.73      51.79 us      22.00 us    2574.00 us          39822    FXATTROP
> 29.52      53.87 us      16.00 us    3290.00 us          39328      WRITE
> 40.92      24.71 us      10.00 us    3224.00 us        118864    FINODELK
>
> Duration: 189 seconds
> Data Read: 0 bytes
> Data Written: 39328 bytes
>
> Brick: ovirt2.localdomain:/gluster_bricks/data/data
> ---------------------------------------------------
> Cumulative Stats:
> Block Size:                512b+              131072b+
> No. of Reads:                    0                    0
> No. of Writes:                  36                76758
> %-latency  Avg-latency  Min-Latency  Max-Latency  No. of calls        Fop
> ---------  -----------  -----------  -----------  ------------        ----
> 0.00      0.00 us      0.00 us      0.00 us              6      FORGET
> 0.00      0.00 us      0.00 us      0.00 us            87    RELEASE
> 0.00      0.00 us      0.00 us      0.00 us            96  RELEASEDIR
> 0.00    100.50 us      80.00 us    121.00 us              2 REMOVEXATTR
> 0.00    101.00 us    101.00 us    101.00 us              2    SETXATTR
> 0.00      36.18 us      22.00 us      62.00 us            11      FLUSH
> 0.00      57.44 us      42.00 us      77.00 us              9  FTRUNCATE
> 0.00      82.56 us      59.00 us    138.00 us              9      FSTAT
> 0.00      89.42 us      67.00 us    161.00 us            12    SETATTR
> 0.00    272.40 us    235.00 us    296.00 us              5      CREATE
> 0.01    154.28 us      88.00 us    320.00 us            18    XATTROP
> 0.01      45.29 us      1.00 us    319.00 us            96    OPENDIR
> 0.01      86.69 us      30.00 us    379.00 us            62        STAT
> 0.01      64.30 us      47.00 us    169.00 us            84        OPEN
> 0.02    107.34 us      23.00 us    273.00 us            73    READDIRP
> 0.02    4688.00 us      86.00 us    9290.00 us              2    TRUNCATE
> 0.02      59.29 us      13.00 us    394.00 us            165    GETXATTR
> 0.03    128.51 us      27.00 us    338.00 us            96      FSYNC
> 0.03    240.75 us      14.00 us    1943.00 us            52    READDIR
> 0.04      65.59 us      26.00 us    293.00 us            279      STATFS
> 0.06    180.77 us    118.00 us    306.00 us            148      MKNOD
> 0.14      37.98 us      17.00 us    192.00 us          1598    INODELK
> 0.67      91.68 us      12.00 us    1141.00 us          3186      LOOKUP
> 10.10      55.92 us      28.00 us    1658.00 us          78608    FXATTROP
> 11.89    6814.76 us      18.00 us  301246.00 us            760    ENTRYLK
> 19.44      36.55 us      14.00 us    2353.00 us        231535    FINODELK
> 25.21    142.92 us      62.00 us    593.00 us          76794      WRITE
> 32.28  91283.68 us      28.00 us  316658.00 us            154      UNLINK
>
> Duration: 1206 seconds
> Data Read: 0 bytes
> Data Written: 10060843008 bytes
>
> Interval 0 Stats:
> Block Size:                512b+              131072b+
> No. of Reads:                    0                    0
> No. of Writes:                  36                76758
> %-latency  Avg-latency  Min-Latency  Max-Latency  No. of calls        Fop
> ---------  -----------  -----------  -----------  ------------        ----
> 0.00      0.00 us      0.00 us      0.00 us              6      FORGET
> 0.00      0.00 us      0.00 us      0.00 us            87    RELEASE
> 0.00      0.00 us      0.00 us      0.00 us            96  RELEASEDIR
> 0.00    100.50 us      80.00 us    121.00 us              2 REMOVEXATTR
> 0.00    101.00 us    101.00 us    101.00 us              2    SETXATTR
> 0.00      36.18 us      22.00 us      62.00 us            11      FLUSH
> 0.00      57.44 us      42.00 us      77.00 us              9  FTRUNCATE
> 0.00      82.56 us      59.00 us    138.00 us              9      FSTAT
> 0.00      89.42 us      67.00 us    161.00 us            12    SETATTR
> 0.00    272.40 us    235.00 us    296.00 us              5      CREATE
> 0.01    154.28 us      88.00 us    320.00 us            18    XATTROP
> 0.01      45.29 us      1.00 us    319.00 us            96    OPENDIR
> 0.01      86.69 us      30.00 us    379.00 us            62        STAT
> 0.01      64.30 us      47.00 us    169.00 us            84        OPEN
> 0.02    107.34 us      23.00 us    273.00 us            73    READDIRP
> 0.02    4688.00 us      86.00 us    9290.00 us              2    TRUNCATE
> 0.02      59.29 us      13.00 us    394.00 us            165    GETXATTR
> 0.03    128.51 us      27.00 us    338.00 us            96      FSYNC
> 0.03    240.75 us      14.00 us    1943.00 us            52    READDIR
> 0.04      65.59 us      26.00 us    293.00 us            279      STATFS
> 0.06    180.77 us    118.00 us    306.00 us            148      MKNOD
> 0.14      37.98 us      17.00 us    192.00 us          1598    INODELK
> 0.67      91.66 us      12.00 us    1141.00 us          3186      LOOKUP
> 10.10      55.92 us      28.00 us    1658.00 us          78608    FXATTROP
> 11.89    6814.76 us      18.00 us  301246.00 us            760    ENTRYLK
> 19.44      36.55 us      14.00 us    2353.00 us        231535    FINODELK
> 25.21    142.92 us      62.00 us    593.00 us          76794      WRITE
> 32.28  91283.68 us      28.00 us  316658.00 us            154      UNLINK
>
> Duration: 1206 seconds
> Data Read: 0 bytes
> Data Written: 10060843008 bytes
>
>
>
> This indicates to me that it's not a problem in Disk/LVM/FileSystem layout.
>
> Most probably I haven't created the volume properly or some option/feature
> is disabled ?!?
> Network shows OK for a gigabit:
> [root at ovirt1 data]# dd if=/dev/zero status=progress | nc ovirt2 9999
> 3569227264 bytes (3.6 GB) copied, 29.001052 s, 123 MB/s^C
> 7180980+0 records in
> 7180979+0 records out
> 3676661248 bytes (3.7 GB) copied, 29.8739 s, 123 MB/s
>
>
> I'm looking for any help... you can share your volume info also.
>
> Thanks in advance.
>
> Best Regards,
> Strahil Nikolov
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>


-- 
Amar Tumballi (amarts)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190123/ff93813c/attachment.html>


More information about the Gluster-users mailing list