<div dir="ltr">Hi All,<br>We have set up a new gluster volume, server version 9.3 with clients running 9.0. In this setup we are facing issues where the clients get randomly disconnected and there is no relevant log around that time in client, server and brick logs. One difference we have noticed in this setup vs the other setups which we have in production is that there are many small read/writes happening in this.<br>Wondering if anyone can help with the performance tuning parameters which we can set/unset to optimize our setup.<br>Any suggestions are welcome and appreciated. <br><br>Adding the current server parameters:<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><b>Option                  Value<br></b><b>------                  -----<br></b><b>cluster.lookup-unhashed          on (DEFAULT)<br></b><b>cluster.lookup-optimize          on (DEFAULT)<br></b><b>cluster.min-free-disk           10% (DEFAULT)<br></b><b>cluster.min-free-inodes          5% (DEFAULT)<br></b><b>cluster.rebalance-stats          off (DEFAULT)<br></b><b>cluster.subvols-per-directory       (null) (DEFAULT)<br></b><b>cluster.readdir-optimize         off (DEFAULT)<br></b><b>cluster.rsync-hash-regex         (null) (DEFAULT)<br></b><b>cluster.extra-hash-regex         (null) (DEFAULT)<br></b><b>cluster.dht-xattr-name          trusted.glusterfs.dht (DEFAULT)<br></b><b>cluster.randomize-hash-range-by-gfid   off (DEFAULT)<br></b><b>cluster.rebal-throttle          normal (DEFAULT)<br></b><b>cluster.lock-migration          off<br></b><b>cluster.force-migration          off<br></b><b>cluster.local-volume-name         (null) (DEFAULT)<br></b><b>cluster.weighted-rebalance        on (DEFAULT)<br></b><b>cluster.switch-pattern          (null) (DEFAULT)<br></b><b>cluster.entry-change-log         on (DEFAULT)<br></b><b>cluster.read-subvolume          (null) (DEFAULT)<br></b><b>cluster.read-subvolume-index       -1 (DEFAULT)<br></b><b>cluster.read-hash-mode          1 (DEFAULT)<br></b><b>cluster.background-self-heal-count    8 (DEFAULT)<br></b><b>cluster.metadata-self-heal        off (DEFAULT)<br></b><b>cluster.data-self-heal          off (DEFAULT)<br></b><b>cluster.entry-self-heal          off (DEFAULT)<br></b><b>cluster.self-heal-daemon         on (DEFAULT)<br></b><b>cluster.heal-timeout           600 (DEFAULT)<br></b><b>cluster.self-heal-window-size       8 (DEFAULT)<br></b><b>cluster.data-change-log          on (DEFAULT)<br></b><b>cluster.metadata-change-log        on (DEFAULT)<br></b><b>cluster.data-self-heal-algorithm     (null) (DEFAULT)<br></b><b>cluster.eager-lock            on (DEFAULT)<br></b><b>disperse.eager-lock            on (DEFAULT)<br></b><b>disperse.other-eager-lock         on (DEFAULT)<br></b><b>disperse.eager-lock-timeout        1 (DEFAULT)<br></b><b>disperse.other-eager-lock-timeout     1 (DEFAULT)<br></b><b>cluster.quorum-type            none (DEFAULT)<br></b><b>cluster.quorum-count           (null) (DEFAULT)<br></b><b>cluster.choose-local           true (DEFAULT)<br></b><b>cluster.self-heal-readdir-size      1KB (DEFAULT)<br></b><b>cluster.post-op-delay-secs        1 (DEFAULT)<br></b><b>cluster.ensure-durability         on (DEFAULT)<br></b><b>cluster.consistent-metadata        no (DEFAULT)<br></b><b>cluster.heal-wait-queue-length      128 (DEFAULT)<br></b><b>cluster.favorite-child-policy       none (DEFAULT)<br></b><b>cluster.full-lock             yes (DEFAULT)<br></b><b>cluster.optimistic-change-log       on (DEFAULT)<br></b><b>diagnostics.latency-measurement      off<br></b><b>diagnostics.dump-fd-stats         off (DEFAULT)<br></b><b>diagnostics.count-fop-hits        off<br></b><b>diagnostics.brick-log-level        INFO<br></b><b>diagnostics.client-log-level       INFO<br></b><b>diagnostics.brick-sys-log-level      CRITICAL (DEFAULT)<br></b><b>diagnostics.client-sys-log-level     CRITICAL (DEFAULT)<br></b><b>diagnostics.brick-logger         (null) (DEFAULT)<br></b><b>diagnostics.client-logger         (null) (DEFAULT)<br></b><b>diagnostics.brick-log-format       (null) (DEFAULT)<br></b><b>diagnostics.client-log-format       (null) (DEFAULT)<br></b><b>diagnostics.brick-log-buf-size      5 (DEFAULT)<br></b><b>diagnostics.client-log-buf-size      5 (DEFAULT)<br></b><b>diagnostics.brick-log-flush-timeout    120 (DEFAULT)<br></b><b>diagnostics.client-log-flush-timeout   120 (DEFAULT)<br></b><b>diagnostics.stats-dump-interval      0 (DEFAULT)<br></b><b>diagnostics.fop-sample-interval      0 (DEFAULT)<br></b><b>diagnostics.stats-dump-format       json (DEFAULT)<br></b><b>diagnostics.fop-sample-buf-size      65535 (DEFAULT)<br></b><b>diagnostics.stats-dnscache-ttl-sec    86400 (DEFAULT)<br></b><b>performance.cache-max-file-size      0 (DEFAULT)<br></b><b>performance.cache-min-file-size      0 (DEFAULT)<br></b><b>performance.cache-refresh-timeout     1 (DEFAULT)<br></b><b>performance.cache-priority         (DEFAULT)<br></b><b>performance.io-cache-size         32MB (DEFAULT)<br></b><b>performance.cache-size          256MB<br></b><b>performance.io-thread-count        16 (DEFAULT)<br></b><b>performance.high-prio-threads       16 (DEFAULT)<br></b><b>performance.normal-prio-threads      16 (DEFAULT)<br></b><b>performance.low-prio-threads       16 (DEFAULT)<br></b><b>performance.least-prio-threads      1 (DEFAULT)<br></b><b>performance.enable-least-priority     on (DEFAULT)<br></b><b>performance.iot-watchdog-secs       (null) (DEFAULT)<br></b><b>performance.iot-cleanup-disconnected-reqs off (DEFAULT)<br></b><b>performance.iot-pass-through       false (DEFAULT)<br></b><b>performance.io-cache-pass-through     false (DEFAULT)<br></b><b>performance.quick-read-cache-size     128MB (DEFAULT)<br></b><b>performance.cache-size          256MB<br></b><b>performance.quick-read-cache-timeout   1 (DEFAULT)<br></b><b>performance.qr-cache-timeout       600<br></b><b>performance.quick-read-cache-invalidation false (DEFAULT)<br></b><b>performance.ctime-invalidation      false (DEFAULT)<br></b><b>performance.flush-behind         on (DEFAULT)<br></b><b>performance.nfs.flush-behind       on (DEFAULT)<br></b><b>performance.write-behind-window-size   1MB (DEFAULT)<br></b><b>performance.resync-failed-syncs-after-fsync off (DEFAULT)<br></b><b>performance.nfs.write-behind-window-size 1MB (DEFAULT)<br></b><b>performance.strict-o-direct        off (DEFAULT)<br></b><b>performance.nfs.strict-o-direct      off (DEFAULT)<br></b><b>performance.strict-write-ordering     off (DEFAULT)<br></b><b>performance.nfs.strict-write-ordering   off (DEFAULT)<br></b><b>performance.write-behind-trickling-writes on (DEFAULT)<br></b><b>performance.aggregate-size        128KB (DEFAULT)<br></b><b>performance.nfs.write-behind-trickling-writes on (DEFAULT)<br></b><b>performance.lazy-open           yes (DEFAULT)<br></b><b>performance.read-after-open        yes (DEFAULT)<br></b><b>performance.open-behind-pass-through   false (DEFAULT)<br></b><b>performance.read-ahead-page-count     4 (DEFAULT)<br></b><b>performance.read-ahead-pass-through    false (DEFAULT)<br></b><b>performance.readdir-ahead-pass-through  false (DEFAULT)<br></b><b>performance.md-cache-pass-through     false (DEFAULT)<br></b><b>performance.write-behind-pass-through   false (DEFAULT)<br></b><b>performance.md-cache-timeout       1 (DEFAULT)<br></b><b>performance.cache-swift-metadata     false (DEFAULT)<br></b><b>performance.cache-samba-metadata     false (DEFAULT)<br></b><b>performance.cache-capability-xattrs    true (DEFAULT)<br></b><b>performance.cache-ima-xattrs       true (DEFAULT)<br></b><b>performance.md-cache-statfs        off (DEFAULT)<br></b><b>performance.xattr-cache-list        (DEFAULT)<br></b><b>performance.nl-cache-pass-through     false (DEFAULT)<br></b><b>network.frame-timeout           1800 (DEFAULT)<br></b><b>network.ping-timeout           42 (DEFAULT)<br></b><b>network.tcp-window-size          (null) (DEFAULT)<br></b><b>client.ssl                off<br></b><b>network.remote-dio            disable (DEFAULT)<br></b><b>client.event-threads           12<br></b><b>client.tcp-user-timeout          0<br></b><b>client.keepalive-time           20000<br></b><b>client.keepalive-interval         2000<br></b><b>client.keepalive-count          9<br></b><b>client.strict-locks            off<br></b><b>network.tcp-window-size          (null) (DEFAULT)<br></b><b>network.inode-lru-limit          50000<br></b><b>auth.allow                *<br></b><b>auth.reject                (null) (DEFAULT)<br></b><b>transport.keepalive            1<br></b><b>server.allow-insecure           on (DEFAULT)<br></b><b>server.root-squash            off (DEFAULT)<br></b><b>server.all-squash             off (DEFAULT)<br></b><b>server.anonuid              65534 (DEFAULT)<br></b><b>server.anongid              65534 (DEFAULT)<br></b><b>server.statedump-path           /var/run/gluster (DEFAULT)<br></b><b>server.outstanding-rpc-limit       128<br></b><b>server.ssl                off<br></b><b>auth.ssl-allow              *<br></b><b>server.manage-gids            off (DEFAULT)<br></b><b>server.dynamic-auth            on (DEFAULT)<br></b><b>client.send-gids             on (DEFAULT)<br></b><b>server.gid-timeout            300 (DEFAULT)<br></b><b>server.own-thread             (null) (DEFAULT)<br></b><b>server.event-threads           12<br></b><b>server.tcp-user-timeout          42 (DEFAULT)<br></b><b>server.keepalive-time           20<br></b><b>server.keepalive-interval         2<br></b><b>server.keepalive-count          9<br></b><b>transport.listen-backlog         1024<br></b><b>ssl.own-cert               (null) (DEFAULT)<br></b><b>ssl.private-key              (null) (DEFAULT)<br></b><b>ssl.ca-list                (null) (DEFAULT)<br></b><b>ssl.crl-path               (null) (DEFAULT)<br></b><b>ssl.certificate-depth           (null) (DEFAULT)<br></b><b>ssl.cipher-list              (null) (DEFAULT)<br></b><b>ssl.dh-param               (null) (DEFAULT)<br></b><b>ssl.ec-curve               (null) (DEFAULT)<br></b><b>transport.address-family         inet<br></b><b>performance.write-behind         on<br></b><b>performance.read-ahead          off<br></b><b>performance.readdir-ahead         off<br></b><b>performance.io-cache           off<br></b><b>performance.open-behind          on<br></b><b>performance.quick-read          on<br></b><b>performance.nl-cache           on<br></b><b>performance.stat-prefetch         on<br></b><b>performance.client-io-threads       on<br></b><b>performance.nfs.write-behind       on<br></b><b>performance.nfs.read-ahead        off<br></b><b>performance.nfs.io-cache         off<br></b><b>performance.nfs.quick-read        off<br></b><b>performance.nfs.stat-prefetch       off<br></b><b>performance.nfs.io-threads        off<br></b><b>performance.force-readdirp        true (DEFAULT)<br></b><b>performance.cache-invalidation      on<br></b><b>performance.global-cache-invalidation   true (DEFAULT)<br></b><b>features.uss               off<br></b><b>features.snapshot-directory        .snaps<br></b><b>features.show-snapshot-directory     off<br></b><b>features.tag-namespaces          off<br></b><b>network.compression            off<br></b><b>network.compression.window-size      -15 (DEFAULT)<br></b><b>network.compression.mem-level       8 (DEFAULT)<br></b><b>network.compression.min-size       0 (DEFAULT)<br></b><b>network.compression.compression-level   -1 (DEFAULT)<br></b><b>network.compression.debug         false (DEFAULT)<br></b><b>features.default-soft-limit        80% (DEFAULT)<br></b><b>features.soft-timeout           60 (DEFAULT)<br></b><b>features.hard-timeout           5 (DEFAULT)<br></b><b>features.alert-time            86400 (DEFAULT)<br></b><b>features.quota-deem-statfs        off<br></b><b>geo-replication.indexing         off<br></b><b>geo-replication.indexing         off<br></b><b>geo-replication.ignore-pid-check     off<br></b><b>geo-replication.ignore-pid-check     off<br></b><b>features.quota              off<br></b><b>features.inode-quota           off<br></b><b>features.bitrot              disable<br></b><b>debug.trace                off<br></b><b>debug.log-history             no (DEFAULT)<br></b><b>debug.log-file              no (DEFAULT)<br></b><b>debug.exclude-ops             (null) (DEFAULT)<br></b><b>debug.include-ops             (null) (DEFAULT)<br></b><b>debug.error-gen              off<br></b><b>debug.error-failure            (null) (DEFAULT)<br></b><b>debug.error-number            (null) (DEFAULT)<br></b><b>debug.random-failure           off (DEFAULT)<br></b><b>debug.error-fops             (null) (DEFAULT)<br></b><b>nfs.disable                on<br></b><b>features.read-only            off (DEFAULT)<br></b><b>features.worm               off<br></b><b>features.worm-file-level         off<br></b><b>features.worm-files-deletable       on<br></b><b>features.default-retention-period     120 (DEFAULT)<br></b><b>features.retention-mode          relax (DEFAULT)<br></b><b>features.auto-commit-period        180 (DEFAULT)<br></b><b>storage.linux-aio             off (DEFAULT)<br></b><b>storage.linux-io_uring          off (DEFAULT)<br></b><b>storage.batch-fsync-mode         reverse-fsync (DEFAULT)<br></b><b>storage.batch-fsync-delay-usec      0 (DEFAULT)<br></b><b>storage.owner-uid             -1 (DEFAULT)<br></b><b>storage.owner-gid             -1 (DEFAULT)<br></b><b>storage.node-uuid-pathinfo        off (DEFAULT)<br></b><b>storage.health-check-interval       30 (DEFAULT)<br></b><b>storage.build-pgfid            off (DEFAULT)<br></b><b>storage.gfid2path             on (DEFAULT)<br></b><b>storage.gfid2path-separator        : (DEFAULT)<br></b><b>storage.reserve              1 (DEFAULT)<br></b><b>storage.health-check-timeout       20 (DEFAULT)<br></b><b>storage.fips-mode-rchecksum        off<br></b><b>storage.force-create-mode         0000 (DEFAULT)<br></b><b>storage.force-directory-mode       0000 (DEFAULT)<br></b><b>storage.create-mask            0777 (DEFAULT)<br></b><b>storage.create-directory-mask       0777 (DEFAULT)<br></b><b>storage.max-hardlinks           100 (DEFAULT)<br></b><b>features.ctime              on (DEFAULT)<br></b><b>config.gfproxyd              off<br></b><b>cluster.server-quorum-type        off
</b><b>cluster.server-quorum-ratio        51<br></b><b>changelog.changelog            off (DEFAULT)<br></b><b>changelog.changelog-dir          {{ brick.path }}/.glusterfs/changelogs (DEFAULT)<br></b><b>changelog.encoding            ascii (DEFAULT)<br></b><b>changelog.rollover-time          15 (DEFAULT)<br></b><b>changelog.fsync-interval         5 (DEFAULT)<br></b><b>changelog.changelog-barrier-timeout    120<br></b><b>changelog.capture-del-path        off (DEFAULT)<br></b><b>features.barrier             disable<br></b><b>features.barrier-timeout         120<br></b><b>features.trash              off (DEFAULT)<br></b><b>features.trash-dir            .trashcan (DEFAULT)<br></b><b>features.trash-eliminate-path       (null) (DEFAULT)<br></b><b>features.trash-max-filesize        5MB (DEFAULT)<br></b><b>features.trash-internal-op        off (DEFAULT)<br></b><b>cluster.enable-shared-storage       disable<br></b><b>locks.trace                off (DEFAULT)<br></b><b>locks.mandatory-locking          off (DEFAULT)<br></b><b>cluster.disperse-self-heal-daemon     enable (DEFAULT)<br></b><b>cluster.quorum-reads           no (DEFAULT)<br></b><b>client.bind-insecure           (null) (DEFAULT)<br></b><b>features.shard              off<br></b><b>features.shard-block-size         64MB (DEFAULT)<br></b><b>features.shard-lru-limit         16384 (DEFAULT)<br></b><b>features.shard-deletion-rate       100 (DEFAULT)<br></b><b>features.scrub-throttle          lazy<br></b><b>features.scrub-freq            biweekly<br></b><b>features.scrub              false (DEFAULT)<br></b><b>features.expiry-time           120<br></b><b>features.signer-threads          4<br></b><b>features.cache-invalidation        on<br></b><b>features.cache-invalidation-timeout    600<br></b><b>ganesha.enable              off<br></b><b>features.leases              off<br></b><b>features.lease-lock-recall-timeout    60 (DEFAULT)<br></b><b>disperse.background-heals         8 (DEFAULT)<br></b><b>disperse.heal-wait-qlength        128 (DEFAULT)<br></b><b>cluster.heal-timeout           600 (DEFAULT)<br></b><b>dht.force-readdirp            on (DEFAULT)<br></b><b>disperse.read-policy           gfid-hash (DEFAULT)<br></b><b>cluster.shd-max-threads          1 (DEFAULT)<br></b><b>cluster.shd-wait-qlength         1024 (DEFAULT)<br></b><b>cluster.locking-scheme          full (DEFAULT)<br></b><b>cluster.granular-entry-heal        no (DEFAULT)<br></b><b>features.locks-revocation-secs      0 (DEFAULT)<br></b><b>features.locks-revocation-clear-all    false (DEFAULT)<br></b><b>features.locks-revocation-max-blocked   0 (DEFAULT)<br></b><b>features.locks-monkey-unlocking      false (DEFAULT)<br></b><b>features.locks-notify-contention     yes (DEFAULT)<br></b><b>features.locks-notify-contention-delay  5 (DEFAULT)<br></b><b>disperse.shd-max-threads         1 (DEFAULT)<br></b><b>disperse.shd-wait-qlength         1024 (DEFAULT)<br></b><b>disperse.cpu-extensions          auto (DEFAULT)<br></b><b>disperse.self-heal-window-size      32 (DEFAULT)<br></b><b>cluster.use-compound-fops         off<br></b><b>performance.parallel-readdir       on<br></b><b>performance.rda-request-size       131072<br></b><b>performance.rda-low-wmark         4096 (DEFAULT)<br></b><b>performance.rda-high-wmark        128KB (DEFAULT)<br></b><b>performance.rda-cache-limit        10MB<br></b><b>performance.nl-cache-positive-entry    false (DEFAULT)<br></b><b>performance.nl-cache-limit        10MB<br></b><b>performance.nl-cache-timeout       600<br></b><b>cluster.brick-multiplex          disable<br></b><b>glusterd.vol_count_per_thread       100<br></b><b>cluster.max-bricks-per-process      250<br></b><b>disperse.optimistic-change-log      on (DEFAULT)<br></b><b>disperse.stripe-cache           4 (DEFAULT)<br></b><b>cluster.halo-enabled           False (DEFAULT)<br></b><b>cluster.halo-shd-max-latency       99999 (DEFAULT)<br></b><b>cluster.halo-nfsd-max-latency       5 (DEFAULT)<br></b><b>cluster.halo-max-latency         5 (DEFAULT)<br></b><b>cluster.halo-max-replicas         99999 (DEFAULT)<br></b><b>cluster.halo-min-replicas         2 (DEFAULT)<br></b><b>features.selinux             on<br></b><b>cluster.daemon-log-level         INFO<br></b><b>debug.delay-gen              off<br></b><b>delay-gen.delay-percentage        10% (DEFAULT)<br></b><b>delay-gen.delay-duration         100000 (DEFAULT)<br></b><b>delay-gen.enable              (DEFAULT)<br></b><b>disperse.parallel-writes         on (DEFAULT)<br></b><b>disperse.quorum-count           0 (DEFAULT)<br></b><b>features.sdfs               off<br></b><b>features.cloudsync            off<br></b><b>features.ctime              on<br></b><b>ctime.noatime               on<br></b><b>features.cloudsync-storetype       (null) (DEFAULT)<br></b><b>features.enforce-mandatory-lock      off<br></b><b>config.global-threading          off<br></b><b>config.client-threads           16<br></b><b>config.brick-threads           16<br></b><b>features.cloudsync-remote-read      off<br></b><b>features.cloudsync-store-id        (null) (DEFAULT)<br></b><b>features.cloudsync-product-id       (null) (DEFAULT)<br></b><b>features.acl               enable<br></b><b>cluster.use-anonymous-inode        yes</b></blockquote><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><font face="arial, helvetica, sans-serif">Regards,<br>Shreyansh Shah</font><br><div style="margin:0in 0in 0.0001pt"><font><font color="#0b5394" style="font-weight:bold">Alpha</font><font color="#666666" style="font-weight:bold">Grep</font><font color="#000000"><b> Securities Pvt. Ltd.</b><br></font></font></div></div></div></div></div></div>