<!DOCTYPE html>
<html>
<head>
<title></title>
</head>
<body><div style="font-family:helvetica, arial, sans-serif;">Hello,<br></div>
<div style="font-family:helvetica, arial, sans-serif;"><br></div>
<div style="font-family:helvetica, arial, sans-serif;">I'm currently trialling a deployment of GlusterFS (currently 3.10) for our Kubernetes storage and am hitting performance problems with the gluster command line interface used to provision and manage volumes.<br></div>
<div style="font-family:helvetica, arial, sans-serif;"><br></div>
<div style="font-family:helvetica, arial, sans-serif;">When creating, modifying or setting options on volumes - gluster takes almost exactly 1 second to issue any command.<br></div>
<div style="font-family:helvetica, arial, sans-serif;"><br></div>
<div style="font-family:helvetica, arial, sans-serif;">This means - if for example you have 60 volumes and 10 settings on each volume it's going to take over 16 minutes to create the volumes and set the options on each volume.<br></div>
<div style="font-family:helvetica, arial, sans-serif;"><br></div>
<div style="font-family:helvetica, arial, sans-serif;">Starting a volume with '<span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">gluster volume start</span>' takes 5 seconds.<br></div>
<div style="font-family:helvetica, arial, sans-serif;"><br></div>
<div style="font-family:helvetica, arial, sans-serif;">I can't see any technical reason why setting options on volumes should take any longer than writing the option to disk on the local node, issuing the change to other nodes in the cluster and verifying the success of the change, I would expect that this would only take a few milliseconds as long as disk and network performance was sufficient.<br></div>
<div style="font-family:helvetica, arial, sans-serif;"><br></div>
<div style="font-family:helvetica, arial, sans-serif;">- I'm wondering if there's some sort of sleep like behaviour within the gluster command?<br></div>
<div style="font-family:helvetica, arial, sans-serif;"><br></div>
<div style="font-family:helvetica, arial, sans-serif;">Any assistance would be appreciated.<br></div>
<div style="font-family:helvetica, arial, sans-serif;"><br></div>
<div style="font-family:helvetica, arial, sans-serif;">---<br></div>
<div style="font-family:helvetica, arial, sans-serif;"><br></div>
<div style="font-family:helvetica, arial, sans-serif;">Cluster information:<br></div>
<div style="font-family:helvetica, arial, sans-serif;"><br></div>
<div style="font-family:helvetica, arial, sans-serif;">- OS: CentOS 7<br></div>
<div style="font-family:helvetica, arial, sans-serif;">- Gluster: 3.10 (newest available to CentOS 7)<br></div>
<div style="font-family:helvetica, arial, sans-serif;">- Storage: SSD (40K random read/write IOPs, 300MB/s per node)<br></div>
<div style="font-family:helvetica, arial, sans-serif;">- Memory: 16GB per node<br></div>
<div style="font-family:helvetica, arial, sans-serif;">- Network: 10Gbit (switched, same VLAN)<br></div>
<div style="font-family:helvetica, arial, sans-serif;">- Data size: none, we haven't loaded data yet<br></div>
<div style="font-family:helvetica, arial, sans-serif;">- Cluster configured with replica 3, arbiter 1<br></div>
<div style="font-family:helvetica, arial, sans-serif;"><br></div>
<div style="font-family:helvetica, arial, sans-serif;"><br></div>
<div style="font-family:helvetica, arial, sans-serif;">Example volume configuration:<span class="font" style="font-family: menlo, consolas, monospace, sans-serif;"><br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;"><br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">Status of volume: dashingdev_storage<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">Gluster process TCP Port RDMA Port Online Pid<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">------------------------------------------------------------------------------<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">Brick int-gluster-01:/mn<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">t/gluster-storage/dashingdev_storage 49192 0 Y 16798<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">Brick int-gluster-02:/mn<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">t/gluster-storage/dashingdev_storage 49192 0 Y 19886<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">Brick int-gluster-03:/mn<br>t/gluster-storage/dashingdev_storage 49192 0 Y 20450<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">Self-heal Daemon on localhost N/A N/A N N/A<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">Self-heal Daemon on int-gluster-03</span><br></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;"> N/A N/A Y 32695<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">Self-heal Daemon on int-gluster-02<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;"> N/A N/A N N/A<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;"><br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">Task Status of Volume dashingdev_storage<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">------------------------------------------------------------------------------<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">There are no active volume tasks<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;"><br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;"><br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">Volume Name: dashingdev_storage<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">Type: Replicate<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">Volume ID: 22937936-2e28-47a3-b65d-cfc9b2c0d069<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">Status: Started<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">Snapshot Count: 0<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">Number of Bricks: 1 x 3 = 3<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">Transport-type: tcp<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">Bricks:<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">Brick1: int-gluster-01:/mnt/gluster-storage/dashingdev_storage<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">Brick2: int-gluster-02:/mnt/gluster-storage/dashingdev_storage<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">Brick3: int-gluster-03:/mnt/gluster-storage/dashingdev_storage<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">Options Reconfigured:<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">server.event-threads: 3<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">performance.write-behind-window-size: 1MB<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">performance.stat-prefetch: true<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">performance.readdir-ahead: true<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">performance.rda-cache-limit: 32MB<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">performance.parallel-readdir: true<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">performance.md-cache-timeout: 600<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">performance.io-thread-count: 8<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">performance.io-cache: true<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">performance.client-io-threads: true<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">performance.cache-size: 64MB<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">performance.cache-refresh-timeout: 4<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">performance.cache-invalidation: true<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">performance.cache-ima-xattrs: true<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">network.ping-timeout: 5<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">network.inode-lru-limit: 50000<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">features.cache-invalidation: true<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">features.cache-invalidation-timeout: 600<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">diagnostics.client-log-level: WARNING<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">diagnostics.brick-log-level: WARNING<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">cluster.readdir-optimize: true<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">cluster.lookup-optimize: true<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">client.event-threads: 3<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">transport.address-family: inet<br></span></div>
<div style="font-family:helvetica, arial, sans-serif;"><span class="font" style="font-family: menlo, consolas, monospace, sans-serif;">nfs.disable: true</span><br></div>
<div style="font-family:helvetica, arial, sans-serif;"><br></div>
<div id="sig67788344"><div class="signature">--<br></div>
<div class="signature">Sam McLeod<br></div>
<div class="signature"><a href="https://twitter.com/s_mcleod">@s_mcleod</a> | <a href="http://smcleod.net">smcleod.net</a><br></div>
</div>
<div style="font-family:helvetica, arial, sans-serif;"><br></div>
</body>
</html>