<div dir="ltr">I have a pair of nodes configured with GlusterFS. I am trying to otimize the read performance of the setup. with smallfiles_cli.py test I'm getting 4MiB/s on the CREATE and 14MiB/s on the READ<div><br></div><div>I'm using the FUSE mount, mounting to the local GlustertFS server:</div><div>localhost:/gv_wordpress_std_var<span style="white-space:pre"> </span>/var/www/wordpress<span style="white-space:pre"> </span>glusterfs<span style="white-space:pre"> </span>defaults,_netdev,noauto,x-systemd.automount,noatime,log-level=ERROR<span style="white-space:pre"> </span>0<span style="white-space:pre"> </span>0<br></div><div><br></div><div>
<div style="font-size:small;text-decoration-style:initial;text-decoration-color:initial">I'm aware from reading that small files aren't GlusterFS ideal workload, but these smallfiles results seem very low. Both machines are VMs on a Rackspace dedicated host.</div><div style="font-size:small;text-decoration-style:initial;text-decoration-color:initial"><br></div>---- </div><div><br></div><div><div>Volume Name: gv_wordpress_std_var</div><div>Type: Replicate</div><div>Volume ID: c02ce2a4-4953-4228-9abe-8e3a55f8cdfd</div><div>Status: Started</div><div>Snapshot Count: 0</div><div>Number of Bricks: 1 x 2 = 2</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: rs-wordpress-std-a:/data/glusterfs/bricks/wordpress_std_var-a</div><div>Brick2: rs-wordpress-std-b:/data/glusterfs/bricks/wordpress_std_var-b</div><div>Options Reconfigured:</div><div>client.event-threads: 4</div><div>cluster.lookup-optimize: on</div><div>diagnostics.count-fop-hits: on</div><div>diagnostics.latency-measurement: on</div><div>performance.cache-size: 1GB</div><div>network.inode-lru-limit: 50000</div><div>performance.md-cache-timeout: 600</div><div>performance.cache-invalidation: on</div><div>performance.stat-prefetch: on</div><div>features.cache-invalidation-timeout: 600</div><div>features.cache-invalidation: on</div><div>auth.allow: 172.24.16.*</div><div>transport.address-family: inet</div><div>nfs.disable: on</div><div>performance.client-io-threads: on</div><div>performance.cache-max-file-size: 128MB</div></div><div><br></div><div>----</div><div><br></div><div><div> version : 3.1</div><div> hosts in test : None</div><div> top test directory(s) : ['/var/www/wordpress/smf']</div><div> operation : create</div><div> files/thread : 10000</div><div> threads : 8</div><div> record size (KB, 0 = maximum) : 0</div><div> file size (KB) : 4</div><div> file size distribution : fixed</div><div> files per dir : 100</div><div> dirs per dir : 10</div><div> threads share directories? : N</div><div> filename prefix : </div><div> filename suffix : </div><div> hash file number into dir.? : N</div><div> fsync after modify? : N</div><div> pause between files (microsec) : 0</div><div> minimum directories per sec : 50</div><div> finish all requests? : Y</div><div> stonewall? : Y</div><div> measure response times? : Y</div><div> verify read? : Y</div><div> verbose? : False</div><div> log to stderr? : False</div><div> ext.attr.size : 0</div><div> ext.attr.count : 0</div><div>host = wordpress-std-b,thr = 00,elapsed = 77.843555,files = 10000,records = 10000,status = ok</div><div>host = wordpress-std-b,thr = 01,elapsed = 77.789082,files = 10000,records = 10000,status = ok</div><div>host = wordpress-std-b,thr = 02,elapsed = 77.887903,files = 10000,records = 10000,status = ok</div><div>host = wordpress-std-b,thr = 03,elapsed = 77.739701,files = 10000,records = 10000,status = ok</div><div>host = wordpress-std-b,thr = 04,elapsed = 77.943002,files = 10000,records = 10000,status = ok</div><div>host = wordpress-std-b,thr = 05,elapsed = 77.632973,files = 10000,records = 10000,status = ok</div><div>host = wordpress-std-b,thr = 06,elapsed = 77.547384,files = 10000,records = 10000,status = ok</div><div>host = wordpress-std-b,thr = 07,elapsed = 77.917453,files = 10000,records = 10000,status = ok</div><div>total threads = 8</div><div>total files = 80000</div><div>total IOPS = 80000</div><div>total data = 0.305 GiB</div><div>100.00% of requested files processed, minimum is 90.00</div><div>elapsed time = 77.943</div><div>files/sec = 1026.391055</div><div>IOPS = 1026.391055</div><div>MiB/sec = 4.009340</div></div><div><br></div><div>----</div><div><br></div><div><div> version : 3.1</div><div> hosts in test : None</div><div> top test directory(s) : ['/var/www/wordpress/smf']</div><div> operation : read</div><div> files/thread : 10000</div><div> threads : 8</div><div> record size (KB, 0 = maximum) : 0</div><div> file size (KB) : 4</div><div> file size distribution : fixed</div><div> files per dir : 100</div><div> dirs per dir : 10</div><div> threads share directories? : N</div><div> filename prefix : </div><div> filename suffix : </div><div> hash file number into dir.? : N</div><div> fsync after modify? : N</div><div> pause between files (microsec) : 0</div><div> minimum directories per sec : 50</div><div> finish all requests? : Y</div><div> stonewall? : Y</div><div> measure response times? : Y</div><div> verify read? : Y</div><div> verbose? : False</div><div> log to stderr? : False</div><div> ext.attr.size : 0</div><div> ext.attr.count : 0</div><div>host = wordpress-std-b,thr = 00,elapsed = 20.905861,files = 9900,records = 9900,status = ok</div><div>host = wordpress-std-b,thr = 01,elapsed = 20.851949,files = 10000,records = 10000,status = ok</div><div>host = wordpress-std-b,thr = 02,elapsed = 20.791474,files = 10000,records = 10000,status = ok</div><div>host = wordpress-std-b,thr = 03,elapsed = 20.895812,files = 9900,records = 9900,status = ok</div><div>host = wordpress-std-b,thr = 04,elapsed = 20.869731,files = 9900,records = 9900,status = ok</div><div>host = wordpress-std-b,thr = 05,elapsed = 20.795042,files = 9900,records = 9900,status = ok</div><div>host = wordpress-std-b,thr = 06,elapsed = 20.809913,files = 10000,records = 10000,status = ok</div><div>host = wordpress-std-b,thr = 07,elapsed = 20.981664,files = 10000,records = 10000,status = ok</div><div>total threads = 8</div><div>total files = 79600</div><div>total IOPS = 79600</div><div>total data = 0.304 GiB</div><div> 99.50% of requested files processed, minimum is 90.00</div><div>elapsed time = 20.982</div><div>files/sec = 3793.788677</div><div>IOPS = 3793.788677</div><div>MiB/sec = 14.819487</div></div><div><br></div></div>