<div dir="ltr">With attachments :)<div class="gmail_extra"><br><div class="gmail_quote">2017-05-08 14:57 GMT+02:00 Ingard Mevåg <span dir="ltr"><<a href="mailto:ingard@jotta.no" target="_blank">ingard@jotta.no</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi<div><br><div class="m_-3225906951752204003gmail_signature"><div dir="ltr"><div><div dir="ltr"></div></div></div></div>
<div>We've got 3 servers with 60 drives each setup with an EC volume running on gluster 3.10.0</div></div><div>The servers are connected via 10gigE.</div><div><br></div><div>We've done the changes recommended here : <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1349953#c17" target="_blank">https://bugzilla.redhat.com/<wbr>show_bug.cgi?id=1349953#c17</a> and we're able to max out the network with the iozone tests referenced in the same ticket.</div><div><br></div><div>However for small files we are getting 3-5 MB/s with the smallfile_cli.py tool. For instance:</div><div>python smallfile_cli.py --operation create --threads 32 --file-size 100 --files 1000 --top /tmp/dfs-archive-001/<br></div><div>.</div><div>.</div><div><div>total threads = 32</div><div>total files = 31294</div><div>total data = 2.984 GB</div><div> 97.79% of requested files processed, minimum is 90.00</div><div>785.542908 sec elapsed time</div><div>39.837416 files/sec</div><div>39.837416 IOPS</div><div>3.890373 MB/sec</div></div><div>.</div><div><br></div><div>We're going to use these servers for archive purposes, so the files will be moved there and accessed very little. After noticing our migration tool performing very badly we did some analyses on the data actually being moved :</div><div><br></div><div><div>Bucket 31808791 (16.27 GB) :: 0 bytes - 1.00 KB</div><div>Bucket 49448258 (122.89 GB) :: 1.00 KB - 5.00 KB</div><div>Bucket 13382242 (96.92 GB) :: 5.00 KB - 10.00 KB</div><div>Bucket 13557684 (195.15 GB) :: 10.00 KB - 20.00 KB</div><div>Bucket 22735245 (764.96 GB) :: 20.00 KB - 50.00 KB</div><div>Bucket 15101878 (1041.56 GB) :: 50.00 KB - 100.00 KB</div><div>Bucket 10734103 (1558.35 GB) :: 100.00 KB - 200.00 KB</div><div>Bucket 17695285 (5773.74 GB) :: 200.00 KB - 500.00 KB</div><div>Bucket 13632394 (10039.92 GB) :: 500.00 KB - 1.00 MB</div><div>Bucket 21815815 (32641.81 GB) :: 1.00 MB - 2.00 MB</div><div>Bucket 36940815 (117683.33 GB) :: 2.00 MB - 5.00 MB</div><div>Bucket 13580667 (91899.10 GB) :: 5.00 MB - 10.00 MB</div><div>Bucket 10945768 (232316.33 GB) :: 10.00 MB - 50.00 MB</div><div>Bucket 1723848 (542581.89 GB) :: 50.00 MB - 9223372036.85 GB</div></div><div><br></div><div>So it turns out we've got a very large number of very small files being written to this volume.</div><div>I've attached the volume config and 2 profiling runs so if someone wants to take a look and maybe give us some hints in terms of what volume settings will be best for writing a lot of small files that would be much appreciated.</div><div><br></div><div>kind regards</div><span class="HOEnZb"><font color="#888888"><div>ingard </div></font></span></div>
</blockquote></div><br><div><br></div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"></div></div></div></div>
</div></div>