<div dir="ltr">Hi,<div><br></div><div>I have been working on setting up a 4 replica gluster with over a million files (~250GB total), and I've seen some really weird stuff happen, even after trying to optimize for small files. I've set up a 4-brick replicate volume (gluster 3.13.2).</div><div><br></div><div>It took almost 2 days to rsync the data from the local drive to the gluster volume, and now I'm running a 2nd rsync that just looks for changes in case more files have been written. I'd like to concentrate this email on a very specific and odd issue.</div><div><br></div><div><br></div><div>The dir structure is</div><div>YYYY/</div><div> MM/</div><div> 10k+files in each month folder</div><div><br></div><div><br></div><div>rsyncing each month folder cold can take 2+ minutes.</div><div><br></div><div>However, if I ls the destination folder first, or use find (both of which finish within 5 seconds), the rsync is almost instant.</div><div><br></div><div><br></div><div>Here's a log with time calls that shows you what happens.:</div><div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div>box:/mnt/gluster/uploads/2017 # time rsync -aPr /srv/www/htdocs/uploads/2017/08/ 08/ </div></div><div><div>sending incremental file list</div></div><div><div>^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(637) [sender=3.1.0]</div></div><div><div><br></div></div><div><div>real 1m39.848s</div></div><div><div>user 0m0.010s</div></div><div><div>sys 0m0.030s</div></div><div><div>box:/mnt/gluster/uploads/2017 # time find 08 | wc -l </div></div><div><div>14254</div></div><div><div><br></div></div><div><div>real 0m0.726s</div></div><div><div>user 0m0.013s</div></div><div><div>sys 0m0.033s</div></div><div><div>box:/mnt/gluster/uploads/2017 # time rsync -aPr /srv/www/htdocs/uploads/2017/08/ 08/</div></div><div><div>sending incremental file list</div></div><div><div><br></div></div><div><div>real 0m0.562s</div></div><div><div>user 0m0.057s</div></div><div><div>sys 0m0.137s</div></div><div><div>box:/mnt/gluster/uploads/2017 # time find 07 | wc -l </div></div><div><div>10103</div></div><div><div><br></div></div><div><div>real 0m4.550s</div></div><div><div>user 0m0.010s</div></div><div><div>sys 0m0.033s</div></div><div><div>box:/mnt/gluster/uploads/2017 # time rsync -aPr /srv/www/htdocs/uploads/2017/07/ 07/ </div></div><div><div>sending incremental file list</div></div><div><div><br></div></div><div><div>real 0m0.428s</div></div><div><div>user 0m0.030s</div></div><div><div>sys 0m0.083s</div></div><div><div>box:/mnt/gluster/uploads/2017 # time ls 06 | wc -l </div></div><div><div>11890</div></div><div><div><br></div></div><div><div>real 0m1.850s</div></div><div><div>user 0m0.077s</div></div><div><div>sys 0m0.040s</div></div><div><div>box:/mnt/gluster/uploads/2017 # time rsync -aPr /srv/www/htdocs/uploads/2017/06/ 06/ </div></div><div><div>sending incremental file list</div></div><div><div><br></div></div><div><div>real 0m0.627s</div></div><div><div>user 0m0.073s</div></div><div><div>sys 0m0.107s</div></div><div><div>box:/mnt/gluster/uploads/2017 # time rsync -aPr /srv/www/htdocs/uploads/2017/05/ 05/ </div></div><div><div>sending incremental file list</div></div><div><div><br></div></div><div><div>real 2m24.382s</div></div><div><div>user 0m0.127s</div></div><div><div>sys 0m0.357s</div></div></blockquote><div><div><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><br></div><div>Note how if I precede the rsync call with ls or find, the rsync completes in less than a second (finding no files to sync because they've already been synced). Otherwise, it takes over 2 minutes (I interrupted the first call before the 2 minutes because it was already taking too long).</div><div><br></div><div>What could be causing rsync to work so slowly unless the dir is primed?</div><div><br></div><div>Volume config:</div></div></div></div></div></div></div></div></div></div></div></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><div class="gmail_signature"><div><div><div><div><div><div><div><div><div><div>Volume Name: gluster</div></div></div></div></div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><div><div><div>Type: Replicate</div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><div><div><div>Volume ID: XXXXXXXXXXXXXXXXXXXXXXXXX</div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><div><div><div>Status: Started</div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><div><div><div>Snapshot Count: 0</div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><div><div><div>Number of Bricks: 1 x 4 = 4</div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><div><div><div>Transport-type: tcp</div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><div><div><div>Bricks:</div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><div><div><div>Brick1: server1 :/mnt/server1_block4/<span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">gluster</span>
</div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><div><div><div>Brick2:
<span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">server2</span> :/mnt/<span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">server2</span>_block4/g<span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">luster</span>
</div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><div><div><div>Brick3:
<span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">server3</span> :/mnt/<span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">server3</span>_block4/<span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">gluster</span>
</div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><div><div><div>Brick4:
<span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">server4 </span>:/mnt/server4_block4/gluster</div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><div><div><div>Options Reconfigured:</div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><div><div><div>performance.parallel-readdir: off</div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><div><div><div>transport.address-family: inet</div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><div><div><div>nfs.disable: on</div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><div><div><div>cluster.self-heal-daemon: enable</div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><div><div><div>performance.cache-size: 1GB</div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><div><div><div>network.ping-timeout: 5</div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><div><div><div>cluster.quorum-type: fixed</div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><div><div><div>cluster.quorum-count: 1</div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><div><div><div>features.cache-invalidation: on</div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><div><div><div>features.cache-invalidation-timeout: 600</div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><div><div><div>performance.cache-invalidation: on</div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><div><div><div>performance.md-cache-timeout: 600</div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><div><div><div>network.inode-lru-limit: 500000</div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><div><div><div>performance.rda-cache-limit: 256MB</div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><div><div><div>performance.read-ahead: off</div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><div><div><div>client.event-threads: 4</div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature"><div><div><div><div><div><div>server.event-threads: 4</div></div></div></div></div></div></div></div></div></blockquote><div><div><div class="gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><br></div><div><br></div><div>Thank you for any insight.</div><div dir="ltr"><br>Sincerely,<br>Artem<br></div></div></div></div></div></div></div>
</div></div>