<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi Niklas,<div class=""><br class=""></div><div class="">Out of interest have you tried testing performance with performance.stat-prefetch enabled?</div><div class=""><div class="">
<div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br class="">--<br class="">Sam McLeod <br class="">@s_mcleod<br class=""><a href="https://smcleod.net" class="">https://smcleod.net</a></div></div>
</div>
<div><br class=""><blockquote type="cite" class=""><div class="">On 14 Sep 2017, at 10:42 pm, Niklas Hambüchen <<a href="mailto:mail@nh2.me" class="">mail@nh2.me</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">Hi,<br class=""><br class="">I have a gluster 3.10 volume with a dir with ~1 million small files in<br class="">them, say mounted at /mnt/dir with FUSE, and I'm observing something weird:<br class=""><br class="">When I list and stat them all using rsync, then the lstat() calls that<br class="">rsync does are incredibly fast (23 microseconds per call on average,<br class="">definitely faster than a network roundtrip between my 3-machine bricks<br class="">connected via Ethernet).<br class="">But when I try to back them up with the `bup` tool<br class="">(<a href="https://github.com/bup/bup" class="">https://github.com/bup/bup</a>), which (at least according to strace) does<br class="">the same syscalls as rsync to stat all files, it takes 1700 microseconds<br class="">per lstat() call, and the total time to go over all files is 50x higher.<br class=""><br class="">These are my strace statistics:<br class=""><br class="">rsync strace:<br class=""><br class=""> $ strace -f -w -c rsync -a --dry-run /mnt/dir /tmp/nonexisting<br class=""><br class=""> % time seconds usecs/call calls errors syscall<br class=""> ------ ----------- ----------- --------- --------- ----------------<br class=""> 52.35 69.682773 26617 2618 1 select<br class=""> 39.60 52.712292 1056 49907 getdents<br class=""> 8.00 10.655949 11 998102 lstat<br class=""> 0.02 0.022890 12 1900 read<br class=""> 0.01 0.017219 21 829 write<br class=""> 0.01 0.012093 14 868 munmap<br class=""> 0.01 0.006656 11 606 mmap<br class=""> 0.00 0.004168 1389 3 readlink<br class=""> 0.00 0.001796 1796 1 chdir<br class=""> 0.00 0.001019 510 2 clone<br class=""> 0.00 0.000841 44 19 13 open<br class=""><br class="">Took ~50 seconds real time to complete.<br class=""><br class="">bup strace (I interrupted it after a while):<br class=""><br class=""> strace: Process 10749 attached<br class=""> strace: Process 10750 attached<br class=""> Indexing: 25600 (566 paths/s)<br class=""> ^C<br class=""> % time seconds usecs/call calls errors syscall<br class=""> ------ ----------- ----------- --------- --------- ----------------<br class=""> 89.55 1140.837325 1700 671016 lstat<br class=""> 3.92 49.875376 934 53387 getdents<br class=""> 3.58 45.655985 52238 874 read<br class=""> 2.14 27.293944 789 34588 5799 open<br class=""> 0.57 7.266342 141 51384 llistxattr<br class=""> 0.09 1.090689 42 25692 25692 getxattr<br class=""> 0.06 0.780977 26 29825 1019 close<br class=""> 0.05 0.601806 23 25739 25722 ioctl<br class=""> 0.03 0.373894 1851 202 select<br class=""> 0.00 0.055953 14 3879 brk<br class=""><br class=""> real 20m52.150s<br class=""> user 0m2.608s<br class=""> sys 0m11.456s<br class=""><br class="">Note I passed `-c -w` to strace to measure wall time of the syscalls<br class="">spend, not system CPU time.<br class=""><br class="">Using<br class=""><br class=""> time strace -f -c -w ls -lU /mnt/dir > /dev/null<br class=""><br class="">shows that ls is as fast as rsync (also in 50 seconds).<br class=""><br class="">(Aside: I've filed the large number of getdents() as<br class=""><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1478411" class="">https://bugzilla.redhat.com/show_bug.cgi?id=1478411</a> but that's not the<br class="">core question of this email.)<br class=""><br class="">My volume info:<br class=""><br class=""> # gluster vol info<br class=""><br class=""> Volume Name: myvol<br class=""> Type: Replicate<br class=""> Volume ID: ...<br class=""> Status: Started<br class=""> Snapshot Count: 0<br class=""> Number of Bricks: 1 x 3 = 3<br class=""> Transport-type: tcp<br class=""> Bricks:<br class=""> Brick1: 10.0.0.1:/data/glusterfs/brick<br class=""> Brick2: 10.0.0.2:/data/glusterfs/brick<br class=""> Brick3: 10.0.0.3:/data/glusterfs/brick<br class=""> Options Reconfigured:<br class=""> changelog.capture-del-path: on<br class=""> changelog.changelog: on<br class=""> storage.build-pgfid: on<br class=""> nfs.disable: on<br class=""> transport.address-family: inet<br class=""> client.ssl: on<br class=""> server.ssl: on<br class=""> storage.linux-aio: on<br class=""> <a href="http://performance.io" class="">performance.io</a>-thread-count: 64<br class=""> performance.readdir-ahead: on<br class=""> server.event-threads: 32<br class=""> client.event-threads: 32<br class=""> server.outstanding-rpc-limit: 64<br class=""> cluster.lookup-unhashed: auto<br class=""> performance.flush-behind: on<br class=""> performance.strict-write-ordering: off<br class=""> performance.high-prio-threads: 64<br class=""> performance.normal-prio-threads: 64<br class=""> performance.low-prio-threads: 64<br class=""> performance.write-behind-window-size: 10MB<br class=""> cluster.ensure-durability: on<br class=""> performance.lazy-open: yes<br class=""> cluster.use-compound-fops: off<br class=""> performance.open-behind: on<br class=""> features.cache-invalidation: off<br class=""> performance.quick-read: off<br class=""> performance.read-ahead: off<br class=""> performance.stat-prefetch: off<br class=""> changelog.rollover-time: 1<br class=""> cluster.self-heal-daemon: enable<br class=""><br class="">Questions:<br class=""><br class="">What could explain why bup's lstat()s are slow and rsync's lstat()s are<br class="">fast?<br class=""><br class="">Also, how comes rsync's lstat()s can be faster than a network roundtrip<br class="">at all, given that I have all caching disabled (e.g. stat-prefetch: off,<br class="">cache-invalidation: off)?<br class=""><br class="">Is there some caching going on? Might bup issue its syscalls in an order<br class="">that might destroy this caching, while rsync and ls use a favourable order?<br class=""><br class="">Thanks!<br class="">_______________________________________________<br class="">Gluster-users mailing list<br class=""><a href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class="">http://lists.gluster.org/mailman/listinfo/gluster-users<br class=""></div></div></blockquote></div><br class=""></div></body></html>