[Bugs] [Bug 1524048] New: gluster volume set is very slow

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1524048

            Bug ID: 1524048
           Summary: gluster volume set is very slow
           Product: GlusterFS
           Version: mainline
         Component: core
          Assignee: bugs at gluster.org
          Reporter: nh2-redhatbugzilla at deditus.de
                CC: bugs at gluster.org



Description of problem:

Running a `gluster volume set` command takes almost 2.5 seconds for my
3-machine cluster, no matter if these are physical machines in a LAN or VMs or
containers on the same host (so there is negligible network latency).

I have a script that needs to set 22 gluster options. As a result, this takes 1
minute. It slows down my automatic deployment, so I'd like it to to be faster.

With some investigation, I found that it's mainly fsync()s that make it this
slow.

Example:

    [root at node-3:~]# time gluster vol set myvol rollover-time 1
    volume set: success

    real    0m2.120s

Now we set nobarrier on the filesystem, which makes fsyncs no-ops:

    [all-hosts] # mount -o remount,nobarrier /

    [root at node-3:~]# time gluster vol set myvol rollover-time 1
    volume set: success

    real    0m0.242s

That's 10x faster.

`strace` confirms that fsync is at play here: In normal gluster idle operation,
glusterfsd shows little to no fsync calls. When I execute the `volume set`, I
get 3 seconds in `strace -fp "$(pidof glusterfsd)" -wc`:

    % time     seconds  usecs/call     calls    errors syscall
    ------ ----------- ----------- --------- --------- ----------------
      1.46    3.028789       84133        36           fsync

How reproducible: Always

Steps to Reproduce:
1. setup any gluster cluster
2. run `time gluster volume set` with some option
3. compare run time to when the file system that carries gluster stuf is
mounted nobarrier

Additional info: Using gluster 3.12.3 on Linux

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.