[Gluster-users] Understanding gluster performance

[Gionatan Danti]

Il 21-01-2020 11:40 Yaniv Kaul ha scritto:
> How did you fix this?
> How did you spot this?

I used iperf3 between the two hosts. It shows that, albeit bandwidth was 
near the 1 Gbps limit, there were frequent retransmissions. "netstat -s 
| grep retran" confirmed that retransmissions happened during my gluster 
tests.

> fsync requires each write to land on disk and get an ack on it - it's
> probably the slowest kind of write you can imagine, and you seem be
> doing it for every small (4K?) block.
> This is not very realistic. But in your case you say you are writing
> to /dev/shm, so that is strange. Can you try with a different fio
> engine and see what you get?

True, but putting my briks on /dev/shm means they are actually are 
in-memory, with no disks/seeks slowing down the syncs. I also tried with 
a very simple "dd if=/dev/urandom of=test.img bs=4k count=1024" and the 
results were idential (about 250 IOPs).

To exclude network latency, I did a local, two-brick /dev/shm backed 
volume (both bricks were on the same machine). Locally mounting the 
gluser volume via fuse, I only get about 500-600 IOPs.

> What are you trying to test here?

Database and virtual machine performance (which are both fsync and 4k 
heavy).

> Good question. Perhaps profiling would help here. Perhaps too many
> threads are contending for CPU? Some lock contention?

What kind or profiling should I do? strace on glusterd process? perf 
top/stat?

Thanks.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti at assyoma.it - info at assyoma.it
GPG public key ID: FF5F32A8