[Gluster-devel] Slow write speed with replica vol when link has latency

[Jan Engelhardt]

On Saturday 2012-09-29 22:30, Jan Engelhardt wrote:

>Hi,
>
>in a 2-brick "replica 2" volume (naturally spanning 2 hosts via TCP),
>write speed floats at a measly 1 Mbyte/s, despite the hosts having a
>proven faster connection (simply socat /dev/zero to a remote dest).
>This occurs with gluster-3.3.1qa3, and also has with versions before
>it.

In trying to reproduce this issue at this week's Glusterfs Workshop,
I discovered that replication seems to be (also) highly dependent on the 
link latency/RTT. I am not sure what went on on Sep 29, as the ping RTT 
was
>rtt min/avg/max/mdev = 0.412/0.578/1.012/0.052 ms, ipg/ewma 0.607/0.573 ms


Anyway, the following findings are quite interesting nevertheless. 
Basically, if the latency is only 100 ms, the transfer rates drop by 
roughly 90%.

Below is a table of induced delay vs. transfer speed in bytes/sec as 
reported by rsync (and rsync is also the culprit of the little upward 
spikes in the curve, as it seems to use full seconds only in its 
bytes/time calculation) - the downard trend is visible.

for ((ms = 0; ms < 100; ++ms)); do
	tc qdisc add dev eth0 root netem delay "${ms}ms";
	rsync -HPav /tmp/foo.iso /glustermnt/
	rm -f /glustermnt/foo.iso
	tc qdisc del dev eth0 root
done;

Not quite sure why the transfer completes faster on host M (brick 0) than host
B (brick 1), but that's not important to me right now.

To visualize the values below, save to text file and use gnuplot.

$ gnuplot
gnuplot> plot "values.txt" using 1:2 with lines, "values.txt" using 1:3 
with lines


---8<---
#ms on_hostB on_hostM
0 8660273.03 9943276.44
1 7670527.54 9943276.44
2 7670527.54 9943276.44
3 6883806.77 9257533.24
4 6883806.77 8660273.03
5 5712094.98 8660273.03
6 5965965.87 8135408.00
7 5712094.98 8660273.03
8 5065442.72 8660273.03
9 4550312.95 8135408.00
10 4006992.00 7670527.54
11 4261404.19 7670527.54
12 3781245.97 7255904.43
13 3486603.43 7255904.43
14 3398334.99 7255904.43
15 3398334.99 6883806.77
16 3314425.48 5478948.24
17 3016499.60 6548011.32
18 3085844.41 6548011.32
19 2950202.90 6243452.65
20 2606489.94 6243452.65
21 2825983.83 5965965.87
22 2509051.07 5965965.87
23 2375827.12 5712094.98
24 2334508.38 5478948.24
25 2182670.44 5065442.72
26 2182670.44 4550312.95
27 2049377.59 4550312.95
28 2018559.88 4550312.95
29 2418634.81 4006992.00
30 1851506.65 3234559.81
31 1988655.29 4401122.36
32 1851506.65 3781245.97
33 2182670.44 2767716.12
34 1688480.91 3890847.30
35 1754695.84 3890847.30
36 1647045.79 2950202.90
37 1551840.83 3158452.52
38 1451180.89 3314425.48
39 1607595.59 3486603.43
40 1688480.91 3441903.56
41 1391028.31 3121729.12
42 1588570.79 3274008.59
43 1405594.05 2856050.04
44 1591710.34 3121729.12
45 1329052.99 3274008.59
46 1235285.57 2856050.04
47 1220314.47 2739476.57
48 1137581.29 2856050.04
49 1212963.42 2440624.58
50 958821.94 2065143.88
51 1082540.90 2632046.12
52 958821.94 2739476.57
53 1016932.36 2632046.12
54 1016932.36 2739476.57
55 958821.94 2275158.51
56 958821.94 2200563.15
57 958821.94 2275158.51
58 958821.94 2532723.62
59 745750.40 2440624.58
60 906993.73 2275158.51
61 860481.23 2130704.00
62 745750.40 2200563.15
63 818506.54 2200563.15
64 818506.54 2275158.51
65 714016.34 2003497.79
66 745750.40 1945425.39
67 780436.47 1743303.27
68 714016.34 2130704.00
69 745750.40 1954869.40
70 745750.40 1266364.08
71 780436.47 1316025.41
72 745750.40 1813980.97
73 745750.40 1720956.31
74 745750.40 1428027.57
75 780436.47 1137581.29
76 1157198.90 1316025.41
77 1157198.90 1369740.73
78 1157198.90 1548860.98
79 958821.94 1242917.33
80 1082540.90 1342350.72
81 1082540.90 1242917.33
82 1082540.90 1157198.90
83 1016932.36 1242917.33
84 1016932.36 1242917.33
85 1016932.36 1242917.33
86 714016.34 588750.32
87 633184.30 860481.23
88 568792.68 958821.94
89 610159.42 1157198.90
90 610159.42 1157198.90
91 610159.42 860481.23
92 610159.42 1016932.36
93 958821.94 906993.73
94 818506.54 958821.94
95 568792.68 958821.94
96 568792.68 1157198.90
97 568792.68 1157198.90
98 588750.32 860481.23
99 610159.42 780436.47
100 550143.74 958821.94