[Bugs] [Bug 1601166] performance.read-ahead causes huge increase in unnecessary network traffic

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1601166



--- Comment #1 from Raghavendra G <rgowdapp at redhat.com> ---
Turning on performance.read-ahead results in a huge increase in network traffic
during reads, up to 2.5X the amount of traffic expected.

Red Hat Enterprise Linux Server release 6.6 (Santiago)
Red Hat Storage Server 3.0 Update 2

8-node cluster, 10G network, 6 JBOD bricks per node, replication=2

This is the test protocol:

1. Run: gluster volume set HadoopVol performance.read-ahead on

2. Write a 1 GB file to gluster. We monitor performance stats during tests with
collectl and colmux (collectl.sourceforge.net). When writing a 1 GB file, we
see 2GB leave the client node over the network. Each server node
(replication=2) receives 1GB over the net and writes 1GB to disk, as expected.

3. Drop the linux page cache on all nodes.

4. Read the file sequentially on the node that wrote it, with an I/O size of
256KB. We see one of the two servers read 1GB from disk and send 2.5 GB to the
client node. That's 2.5 times the amount of network traffic expected.


The factor of 2.5X does not depend on the file size, from 10 MB up to 10 GB.

The factor does depend on the I/O size. For a read size of 256KB or less, the
factor is about 2.5X. For a 1 MB read size, the factor is 1.6X. For a read size
of 16 MB, the extra traffic is negligible. It looks like each read causes an
unnecessary 500-600 KB of traffic.

When we turn off performance.read-ahead, this problem goes away.

Just in case there was a problem with the the counters used by collectl, we
captured tcpdump traces during the tests, and added up the packet sizes. These
results agree with the collectl data.


Hank

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.