[Gluster-users] Is the read speed affected by brick order?

[Jin Li]

Hi,

I am currently trying to test the read performance of a glusterfs file
system and understand its speed characteristics.

This glusterfs file system is made of 4 machines with identical
hardware and OS/software configuration. This filesystem is mounted
onto 4 clients again with identical hardware and OS/software
configuration.

The gluster volume has the following configuration.
~~~
$ sudo gluster volume info gvd2_cache_128MB

Volume Name: gvd2_cache_128MB
Type: Distribute
Volume ID: ee88b666-3130-44c5-b055-d19c42cf7b5b
Status: Started
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: eridani:/mnt/raid6/glusterfs_distributed_export_cache_128MB
Brick2: sargas:/mnt/raid6/glusterfs_distributed_export_cache_128MB
Brick3: toliman:/mnt/raid6/glusterfs_distributed_export_cache_128MB
Brick4: alphard:/mnt/raid6/glusterfs_distributed_export_cache_128MB
Options Reconfigured:
performance.cache-size: 128MB
~~~

I have selected 3 files from each of four machines, and all 12 files
have the same file size, as.
~~~
eridani /mnt/raid6/glusterfs_distributed_export_cache_128MB/benchmark/sync/ddfile_6
eridani /mnt/raid6/glusterfs_distributed_export_cache_128MB/benchmark/sync/ddfile_29
eridani /mnt/raid6/glusterfs_distributed_export_cache_128MB/benchmark/sync/ddfile_7
sargas  /mnt/raid6/glusterfs_distributed_export_cache_128MB/benchmark/sync/ddfile_31
sargas  /mnt/raid6/glusterfs_distributed_export_cache_128MB/benchmark/sync/ddfile_27
sargas  /mnt/raid6/glusterfs_distributed_export_cache_128MB/benchmark/sync/ddfile_5
toliman /mnt/raid6/glusterfs_distributed_export_cache_128MB/benchmark/sync/ddfile_12
toliman /mnt/raid6/glusterfs_distributed_export_cache_128MB/benchmark/sync/ddfile_16
toliman /mnt/raid6/glusterfs_distributed_export_cache_128MB/benchmark/sync/ddfile_32
alphard /mnt/raid6/glusterfs_distributed_export_cache_128MB/benchmark/sync/ddfile_25
alphard /mnt/raid6/glusterfs_distributed_export_cache_128MB/benchmark/sync/ddfile_23
alphard /mnt/raid6/glusterfs_distributed_export_cache_128MB/benchmark/sync/ddfile_22
~~~

Then I run `dd` in client machines, and I got the following read speed.
~~~
ddfile_5   55.3  MB/s  ddfile_16  63.2  MB/s  ddfile_12  76.0  MB/s
ddfile_16  61.8  MB/s
ddfile_27  55.4  MB/s  ddfile_23  63.2  MB/s  ddfile_32  76.0  MB/s
ddfile_12  61.9  MB/s
ddfile_31  55.4  MB/s  ddfile_25  63.2  MB/s  ddfile_16  76.1  MB/s
ddfile_27  61.9  MB/s
ddfile_12  56.2  MB/s  ddfile_12  63.3  MB/s  ddfile_22  76.1  MB/s
ddfile_31  61.9  MB/s
ddfile_16  56.2  MB/s  ddfile_22  63.4  MB/s  ddfile_25  76.1  MB/s
ddfile_5   61.9  MB/s
ddfile_22  56.2  MB/s  ddfile_27  63.4  MB/s  ddfile_27  76.1  MB/s
ddfile_22  62.0  MB/s
ddfile_25  56.2  MB/s  ddfile_32  63.4  MB/s  ddfile_5   76.1  MB/s
ddfile_23  62.0  MB/s
ddfile_32  56.2  MB/s  ddfile_31  63.5  MB/s  ddfile_23  76.2  MB/s
ddfile_25  62.0  MB/s
ddfile_23  56.3  MB/s  ddfile_5   63.6  MB/s  ddfile_31  76.2  MB/s
ddfile_32  62.1  MB/s
ddfile_7   57.7  MB/s  ddfile_29  63.8  MB/s  ddfile_29  77.3  MB/s
ddfile_7   63.9  MB/s
ddfile_29  58.1  MB/s  ddfile_7   64.0  MB/s  ddfile_7   77.4  MB/s
ddfile_29  64.0  MB/s
ddfile_6   59.3  MB/s  ddfile_6   65.0  MB/s  ddfile_6   79.1  MB/s
ddfile_6   67.4  MB/s
~~~

I found that files from machine `eridani` have the fastest read speed
in all 4 client machines, and roughly the read speed is ordered with
the ordered bricks in the server machines.

Do you know whether the order of bricks affects the read speed of the
files in the volume?

Best regards,
Jin