[Gluster-users] Gluster-users Digest, Vol 85, Issue 22 - 9. Re: seq read performance comparion between libgfapi and fuse

[Ben England]

Paul, I don't check this list every day, I would expect you can get more than half of minimum of network line speed or storage block device speed using a single libgfapi sequential read thread.  I did not see any throughput calculation or file size in your e-mail.

HTH, inline below...

-ben e

----- Original Message -----
> From: gluster-users-request at gluster.org
> To: gluster-users at gluster.org
> Sent: Friday, May 22, 2015 8:00:02 AM
> Subject: Gluster-users Digest, Vol 85, Issue 22
> 
> Message: 8
> Date: Fri, 22 May 2015 18:50:40 +0800
> From: Paul Guo <bigpaulguo at foxmail.com>
> To: gluster-users at gluster.org
> Subject: [Gluster-users] seq read performance comparion between
> 	libgfapi and	fuse
> Message-ID: <555F0A00.2060104 at foxmail.com>
> Content-Type: text/plain; charset=gbk; format=flowed
> 
> Hello,
> 
> I wrote two simple single-process seq read test case to compare libgfapi
> and fuse. The logic looks like this.
> 
> char buf[32768];
> while (1) {
>                cnt = read(fd, buf, sizeof(buf));
>                  if (cnt == 0)
>                          break;
>                  else if (cnt > 0)
>                          total += cnt;
>                   // No "cnt < 0" was found during testing.
> }
> 
> Following is the time which is needed to finish reading a large file.
> 
>                     fuse         libgfapi
> direct io:         40s          51s
> non direct io:     40s          47s
> 
> The version is 3.6.3 on centos6.5. The result shows that libgfapi is
> obviously slower than the fuse interface although the cpu cycles were
> saved a lot during libgfapi testing. Each test was run before cleaning
> up all kernel pageche&inode&dentry caches and stopping and then starting
> glusterd&gluster (to clean up gluster cache).

so if you use libgfapi in a single-threaded app, you may need to tune gluster volume parameter read-ahead-page-count (defaults to 4).  The default is intended to trade-off single-thread performance for better aggregate performance and response time.  Here is a example of how to tune it for a single-thread use case, don't do this all the time. 

gluster volume set your-volume performance.read-ahead-page-count 16

As a debugging tool, you can try disabling readahead translator altogether 

# gluster v set your-volume read-ahead off

To reset parameters to defaults:

# gluster v set your-volume read-ahead
# gluster v set your-volume read-ahead-page-count

I have a benchmark for libgfapi testing in case this is useful to you:

https://github.com/bengland2/parallel-libgfapi

please e-mail me direct if problems with it.

> 
> I tested direct io because I suspected that fuse kernel readahead
> helped more than the read optimization solutions in gluster. I searched
> a lot but I did not find much about the comparison between fuse and
> libgfapi. Anyone has known about this and known why?
> 

If you use O_DIRECT you may be  bypassing readahead translator in Gluster and this may account for your problem.  Try NOT using O_DIRECT, and try above tuning.  Or if you really need O_DIRECT on client, try this command, which disables O_DIRECT on the server side but not the client, it's equivalent of NFS behavior.

# gluster v set your-volume network.remote-dio on

Also try turning off io-cache translator which will not help you here.

# gluster v set your-volume io-cache off

Also, O_DIRECT is passed all the way to the server by Gluster so your disk reads will ALSO use O_DIRECT, this is terrible for performance.  You want to have block device readahead when doing this test.  Suggest you set it to at least 4096 KB for block devices used for Gluster brick mountpoints.