[Gluster-devel] Unfair scheduling in unify/AFR
cc at avaxio.hu
Wed Nov 21 18:51:51 UTC 2007
Anand Avati wrote:
> I'm using io-threads on client side. At this time, only one client
> accesses a storage brick at a time (on a given server), so I thought
> io-threads won't help there. But on client side, waiting for a read for
> one thread shouldn't block the whole client (because there can be more
> threads), so I loaded io-threads on the client.
> The client does not block for any operation and is completely
> asynchronous. The synchronous parts in glusterfs are network writes in
> tcp (we are working on a non-blocking socket model design) and posix
> (linux-aio translator is on the way). So io-threads would help for reads
> on the server side alone since tcp writes of read requests is not really
So am I right when I think that io-threads would help only if loaded on
the server side, just above the storage/posix brick and more clients
access the server in parallel?
But from the application's point of view, reads are synchronous in the
sense that a read() blocks until the corresponding block is fetched from
the server. And this is why we can't utilize the full bandwidth, but
this is outside the scope of GlusterFS, this is a kernel "feature". We
cannot (and do not) blame GLusterFS for this. Just thinking. Am I right?
(I try to find a reason for GlusterFS not being able to saturate the
link.) Is this true? If not, why do I get a performance of 540 MB/s not
900 or 1000? Interconnect speed is far above this, see below.
> > When you copy, are the source and destination files both on the
> > mount point?
> No, since we are testing pure read performance, and only GlusterFS
> performance. So I copy sparse files over GlsuterFS into /dev/null with
> `dd bs=1M`.
> Currently, a single and only thread has a read performance about
> MB/s. What I would like to see is two threads reading two files from two
> servers, with a performance of at least 540 MB/s *each*.
> Are these two dd's from the same machine? If so I suspect that fuse is
> becoming a bottleneck. What is your interconnect? 10Gig/E ?
Yes, they are from the same host, so they use one GlusterFS client. My
interconnect is 20 Gbps ib-verbs (of which we can utilize about 11 Gbps,
due to architectural limitations, I guess).
> You might be
> interested in the new booster module which is on the way which will
> short cut the IO path directly to the server from the application's
> address space (via LD_PRELOAD). You might also want to try the next few
> TLA commits which are on the way, which will help parallel file reads
> going via fuse.
I have tried the booster module, it does the same. Please let me know if
there are new commits that may help.
Thanks in advance,
More information about the Gluster-devel