[Gluster-users] very bad performance on small files
Joe Landman
landman at scalableinformatics.com
Fri Jan 14 23:51:39 UTC 2011
On 01/14/2011 06:26 PM, Marcus Bointon wrote:
>> Our own experience has been generally that you are IOP constrained
>> because of the stack you have to traverse. If you add more latency
>> into this stack, you have more to traverse, and therefore, you have
>> more you need to wait. Which will have a magnification effect upon
>> times for small IO ops which are seeky (stat, small writes, random
>> ops).
>
> Sure, and all that applies equally to both NFS and gluster, yet in
> Max's example NFS was ~50x faster than gluster for an identical
> small-file workload. So what's gluster doing over and above what NFS
> is doing that's taking so long, given that network and disk factors
> are equal? I'd buy a factor of 2 for replication, but not 50.
If the NFS was doing attribute caching and the GlusterFS implementation
had stat prefetch and other caching turned off, this could explain it.
> In case you missed what I'm on about, it was these stats that Max
> posted:
>
>> Here is the results per command: dd if=/dev/zero of=M/tmp bs=1M
>> count=16384 69.2 MB/se (Native) 69.2 MB/sec(FUSE) 52 MB/sec (NFS)
>> dd if=/dev/zero of=M/tmp bs=1K count=163840000 88.1 MB/sec
>> (Native) 1.1MB/sec (FUSE) 52.4 MB/sec (NFS) time tar cf - M | pv>
>> /dev/null 15.8 MB/sec (native) 3.48MB/sec (FUSE) 254 Kb/sec (NFS)
Ok, I am not sure if I saw the numbers before. Thanks.
>
> In my case I'm running 30kiops SSDs over gigabit. At the moment my
> problem (running 3.0.6) isn't performance but reliability - files are
> occasionally reported as 'vanished' by front-end apps (like rsync)
> even though they are present on both backing stores; no errors in
> gluster logs, self-heal doesn't help.
Check your stat-prefetch settings, and your time base. We've had some
strange issues that seem to be correlated with time bases drifting.
Including files disappearing. We have a few open tickets on this.
The way we've worked around this problem is to abandon the NFS client
and use the glusterfs client. Not our preferred option, but it provides
a workaround for the moment. The NFS translator does appear to have a
few issues. I am hoping we get more tuning knobs for it soon so we can
see if we can work around this.
Regards,
Joe
>
> Marcus _______________________________________________ Gluster-users
> mailing list Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: landman at scalableinformatics.com
web : http://scalableinformatics.com
http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615
More information about the Gluster-users
mailing list