[Gluster-users] very bad performance on small files

[Joe Landman]

On 01/14/2011 06:26 PM, Marcus Bointon wrote:

>> Our own experience has been generally that you are IOP constrained
>> because of the stack you have to traverse.  If you add more latency
>> into this stack, you have more to traverse, and therefore, you have
>> more you need to wait.  Which will have a magnification effect upon
>> times for small IO ops which are seeky (stat, small writes, random
>> ops).
>
> Sure, and all that applies equally to both NFS and gluster, yet in
> Max's example NFS was ~50x faster than gluster for an identical
> small-file workload. So what's gluster doing over and above what NFS
> is doing that's taking so long, given that network and disk factors
> are equal? I'd buy a factor of 2 for replication, but not 50.

If the NFS was doing attribute caching and the GlusterFS implementation 
had stat prefetch and other caching turned off, this could explain it.

> In case you missed what I'm on about, it was these stats that Max
> posted:
>
>> Here is the results per command: dd if=/dev/zero of=M/tmp bs=1M
>> count=16384 69.2 MB/se (Native) 69.2 MB/sec(FUSE) 52 MB/sec (NFS)
>> dd if=/dev/zero of=M/tmp bs=1K count=163840000  88.1 MB/sec
>> (Native) 1.1MB/sec (FUSE) 52.4 MB/sec (NFS) time tar cf - M | pv>
>> /dev/null 15.8 MB/sec (native) 3.48MB/sec (FUSE) 254 Kb/sec (NFS)

Ok, I am not sure if I saw the numbers before.  Thanks.

>
> In my case I'm running 30kiops SSDs over gigabit. At the moment my
> problem (running 3.0.6) isn't performance but reliability - files are
> occasionally reported as 'vanished' by front-end apps (like rsync)
> even though they are present on both backing stores; no errors in
> gluster logs, self-heal doesn't help.

Check your stat-prefetch settings, and your time base.  We've had some 
strange issues that seem to be correlated with time bases drifting. 
Including files disappearing.  We have a few open tickets on this.

The way we've worked around this problem is to abandon the NFS client 
and use the glusterfs client.  Not our preferred option, but it provides 
a workaround for the moment.  The NFS translator does appear to have a 
few issues.  I am hoping we get more tuning knobs for it soon so we can 
see if we can work around this.

Regards,

Joe

>
> Marcus _______________________________________________ Gluster-users
> mailing list Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users


-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615