[Gluster-users] performance problem with large directory

[Raghavendra G]

Hi Keith,

Comments are inlined.

On Mon, Sep 8, 2008 at 3:11 AM, Keith Freedman <freedman at freeformit.com>wrote:

> I have an application which stores old copies of files whenever new
> ones are uploaded.
>
> it does this by checking if file_exists in the archive folder
> this has roughly 30000 items in it.


How are you checking whether a file exists or not? do you  use  ls? Can you
make sure, that  ls doesn't  send a stat call on each of the files in the
directory (as in ls -l )? if ls just sends readdir calls and still the
performance is low, let us know about it.

If you need stat calls, you might need stat-prefetch, which is currently not
funcitonal.


>
> I've got an AFR configuration.
>
> so, when someone uploads 10 files, it checks for the file existing,
> which does a directory scan which causes afr to check the other
> server, however this takes a reallllly long time.
> and it does this for each file, which is a problem.
>
> which performance translator would help the most with this.
>
> the AFR configuration uses local read-volume.
>
> will any performance translators help given it's an AFR config?  I
> understand afr wants to maintain the directory integrity, but it's
> kind of irritating.
>
> the directory filesize is 2.2 mb so I'm not sure if it's actually
> moving this volume of data constantly or not, but it's painfully slow
> and things seem to timeout constantly.
>
> any thoughts are appreciated.
>
> Keith
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>



-- 
Raghavendra G

A centipede was happy quite, until a toad in fun,
Said, "Prey, which leg comes after which?",
This raised his doubts to such a pitch,
He fell flat into the ditch,
Not knowing how to run.
-Anonymous
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20080908/b186d09d/attachment.html>