[Gluster-devel] Doing LS with a lot of directory, files

[gordan at bobich.net]

Can you explain what the principal change was that made it go much faster? 
Splitting up GlusterFS and non-GlusterFS traffic? Increasing the MTU? I 
cannot see either of those making that big a difference...

Gordan

On Thu, 24 Apr 2008, Einar Gautun wrote:

> Hello,
> I've set up a split network, with one switch(Gbit) having the official
> ip for ssh, ntp, nis and so on - mtu 1500. Then I have another
> switch(Gbit) with 10.0.0.0/24 - mtu 9000 - for the file transport. Here
> I use trunking. This gave so much better responce with ls, so now I have
> 3029 files in a directory and ls -l takes only 2-3 seconds(test setup).
>
> Even under load this setup works just so much better, going from ls -l
> with up to 1 min responce to 1-2 seconds under same load and same
> equipment(running system). I have ext3 as filesystem, and io-threads
> only on server, unify on client, no other translators serverside or
> clientside.
>
> Regards,
>
> Einar
>
> On Thu, 2008-04-24 at 12:47 +0200, Tom Myny wrote:
>> Hello,
>>
>> I'm running afr on two storage servers, with three clients.
>> For the moment, we have copied over 500 million small files on it, splitting
>> into each directory which contains 1000 files.
>>
>> When doing ls in directory containing 1000 directory's we have the following
>> issue:
>>
>>
>> - Ls is taking more then 15 minutes to complete in a directory with 1000
>> folders. (this will be split also to 100 folders later, but it's now a big
>> problem)
>> 	-> Yes, for now its ls --color=auto by default on debian :D
>> - When doing copies from other clients, those copies halt until that ls is
>> complete.
>>
>>
>> Is there a way to
>>
>> 1) Do a ls faster (ok, I know it can be that fast like on the filesystem
>> itself, but on the filesystem (or an nfs system) it takes max 15 seconds)
>> 2) When someone is doing an ls, the other processes are not freesing.
>> (checking on the storage servers, we have a load of 0.00)
>>
>> The filesystems we use are based on xfs.
>> An example of a server config:
>>
>> volume sas-ds
>>         type storage/posix
>>         option directory /sas/data
>> end-volume
>>
>> volume sas-ns
>>         type storage/posix
>>         option directory /sas/ns
>> end-volume
>>
>> volume sata-ds
>>         type storage/posix
>>         option directory /sata/data
>> end-volume
>>
>> volume sata-ns
>>         type storage/posix
>>         option directory /sata/ns
>> end-volume
>>
>> volume sas-backup-ds
>>         type protocol/client
>>         option transport-type tcp/client
>>         option remote-host x.x.x.x
>>         option remote-subvolume sas-ds
>> end-volume
>>
>> volume sas-backup-ns
>>         type protocol/client
>>         option transport-type tcp/client
>>         option remote-host x.x.x.x
>>         option remote-subvolume sas-ns
>> end-volume
>>
>> ...
>>
>> volume sas-unify
>>         type cluster/unify
>>         subvolumes sas-ds-afr
>>         option namespace sas-ns-afr
>>         option scheduler rr
>> end-volume
>>
>> volume sata-unify
>>         type cluster/unify
>>         subvolumes sata-ds-afr
>>         option namespace sata-ns-afr
>>         option scheduler rr
>> end-volume
>>
>> volume sas
>>         type performance/io-threads
>>         option thread-count 16
>>         option cache-size 256MB
>>         subvolumes sas-unify
>> end-volume
>>
>> volume sata
>>         type performance/io-threads
>>         option thread-count 16
>>         option cache-size 256MB
>>         subvolumes sata-unify
>> end-volume
>>
>> ..
>>
>> I hope to fix this, because we want to double this next year :)
>>
>>
>> Regards,
>> Tom
>>
>>
>>
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at nongnu.org
>> http://lists.nongnu.org/mailman/listinfo/gluster-devel
> -- 
> Einar Gautun                     einar.gautun at statkart.no
>
> Statens kartverk            | Norwegian Mapping Authority
> 3507 Hønefoss               |    NO-3507 Hønefoss, Norway
>
> Ph +47 32118372   Fax +47 32118101       Mob +47 92692662
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>