[Gluster-users] How many subfolders in parent folders ?

[alain meunier]

Hello !

Many thanks for thoses answsers.

My concern is 
1) that it is easy to create a path using c++ without waiting lots of seconds.
2) fast to retrieve a data with a known path

Ideally, I should be able to make a "ls /examplefolder/a/b/c/.../9/myfile.txt" in only a second maximum.

I don't care to find a file, nor to layout it. I just want that a known path goes straight to the goal for the asked file as fast as possible.

Do you think I am misleading myself at some point ?

Many thanks,

Regards,

Alain

> From: laurent.chouinard at ubisoft.com
> To: gluster-users at gluster.org
> Date: Wed, 8 May 2013 17:56:17 -0400
> Subject: Re: [Gluster-users] How many subfolders in parent folders ?
> 
> Hi Alain,
> 
> The performance of this is going to be very dependent on the hardware that your cluster uses.
> 
> For example, I have a cluster here of 4 nodes with 4 SSDs each, and a replication factor set to 2.  Bricks are using XFS. Nodes are inter-connected with 10 GigE networking.
> 
> The folder structure we use is based on hexadecimal representation and we take the first two bytes of the file name to decide where it goes. This way, we end up with:
> \00\00\...
> \00\01\...
> \00\0f\... etc
> 
> That makes a total of only 65 792 folders (256 x 256 + 256), and if you spread 100 million different files in that, the result is a bit over 1500 files per folder, which is very reasonable.
> 
> Now for the speed that crawling to such a layout happens, my cluster here has 3 million files at the moment. If I find a "find ." command from inside one of the 256 top folders, it takes 14 seconds. I can extrapolate that to just over 59 minutes if I were to crawl through all of them.
> 
> I would advise against going too deep in the folder-in-folder-in-folder, because as you multiply the possibilities, you'll end up with a file system with millions of entries just for folders themselves.
> 
> Laurent Chouinard
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130509/3402e957/attachment.html>