[Gluster-users] glusterfs for cloud storage

[Wei Dong]

Hi All,

We are using glusterfs on our lab cluster for a shared storage to save a 
large number of image files, about 30 million at the moment.  We use 
Hadoop for distributed computing, but we are reluctant to store small 
files on hadoop for it's low throughput on small files and also the 
non-standard filesystem interface (e.g. we won't be able to run convert 
on each image to produce a thumbnail if the files are stored in 
hadoop).  What we do now is to store a list of paths to all images in 
hadoop, and use Hadoop streaming to pipe the paths to some script, which 
will then read the images from glusterfs filesystem and do the 
processing.  This has been working for a while so long as glusterfs 
doesn't hang, but the problem is that we basically lose all data 
locality.  We have 66 nodes and the chance that a needed file is on 
local disk is only 1/66, and 55/66 of file I/O has to go through 
network, which make me very uncomfortable.  I'm wondering if there's a 
better way of making glusterfs and Hadoop work together to take the 
advantage of data locality.

I know that there's a nufa translator which gives high preference to 
local drive.  This is good enough if the assignment of files to nodes is 
fixed.  But if we want to assign files to nodes according to the 
location of the file, what interface should we use to get the physical 
location of the file?

I appreciate all your suggestions.

- Wei Dong