[Gluster-users] Can Hadoop run on gluster in 1 JT, N TT setup or only works for 1 JT+TT?

[Fermín Galán Márquez]

Hi,

Recently I've set up a Gluster cluster to run Hadoop M/R jobs, following
the document at
http://download.gluster.com/pub/gluster/glusterfs/qa-releases/3.3-beta-2/Gluster_Hadoop_Compatible_Storage.pdf.

As long as I check in my tests, what the gluster_hadoop.jar plugin is
doing is to automatically mount the gluster volumen at the JT node, then
the TT (in the same node) uses that mountpoint to do its work. That's ok
if JT and TT are runing in the same node (i.e. a one-node setup (*)).
However, when I test with a 2-nodes (*) setup in which the JT runs in a
node and TT in another node it doesn't work (e.g. hadoop jar gets
stalled in the "INFO mapred.JobClient:  map 0% reduce 0%" with no
progress after that), which at the end makes sense, given that the
gluster volume is not mounted in the TT node (it's only mounted in the
JT node).

This is a bit annoying to me, given I was expecting that the gluster
volume gets mounted in the TT nodes, which are the ones that actually
need to access to data in the filesystem.

Thus, is not possible to run a 1 JT, N TT Hadoop cluster with gluster?
It only works on a 1 JT+TT?

Or maybe I'm doing something wrong or maybe I'm not understanding
correctly the document at
http://download.gluster.com/pub/gluster/glusterfs/qa-releases/3.3-beta-2/Gluster_Hadoop_Compatible_Storage.pdf
(any piece of information about Hadoop running on gluster is highly
welcome, please).

I'm using Hadoop 0.20.2 and Gluster 3.3beta2. If you need to know any
other information about my setup, don't hesitate to ask for it!

Thanks in advance!

Best regards,

------
Fermín

(*) I refer to nodes in the Hadoop cluster, no matter how many nodes are
implementing the gluster cluster (latter ones are "abstracted" by the
mountpoint, as far as I understand)

Este mensaje se dirige exclusivamente a su destinatario. Puede consultar nuestra política de envío y recepción de correo electrónico en el enlace situado más abajo.
This message is intended exclusively for its addressee. We only send and receive email on the basis of the terms set out at
http://www.tid.es/ES/PAGINAS/disclaimer.aspx