[Gluster-devel] non-blocking connect() returned: 111 (Connection refused)

[Jordi Moles Blanco]

En/na Jordi Moles Blanco ha escrit:
> Hi,
>
> i've got 6 nodes providing a storage unit with gluster 2.5 patch 800. 
> They are set in 2 groups of 3 nodes each.
>
> On top of that, i've got a Xen 3.2 machine storing its virtual 
> machines in gluster mount point.
>
> The thing is that i used to have only 2 nodes for group, that's 4 
> nodes in total, and today I'm trying to add 1 extra node for each group.
>
> This is the final setting on Xen's Side:
>
>
> **************
>
> volume espai1
>        type protocol/client
>        option transport-type tcp/client
>        option remote-host 10.0.0.3
>        option remote-subvolume espai
> end-volume
>
> volume espai2
>        type protocol/client
>        option transport-type tcp/client
>        option remote-host 10.0.0.4
>        option remote-subvolume espai
> end-volume
>
> volume espai3
>        type protocol/client
>        option transport-type tcp/client
>        option remote-host 10.0.0.5
>        option remote-subvolume espai
> end-volume
>
> volume espai4
>    type protocol/client
>    option transport-type tcp/client
>    option remote-host 10.0.0.6
>    option remote-subvolume espai
> end-volume
>
> volume espai5
>    type protocol/client
>    option transport-type tcp/client
>    option remote-host 10.0.0.7
>    option remote-subvolume espai
> end-volume
>
> volume espai6
>    type protocol/client
>    option transport-type tcp/client
>    option remote-host 10.0.0.8
>    option remote-subvolume espai
> end-volume
>
> volume namespace1
>        type protocol/client
>        option transport-type tcp/client
>        option remote-host 10.0.0.4
>        option remote-subvolume nm
> end-volume
>
> volume namespace2
>        type protocol/client
>        option transport-type tcp/client
>        option remote-host 10.0.0.5
>        option remote-subvolume nm
> end-volume
>
> volume grup1
>        type cluster/afr
>        subvolumes espai1 espai3 espai5
> end-volume
>
> volume grup2
>        type cluster/afr
>        subvolumes espai2 espai4 espai6
> end-volume
>
> volume nm
>        type cluster/afr
>        subvolumes namespace1 namespace2
> end-volume
>
> volume g01
>        type cluster/unify
>        subvolumes grup1 grup2
>        option scheduler rr
>        option namespace nm
> end-volume
>
> volume io-cache
>        type performance/io-cache
>        option cache-size 512MB
>        option page-size 1MB
>        option force-revalidate-timeout 2
>        subvolumes g01
> end-volume 
>
> **************
>
> so... i stopped all virtual machines, unmounted gluster on Xen, 
> updated the spec file (the one above) and ran gluster again in Xen.
>
> I've set different gluster environments but i had never tried this, 
> and now i'm facing some problems.
>
> For what i had read before this... i used to think that when adding 
> and extra node to a group and "remounting" on client's side, the 
> Healing feature would copy all the content of the other nodes already 
> present in the group to the "new one". That hasn't happened, even when 
> I've tried to force the file system, by listing the files or doing 
> what you suggest in you documentation:
>
> **********
>
> find /mnt/glusterfs -type f -print0 | xargs -0 head -c1 >/dev/null
>
> **********
>
> so... my first question would be... does "self-healing" work this way? 
> If it doesn't.... which is the best way to add a node to a group? Do i 
> have to run a "copy" command manually to get the new node ready?
> I've also noticed that i have necessarily to umount gluster from Xen. 
> Is there a way to avoid stopping all the virtual machines, umounting 
> and mounting again? Is there a feature like "refresh config file"?
>
> And finally... i looked into the logs to see why self-healing wasn't 
> working, and i found this on Xen's Side:
>
> **********
> 2008-12-17 12:08:30 E [tcp-client.c:190:tcp_connect] espai6: 
> non-blocking connect() returned: 111 (Connection refused)
> **********
>
> and it keeps saying this when i want to access  files which were 
> created in the "old nodes".
>
> is this a bug? how can i work around this?
>
> If i create new stuff, though, it replicates to the 3 nodes, no 
> problem with that.... the only problem is with the old files that were 
> already present before i added the new node.
>
> Thanks for your help in advance, and let me know if you need any 
> further information.
>
>
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel


well... i've found out something else...


in my previous mail i said that if i create new files,the replicate into 
all the nodes of the group. I said this because if i do "ls" on the 
"shared folder" on the new nodes, they "apparently" have the new content 
as well.
However, if i shutdown the 2 old nodes, Xen can't see anything at all 
from gluster, the mount point is like "stale" and even the new content 
is not available for Xen. If i look at the log files from nodes (the new 
ones), they log "activity" in the file system, but Xen can't get to the 
data.

Thanks for you help.