Fwd: [Gluster-devel] gluster working, but error appearing every two seconds in logs - NEW INFO
Jordi Moles Blanco
jordi at cdmon.com
Fri Feb 20 09:16:53 UTC 2009
En/na Basavanagowda Kanur ha escrit:
> Forwarding to list
>
> ---------- Forwarded message ----------
> From: *Basavanagowda Kanur* <gowda at gluster.com <mailto:gowda at gluster.com>>
> Date: Thu, Feb 19, 2009 at 11:56 PM
> Subject: Re: [Gluster-devel] gluster working, but error appearing
> every two seconds in logs - NEW INFO
> To: jordi at cdmon.com <mailto:jordi at cdmon.com>
> Cc: Anand Avati <avati at zresearch.com <mailto:avati at zresearch.com>>,
> Gluster List <gluster-devel at nongnu.org <mailto:gluster-devel at nongnu.org>>
>
>
> Jordi,
> Please find replies inline.
>
>
> Hi,
>
> since the last time we were in contact I've been trying to track
> down where the problem is. I've been monitoring almost every
> possible thing related to network traffic, and... eventually.... i
> found out what the problem is by chance!!
>
> Did you try 'telnet <server> <port>' at the time when you were getting
> 'Connection refused' errors?
>
>
> It turns out that when in a client-server mounting gluster i run
> "df -h", i get this:
>
> ***********
> 2009-02-19 17:15:43 E [tcp-client.c:190:tcp_connect] espai1:
> non-blocking connect() returned: 111 (Connection refused)
> 2009-02-19 17:15:43 W [client-protocol.c:332:client_protocol_xfer]
> espai1: not connected at the moment to submit frame type(1) op(15)
> 2009-02-19 17:15:43 E [client-protocol.c:3891:client_statfs_cbk]
> espai1: no proper reply from server, returning ENOTCONN
> 2009-02-19 17:15:43 E [tcp-client.c:190:tcp_connect] espai5:
> non-blocking connect() returned: 111 (Connection refused)
> 2009-02-19 17:15:43 W [client-protocol.c:332:client_protocol_xfer]
> espai5: not connected at the moment to submit frame type(1) op(15)
> 2009-02-19 17:15:43 E [client-protocol.c:3891:client_statfs_cbk]
> espai5: no proper reply from server, returning ENOTCONN
> 2009-02-19 17:15:43 E [tcp-client.c:190:tcp_connect] espai2:
> non-blocking connect() returned: 111 (Connection refused)
> 2009-02-19 17:15:43 W [client-protocol.c:332:client_protocol_xfer]
> espai2: not connected at the moment to submit frame type(1) op(15)
> 2009-02-19 17:15:43 E [client-protocol.c:3891:client_statfs_cbk]
> espai2: no proper reply from server, returning ENOTCONN
> 2009-02-19 17:15:43 E [tcp-client.c:190:tcp_connect] espai6:
> non-blocking connect() returned: 111 (Connection refused)
> 2009-02-19 17:15:43 W [client-protocol.c:332:client_protocol_xfer]
> espai6: not connected at the moment to submit frame type(1) op(15)
> 2009-02-19 17:15:43 E [client-protocol.c:3891:client_statfs_cbk]
> espai6: no proper reply from server, returning ENOTCONN
>
> ************
>
> so... the reason why it is appearing so often is that i've got
> munin monitoring this gluster environment, and it performs a "df"
> command to check the disk space of all the servers, including, of
> course, the gluster mount point. When this happens... the error
> log shown above these lines is reported and eventually.... the
> mount point in that server fails. No data is lost, but i have to
> remount glusterfs as it becomes stale and data is not accessible.
>
> is this a normal behaviour?
>
> No. This is not normal behaviour. 'Connection refused' error comes
> when either the server is not running or there is a packet filtering
> between server and client.
>
>
>
>
> i could stop munin from running "df" every 5 minutes... but
> still... is there any problem in my setup or is this what gluster
> is supposed to do?
>
> The 'Connection refused' problem is not caused by any application
> accessing the mount-point. 'Connection refused' can come only if
> server is dead or packet is being filtered out.
>
>
>
> Thanks.
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org <mailto:Gluster-devel at nongnu.org>
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>
>
>
>
> --
> gowda
>
>
>
> --
> hard work often pays off after time, but laziness always pays off now
> ------------------------------------------------------------------------
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>
hi,
********
Did you try 'telnet <server> <port>' at the time when you were getting
'Connection refused' errors?
*******
no, i hadn't tried that, i tried this morning. I ran a infinite loop in
shellscript with "df -h" while i was doing some telnet and ssh attempts
to the node servers, and it worked fine, to all of them. It doesn't seem
there's any network problem between clients and nodes, even when i'm
running "df" constantly on the gluster mount point.
I know you told me that...
" 'Connection refused' error comes when either the server is not running
or there is a packet filtering between server and client."
and it makes sense, but i've even tried to replace the switch device
which links clients and nodes, and this error message keeps appearing.
The nodes, however, don't log anything, they "don't see" any kind of
problem, it's only the client server which reports de problem. What's
more, this problem keeps appearing through the whole day, but the mount
point keeps beeing accessible. It's just after some time of this problem
occurring that it eventually fails and the mount point gets stale. After
that, i simply umount/mount and everything works fine again for some
more time.
I can always reproduce this problem, i only have to run a simple "df -h"
command.
This error messages have been reported using WARNING log level. This
morning i also tried with DEBUG log level. Nodes don't report anything
special, but clients do, this is the full log file when it loads the
gluster mount point
****************************************
2009-02-20 09:50:50 D [glusterfs.c:167:get_spec_fp] glusterfs: loading spec from /etc/glusterfs/glusterfs-client.vol
2009-02-20 09:50:50 D [spec.y:107:new_section] parser: New node for 'espai1'
2009-02-20 09:50:50 D [xlator.c:115:xlator_set_type] xlator: attempt to load file //lib/glusterfs/1.3.12a/xlator/protocol/client.so
2009-02-20 09:50:50 D [spec.y:127:section_type] parser: Type:espai1:protocol/client
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:espai1:transport-type:tcp/client
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:espai1:remote-host:10.0.0.3
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:espai1:remote-subvolume:espai
2009-02-20 09:50:50 D [spec.y:209:section_end] parser: end:espai1
2009-02-20 09:50:50 D [spec.y:107:new_section] parser: New node for 'espai2'
2009-02-20 09:50:50 D [xlator.c:115:xlator_set_type] xlator: attempt to load file //lib/glusterfs/1.3.12a/xlator/protocol/client.so
2009-02-20 09:50:50 D [spec.y:127:section_type] parser: Type:espai2:protocol/client
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:espai2:transport-type:tcp/client
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:espai2:remote-host:10.0.0.4
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:espai2:remote-subvolume:espai
2009-02-20 09:50:50 D [spec.y:209:section_end] parser: end:espai2
2009-02-20 09:50:50 D [spec.y:107:new_section] parser: New node for 'espai3'
2009-02-20 09:50:50 D [xlator.c:115:xlator_set_type] xlator: attempt to load file //lib/glusterfs/1.3.12a/xlator/protocol/client.so
2009-02-20 09:50:50 D [spec.y:127:section_type] parser: Type:espai3:protocol/client
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:espai3:transport-type:tcp/client
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:espai3:remote-host:10.0.0.5
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:espai3:remote-subvolume:espai
2009-02-20 09:50:50 D [spec.y:209:section_end] parser: end:espai3
2009-02-20 09:50:50 D [spec.y:107:new_section] parser: New node for 'espai4'
2009-02-20 09:50:50 D [xlator.c:115:xlator_set_type] xlator: attempt to load file //lib/glusterfs/1.3.12a/xlator/protocol/client.so
2009-02-20 09:50:50 D [spec.y:127:section_type] parser: Type:espai4:protocol/client
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:espai4:transport-type:tcp/client
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:espai4:remote-host:10.0.0.6
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:espai4:remote-subvolume:espai
2009-02-20 09:50:50 D [spec.y:209:section_end] parser: end:espai4
2009-02-20 09:50:50 D [spec.y:107:new_section] parser: New node for 'espai5'
2009-02-20 09:50:50 D [xlator.c:115:xlator_set_type] xlator: attempt to load file //lib/glusterfs/1.3.12a/xlator/protocol/client.so
2009-02-20 09:50:50 D [spec.y:127:section_type] parser: Type:espai5:protocol/client
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:espai5:transport-type:tcp/client
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:espai5:remote-host:10.0.0.7
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:espai5:remote-subvolume:espai
2009-02-20 09:50:50 D [spec.y:209:section_end] parser: end:espai5
2009-02-20 09:50:50 D [spec.y:107:new_section] parser: New node for 'espai6'
2009-02-20 09:50:50 D [xlator.c:115:xlator_set_type] xlator: attempt to load file //lib/glusterfs/1.3.12a/xlator/protocol/client.so
2009-02-20 09:50:50 D [spec.y:127:section_type] parser: Type:espai6:protocol/client
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:espai6:transport-type:tcp/client
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:espai6:remote-host:10.0.0.8
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:espai6:remote-subvolume:espai
2009-02-20 09:50:50 D [spec.y:209:section_end] parser: end:espai6
2009-02-20 09:50:50 D [spec.y:107:new_section] parser: New node for 'namespace1'
2009-02-20 09:50:50 D [xlator.c:115:xlator_set_type] xlator: attempt to load file //lib/glusterfs/1.3.12a/xlator/protocol/client.so
2009-02-20 09:50:50 D [spec.y:127:section_type] parser: Type:namespace1:protocol/client
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:namespace1:transport-type:tcp/client
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:namespace1:remote-host:10.0.0.3
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:namespace1:remote-subvolume:nm
2009-02-20 09:50:50 D [spec.y:209:section_end] parser: end:namespace1
2009-02-20 09:50:50 D [spec.y:107:new_section] parser: New node for 'namespace2'
2009-02-20 09:50:50 D [xlator.c:115:xlator_set_type] xlator: attempt to load file //lib/glusterfs/1.3.12a/xlator/protocol/client.so
2009-02-20 09:50:50 D [spec.y:127:section_type] parser: Type:namespace2:protocol/client
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:namespace2:transport-type:tcp/client
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:namespace2:remote-host:10.0.0.4
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:namespace2:remote-subvolume:nm
2009-02-20 09:50:50 D [spec.y:209:section_end] parser: end:namespace2
2009-02-20 09:50:50 D [spec.y:107:new_section] parser: New node for 'namespace3'
2009-02-20 09:50:50 D [xlator.c:115:xlator_set_type] xlator: attempt to load file //lib/glusterfs/1.3.12a/xlator/protocol/client.so
2009-02-20 09:50:50 D [spec.y:127:section_type] parser: Type:namespace3:protocol/client
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:namespace3:transport-type:tcp/client
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:namespace3:remote-host:10.0.0.5
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:namespace3:remote-subvolume:nm
2009-02-20 09:50:50 D [spec.y:209:section_end] parser: end:namespace3
2009-02-20 09:50:50 D [spec.y:107:new_section] parser: New node for 'namespace4'
2009-02-20 09:50:50 D [xlator.c:115:xlator_set_type] xlator: attempt to load file //lib/glusterfs/1.3.12a/xlator/protocol/client.so
2009-02-20 09:50:50 D [spec.y:127:section_type] parser: Type:namespace4:protocol/client
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:namespace4:transport-type:tcp/client
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:namespace4:remote-host:10.0.0.6
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:namespace4:remote-subvolume:nm
2009-02-20 09:50:50 D [spec.y:209:section_end] parser: end:namespace4
2009-02-20 09:50:50 D [spec.y:107:new_section] parser: New node for 'namespace5'
2009-02-20 09:50:50 D [xlator.c:115:xlator_set_type] xlator: attempt to load file //lib/glusterfs/1.3.12a/xlator/protocol/client.so
2009-02-20 09:50:50 D [spec.y:127:section_type] parser: Type:namespace5:protocol/client
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:namespace5:transport-type:tcp/client
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:namespace5:remote-host:10.0.0.7
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:namespace5:remote-subvolume:nm
2009-02-20 09:50:50 D [spec.y:209:section_end] parser: end:namespace5
2009-02-20 09:50:50 D [spec.y:107:new_section] parser: New node for 'namespace6'
2009-02-20 09:50:50 D [xlator.c:115:xlator_set_type] xlator: attempt to load file //lib/glusterfs/1.3.12a/xlator/protocol/client.so
2009-02-20 09:50:50 D [spec.y:127:section_type] parser: Type:namespace6:protocol/client
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:namespace6:transport-type:tcp/client
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:namespace6:remote-host:10.0.0.8
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:namespace6:remote-subvolume:nm
2009-02-20 09:50:50 D [spec.y:209:section_end] parser: end:namespace6
2009-02-20 09:50:50 D [spec.y:107:new_section] parser: New node for 'grup1'
2009-02-20 09:50:50 D [xlator.c:115:xlator_set_type] xlator: attempt to load file //lib/glusterfs/1.3.12a/xlator/cluster/afr.so
2009-02-20 09:50:50 D [spec.y:127:section_type] parser: Type:grup1:cluster/afr
2009-02-20 09:50:50 D [spec.y:196:section_sub] parser: child:grup1->espai1
2009-02-20 09:50:50 D [spec.y:196:section_sub] parser: child:grup1->espai3
2009-02-20 09:50:50 D [spec.y:196:section_sub] parser: child:grup1->espai5
2009-02-20 09:50:50 D [spec.y:209:section_end] parser: end:grup1
2009-02-20 09:50:50 D [spec.y:107:new_section] parser: New node for 'grup2'
2009-02-20 09:50:50 D [xlator.c:115:xlator_set_type] xlator: attempt to load file //lib/glusterfs/1.3.12a/xlator/cluster/afr.so
2009-02-20 09:50:50 D [spec.y:127:section_type] parser: Type:grup2:cluster/afr
2009-02-20 09:50:50 D [spec.y:196:section_sub] parser: child:grup2->espai2
2009-02-20 09:50:50 D [spec.y:196:section_sub] parser: child:grup2->espai4
2009-02-20 09:50:50 D [spec.y:196:section_sub] parser: child:grup2->espai6
2009-02-20 09:50:50 D [spec.y:209:section_end] parser: end:grup2
2009-02-20 09:50:50 D [spec.y:107:new_section] parser: New node for 'nm'
2009-02-20 09:50:50 D [xlator.c:115:xlator_set_type] xlator: attempt to load file //lib/glusterfs/1.3.12a/xlator/cluster/afr.so
2009-02-20 09:50:50 D [spec.y:127:section_type] parser: Type:nm:cluster/afr
2009-02-20 09:50:50 D [spec.y:196:section_sub] parser: child:nm->namespace1
2009-02-20 09:50:50 D [spec.y:196:section_sub] parser: child:nm->namespace2
2009-02-20 09:50:50 D [spec.y:196:section_sub] parser: child:nm->namespace3
2009-02-20 09:50:50 D [spec.y:196:section_sub] parser: child:nm->namespace4
2009-02-20 09:50:50 D [spec.y:196:section_sub] parser: child:nm->namespace5
2009-02-20 09:50:50 D [spec.y:196:section_sub] parser: child:nm->namespace6
2009-02-20 09:50:50 D [spec.y:209:section_end] parser: end:nm
2009-02-20 09:50:50 D [spec.y:107:new_section] parser: New node for 'g01'
2009-02-20 09:50:50 D [xlator.c:115:xlator_set_type] xlator: attempt to load file //lib/glusterfs/1.3.12a/xlator/cluster/unify.so
2009-02-20 09:50:50 D [spec.y:127:section_type] parser: Type:g01:cluster/unify
2009-02-20 09:50:50 D [spec.y:196:section_sub] parser: child:g01->grup1
2009-02-20 09:50:50 D [spec.y:196:section_sub] parser: child:g01->grup2
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:g01:scheduler:rr
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:g01:namespace:nm
2009-02-20 09:50:50 D [spec.y:209:section_end] parser: end:g01
2009-02-20 09:50:50 D [spec.y:107:new_section] parser: New node for 'io-cache'
2009-02-20 09:50:50 D [xlator.c:115:xlator_set_type] xlator: attempt to load file //lib/glusterfs/1.3.12a/xlator/performance/io-cache.so
2009-02-20 09:50:50 D [xlator.c:145:xlator_set_type] xlator: dlsym(notify) on //lib/glusterfs/1.3.12a/xlator/performance/io-cache.so: undefined symbol: notify -- neglecting
2009-02-20 09:50:50 D [spec.y:127:section_type] parser: Type:io-cache:performance/io-cache
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:io-cache:cache-size:512MB
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:io-cache:page-size:1MB
2009-02-20 09:50:50 D [spec.y:141:section_option] parser: Option:io-cache:force-revalidate-timeout:2
2009-02-20 09:50:50 D [spec.y:196:section_sub] parser: child:io-cache->g01
2009-02-20 09:50:50 D [spec.y:209:section_end] parser: end:io-cache
2009-02-20 09:50:50 D [glusterfs.c:128:fuse_graph] glusterfs: setting option mount-point to /mnt/glusterfs
2009-02-20 09:50:50 D [xlator.c:115:xlator_set_type] xlator: attempt to load file //lib/glusterfs/1.3.12a/xlator/mount/fuse.so
2009-02-20 09:50:50 D [client-protocol.c:5013:init] espai1: defaulting transport-timeout to 42
2009-02-20 09:50:50 D [transport.c:80:transport_load] transport: attempt to load file //lib/glusterfs/1.3.12a/transport/tcp/client.so
2009-02-20 09:50:50 D [client-protocol.c:5040:init] espai1: defaulting limits.transaction-size to 268435456
2009-02-20 09:50:50 D [client-protocol.c:5013:init] espai3: defaulting transport-timeout to 42
2009-02-20 09:50:50 D [transport.c:80:transport_load] transport: attempt to load file //lib/glusterfs/1.3.12a/transport/tcp/client.so
2009-02-20 09:50:50 D [client-protocol.c:5040:init] espai3: defaulting limits.transaction-size to 268435456
2009-02-20 09:50:50 D [client-protocol.c:5013:init] espai5: defaulting transport-timeout to 42
2009-02-20 09:50:50 D [transport.c:80:transport_load] transport: attempt to load file //lib/glusterfs/1.3.12a/transport/tcp/client.so
2009-02-20 09:50:50 D [client-protocol.c:5040:init] espai5: defaulting limits.transaction-size to 268435456
2009-02-20 09:50:50 D [afr.c:6084:init] grup1: self-heal is enabled (default)
2009-02-20 09:50:50 D [afr.c:6112:init] grup1: (default) reads will be scheduled between the children
2009-02-20 09:50:50 D [client-protocol.c:5340:notify] espai1: got GF_EVENT_PARENT_UP, attempting connect on transport
2009-02-20 09:50:50 D [client-protocol.c:5340:notify] espai3: got GF_EVENT_PARENT_UP, attempting connect on transport
2009-02-20 09:50:50 D [client-protocol.c:5340:notify] espai5: got GF_EVENT_PARENT_UP, attempting connect on transport
2009-02-20 09:50:50 D [client-protocol.c:5013:init] espai2: defaulting transport-timeout to 42
2009-02-20 09:50:50 D [transport.c:80:transport_load] transport: attempt to load file //lib/glusterfs/1.3.12a/transport/tcp/client.so
2009-02-20 09:50:50 D [client-protocol.c:5040:init] espai2: defaulting limits.transaction-size to 268435456
2009-02-20 09:50:50 D [client-protocol.c:5013:init] espai4: defaulting transport-timeout to 42
2009-02-20 09:50:50 D [transport.c:80:transport_load] transport: attempt to load file //lib/glusterfs/1.3.12a/transport/tcp/client.so
2009-02-20 09:50:50 D [client-protocol.c:5040:init] espai4: defaulting limits.transaction-size to 268435456
2009-02-20 09:50:50 D [client-protocol.c:5013:init] espai6: defaulting transport-timeout to 42
2009-02-20 09:50:50 D [transport.c:80:transport_load] transport: attempt to load file //lib/glusterfs/1.3.12a/transport/tcp/client.so
2009-02-20 09:50:50 D [client-protocol.c:5040:init] espai6: defaulting limits.transaction-size to 268435456
2009-02-20 09:50:50 D [afr.c:6084:init] grup2: self-heal is enabled (default)
2009-02-20 09:50:50 D [afr.c:6112:init] grup2: (default) reads will be scheduled between the children
2009-02-20 09:50:50 D [client-protocol.c:5340:notify] espai2: got GF_EVENT_PARENT_UP, attempting connect on transport
2009-02-20 09:50:50 D [client-protocol.c:5340:notify] espai4: got GF_EVENT_PARENT_UP, attempting connect on transport
2009-02-20 09:50:50 D [client-protocol.c:5340:notify] espai6: got GF_EVENT_PARENT_UP, attempting connect on transport
2009-02-20 09:50:50 D [unify.c:4018:init] g01: namespace node specified as nm
2009-02-20 09:50:50 D [scheduler.c:42:get_scheduler] scheduler: attempt to load file rr.so
2009-02-20 09:50:50 D [unify.c:4048:init] g01: Child node count is 2
2009-02-20 09:50:50 D [rr.c:48:rr_init] g01: 'option rr.limits.min-free-disk' not specified, defaulting to 5
2009-02-20 09:50:50 D [client-protocol.c:5013:init] namespace1: defaulting transport-timeout to 42
2009-02-20 09:50:50 D [transport.c:80:transport_load] transport: attempt to load file //lib/glusterfs/1.3.12a/transport/tcp/client.so
2009-02-20 09:50:50 D [client-protocol.c:5040:init] namespace1: defaulting limits.transaction-size to 268435456
2009-02-20 09:50:50 D [client-protocol.c:5013:init] namespace2: defaulting transport-timeout to 42
2009-02-20 09:50:50 D [transport.c:80:transport_load] transport: attempt to load file //lib/glusterfs/1.3.12a/transport/tcp/client.so
2009-02-20 09:50:50 D [client-protocol.c:5040:init] namespace2: defaulting limits.transaction-size to 268435456
2009-02-20 09:50:50 D [client-protocol.c:5013:init] namespace3: defaulting transport-timeout to 42
2009-02-20 09:50:50 D [transport.c:80:transport_load] transport: attempt to load file //lib/glusterfs/1.3.12a/transport/tcp/client.so
2009-02-20 09:50:50 D [client-protocol.c:5040:init] namespace3: defaulting limits.transaction-size to 268435456
2009-02-20 09:50:50 D [client-protocol.c:5013:init] namespace4: defaulting transport-timeout to 42
2009-02-20 09:50:50 D [transport.c:80:transport_load] transport: attempt to load file //lib/glusterfs/1.3.12a/transport/tcp/client.so
2009-02-20 09:50:50 D [client-protocol.c:5040:init] namespace4: defaulting limits.transaction-size to 268435456
2009-02-20 09:50:50 D [client-protocol.c:5013:init] namespace5: defaulting transport-timeout to 42
2009-02-20 09:50:50 D [transport.c:80:transport_load] transport: attempt to load file //lib/glusterfs/1.3.12a/transport/tcp/client.so
2009-02-20 09:50:50 D [client-protocol.c:5040:init] namespace5: defaulting limits.transaction-size to 268435456
2009-02-20 09:50:50 D [client-protocol.c:5013:init] namespace6: defaulting transport-timeout to 42
2009-02-20 09:50:50 D [transport.c:80:transport_load] transport: attempt to load file //lib/glusterfs/1.3.12a/transport/tcp/client.so
2009-02-20 09:50:50 D [client-protocol.c:5040:init] namespace6: defaulting limits.transaction-size to 268435456
2009-02-20 09:50:50 D [afr.c:6084:init] nm: self-heal is enabled (default)
2009-02-20 09:50:50 D [afr.c:6112:init] nm: (default) reads will be scheduled between the children
****************************************
and this is when it starts to fail (the same i showed you previously but
with DEBUG level):
****************************************
009-02-20 09:50:50 D [tcp-client.c:77:tcp_connect] espai1: socket fd = 6
2009-02-20 09:50:50 D [tcp-client.c:107:tcp_connect] espai1: finalized on port `1019'
2009-02-20 09:50:50 D [tcp-client.c:128:tcp_connect] espai1: defaulting remote-port to 6996
2009-02-20 09:50:50 D [common-utils.c:179:gf_resolve_ip] resolver: DNS cache not present, freshly probing hostname: 10.0.0.3
2009-02-20 09:50:50 D [common-utils.c:204:gf_resolve_ip] resolver: returning IP:10.0.0.3[0] for hostname: 10.0.0.3
2009-02-20 09:50:50 D [common-utils.c:212:gf_resolve_ip] resolver: flushing DNS cache
2009-02-20 09:50:50 D [tcp-client.c:161:tcp_connect] espai1: connect on 6 in progress (non-blocking)
2009-02-20 09:50:50 E [tcp-client.c:190:tcp_connect] espai1: non-blocking connect() returned: 111 (Connection refused)
2009-02-20 09:50:50 D [client-protocol.c:4757:client_protocol_reconnect] espai3: attempting reconnect
2009-02-20 09:50:50 D [tcp-client.c:77:tcp_connect] espai3: socket fd = 6
2009-02-20 09:50:50 D [tcp-client.c:107:tcp_connect] espai3: finalized on port `1019'
2009-02-20 09:50:50 D [tcp-client.c:128:tcp_connect] espai3: defaulting remote-port to 6996
2009-02-20 09:50:50 D [common-utils.c:179:gf_resolve_ip] resolver: DNS cache not present, freshly probing hostname: 10.0.0.5
2009-02-20 09:50:50 D [common-utils.c:204:gf_resolve_ip] resolver: returning IP:10.0.0.5[0] for hostname: 10.0.0.5
2009-02-20 09:50:50 D [common-utils.c:212:gf_resolve_ip] resolver: flushing DNS cache
****************************************
Just let me know if i can provide any extra information.
Thanks.
More information about the Gluster-devel
mailing list