[Gluster-devel] AFR: machine crash hangs other mounts ortransport endpoint not connected
Christopher Hawkins
chawkins at veracitynetworks.com
Tue Apr 29 12:51:03 UTC 2008
Oops - there is a "brick ns" in my server configs that I should have removed
- will do so now and verify results are the same.
> Thanks Krishna! I moved the setup out of the diskless boot
> cluster and am able to reproduce on regular machines. All
> version and config information is below. The scenario is:
>
> 1. Start glusterfsd on both servers
> 2. On client, mount gluster at /mnt/gluster 3. Run little
> testing script on client to show status of the mount:
> #!/bin/bash
> while true
> do
> echo $(date)
> sleep 1
> cat /mnt/gluster/etc/fstab
> done
> 4-A. ifconfig down on server1 - client logs no errors, no
> delays (must be reading from server2) 4-B ifconfig down on
> server2 - first time I tried = recovery in 5 seconds
> 4-B " " - 2nd and 3rd times = client
> hangs until I
> manually kill the process
> 4-C. Hard power off on server1 - client logs no errors, no
> delays 4-D. Hard power off on server2, client hangs until I
> manually kill the process
>
> The client logs the following in situation 4-B during a hang:
>
> 2008-04-29 08:36:50 W [client-protocol.c:204:call_bail]
> master1: activating bail-out. pending frames = 1. last sent =
> 2008-04-29 08:36:41. last received = 2008-04-29 08:36:40
> transport-timeout = 5
> 2008-04-29 08:36:50 C [client-protocol.c:211:call_bail]
> master1: bailing transport
> 2008-04-29 08:36:50 W [client-protocol.c:204:call_bail]
> master2: activating bail-out. pending frames = 1. last sent =
> 2008-04-29 08:36:41. last received = 2008-04-29 08:36:40
> transport-timeout = 5
> 2008-04-29 08:36:50 C [client-protocol.c:211:call_bail]
> master2: bailing transport
> 2008-04-29 08:36:50 W [client-protocol.c:4759:client_protocol_cleanup]
> master2: cleaning up state in transport object 0x8e11968
> 2008-04-29 08:36:50 E [client-protocol.c:4809:client_protocol_cleanup]
> master2: forced unwinding frame type(1) op(34) reply=@0x8e4a8e0
> 2008-04-29 08:36:50 E
> [client-protocol.c:4405:client_lookup_cbk] master2: no proper
> reply from server, returning ENOTCONN
> 2008-04-29 08:36:50 W [client-protocol.c:4759:client_protocol_cleanup]
> master1: cleaning up state in transport object 0x8e103b8
> 2008-04-29 08:36:50 E [client-protocol.c:4809:client_protocol_cleanup]
> master1: forced unwinding frame type(1) op(34) reply=@0x8e4a9b0
> 2008-04-29 08:36:50 E
> [client-protocol.c:4405:client_lookup_cbk] master1: no proper
> reply from server, returning ENOTCONN
> 2008-04-29 08:36:50 E [fuse-bridge.c:459:fuse_entry_cbk]
> glusterfs-fuse:
> 362: (34) /etc => -1 (107)
> 2008-04-29 08:36:50 E
> [client-protocol.c:324:client_protocol_xfer] master1:
> transport_submit failed
> 2008-04-29 08:36:50 W
> [client-protocol.c:331:client_protocol_xfer] master2:
> not connected at the moment to submit frame type(1) op(34)
> 2008-04-29 08:36:50 E
> [client-protocol.c:4405:client_lookup_cbk] master2: no proper
> reply from server, returning ENOTCONN
> 2008-04-29 08:37:43 E [tcp-client.c:190:tcp_connect] master2:
> non-blocking
> connect() returned: 113 (No route to host)
> 2008-04-29 08:37:43 E [tcp-client.c:190:tcp_connect] master1:
> non-blocking
> connect() returned: 113 (No route to host)
> 2008-04-29 08:39:13 E [tcp-client.c:190:tcp_connect] master2:
> non-blocking
> connect() returned: 113 (No route to host)
> 2008-04-29 08:39:13 E [tcp-client.c:190:tcp_connect] master1:
> non-blocking
> connect() returned: 113 (No route to host)
>
> Version:
> glusterfs 1.3.8pre6 built on Apr 28 2008 21:20:10 Repository
> revision: glusterfs--mainline--2.5--patch-748
>
> Config file on server1 and server2:
> -------------------------
> volume storage1
> type storage/posix # POSIX FS translator
> option directory / # Export this directory
> end-volume
> #
> volume brick-ns
> type storage/posix
> option directory /ns
> end-volume
> #
> volume server
> type protocol/server
> option transport-type tcp/server # For TCP/IP transport
> subvolumes storage1
> option auth.ip.storage1.allow 192.168.20.* # Allow access
> to "storage1"
> volume
> end-volume
> -------------------------
>
> Config file on client:
> -------------------------
> volume master1
> type protocol/client
> option transport-type tcp/client # for TCP/IP transport
> option remote-host 192.168.20.140 # IP address of the
> remote brick
> option transport-timeout 5
> option remote-subvolume storage1 # name of the remote volume
> end-volume
>
> volume master2
> type protocol/client
> option transport-type tcp/client # for TCP/IP transport
> option remote-host 192.168.20.141 # IP address of the
> remote brick
> option transport-timeout 5
> option remote-subvolume storage1 # name of the remote volume
> end-volume
>
> volume data-afr
> type cluster/afr
> subvolumes master1 master2
> end-volume
> -------------------------
>
>
> > Gerry, Christopher,
> >
> > Here is what I tried to do. Two servers, one client, simple
> setup, afr
> > on the client side. I did "ls" on client mount point, it
> works, now I
> > do "ifconfig eth0 down"
> > on the server, next I do "ls" on client, it hangs for 10
> secs (timeout
> > value) and fails over and starts working again without any problem.
> >
> > I guess few users are facing the problem you guys are facing.
> > Can you give us your setup details and mention the exact steps to
> > reproduce. Also try to come up with minimal config details
> which can
> > still reproduce the problem
> >
> > Thanks!
> > Krishna
> >
> > On Sat, Apr 26, 2008 at 7:01 AM, Christopher Hawkins
> > <chawkins at veracitynetworks.com> wrote:
> > > I am having the same issue. I'm working on a diskless
> node cluster
> > > and figured the issue was related to that since AFR
> seems to fail
> > > over nicely for everyone else...
> > > But it seems I am not alone, so what can I do to help
> troubleshoot?
> > >
> > > I have two servers exporting a brick each, and a client mounting
> > > them both with AFR and no unify. Transport timeout
> settings don't
> > > seem to make a difference - client is just hung if I
> power off or
> > > just stop glusterfsd. There is nothing logged on the server side.
> > > I'll use a usb thumb drive for client side logging since
> > any logs in
> > > the ramdisk obviously disappear after the reboot which
> > fixes the hang...
> > > If I get any insight from this I'll report it asap.
> > >
> > > Thanks,
> > > Chris
> > >
> > >
> > >
>
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>
More information about the Gluster-devel
mailing list