[Gluster-devel] AFR setup with Virtual Servers crashes
Urban Loesch
ul at enas.net
Thu May 10 10:39:20 UTC 2007
Hi Avati,
thanks for your fast answer.
I use the version glusterfs-1.3.0-pre3 downloaded form your server
(http://ftp.zresearch.com/pub/gluster/glusterfs/1.3-pre/).
I will try the latest version from TLA today afternoon and let you know
what happens.
Here's the backtrace from the core dump
# gdb glusterfsd -c core.15160
..
Core was generated by `glusterfsd --no-daemon --log-file=/dev/stdout
--log-level=DEBUG'.
Program terminated with signal 11, Segmentation fault.
#0 0xb75d8fd3 in posix_locks_flush () from
/usr/lib/glusterfs/1.3.0-pre3/xlator/features/posix-locks.so
(gdb) bt
#0 0xb75d8fd3 in posix_locks_flush () from
/usr/lib/glusterfs/1.3.0-pre3/xlator/features/posix-locks.so
#1 0xb75d1192 in fop_flush () from
/usr/lib/glusterfs/1.3.0-pre3/xlator/protocol/server.so
#2 0xb75cded7 in proto_srv_notify () from
/usr/lib/glusterfs/1.3.0-pre3/xlator/protocol/server.so
#3 0xb7f54ecd in transport_notify (this=0x804b1a0, event=1) at
transport.c:148
#4 0xb7f55b79 in sys_epoll_iteration (ctx=0xbfbc2ff0) at epoll.c:53
#5 0xb7f54f7d in poll_iteration (ctx=0xbfbc2ff0) at transport.c:251
#6 0x0804924e in main ()
Yes it is reproducible. It happens every time when I try to start my
virtual server.
Thanks
Urban
Anand Avati wrote:
> Urban,
> which version of glusterfs are you using? if it is from TLA checkout
> what is the patchset number?
>
> you have a core dump generated from the segfault, can you please get a
> backtrace from it? (gdb glusterfsd -c core.<pid> or gdb glusterfsd -c
> core, type 'bt' command and paste the output) please.
>
> is this easily reproducible? have you checked with the latest TLA
> checkout?
>
> thanks,
> avati
>
> 2007/5/10, Urban Loesch <ul at enas.net>:
>> Hi,
>>
>> I'm new to this list.
>> First: sorry for my bad english.
>>
>> I was searching for some easy and transparent Clusterfilesystem with
>> failover feature and I found on Wikipedia the GlusterFS project.
>> It's a nice project and tried it on my test environment. I thought when
>> it works good I use it in production too.
>>
>> A very nice feature for me is the AFR setup. So I can replicate all the
>> data over 2 Servers in RAID-1 Mode.
>> But it seems that I make something wrong, because the "glusterfsd"
>> crashes on both nodes.
>> But let me explain form the beginning.
>>
>> Here's my setup:
>> Hardware:
>> 2 different servers for storage
>> 1 server as client
>> On top of the server I use a virtual server setup (details
>> http://linux-vserver.org).
>>
>> OS:
>> Debian Sarge with self compiled 2.6.19.2 (uname -r 2.6.19.2-vs2.2.0) and
>> latest stable virtual server patch.
>> glusterfs-1.3.0-pre3.tar.gz
>>
>> What I'm trying to do:
>> - Create a AFR Mirror over the 2 Servers.
>> - Mount the Volume on Server 3 (Client).
>> - Install on the mounted volume the hole virtual Server with Apache,
>> MySql and so on.
>> So I have a full redundant Virtual Server mirrored over two bricks .
>>
>> Here my current confuguration:
>> - Serverconfig on Server 1 (brick)
>>
>> ### Export volume "brick" with the contents of "/home/export" directory.
>> volume brick
>> type storage/posix # POSIX FS translator
>> option directory /gluster # Export this directory
>> end-volume
>>
>> ### File Locking
>> volume locks
>> type features/posix-locks
>> subvolumes brick
>> end-volume
>>
>> ### Add network serving capability to above brick.
>> volume server
>> type protocol/server
>> option transport-type tcp/server # For TCP/IP transport
>> option listen-port 6996 # Default is 6996
>> subvolumes locks
>> option auth.ip.locks.allow * # access to "brick" volume
>> end-volume
>>
>> - Serverconfig on Server 2 (brick-afr)
>> ### Export volume "brick" with the contents of "/home/export" directory.
>> volume brick-afr
>> type storage/posix # POSIX FS translator
>> option directory /gluster-afr # Export this directory
>> end-volume
>>
>> ### File Locking
>> volume locks-afr
>> type features/posix-locks
>> subvolumes brick-afr
>> end-volume
>>
>> ### Add network serving capability to above brick.
>> volume server
>> type protocol/server
>> option transport-type tcp/server # For TCP/IP transport
>> option listen-port 6996 # Default is 6996
>> subvolumes locks-afr
>> option auth.ip.locks-afr.allow * # access to "brick" volume
>> end-volume
>>
>> - Clientconfiguration on Server 3 (
>> ### Add client feature and attach to remote subvolume of server1
>> volume brick
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> option remote-host 192.168.0.1 # IP address of the remote brick
>> option remote-port 6996 # default server port is 6996
>> option remote-subvolume locks # name of the remote volume
>> end-volume
>>
>> ### Add client feature and attach to remote subvolume of brick1
>> volume brick-afr
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> option remote-host 192.168.0.2 # IP address of the remote brick
>> option remote-port 6996 # default server port is 6996
>> option remote-subvolume locks-afr # name of the remote volume
>> end-volume
>>
>> ### Add AFR feature to brick
>> volume afr
>> type cluster/afr
>> subvolumes brick brick-afr
>> option replicate *:2 # All files 2 copies (RAID-1)
>> end-volume
>>
>> ----------------------------------------------------------------------------------------------------------------------
>>
>> I started the two Bricks in debug mode and it starts without problems.
>>
>> - Server1
>> glusterfsd --no-daemon --log-file=/dev/stdout --log-level=DEBUG
>> ....
>> [May 10 11:52:11] [DEBUG/proto-srv.c:2919/init()]
>> protocol/server:protocol/server xlator loaded
>> [May 10 11:52:11] [DEBUG/transport.c:83/transport_load()]
>> libglusterfs/transport:attempt to load type tcp/server
>> [May 10 11:52:11] [DEBUG/transport.c:88/transport_load()]
>> libglusterfs/transport:attempt to load file
>> /usr/lib/glusterfs/1.3.0-pre3/transport/tcp/server.so
>>
>> - Server2
>> glusterfsd --no-daemon --log-file=/dev/stdout --log-level=DEBUG
>> ....
>> [May 10 11:51:44] [DEBUG/proto-srv.c:2919/init()]
>> protocol/server:protocol/server xlator loaded
>> [May 10 11:51:44] [DEBUG/transport.c:83/transport_load()]
>> libglusterfs/transport:attempt to load type tcp/server
>> [May 10 11:51:44] [DEBUG/transport.c:88/transport_load()]
>> libglusterfs/transport:attempt to load file
>> /usr/lib/glusterfs/1.3.0-pre3/transport/tcp/server.so
>> ------------------------------------------------------------------------------------------------------------------------------
>>
>>
>> So far so good.
>>
>> After I mounted the volume on server 3 (client). It mounts without any
>> problems.
>> glusterfs --no-daemon --log-file=/dev/stdout --log-level=DEBUG
>> --spec-file=/etc/glusterfs/glusterfs-client.vol
>> /var/lib/vservers/mastersql
>> ...
>> [May 10 13:59:00] [DEBUG/client-protocol.c:2796/init()]
>> protocol/client:defaulting transport-timeout to 120
>> [May 10 13:59:00] [DEBUG/transport.c:83/transport_load()]
>> libglusterfs/transport:attempt to load type tcp/client
>> [May 10 13:59:00] [DEBUG/transport.c:88/transport_load()]
>> libglusterfs/transport:attempt to load file
>> /usr/lib/glusterfs/1.3.0-pre3/transport/tcp/client.so
>> [May 10 13:59:00] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
>> :try_connect: socket fd = 8
>> [May 10 13:59:00] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
>> :try_connect: finalized on port `1022'
>> [May 10 13:59:00] [DEBUG/tcp-client.c:255/tcp_connect()]
>> tcp/client:connect on 8 in progress (non-blocking)
>> [May 10 13:59:00] [DEBUG/tcp-client.c:293/tcp_connect()]
>> tcp/client:connection on 8 still in progress - try later
>>
>> OK. Nice.
>> A short check on the client:
>> df -HT
>> Filesystem Type Size Used Avail Use% Mounted on
>> /dev/sda1 ext3 13G 2.6G 8.9G 23% /
>> tmpfs tmpfs 1.1G 0 1.1G 0% /lib/init/rw
>> udev tmpfs 11M 46k 11M 1% /dev
>> tmpfs tmpfs 1.1G 0 1.1G 0% /dev/shm
>> glusterfs:24914
>> fuse 9.9G 2.5G 6.9G 27%
>> /var/lib/vservers/mastersql
>>
>> Wow it works. Now I can add, remove or edit files and directories
>> without problems. The file are written to all two bricks without
>> problems. Performance is good too.
>>
>> But then I tried to start my virtual Server (called mastersql).
>> The virtual server not starts and I get the a lot of following debug
>> output on the client:
>>
>> [May 10 14:04:43] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
>> :try_connect: socket fd = 4
>> [May 10 14:04:43] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
>> :try_connect: finalized on port `1023'
>> [May 10 14:04:43] [DEBUG/tcp-client.c:255/tcp_connect()]
>> tcp/client:connect on 4 in progress (non-blocking)
>> [May 10 14:04:43] [DEBUG/tcp-client.c:293/tcp_connect()]
>> tcp/client:connection on 4 still in progress - try later
>> [May 10 14:04:43] [ERROR/client-protocol.c:204/client_protocol_xfer()]
>> protocol/client:transport_submit failed
>> [May 10 14:04:43]
>> [DEBUG/client-protocol.c:2604/client_protocol_cleanup()]
>> protocol/client:cleaning up state in transport object 0x8076cf0
>> [May 10 14:04:43] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp:
>> :try_connect: socket fd = 7
>> [May 10 14:04:43] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp:
>> :try_connect: finalized on port `1022'
>> [May 10 14:04:43] [DEBUG/tcp-client.c:255/tcp_connect()]
>> tcp/client:connect on 7 in progress (non-blocking)
>> [May 10 14:04:43] [DEBUG/tcp-client.c:293/tcp_connect()]
>> tcp/client:connection on 7 still in progress - try later
>> [May 10 14:04:43] [ERROR/client-protocol.c:204/client_protocol_xfer()]
>> protocol/client:transport_submit failed
>> [May 10 14:04:43]
>> [DEBUG/client-protocol.c:2604/client_protocol_cleanup()]
>> protocol/client:cleaning up state in transport object 0x80762d0
>>
>> The two mirrorservers are crashing with the following debug code:
>>
>> [May 10 11:54:26] [DEBUG/tcp-server.c:134/tcp_server_notify()]
>> tcp/server:Registering socket (5) for new transport object of
>> 192.168.0.3
>> [May 10 11:55:22] [DEBUG/proto-srv.c:2418/mop_setvolume()]
>> server-protocol:mop_setvolume: received port = 1022
>> [May 10 11:55:22] [DEBUG/proto-srv.c:2434/mop_setvolume()]
>> server-protocol:mop_setvolume: IP addr = *, received ip addr =
>> 192.168.0.3
>> [May 10 11:55:22] [DEBUG/proto-srv.c:2444/mop_setvolume()]
>> server-protocol:mop_setvolume: accepted client from 192.168.0.3
>>
>> Trying to set: READ Is grantable: READ Inserting: READTrying to set:
>> UNLOCK Is grantable: UNLOCK Conflict with: READTrying to set: WRITE
>> Is grantable: WRITE Inserting: WRITETrying to set: UNLOCK Is
>> grantable: UNLOCK Conflict with: WRITETrying to set: WRITE Is
>> grantable: WRITE Inserting: WRITETrying to set: UNLOCK Is grantable:
>> UNLOCK Conflict with: WRITETrying to set: WRITE Is grantable: WRITE
>> Inserting: WRITE[May 10 12:00:09]
>> [CRITICAL/common-utils.c:215/gf_print_trace()] debug-backtrace:Got
>> signal (11), printing backtrace
>> [May 10 12:00:09] [CRITICAL/common-utils.c:217/gf_print_trace()]
>> debug-backtrace:/usr/lib/libglusterfs.so.0(gf_print_trace+0x2e)
>> [0xb7f53a7e]
>> [May 10 12:00:09] [CRITICAL/common-utils.c:217/gf_print_trace()]
>> debug-backtrace:[0xb7f60420]
>> [May 10 12:00:09] [CRITICAL/common-utils.c:217/gf_print_trace()]
>> debug-backtrace:/usr/lib/glusterfs/1.3.0-pre3/xlator/protocol/server.so
>> [0xb75d1192]
>> [May 10 12:00:09] [CRITICAL/common-utils.c:217/gf_print_trace()]
>> debug-backtrace:/usr/lib/glusterfs/1.3.0-pre3/xlator/protocol/server.so
>> [0xb75cded7]
>> [May 10 12:00:09] [CRITICAL/common-utils.c:217/gf_print_trace()]
>> debug-backtrace:/usr/lib/libglusterfs.so.0(transport_notify+0x1d)
>> [0xb7f54ecd]
>> [May 10 12:00:09] [CRITICAL/common-utils.c:217/gf_print_trace()]
>> debug-backtrace:/usr/lib/libglusterfs.so.0(sys_epoll_iteration+0xe9)
>> [0xb7f55b79]
>> [May 10 12:00:09] [CRITICAL/common-utils.c:217/gf_print_trace()]
>> debug-backtrace:/usr/lib/libglusterfs.so.0(poll_iteration+0x1d)
>> [0xb7f54f7d]
>> [May 10 12:00:09] [CRITICAL/common-utils.c:217/gf_print_trace()]
>> debug-backtrace:glusterfsd [0x804924e]
>> [May 10 12:00:09] [CRITICAL/common-utils.c:217/gf_print_trace()]
>> debug-backtrace:/lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xc8)
>> [0xb7e17ea8]
>> [May 10 12:00:09] [CRITICAL/common-utils.c:217/gf_print_trace()]
>> debug-backtrace:glusterfsd [0x8048c51]
>> Segmentation fault (core dumped)
>>
>> It seems that there are come conflicts with "READ, WRITE, UNLOCK". But
>> I'm not an expert on filesystems an locking features.
>>
>> As you can see the filesystem is just mounted but not connected to the
>> two bricks.
>> df -HT
>> Filesystem Type Size Used Avail Use% Mounted on
>> /dev/sda1 ext3 13G 2.6G 8.9G 23% /
>> tmpfs tmpfs 1.1G 0 1.1G 0% /lib/init/rw
>> udev tmpfs 11M 46k 11M 1% /dev
>> tmpfs tmpfs 1.1G 0 1.1G 0% /dev/shm
>> df: `/var/lib/vservers/mastersql': Transport endpoint is not connected
>>
>> I'm not sure if i make something wrong (configuration) or if it is a
>> bug!
>> Can you experts please help me?
>>
>> If you need any further information or something please let me know.
>>
>> Thanks and regards
>> Urban
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at nongnu.org
>> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>>
>
>
More information about the Gluster-devel
mailing list