[Gluster-devel] Random errors with "Transport endpoint is not connected"
Anand Avati
avati at zresearch.com
Sat Jun 23 19:47:17 UTC 2007
Constantin,
The random 'disconnection' is a fixed issue and the fix is available in
the checkout with 'tla get glusterfs--mainline--2.4'.
W.R.T to running databases over glusterfs, we have not performed
extensive analysis of this scenario. As a crude analysis, i would presume
databases would open the files with O_DIRECTIO which disables all
performance enhancements.
I am sure glusterfs is pretty much untuned for running databases over it
at this stage, though it should 'work'. With some feedback from the users
who use glusterfs for databases it would be possible to tune it further to
give best performance.
GlusterFS aims at becoming 'best suited' for all types of applications,
since it is easy with the translator design (you could load an alternative
translator which has code tuned for delivering best performance for that
application) and database is definitely one of the appliations we are
looking at in the long run.
thanks!
avati
2007/6/24, Constantin Teodorescu <teo at flex.ro>:
>
> Hi all,
> first of all, glusterfs is a nice idea, liked it a lot, but it need to
> be a rock-solid product so I made some tests , hope that this bug-report
> will help you.
>
> Configuration:
> 3 identical servers and a client, connected with TCP/IP
> Servers: Mandrake 9.x (compiled, installed without problem) , client
> CentOS-5 x86_64, fuse 2.6.5 compiled, installed from source, everything
> went OK.
>
> Those 3 servers provide 3 simple bricks joined at the client in full
> mirror ( afr x 3) configuration + read-ahead & writebehind translators.
> I made a PostgreSQL tablespace (zone) on the mounted /mnt/gfs and copied
> there a 50 Mb table then stress-it with various operations.
> 10 read and full updates on every row in the table succeeded.
>
> After a while, a simple "vacuum full analyze" gave the error:
> glu=# vacuum full analyze;
> ERROR: could not read block 43155 of relation
> 527933664/527933665/527933666: Transport endpoint is not connected
>
> I repeated the tests many times, after 2,3 minutes of operation, I got
> the same error, in another place but mostly in WRITE operations.
> I deleted the whole mounted client disk and rebuild it with STRIPE
> option instead of AFR translator.
> The behaviour is the same ... after a couple of succeded operations, I
> got a failure.
>
> That were the facts, now ... the logs and configuration files.
>
> The client debug log shows this errors:
> [1:48:22] [CRITICAL/client-protocol.c:218/call_bail()]
> client/protocol:bailing transport
> [1:48:22] [DEBUG/tcp.c:123/cont_hand()] tcp:forcing poll/read/write to
> break on blocked socket (if any)
> [1:48:22] [CRITICAL/client-protocol.c:218/call_bail()]
> client/protocol:bailing transport
> [1:48:22] [ERROR/common-utils.c:110/full_rwv()] libglusterfs:full_rwv:
> 6689 bytes r/w instead of 8539 (Broken pipe)
> [1:48:22] [ERROR/client-protocol.c:204/client_protocol_xfer()]
> protocol/client:transport_submit failed
> [1:48:22] [DEBUG/tcp.c:123/cont_hand()] tcp:forcing poll/read/write to
> break on blocked socket (if any)
> [1:48:22] [CRITICAL/client-protocol.c:218/call_bail()]
> client/protocol:bailing transport
> ...
> ...
> [1:48:22] [DEBUG/tcp.c:123/cont_hand()] tcp:forcing poll/read/write to
> break on blocked socket (if any)
> [1:48:22] [CRITICAL/client-protocol.c:218/call_bail()]
> client/protocol:bailing transport
> [1:48:22] [DEBUG/tcp.c:123/cont_hand()] tcp:forcing poll/read/write to
> break on blocked socket (if any)
> [1:48:22] [DEBUG/client-protocol.c:2708/client_protocol_interpret()]
> protocol/client:frame not found for blk with callid: 139893
> [1:48:22] [DEBUG/client-protocol.c:2605/client_protocol_cleanup()]
> protocol/client:cleaning up state in transport object 0x16595730
> [1:48:22] [CRITICAL/tcp.c:81/tcp_disconnect()] transport/tcp:client1:
> connection to server disconnected
> [1:48:22] [CRITICAL/common-utils.c:215/gf_print_trace()]
> debug-backtrace:Got signal (11), printing backtrace
> [1:48:22] [CRITICAL/common-utils.c:217/gf_print_trace()]
> debug-backtrace:/usr/lib/libglusterfs.so.0(gf_print_trace+0x21)
> [0x2aaaaaccf4a1]
> [1:48:22] [CRITICAL/common-utils.c:217/gf_print_trace()]
> debug-backtrace:/lib64/libc.so.6 [0x2aaaab53b070]
> [1:48:22] [CRITICAL/common-utils.c:217/gf_print_trace()]
> debug-backtrace:/usr/lib/glusterfs/1.3.0-pre4/xlator/performance/read-
> ahead.so(ra_frame_return+0x142)
> [0x2aaaac4da2a2]
> [1:48:22] [CRITICAL/common-utils.c:217/gf_print_trace()]
> debug-backtrace:/usr/lib/glusterfs/1.3.0-pre4/xlator/performance/read-
> ahead.so
> [0x2aaaac4d9daa]
> [1:48:22] [CRITICAL/common-utils.c:217/gf_print_trace()]
> debug-backtrace:[glusterfs] [0x40910b]
> [1:48:22] [CRITICAL/common-utils.c:217/gf_print_trace()]
> debug-backtrace:/usr/lib64/libfuse.so.2 [0x2aaaaaee3059]
> [1:48:22] [CRITICAL/common-utils.c:217/gf_print_trace()]
> debug-backtrace:[glusterfs] [0x402f29]
> [1:48:22] [CRITICAL/common-utils.c:217/gf_print_trace()]
> debug-backtrace:/usr/lib/libglusterfs.so.0(sys_epoll_iteration+0xd4)
> [0x2aaaaacd0ef4]
> [1:48:22] [CRITICAL/common-utils.c:217/gf_print_trace()]
> debug-backtrace:[glusterfs] [0x402898]
> [1:48:22] [CRITICAL/common-utils.c:217/gf_print_trace()]
> debug-backtrace:/lib64/libc.so.6(__libc_start_main+0xf4) [0x2aaaab5288a4]
> [1:48:22] [CRITICAL/common-utils.c:217/gf_print_trace()]
> debug-backtrace:[glusterfs] [0x4025f9]
>
> The server configuration files are like that :
> -------------------
> volume brick
> type storage/posix
> option directory /var/gldata
> end-volume
>
> volume server
> type protocol/server
> option transport-type tcp/server
> option listen-port 6996
> option bind-address 29.11.276.x
> subvolumes brick
> option auth.ip.brick.allow *
> end-volume
> -------------------
>
> The client configuration file is :
> -------------------
> volume clientX #{1,2,3}
> type protocol/client
> option transport-type tcp/client
> option remote-host A.B.C.X
> option remote-port 6996
> option remote-subvolume brick
> end-volume
>
>
> ### Add AFR feature to brick
> volume afr
> type cluster/afr
> subvolumes client1 client2 client3
> option replicate *:3 # All files 3 copies
> end-volume
>
> #volume stripe
> # type cluster/stripe
> # subvolumes client1 client2 client3
> # option block-size *:256kB
> #end-volume
>
> #volume trace
> # type debug/trace
> # subvolumes afr
> # option debug on
> #end-volume
>
> volume writebehind
> type performance/write-behind
> option aggregate-size 131072 # aggregate block size in bytes
> subvolumes afr
> end-volume
>
> volume readahead
> type performance/read-ahead
> option page-size 131072 ### size in bytes
> option page-count 16 ### page-size x page-count is the amount of
> read-ahead data per file
> subvolumes writebehind
> end-volume
> -------------------
>
>
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>
--
Anand V. Avati
More information about the Gluster-devel
mailing list