[Gluster-devel] Segfault in read_ahead in 1.3.0_pre4
Brent A Nelson
brent at phys.ufl.edu
Thu May 24 15:30:16 UTC 2007
I was getting the same behavior in my testing. After I reported it, the
readahead crash was quickly patched, but the random disconnect is still
very much a mystery...
I noticed that you are using stat-prefetch; have you encountered any
issues? I was finding that du's on complex directories could return
abnormal results and/or errors, so it seemed that heavy metadata queries
were occassionally glitchy. Without stat-prefetch, it's been fine. If
you've been having good luck with it, maybe I should try again.
Thanks,
Brent
On Thu, 24 May 2007, Anand Avati wrote:
> Harris,
> this bug was fixed a few days back and the fix is available in the
> glusterfs--mainline--2.4 repository latest checkout.
>
> thanks,
> avati
>
> 2007/5/24, Harris Landgarten <harrisl at lhjonline.com>:
>> I am running glusterfs in a very basic configuration on Amazon EC
>> instances. I have a 2 brick cluster and 2 clients. One of the clients is
>> running Zimbra and I am using the cluster as secondary storage for the mail
>> store. I have repeatedly tried to reindex a mailbox with 31000 items. Most
>> of the email is on the cluster. The entire process takes about 2 hours.
>> Part way through I get at least one TCP Disconnect which seems random. With
>> read_ahead enabled on the client, the disconnect results in a segfault and
>> the mount point disappears. When I disabled read_ahead on the client, the
>> disconnect was recovered from, and the process completed. This is the
>> backtrace from the read_ahead segfault:
>>
>> [May 23 20:00:37] [CRITICAL/client-protocol.c:218/call_bail()]
>> client/protocol:bailing transport
>> [May 23 20:00:37] [DEBUG/tcp.c:123/cont_hand()] tcp:forcing poll/read/write
>> to break on blocked socket (if any)
>> [May 23 20:00:37] [ERROR/common-utils.c:55/full_rw()] libglusterfs:full_rw:
>> 0 bytes r/w instead of 113 (errno=115)
>> [May 23 20:00:37] [DEBUG/protocol.c:244/gf_block_unserialize_transport()]
>> libglusterfs/protocol:gf_block_unserialize_transport: full_read of header
>> failed
>> [May 23 20:00:37] [DEBUG/client-protocol.c:2605/client_protocol_cleanup()]
>> protocol/client:cleaning up state in transport object 0x8078a08
>> [May 23 20:00:37] [CRITICAL/common-utils.c:215/gf_print_trace()]
>> debug-backtrace:Got signal (11), printing backtrace
>> [May 23 20:00:37] [CRITICAL/common-utils.c:217/gf_print_trace()]
>> debug-backtrace:/usr/lib/libglusterfs.so.0(gf_print_trace+0x2d)
>> [0xb7f2584d]
>> [May 23 20:00:37] [CRITICAL/common-utils.c:217/gf_print_trace()]
>> debug-backtrace:[0xbfffe420]
>> [May 23 20:00:37] [CRITICAL/common-utils.c:217/gf_print_trace()]
>> debug-backtrace:/usr/lib/glusterfs/1.3.0-pre4/xlator/performance/read-ahead.so(ra_page_error+0x47)
>> [0xb755e587]
>> [May 23 20:00:37] [CRITICAL/common-utils.c:217/gf_print_trace()]
>> debug-backtrace:/usr/lib/glusterfs/1.3.0-pre4/xlator/performance/read-ahead.so
>> [0xb755ecf0]
>> [May 23 20:00:37] [CRITICAL/common-utils.c:217/gf_print_trace()]
>> debug-backtrace:/usr/lib/glusterfs/1.3.0-pre4/xlator/performance/write-behind.so
>> [0xb7561809]
>> [May 23 20:00:37] [CRITICAL/common-utils.c:217/gf_print_trace()]
>> debug-backtrace:/usr/lib/glusterfs/1.3.0-pre4/xlator/cluster/unify.so
>> [0xb7564919]
>> [May 23 20:00:37] [CRITICAL/common-utils.c:217/gf_print_trace()]
>> debug-backtrace:/usr/lib/glusterfs/1.3.0-pre4/xlator/protocol/client.so
>> [0xb756d17b]
>> [May 23 20:00:37] [CRITICAL/common-utils.c:217/gf_print_trace()]
>> debug-backtrace:/usr/lib/glusterfs/1.3.0-pre4/xlator/protocol/client.so
>> [0xb75717a5]
>> [May 23 20:00:37] [CRITICAL/common-utils.c:217/gf_print_trace()]
>> debug-backtrace:/usr/lib/libglusterfs.so.0(transport_notify+0x1d)
>> [0xb7f26d2d]
>> [May 23 20:00:37] [CRITICAL/common-utils.c:217/gf_print_trace()]
>> debug-backtrace:/usr/lib/libglusterfs.so.0(sys_epoll_iteration+0xe7)
>> [0xb7f279d7]
>> [May 23 20:00:37] [CRITICAL/common-utils.c:217/gf_print_trace()]
>> debug-backtrace:/usr/lib/libglusterfs.so.0(poll_iteration+0x1d)
>> [0xb7f26ddd]
>> [May 23 20:00:37] [CRITICAL/common-utils.c:217/gf_print_trace()]
>> debug-backtrace:glusterfs [0x804a15e]
>> [May 23 20:00:37] [CRITICAL/common-utils.c:217/gf_print_trace()]
>> debug-backtrace:/lib/libc.so.6(__libc_start_main+0xdc) [0xb7dca8cc]
>> [May 23 20:00:37] [CRITICAL/common-utils.c:217/gf_print_trace()]
>> debug-backtrace:glusterfs [0x8049e71]
>> Segmentation fault (core dumped)
>>
>> This is a sample of the debug log with read_ahead turned off
>>
>> [May 24 05:35:05] [CRITICAL/client-protocol.c:218/call_bail()]
>> client/protocol:bailing transport
>> [May 24 05:35:05] [DEBUG/tcp.c:123/cont_hand()] tcp:forcing poll/read/write
>> to break on blocked socket (if any)
>> [May 24 05:35:05] [ERROR/common-utils.c:55/full_rw()] libglusterfs:full_rw:
>> 0 bytes r/w instead of 113 (errno=115)
>> [May 24 05:35:05] [DEBUG/protocol.c:244/gf_block_unserialize_transport()]
>> libglusterfs/protocol:gf_block_unserialize_transport: full_read of header
>> failed
>> [May 24 05:35:05] [DEBUG/client-protocol.c:2605/client_protocol_cleanup()]
>> protocol/client:cleaning up state in transport object 0x80783d0
>> [May 24 05:35:05] [CRITICAL/tcp.c:81/tcp_disconnect()]
>> transport/tcp:client1: connection to server disconnected
>> [May 24 05:35:05] [DEBUG/tcp-client.c:180/tcp_connect()] transport: tcp:
>> :try_connect: socket fd = 4
>> [May 24 05:35:05] [DEBUG/tcp-client.c:202/tcp_connect()] transport: tcp:
>> :try_connect: finalized on port `1022'
>> [May 24 05:35:05] [DEBUG/tcp-client.c:226/tcp_connect()]
>> tcp/client:try_connect: defaulting remote-port to 6996
>> [May 24 05:35:05] [DEBUG/tcp-client.c:262/tcp_connect()] tcp/client:connect
>> on 4 in progress (non-blocking)
>> [May 24 05:35:05] [DEBUG/tcp-client.c:301/tcp_connect()]
>> tcp/client:connection on 4 still in progress - try later
>> [May 24 05:35:05] [ERROR/client-protocol.c:204/client_protocol_xfer()]
>> protocol/client:transport_submit failed
>> [May 24 05:35:05] [DEBUG/client-protocol.c:2605/client_protocol_cleanup()]
>> protocol/client:cleaning up state in transport object 0x80783d0
>> [May 24 05:35:11] [DEBUG/tcp-client.c:310/tcp_connect()]
>> tcp/client:connection on 4 success, attempting to handshake
>> [May 24 05:35:11] [DEBUG/tcp-client.c:54/do_handshake()]
>> transport/tcp-client:dictionary length = 50
>> [May 24 07:20:10] [DEBUG/stat-prefetch.c:58/stat_prefetch_cache_flush()]
>> stat-prefetch:flush on: /
>> [May 24 07:20:20] [DEBUG/stat-prefetch.c:58/stat_prefetch_cache_flush()]
>> stat-prefetch:flush on: /backups/sessions
>> [May 24 07:57:12] [CRITICAL/client-protocol.c:218/call_bail()]
>> client/protocol:bailing transport
>> [May 24 07:57:12] [DEBUG/tcp.c:123/cont_hand()] tcp:forcing poll/read/write
>> to break on blocked socket (if any)
>> [May 24 07:57:12] [ERROR/common-utils.c:55/full_rw()] libglusterfs:full_rw:
>> 0 bytes r/w instead of 113 (errno=115)
>> [May 24 07:57:12] [DEBUG/protocol.c:244/gf_block_unserialize_transport()]
>> libglusterfs/protocol:gf_block_unserialize_transport: full_read of header
>> failed
>> [May 24 07:57:12] [DEBUG/client-protocol.c:2605/client_protocol_cleanup()]
>> protocol/client:cleaning up state in transport object 0x80783d0
>> [May 24 07:57:12] [CRITICAL/tcp.c:81/tcp_disconnect()]
>> transport/tcp:client1: connection to server disconnected
>> [May 24 07:57:12] [DEBUG/tcp-client.c:180/tcp_connect()] transport: tcp:
>> :try_connect: socket fd = 4
>> [May 24 07:57:12] [DEBUG/tcp-client.c:202/tcp_connect()] transport: tcp:
>> :try_connect: finalized on port `1023'
>> [May 24 07:57:12] [DEBUG/tcp-client.c:226/tcp_connect()]
>> tcp/client:try_connect: defaulting remote-port to 6996
>> [May 24 07:57:12] [DEBUG/tcp-client.c:262/tcp_connect()] tcp/client:connect
>> on 4 in progress (non-blocking)
>> [May 24 07:57:12] [DEBUG/tcp-client.c:301/tcp_connect()]
>> tcp/client:connection on 4 still in progress - try later
>> [May 24 07:57:12] [ERROR/client-protocol.c:204/client_protocol_xfer()]
>> protocol/client:transport_submit failed
>> [May 24 07:57:12] [DEBUG/client-protocol.c:2605/client_protocol_cleanup()]
>> protocol/client:cleaning up state in transport object 0x80783d0
>> [May 24 07:57:12] [DEBUG/tcp-client.c:310/tcp_connect()]
>> tcp/client:connection on 4 success, attempting to handshake
>> [May 24 07:57:12] [DEBUG/tcp-client.c:54/do_handshake()]
>> transport/tcp-client:dictionary length = 50
>>
>> This is the client config with read_ahead
>>
>> ### Add client feature and attach to remote subvolume
>> volume client1
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option ibv-send-work-request-size 131072
>> # option ibv-send-work-request-count 64
>> # option ibv-recv-work-request-size 131072
>> # option ibv-recv-work-request-count 64
>> # option transport-type ib-sdp/client # for Infiniband transport
>> # option transport-type ib-verbs/client # for ib-verbs transport
>> option remote-host xx.xxx.xx.xxx # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>>
>> # option transport-timeout 30 # seconds to wait for a reply
>> # from server for each request
>> option remote-subvolume brick # name of the remote volume
>> end-volume
>>
>> ### Add client feature and attach to remote subvolume
>> volume client2
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option ibv-send-work-request-size 131072
>> # option ibv-send-work-request-count 64
>> # option ibv-recv-work-request-size 131072
>> # option ibv-recv-work-request-count 64
>> # option transport-type ib-sdp/client # for Infiniband transport
>> # option transport-type ib-verbs/client # for ib-verbs transport
>> option remote-host yy.yyy.yy.yyy # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>>
>> # option transport-timeout 30 # seconds to wait for a reply
>> # from server for each request
>> option remote-subvolume brick # name of the remote volume
>> end-volume
>>
>> volume bricks
>> type cluster/unify
>> subvolumes client1 client2
>> option scheduler alu
>> option alu.limits.min-free-disk 4GB
>> option alu.limits.max-open-files 10000
>>
>> option alu.order disk-usage:read-usage:write-usage:open-files-usage
>> option alu.disk-usage.entry-threshold 2GB
>> option alu.disk-usage.exit-threshold 10GB
>> option alu.open-files-usage.entry-threshold 1024
>> option alu.open-files-usage.exit-threshold 32
>> option alu.stat-refresh.interval 10sec
>>
>> end-volume
>> #
>>
>> ### Add writeback feature
>> volume writeback
>> type performance/write-behind
>> option aggregate-size 131072 # unit in bytes
>> subvolumes bricks
>> end-volume
>>
>> ### Add readahead feature
>> volume readahead
>> type performance/read-ahead
>> option page-size 65536 # unit in bytes
>> option page-count 16 # cache per file = (page-count x page-size)
>> subvolumes writeback
>> end-volume
>>
>> ### Add stat-prefetch feature
>> ### If you are not concerned about performance of interactive commands
>> ### like "ls -l", you wouln't need this translator.
>> volume statprefetch
>> type performance/stat-prefetch
>> option cache-seconds 2 # timeout for stat cache
>> subvolumes readahead
>> end-volume
>>
>> This is the brick config:
>>
>> ### Export volume "brick" with the contents of "/home/export" directory.
>> volume brick
>> type storage/posix # POSIX FS translator
>> option directory /mnt/export # Export this directory
>> end-volume
>>
>> volume iothreads
>> type performance/io-threads
>> option thread-count 8
>> subvolumes brick
>> end-volume
>>
>> ### Add network serving capability to above brick.
>> volume server
>> type protocol/server
>> option transport-type tcp/server # For TCP/IP transport
>> # option ibv-send-work-request-size 131072
>> # option ibv-send-work-request-count 64
>> # option ibv-recv-work-request-size 131072
>> # option ibv-recv-work-request-count 64
>> # option transport-type ib-sdp/server # For Infiniband transport
>> # option transport-type ib-verbs/server # For ib-verbs transport
>> # option bind-address 192.168.1.10 # Default is to listen on all
>> interfaces
>> # option listen-port 6996 # Default is 6996
>> # option client-volume-filename /etc/glusterfs/glusterfs-client.vol
>> subvolumes iothreads
>> # NOTE: Access to any volume through protocol/server is denied by
>> # default. You need to explicitly grant access through # "auth"
>> # option.
>> option auth.ip.brick.allow * # Allow access to "brick" volume
>> end-volume
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at nongnu.org
>> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>>
>
>
> --
> Anand V. Avati
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>
More information about the Gluster-devel
mailing list