[Gluster-users] GlusterFS 3.1.2 crash with NFS

Thu Feb 17 00:06:48 UTC 2011

Anand,

Thanks for quick response, I've described situation that occured with
some more details here
http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=2438 (gdb
output included).

Best regards
Piotr Kandziora

On Thu, Feb 17, 2011 at 12:06 AM, Anand Avati <anand.avati at gmail.com> wrote:
> Can you please file a bug with the gdb backtrace on the core dump file?
> Avati
>
> On Wed, Feb 16, 2011 at 3:03 PM, Piotr Kandziora <raveenpl at gmail.com> wrote:
>>
>> Hi,
>>
>> I am testing GlusterFS 3.1.2 on Ubuntu with NFS support.
>>
>> After a few weeks of error-free working, unfortunately GlusterFS crashed.
>>
>> Maybe this is bug in newest version therefore I am putting here logs
>> to diagnose (maybe some guru from Gluster Team can help here).
>>
>> If more details is needed please let me know.
>>
>> Below logs:
>>
>>
>> [2011-02-16 16:14:09.924988] I
>> [afr-common.c:613:afr_lookup_self_heal_check] test_volume-replicate-2:
>> size differs for /onerepo/50cab8255b4c3a6ee377baf45fc61
>> e7c7cab6b32
>> [2011-02-16 16:14:09.925093] I [afr-common.c:716:afr_lookup_done]
>> test_volume-replicate-2: background  meta-data data self-heal
>> triggered. path: /onerepo/50c
>> ab8255b4c3a6ee377baf45fc61e7c7cab6b32
>> [2011-02-16 16:14:10.35264] I
>> [afr-common.c:613:afr_lookup_self_heal_check] test_volume-replicate-2:
>> size differs for /onerepo/75103af384304fdbffadfa377690f0
>> 344fb532a3
>> [2011-02-16 16:14:10.35350] I [afr-common.c:716:afr_lookup_done]
>> test_volume-replicate-2: background  meta-data data self-heal
>> triggered. path: /onerepo/7510
>> 3af384304fdbffadfa377690f0344fb532a3
>> [2011-02-16 16:14:10.413155] I
>> [afr-common.c:613:afr_lookup_self_heal_check] test_volume-replicate-2:
>> size differs for /onerepo/e70ee7f945ad888426f55e6f0cdf3
>> 9c091241603
>> [2011-02-16 16:14:10.413271] I [afr-common.c:716:afr_lookup_done]
>> test_volume-replicate-2: background  meta-data data self-heal
>> triggered. path: /onerepo/e70
>> ee7f945ad888426f55e6f0cdf39c091241603
>> [2011-02-16 16:14:10.761177] I
>> [afr-common.c:613:afr_lookup_self_heal_check] test_volume-replicate-2:
>> size differs for /onerepo/0eaa253ac3e9dd9b5656828a4f7e7
>> 486d6759850
>> [2011-02-16 16:14:10.761288] I [afr-common.c:716:afr_lookup_done]
>> test_volume-replicate-2: background  meta-data data self-heal
>> triggered. path: /onerepo/0ea
>> a253ac3e9dd9b5656828a4f7e7486d6759850
>> [2011-02-16 16:14:12.354160] I
>> [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
>> test_volume-replicate-2: background  meta-data data self-heal compl
>> eted on /onerepo/75103af384304fdbffadfa377690f0344fb532a3
>> [2011-02-16 16:14:12.797746] I
>> [afr-self-heal-common.c:1526:afr_self_heal_completion_cbk]
>> test_volume-replicate-2: background  meta-data data self-heal compl
>> eted on /onerepo/e70ee7f945ad888426f55e6f0cdf39c091241603
>> [2011-02-16 16:23:31.949740] E [event.c:722:event_select_on_epoll]
>> epoll: index not found for fd=-1 (idx_hint=0)
>> [2011-02-16 16:24:56.959805] E
>> [rpcsvc.c:1693:nfs_rpcsvc_submit_generic] nfsrpc: Failed to submit
>> message
>> [2011-02-16 16:24:56.959940] E [nfs3.c:522:nfs3svc_submit_reply]
>> nfs-nfsv3: Reply submission failed
>> [2011-02-16 16:24:56.961438] E
>> [rpcsvc.c:1693:nfs_rpcsvc_submit_generic] nfsrpc: Failed to submit
>> message
>> [2011-02-16 16:24:56.961553] E [nfs3.c:522:nfs3svc_submit_reply]
>> nfs-nfsv3: Reply submission failed
>> [2011-02-16 16:24:56.974049] E
>> [rpcsvc.c:1693:nfs_rpcsvc_submit_generic] nfsrpc: Failed to submit
>> message
>> [2011-02-16 16:24:56.974116] E [nfs3.c:522:nfs3svc_submit_reply]
>> nfs-nfsv3: Reply submission failed
>> [2011-02-16 16:24:57.46931] E [event.c:722:event_select_on_epoll]
>> epoll: index not found for fd=-1 (idx_hint=0)
>> [2011-02-16 16:24:57.47068] E [event.c:722:event_select_on_epoll]
>> epoll: index not found for fd=-1 (idx_hint=-1)
>> [2011-02-16 16:24:57.47133] E [event.c:722:event_select_on_epoll]
>> epoll: index not found for fd=-1 (idx_hint=-1)
>> [2011-02-16 16:24:57.56965] E [event.c:722:event_select_on_epoll]
>> epoll: index not found for fd=-1 (idx_hint=-1)
>> [2011-02-16 16:24:57.57052] E [event.c:722:event_select_on_epoll]
>> epoll: index not found for fd=-1 (idx_hint=-1)
>> [2011-02-16 16:24:57.60621] E [event.c:722:event_select_on_epoll]
>> epoll: index not found for fd=-1 (idx_hint=-1)
>> [2011-02-16 16:24:57.60708] E [event.c:722:event_select_on_epoll]
>> epoll: index not found for fd=-1 (idx_hint=-1)
>> [2011-02-16 16:24:57.66769] E [event.c:722:event_select_on_epoll]
>> epoll: index not found for fd=-1 (idx_hint=-1)
>> pending frames:
>>
>> patchset: v3.1.1-64-gf2a067c
>> signal received: 11
>> time of crash: 2011-02-16 16:24:57
>> configuration details:
>> argp 1
>> backtrace 1
>> dlfcn 1
>> fdatasync 1
>> libpthread 1
>> llistxattr 1
>> setfsid 1
>> spinlock 1
>> epoll.h 1
>> xattr.h 1
>> st_atim.tv_nsec 1
>> package-string: glusterfs 3.1.2
>> /lib/libc.so.6(+0x33af0)[0x7f002031daf0]
>>
>> /usr/lib/glusterfs/3.1.2/xlator/nfs/server.so(nfs_rpcsvc_submit_vectors+0x160)[0x7f001d89ef70]
>>
>> /usr/lib/glusterfs/3.1.2/xlator/nfs/server.so(nfs3svc_submit_vector_reply+0xa4)[0x7f001d885b14]
>>
>> /usr/lib/glusterfs/3.1.2/xlator/nfs/server.so(nfs3_read_reply+0xf5)[0x7f001d886915]
>>
>> /usr/lib/glusterfs/3.1.2/xlator/nfs/server.so(nfs3svc_read_cbk+0x8a)[0x7f001d8898aa]
>>
>> /usr/lib/glusterfs/3.1.2/xlator/nfs/server.so(nfs_fop_readv_cbk+0x53)[0x7f001d87baa3]
>>
>> /usr/lib/glusterfs/3.1.2/xlator/debug/io-stats.so(io_stats_readv_cbk+0x16f)[0x7f001dac4aef]
>>
>> /usr/lib/glusterfs/3.1.2/xlator/performance/quick-read.so(qr_readv_cbk+0xa6)[0x7f001dccd0a6]
>>
>> /usr/lib/glusterfs/3.1.2/xlator/performance/io-cache.so(ioc_frame_return+0x366)[0x7f001dedf356]
>>
>> /usr/lib/glusterfs/3.1.2/xlator/performance/io-cache.so(ioc_waitq_return+0x1c)[0x7f001dedf5ac]
>>
>> /usr/lib/glusterfs/3.1.2/xlator/performance/io-cache.so(ioc_fault_cbk+0x259)[0x7f001dee0a59]
>>
>> /usr/lib/glusterfs/3.1.2/xlator/performance/read-ahead.so(ra_readv_disabled_cbk+0x9b)[0x7f001e0e871b]
>>
>> /usr/lib/glusterfs/3.1.2/xlator/performance/write-behind.so(wb_readv_cbk+0xab)[0x7f001e2f633b]
>>
>> /usr/lib/glusterfs/3.1.2/xlator/cluster/distribute.so(dht_readv_cbk+0xd3)[0x7f001e50d793]
>>
>> /usr/lib/glusterfs/3.1.2/xlator/cluster/replicate.so(afr_readv_cbk+0x372)[0x7f001e73bcf2]
>>
>> /usr/lib/glusterfs/3.1.2/xlator/protocol/client.so(client3_1_readv_cbk+0x367)[0x7f001e995567]
>> /usr/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)[0x7f0020cb1c15]
>> /usr/lib/libgfrpc.so.0(rpc_clnt_notify+0xc9)[0x7f0020cb1e69]
>> /usr/lib/libgfrpc.so.0(rpc_transport_notify+0x2d)[0x7f0020cad02d]
>>
>> /usr/lib/glusterfs/3.1.2/rpc-transport/socket.so(socket_event_poll_in+0x34)[0x7f0017b3f334]
>>
>> /usr/lib/glusterfs/3.1.2/rpc-transport/socket.so(socket_event_handler+0xb3)[0x7f0017b3f403]
>> /usr/lib/libglusterfs.so.0(+0x38592)[0x7f0020ef1592]
>> /usr/sbin/glusterfs(main+0x247)[0x405597]
>> /lib/libc.so.6(__libc_start_main+0xfd)[0x7f0020308c4d]
>> /usr/sbin/glusterfs[0x4032a9]
>> ---------
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
>