[Gluster-devel] Seg 11 with mainline-2.5 patch 185

Thu Jun 21 21:26:10 UTC 2007

Linux (ubuntu dapper) 2.6.15. Fuse 2.6.5. Glusterfs mainline-2.5 patch 185.

glusterfs crashing with signal 11 on 'ls'. Was working for a few hours 
and now it crashes on any 'ls' of the mount. Odd thing, restarting 
glusterfsd and remounting fresh and restarting rsync it seems to be 
working ok (no errors). Running 6 rsyncs in parallel. As long as I dont 
do an 'ls' on the mount.

Things I did (mostly in sequence) leading up to this:
   * I was rsyncing for a few hours (rsync'd 75G).
   * Had aborted and restarted rsync a number of times.
   * Had rebooted once.
After one of these it started crashing. Dunno which cuz they were all 
done about the same time:
   * Tweaked network in sysctl.conf. A number of thing. Reset them back 
and reboot but still crashing.
   * Ran bonnie++ and aborted it.
   * Added write-behind xlator to glusterfs-client.vol. Removed when it 
wouldnt work with 'notify missing' error.

========================================
GDB glusterfs -c /core.5194:
Core was generated by 
`[glusterfs]                                                       '.
Program terminated with signal 11, Segmentation fault.

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /lib/libglusterfs.so.0...done.
Loaded symbols for /lib/libglusterfs.so.0
Reading symbols from /usr/local/lib/libfuse.so.2...done.
Loaded symbols for /usr/local/lib/libfuse.so.2
Reading symbols from /lib/tls/i686/cmov/libdl.so.2...done.
Loaded symbols for /lib/tls/i686/cmov/libdl.so.2
Reading symbols from /lib/tls/i686/cmov/libpthread.so.0...done.
Loaded symbols for /lib/tls/i686/cmov/libpthread.so.0
Reading symbols from /lib/tls/i686/cmov/libc.so.6...done.
Loaded symbols for /lib/tls/i686/cmov/libc.so.6
Reading symbols from /lib/tls/i686/cmov/librt.so.1...done.
Loaded symbols for /lib/tls/i686/cmov/librt.so.1
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from 
/lib/glusterfs/1.3.0-pre5/xlator/protocol/client.so...done.
Loaded symbols for /lib/glusterfs/1.3.0-pre5/xlator/protocol/client.so
Reading symbols from 
/lib/glusterfs/1.3.0-pre5/xlator/cluster/unify.so...done.
Loaded symbols for /lib/glusterfs/1.3.0-pre5/xlator/cluster/unify.so
Reading symbols from 
/lib/glusterfs/1.3.0-pre5/transport/tcp/client.so...done.
Loaded symbols for /lib/glusterfs/1.3.0-pre5/transport/tcp/client.so
Reading symbols from /lib/glusterfs/1.3.0-pre5/scheduler/rr.so...done.
Loaded symbols for /lib/glusterfs/1.3.0-pre5/scheduler/rr.so
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
#0  fuse_xattr_cbk (frame=0x8094e38, cookie=0x8094608, this=0x80521f0, 
op_ret=0, op_errno=17, dict=0x80942b0) at fuse-bridge.c:1442
1442            ret = value_data->len;
(gdb) bt
#0  fuse_xattr_cbk (frame=0x8094e38, cookie=0x8094608, this=0x80521f0, 
op_ret=0, op_errno=17, dict=0x80942b0) at fuse-bridge.c:1442
#1  0xb76564f7 in unify_getxattr_cbk (frame=0xa, cookie=0x8094190, 
this=0x80570b8, op_ret=0, op_errno=0, value=0x0) at unify.c:3287
#2  0xb7663005 in client_getxattr_cbk (frame=0x8094190, args=0x8094ea8) 
at client-protocol.c:3431
#3  0xb7664f4c in notify (this=0x8056e08, event=2, data=0x808e6a0) at 
client-protocol.c:4126
#4  0xb7fd8982 in transport_notify (this=0x8094164, event=1) at 
transport.c:152
#5  0xb7fd90fe in sys_epoll_iteration (ctx=0x0) at epoll.c:54
#6  0xb7fd8b2d in poll_iteration (ctx=0x0) at transport.c:260
#7  0x0804a329 in main (argc=3, argv=0xbfdf9c84) at glusterfs.c:341

========================================
GLUSTERFSD.LOG:
2007-06-21 14:09:47 E [protocol.c:262:gf_block_unserialize_transport] 
libglusterfs/protocol: full_read of header failed: peer (127.0.0.1)
2007-06-21 14:09:47 C [tcp.c:82:tcp_disconnect] transport/tcp: 
clusterfs: connection disconnected
2007-06-21 14:09:47 E [protocol.c:262:gf_block_unserialize_transport] 
libglusterfs/protocol: full_read of header failed: peer (127.0.0.1)
2007-06-21 14:09:47 C [tcp.c:82:tcp_disconnect] transport/tcp: 
clusterfs: connection disconnected

========================================
GLUSTERFS.LOG:
2007-06-21 14:09:47 C [common-utils.c:205:gf_print_trace] 
debug-backtrace: Got signal (11), printing backtrace
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace] 
debug-backtrace: /lib/libglusterfs.so.0(gf_print_trace+0x2d) [0xb7f97499]
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace] 
debug-backtrace: [0xffffe420]
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace] 
debug-backtrace: /lib/glusterfs/1.3.0-pre5/xlator/cluster/unify.so 
[0xb76164f7]
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace] 
debug-backtrace: /lib/glusterfs/1.3.0-pre5/xlator/protocol/client.so 
[0xb7623005]
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace] 
debug-backtrace: 
/lib/glusterfs/1.3.0-pre5/xlator/protocol/client.so(notify+0x855) 
[0xb7624f4c]
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace] 
debug-backtrace: /lib/libglusterfs.so.0(transport_notify+0x37) [0xb7f98982]
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace] 
debug-backtrace: /lib/libglusterfs.so.0(sys_epoll_iteration+0xd7) 
[0xb7f990fe]
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace] 
debug-backtrace: /lib/libglusterfs.so.0(poll_iteration+0x1d) [0xb7f98b2d]
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace] 
debug-backtrace: [glusterfs] [0x804a329]
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace] 
debug-backtrace: /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xd2) 
[0xb7e49ea2]
2007-06-21 14:09:47 C [common-utils.c:207:gf_print_trace] 
debug-backtrace: [glusterfs] [0x8049c71]

========================================
glusterfs-client.vol:
volume server1
         type protocol/client
         option transport-type tcp/client     # for TCP/IP transport
         option remote-host 127.0.0.1     # IP address of the remote brick
         option remote-subvolume volumenamespace
end-volume

volume server1vol1
         type protocol/client
         option transport-type tcp/client     # for TCP/IP transport
         option remote-host 127.0.0.1     # IP address of the remote brick
         option remote-subvolume clusterfs1
end-volume

###################

volume bricks
  type cluster/unify
  option namespace server1
  option readdir-force-success on  # ignore failed mounts
  subvolumes server1vol1
  option scheduler rr
  option rr.limits.min-free-disk 5 #%
end-volume

========================================
glusterfs-server.vol:
volume clusterfs1
  type storage/posix
  option directory /volume1
end-volume

#######

volume volumenamespace
  type storage/posix
  option directory /volume.namespace
end-volume

###

volume clusterfs
  type protocol/server
  option transport-type tcp/server
  subvolumes clusterfs1 volumenamespace
  option auth.ip.clusterfs1.allow *
  option auth.ip.volumenamespace.allow *
end-volume