[Gluster-devel] Was using the file system, and then apparently lost connection GF_EVENT_POLLERR

DeeDee Park deedee6905 at hotmail.com
Sun Jul 1 10:04:36 UTC 2007


everything was working fine, and then i lost access to my file system.
I looked at logs and it appears I lost connectivity to my server bricks,
but i could ping them and telnet to port 6996
They are all on the same subnet, and no network problems and no sort of
glusterfs recovery occurred.

Script started on Sun 01 Jul 2007 02:54:02 AM PDT
/tmp/1 is my glusterfs client mountpoint



[root at dhcp217 1]# ls
ls: cannot open directory .: Transport endpoint is not connected
]0;root at dhcp217:/tmp/1 [root at dhcp217 1]# cd 
/usr/local/var/log/glusterfs/
2007-06-30 08:54:55 E [tcp-client.c:135:tcp_connect] 
device128-500GB-128IDEss: error: not in progress - trace: Network is 
unreachable
2007-06-30 08:54:56 E [tcp-client.c:135:tcp_connect] 
device128-500GB-128IDEss: error: not in progress - trace: Network is 
unreachable
2007-06-30 08:54:56 E [tcp-client.c:135:tcp_connect] 
device128-500GB-128IDEss: error: not in progress - trace: Network is 
unreachable
2007-06-30 08:55:01 E [fuse-bridge.c:1910:fuse_transport_notify] 
glusterfs-fuse: got GF_EVENT_POLLERR
2007-06-30 09:20:07 E [tcp-client.c:135:tcp_connect] 
device128-500GB-128IDEss: error: not in progress - trace: Network is 
unreachable
2007-06-30 09:20:08 E [tcp-client.c:135:tcp_connect] 
device128-500GB-128IDEss: error: not in progress - trace: Network is 
unreachable
2007-06-30 09:20:08 E [tcp-client.c:135:tcp_connect] 
device128-500GB-128IDEss: error: not in progress - trace: Network is 
unreachable
2007-06-30 09:20:15 E [fuse-bridge.c:1910:fuse_transport_notify] 
glusterfs-fuse: got GF_EVENT_POLLERR
2007-06-30 09:20:53 E [fuse-bridge.c:1910:fuse_transport_notify] 
glusterfs-fuse: got GF_EVENT_POLLERR
2007-06-30 09:24:28 E [protocol.c:262:gf_block_unserialize_transport] 
libglusterfs/protocol: full_read of header failed: peer (0.0.0.0)
2007-06-30 09:24:28 C [tcp.c:81:tcp_disconnect] glfsd115-250GB-truecrypt: 
connection disconnected
2007-06-30 09:24:28 E [client-protocol.c:327:client_protocol_xfer] 
glfsd115-250GB-truecrypt: transport_submit failed
2007-06-30 09:24:31 E [tcp-client.c:171:tcp_connect] 
glfsd115-250GB-truecrypt: non-blocking connect() returned: 111 (Connection 
refused)
2007-06-30 09:24:34 E [tcp-client.c:171:tcp_connect] 
glfsd115-250GB-truecrypt: non-blocking connect() returned: 111 (Connection 
refused)
2007-06-30 09:24:39 E [tcp-client.c:171:tcp_connect] 
glfsd115-250GB-truecrypt: non-blocking connect() returned: 111 (Connection 
refused)
2007-06-30 09:24:47 E [tcp-client.c:171:tcp_connect] 
glfsd115-250GB-truecrypt: non-blocking connect() returned: 111 (Connection 
refused)
2007-06-30 09:25:00 E [tcp-client.c:171:tcp_connect] 
glfsd115-250GB-truecrypt: non-blocking connect() returned: 111 (Connection 
refused)
2007-06-30 09:25:21 E [tcp-client.c:171:tcp_connect] 
glfsd115-250GB-truecrypt: non-blocking connect() returned: 111 (Connection 
refused)
2007-06-30 09:25:55 E [tcp-client.c:171:tcp_connect] 
glfsd115-250GB-truecrypt: non-blocking connect() returned: 111 (Connection 
refused)
2007-06-30 09:26:50 E [tcp-client.c:171:tcp_connect] 
glfsd115-250GB-truecrypt: non-blocking connect() returned: 111 (Connection 
refused)
2007-06-30 09:28:19 E [tcp-client.c:171:tcp_connect] 
glfsd115-250GB-truecrypt: non-blocking connect() returned: 111 (Connection 
refused)
2007-06-30 09:30:43 E [tcp-client.c:171:tcp_connect] 
glfsd115-250GB-truecrypt: non-blocking connect() returned: 111 (Connection 
refused)
2007-06-30 09:32:10 E [fuse-bridge.c:1910:fuse_transport_notify] 
glusterfs-fuse: got GF_EVENT_POLLERR
2007-06-30 09:32:31 C [common-utils.c:205:gf_print_trace] debug-backtrace: 
Got signal (11), printing backtrace
2007-06-30 09:32:31 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/usr/local/lib/libglusterfs.so.0(gf_print_trace+0x2d) [0x84d94d]
2007-06-30 09:32:31 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
[0xc33420]
2007-06-30 09:32:31 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/usr/local/lib/libglusterfs.so.0(transport_notify+0x37) [0x84ee47]
2007-06-30 09:32:31 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/usr/local/lib/libglusterfs.so.0(sys_epoll_iteration+0xd9) [0x84f969]
2007-06-30 09:32:31 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/usr/local/lib/libglusterfs.so.0(poll_iteration+0x1d) [0x84ef1d]
2007-06-30 09:32:31 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
[glusterfs] [0x8049e70]
2007-06-30 09:32:31 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/lib/libc.so.6(__libc_start_main+0xe0) [0x125f70]
2007-06-30 09:32:31 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
[glusterfs] [0x8049a41]
2007-06-30 09:35:53 E [fuse-bridge.c:1910:fuse_transport_notify] 
glusterfs-fuse: got GF_EVENT_POLLERR
2007-06-30 09:49:02 E [fuse-bridge.c:1910:fuse_transport_notify] 
glusterfs-fuse: got GF_EVENT_POLLERR
2007-06-30 10:15:14 E [fuse-bridge.c:1910:fuse_transport_notify] 
glusterfs-fuse: got GF_EVENT_POLLERR
2007-07-01 01:54:43 E [tcp-client.c:135:tcp_connect] 
device128-500GB-128IDEss: error: not in progress - trace: Network is 
unreachable
2007-07-01 01:54:43 E [tcp-client.c:135:tcp_connect] 
device128-500GB-128IDEss: error: not in progress - trace: Network is 
unreachable
2007-07-01 01:54:44 E [tcp-client.c:135:tcp_connect] 
device128-500GB-128IDEss: error: not in progress - trace: Network is 
unreachable
2007-07-01 01:54:44 E [tcp-client.c:135:tcp_connect] 
device128-500GB-128IDEss: error: not in progress - trace: Network is 
unreachable
2007-07-01 01:54:45 E [tcp-client.c:135:tcp_connect] 
device128-500GB-128IDEss: error: not in progress - trace: Network is 
unreachable
2007-07-01 01:54:45 E [tcp-client.c:135:tcp_connect] 
device128-500GB-128IDEss: error: not in progress - trace: Network is 
unreachable
2007-07-01 01:54:46 E [tcp-client.c:135:tcp_connect] 
device128-500GB-128IDEss: error: not in progress - trace: Network is 
unreachable
2007-07-01 01:54:46 E [tcp-client.c:135:tcp_connect] 
device128-500GB-128IDEss: error: not in progress - trace: Network is 
unreachable
2007-07-01 01:54:50 E [fuse-bridge.c:1910:fuse_transport_notify] 
glusterfs-fuse: got GF_EVENT_POLLERR
2007-07-01 02:52:06 C [common-utils.c:205:gf_print_trace] debug-backtrace: 
Got signal (11), printing backtrace
2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/usr/local/lib/libglusterfs.so.0(gf_print_trace+0x2d) [0xada94d]
2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
[0xf53420]
2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/usr/local/lib/glusterfs/1.3.0-pre5/xlator/cluster/afr.so [0xf91225]
2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/usr/local/lib/libglusterfs.so.0(default_lookup+0x85) [0xada795]
2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/usr/local/lib/libglusterfs.so.0(default_lookup+0x85) [0xada795]
2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
[glusterfs] [0x804d3d6]
2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/usr/local/lib/libfuse.so.2 [0x8ded61]
2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/usr/local/lib/libfuse.so.2 [0x8df7a3]
2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/usr/local/lib/libfuse.so.2(fuse_session_process+0x26) [0x8e0fa6]
2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
[glusterfs] [0x804a740]
2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/usr/local/lib/libglusterfs.so.0(transport_notify+0x37) [0xadbe47]
2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/usr/local/lib/libglusterfs.so.0(sys_epoll_iteration+0xd9) [0xadc969]
2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/usr/local/lib/libglusterfs.so.0(poll_iteration+0x1d) [0xadbf1d]
2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
[glusterfs] [0x8049e70]
2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/lib/libc.so.6(__libc_start_main+0xe0) [0x58af70]
2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
[glusterfs] [0x8049a41]
]0;root at dhcp217:/usr/local/var/log/glusterfs [root at dhcp217 glusterfs]# vi 
!$
vi glusterfs.log
[?1049h[?1h=[?12;25h[?12l[?25h[?25l"glusterfs.log"
"glusterfs.log" 103L, 13037C2007-06-30 04:29:08 E 
[xlator.c:115:xlator_set_type] libglusterfs/xlator: 
dlopen(/usr/local/llib/glusterfs/1.3.0-pre5/xlator/performance/stat-prefetch.so): 
/usr/local/lib/glusterfs/1.3.0--pre5/xlator/performance/stat-prefetch.so: 
cannot open shared object file: No such file or dirrectory
2007-06-30 04:41:27 C [unify.c:4379:init] bricks: namespace option not 
specified, Exiting
2007-06-30 04:41:27 C [common-utils.c:205:gf_print_trace] debug-backtrace: 
Got signal (11), pprinting backtrace
2007-06-30 04:41:27 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/usr/local/lib/libbglusterfs.so.0(gf_print_trace+0x2d) [0xef994d]
2007-06-30 04:41:27 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
[0xc8d420]
2007-06-30 04:41:27 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/usr/local/lib/libbglusterfs.so.0(default_notify+0x5b) [0xef7bbb]
2007-06-30 04:41:27 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/usr/local/lib/libbglusterfs.so.0(default_notify+0x5b) [0xef7bbb]
2007-06-30 04:41:27 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/usr/local/lib/libbglusterfs.so.0(default_notify+0x5b) [0xef7bbb]
2007-06-30 04:41:27 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/usr/local/lib/libbglusterfs.so.0(xlator_tree_init+0x53) [0xef6c73]
2007-06-30 04:41:27 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
[glusterfs] [0x8044d90d]
2007-06-30 04:41:27 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/usr/local/lib/libbfuse.so.2 [0xe86bef]
2007-06-30 04:41:27 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/usr/local/lib/libbfuse.so.2 [0xe867a3]
2007-06-30 04:41:27 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/usr/local/lib/lib@                                          
                                                   
[?12l[?25h[?25l2007-07-01 02:52:06 C 
[common-utils.c:207:gf_print_trace] debug-backtrace: 
/usr/local/lib/libbglusterfs.so.0(default_lookup+0x85) [0xada795]
2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
[glusterfs] [0x8044d3d6]
2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/usr/local/lib/libbfuse.so.2 [0x8ded61]
2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/usr/local/lib/libbfuse.so.2 [0x8df7a3]
2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/usr/local/lib/libbfuse.so.2(fuse_session_process+0x26) [0x8e0fa6]
2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
[glusterfs] [0x8044a740]
2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/usr/local/lib/libbglusterfs.so.0(transport_notify+0x37) [0xadbe47]
2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/usr/local/lib/libbglusterfs.so.0(sys_epoll_iteration+0xd9) 
[0xadc969]
2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/usr/local/lib/libbglusterfs.so.0(poll_iteration+0x1d) [0xadbf1d]
2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
[glusterfs] [0x80449e70]
2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/lib/libc.so.6(__llibc_start_main+0xe0) [0x58af70]
2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
[glusterfs] [0x80449a41]
~                                                                   
                          
[?12l[?25h[?25l2007-07-01 
01:54:44 E [tcp-client.c:135:tcp_connect] device128-500GB-128IDEss: error: 
not in  progress - trace: Network is unreachable
2007-07-01 01:54:45 E [tcp-client.c:135:tcp_connect] 
device128-500GB-128IDEss: error: not in  progress - trace: Network is 
unreachable
2007-07-01 01:54:45 E [tcp-client.c:135:tcp_connect] 
device128-500GB-128IDEss: error: not in  progress - trace: Network is 
unreachable
2007-07-01 01:54:46 E [tcp-client.c:135:tcp_connect] 
device128-500GB-128IDEss: error: not in  progress - trace: Network is 
unreachable
2007-07-01 01:54:46 E [tcp-client.c:135:tcp_connect] 
device128-500GB-128IDEss: error: not in  progress - trace: Network is 
unreachable
2007-07-01 01:54:50 E [fuse-bridge.c:1910:fuse_transport_notify] 
glusterfs-fuse: got GF_EVENTT_POLLERR
2007-07-01 02:52:06 C [common-utils.c:205:gf_print_trace] debug-backtrace: 
Got signal (11), pprinting backtrace
2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/usr/local/lib/libbglusterfs.so.0(gf_print_trace+0x2d) [0xada94d]
2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
[0xf53420]
2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/usr/local/lib/gluusterfs/1.3.0-pre5/xlator/cluster/afr.so [0xf91225]
2007-07-01 02:52:06 C [common-utils.c:207:gf_print_trace] debug-backtrace: 
/usr/local/lib/libbglusterfs.so.0(default_lookup+0x85) [0xada795]


[?12l[?25h

[?25l:[?12l[?25hq
t at dhcp217:/usr/local/var/log/glusterfs [
root at dhcp217 glusterfs]# ping device128
PING device128 (192.168.1.128) 56(84) bytes of data.
64 bytes from 192.168.1.128: icmp_seq=1 ttl=64 time=2.49 ms
64 bytes from 192.168.1.128: icmp_seq=2 ttl=64 time=0.354 ms

--- device128 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 0.354/1.425/2.497/1.072 ms
]0;root at dhcp217:/usr/local/var/log/glusterfs [root at dhcp217 glusterfs]# 
telnet device128 6996
Trying 192.168.1.128...
Connected to device128.
Escape character is '^]'.

^]

telnet> quit
Connection closed.
]0;root at dhcp217:/usr/local/var/log/glusterfs [root at dhcp217 glusterfs]#
Script done on Sun 01 Jul 2007 02:55:32 AM PDT

_________________________________________________________________
Like puzzles? Play free games & earn great prizes. Play Clink now. 
http://club.live.com/clink.aspx?icid=clink_hotmailtextlink2






More information about the Gluster-devel mailing list