[Gluster-devel] path 651, it keeps hanging

[Jordi Moles Blanco]

hi everyone,

i'm afraid to bring bad news.

after applying patch 650, the system seemed to work smoothly for a whole
under a lot of work.
I moved to patch 651 as you suggested, and it didn't last more than 3
hours :(

the thing is... server didn't hang, but filesystem isn't accessible at
all, not even an "ls" is possible from any of the clients.

this are the node logs:

*************
nothing logged about this matter!!!
************

and these are the client logs:

*************
2008-02-08 14:56:59 E [unify.c:260:unify_lookup_cbk] ultim: Revalidate
failed for /dummy/maildirsize
2008-02-08 14:56:59 E [fuse-bridge.c:431:fuse_entry_cbk] glusterfs-fuse:
202928: /dummy/maildirsize => -1 (2)
2008-02-08 14:56:59 E [afr.c:2398:afr_selfheal_getxattr_cbk] grup2:
(path=/dummy/maildirsize child=espai4) op_ret=-1 op_errno=2
2008-02-08 14:56:59 E [afr.c:2398:afr_selfheal_getxattr_cbk] grup2:
(path=/dummy/maildirsize child=espai3) op_ret=-1 op_errno=2
2008-02-08 14:56:59 E [afr.c:1564:afr_open_cbk] grup2:
(path=/dummy/maildirsize child=espai4) op_ret=-1 op_errno=2
2008-02-08 14:56:59 E [afr.c:1564:afr_open_cbk] grup2:
(path=/dummy/maildirsize child=espai3) op_ret=-1 op_errno=2
2008-02-08 14:56:59 E [unify.c:802:unify_open_cbk] ultim: Open success
on namespace, failed on child node
2008-02-08 14:56:59 E [fuse-bridge.c:670:fuse_fd_cbk] glusterfs-fuse:
203092: /dummy/maildirsize => -1 (2)
2008-02-08 14:56:59 E [afr.c:2398:afr_selfheal_getxattr_cbk] grup2:
(path=/dummy/maildirsize child=espai4) op_ret=-1 op_errno=2
2008-02-08 14:56:59 E [afr.c:2398:afr_selfheal_getxattr_cbk] grup2:
(path=/dummy/maildirsize child=espai3) op_ret=-1 op_errno=2
2008-02-08 14:56:59 E [afr.c:1564:afr_open_cbk] grup2:
(path=/dummy/maildirsize child=espai4) op_ret=-1 op_errno=2
2008-02-08 14:56:59 E [afr.c:1564:afr_open_cbk] grup2:
(path=/dummy/maildirsize child=espai3) op_ret=-1 op_errno=2
2008-02-08 14:56:59 E [unify.c:802:unify_open_cbk] ultim: Open success
on namespace, failed on child node
2008-02-08 14:56:59 E [fuse-bridge.c:670:fuse_fd_cbk] glusterfs-fuse:
203093: /dummy/maildirsize => -1 (2)
2008-02-08 14:56:59 E [afr.c:2398:afr_selfheal_getxattr_cbk] grup2:
(path=/dummy/maildirsize child=espai3) op_ret=-1 op_errno=2
2008-02-08 14:56:59 E [afr.c:2398:afr_selfheal_getxattr_cbk] grup2:
(path=/dummy/maildirsize child=espai4) op_ret=-1 op_errno=2
2008-02-08 14:56:59 E [afr.c:1564:afr_open_cbk] grup2:
(path=/dummy/maildirsize child=espai3) op_ret=-1 op_errno=2
2008-02-08 14:56:59 E [afr.c:1564:afr_open_cbk] grup2:
(path=/dummy/maildirsize child=espai4) op_ret=-1 op_errno=2
2008-02-08 14:56:59 E [unify.c:802:unify_open_cbk] ultim: Open success
on namespace, failed on child node
2008-02-08 14:56:59 E [fuse-bridge.c:670:fuse_fd_cbk] glusterfs-fuse:
203094: /dummy/maildirsize => -1 (2)
2008-02-08 14:57:00 E [unify.c:260:unify_lookup_cbk] ultim: Revalidate
failed for /dummy/maildirsize
2008-02-08 14:57:00 E [fuse-bridge.c:431:fuse_entry_cbk] glusterfs-fuse:
203247: /dummy/maildirsize => -1 (2)
2008-02-08 17:33:05 E [afr.c:2398:afr_selfheal_getxattr_cbk] nm:
(path=/dummy/maildirsize child=namespace2) op_ret=-1 op_errno=2
2008-02-08 17:33:05 E [afr.c:2398:afr_selfheal_getxattr_cbk] nm:
(path=/dummy/maildirsize child=namespace1) op_ret=-1 op_errno=2
2008-02-08 17:33:05 E [afr.c:2398:afr_selfheal_getxattr_cbk] grup2:
(path=/dummy/maildirsize child=espai4) op_ret=-1 op_errno=2
2008-02-08 17:33:05 E [afr.c:2398:afr_selfheal_getxattr_cbk] grup2:
(path=/dummy/maildirsize child=espai3) op_ret=-1 op_errno=2
2008-02-08 17:33:05 E [afr.c:1564:afr_open_cbk] nm:
(path=/dummy/maildirsize child=namespace2) op_ret=-1 op_errno=2
2008-02-08 17:33:05 E [afr.c:1564:afr_open_cbk] nm:
(path=/dummy/maildirsize child=namespace1) op_ret=-1 op_errno=2
2008-02-08 17:33:05 E [afr.c:1564:afr_open_cbk] grup2:
(path=/dummy/maildirsize child=espai4) op_ret=-1 op_errno=2
2008-02-08 17:33:05 E [afr.c:1564:afr_open_cbk] grup2:
(path=/dummy/maildirsize child=espai3) op_ret=-1 op_errno=2
2008-02-08 17:33:05 E [fuse-bridge.c:670:fuse_fd_cbk] glusterfs-fuse:
398639: /dummy/maildirsize => -1 (2)
2008-02-08 17:49:46 E [unify.c:260:unify_lookup_cbk] ultim: Revalidate
failed for /dummy/maildirsize
2008-02-08 17:49:46 E [fuse-bridge.c:431:fuse_entry_cbk] glusterfs-fuse:
517247: /dummy/maildirsize => -1 (2)
2008-02-08 18:03:38 E [afr.c:1564:afr_open_cbk] grup3:
(path=/dummy/maildirsize child=espai6) op_ret=-1 op_errno=2
2008-02-08 18:03:38 E [afr.c:1564:afr_open_cbk] grup3:
(path=/dummy/maildirsize child=espai5) op_ret=-1 op_errno=2
2008-02-08 18:03:38 E [unify.c:802:unify_open_cbk] ultim: Open success
on namespace, failed on child node
2008-02-08 18:03:38 E [fuse-bridge.c:670:fuse_fd_cbk] glusterfs-fuse:
641123: /dummy/maildirsize => -1 (2)
2008-02-08 18:30:33 E [afr.c:1564:afr_open_cbk] grup2:
(path=/dummy/maildirsize child=espai3) op_ret=-1 op_errno=2
2008-02-08 18:30:33 E [afr.c:1564:afr_open_cbk] grup2:
(path=/dummy/maildirsize child=espai4) op_ret=-1 op_errno=2
2008-02-08 18:30:33 E [afr.c:1564:afr_open_cbk] nm:
(path=/dummy/maildirsize child=namespace2) op_ret=-1 op_errno=2
2008-02-08 18:30:33 E [afr.c:1564:afr_open_cbk] nm:
(path=/dummy/maildirsize child=namespace1) op_ret=-1 op_errno=2
2008-02-08 18:30:33 E [fuse-bridge.c:670:fuse_fd_cbk] glusterfs-fuse:
894293: /dummy/maildirsize => -1 (2)
2008-02-08 18:38:09 E [afr.c:1564:afr_open_cbk] nm:
(path=/dummy/maildirsize child=namespace2) op_ret=-1 op_errno=2
2008-02-08 18:38:09 E [afr.c:1564:afr_open_cbk] nm:
(path=/dummy/maildirsize child=namespace1) op_ret=-1 op_errno=2
2008-02-08 18:38:09 E [unify.c:794:unify_open_cbk] ultim: Open success
on child node, failed on namespace
2008-02-08 18:38:09 E [fuse-bridge.c:670:fuse_fd_cbk] glusterfs-fuse:
969868: /dummy/maildirsize => -1 (2)
2008-02-08 19:26:58 E [unify.c:260:unify_lookup_cbk] ultim: Revalidate
failed for /dummy/maildirsize
2008-02-08 19:26:58 E [fuse-bridge.c:431:fuse_entry_cbk] glusterfs-fuse:
1302224: /dummy/maildirsize => -1 (2)
2008-02-08 19:40:03 E [afr.c:1564:afr_open_cbk] nm:
(path=/dummy/maildirsize child=namespace2) op_ret=-1 op_errno=2
2008-02-08 19:40:03 E [afr.c:1564:afr_open_cbk] nm:
(path=/dummy/maildirsize child=namespace1) op_ret=-1 op_errno=2
2008-02-08 19:40:03 E [unify.c:794:unify_open_cbk] ultim: Open success
on child node, failed on namespace
*************

i don't know if that makes any sense, but here you've got come cats

dovecots:

cat /mnt/fusectl/1/waiting >>  6

postfixs:

cat /mnt/fusectl/1/waiting >>  14

these are values at the very moment everything hanged