[Gluster-devel] apache serving files from glusterfs -> High load average -> self heal error

[Sebastien COUPPEY]

Hello,

I have Apache (worker) serving files from glusterfs. After a while
when the simultaneous requests become too high, the load average is
increasing and the server unavailable.

glusterfs/d 1.3.7
AFR 2 servers with client side replicatin


it looks like that some files fail to get heal. 

I found some reference in the mailling list archive concerning the
date/time, but I check and the server are correctly sync with ntp so
it doesn't seem to be the problem.

Does any one faced the problem ?

Thanks a lot



In the logs I have the following error :

2008-07-23 02:25:34 E [afr.c:2063:afr_selfheal_getxattr_cbk] cmt1-afr:
(path=/BR_landing/signup/spl02f/index.php child=cmt1Aps38) op_ret=-1
op_errno=107
2008-07-23 02:25:34 C [tcp.c:81:tcp_disconnect] cmt1Aps38: connection
disconnected
2008-07-23 02:36:22 C [client-protocol.c:218:call_bail] cmt2Aps38:
bailing transport
2008-07-23 02:36:22 C [tcp.c:81:tcp_disconnect] cmt2Aps38: connection
disconnected
2008-07-23 02:36:23 C [client-protocol.c:218:call_bail] nsAps38:
bailing transport
2008-07-23 02:36:23 C [client-protocol.c:218:call_bail] cmt1Aps38:
bailing transport
2008-07-23 02:36:23 E [afr.c:2063:afr_selfheal_getxattr_cbk]
namespace-afr: (path=/BR_landing/signup/spl02f/index.php
child=nsAps38) op_ret=-1 op_errno=107
2008-07-23 02:36:23 E [afr.c:1345:afr_open_cbk] namespace-afr:
(path=/BR_landing/signup/index.php child=nsAps38) op_ret=-1
op_errno=107
2008-07-23 02:36:23 E [afr.c:1345:afr_open_cbk] namespace-afr:
(path=/BR_landing/signup/globalvars.php child=nsAps38) op_ret=-1
op_errno=107
2008-07-23 02:36:23 E [afr.c:2063:afr_selfheal_getxattr_cbk]
namespace-afr: (path=/BR_landing/signup/globalvars.php child=nsAps38)
op_ret=-1 op_errno=107
2008-07-23 02:36:23 E [afr.c:2855:afr_close_cbk] namespace-afr:
(path=/BR_landing/signup/spl02f/index.php child=nsAps38) op_ret=-1
op_errno=107
2008-07-23 02:36:23 E [afr.c:2063:afr_selfheal_getxattr_cbk]
namespace-afr: (path=/BR_landing/signup/index.php child=nsAps38)
op_ret=-1 op_errno=107
2008-07-23 02:36:23 C [tcp.c:81:tcp_disconnect] nsAps38: connection
disconnected
2008-07-23 02:36:23 E [client-protocol.c:335:client_protocol_xfer]
cmt1Aps38: transport_submit failed
2008-07-23 02:36:23 E [afr.c:2777:afr_flush_cbk] cmt1-afr:
(path=/BR_landing/signup/index.php child=cmt1Aps38) op_ret=-1
op_errno=107
2008-07-23 02:36:23 E [afr.c:2855:afr_close_cbk] cmt1-afr:
(path=/BR_landing/signup/spl02f/index.php child=cmt1Aps38) op_ret=-1
op_errno=107
2008-07-23 02:36:23 E [afr.c:2777:afr_flush_cbk] cmt1-afr:
(path=/BR_landing/signup/spl02f/index.php child=cmt1Aps38) op_ret=-1
op_errno=107
2008-07-23 02:36:23 E [afr.c:1345:afr_open_cbk] cmt1-afr:
(path=/BR_landing/signup/index.php child=cmt1Aps38) op_ret=-1
op_errno=107
2008-07-23 02:36:23 C [tcp.c:81:tcp_disconnect] cmt1Aps38: connection
disconnected
2008-07-23 02:36:23 E [afr.c:2777:afr_flush_cbk] cmt2-afr:
(path=/BR_landing/signup/globalvars.php child=cmt2Aps38) op_ret=-1
op_errno=77
2008-07-23 02:38:25 C [client-protocol.c:218:call_bail] cmt1Aps37:
bailing transport
2008-07-23 02:38:25 C [client-protocol.c:218:call_bail] cmt2Aps37:
bailing transport
2008-07-23 02:38:25 C [client-protocol.c:218:call_bail] nsAps37:
bailing transport
2008-07-23 02:38:25 E [afr.c:2239:afr_selfheal_lock_cbk] cmt1-afr:
(path=/BR_landing/signup/spl02f/index.php child=cmt1Aps37) op_ret=-1
op_errno=107
2008-07-23 02:38:25 E [afr.c:2382:afr_open] cmt1-afr: self heal
failed, returning EIO
2008-07-23 02:38:25 E [afr.c:2239:afr_selfheal_lock_cbk] cmt1-afr:
(path=/BR_landing/signup/spl02f/index.php child=cmt1Aps37) op_ret=-1
op_errno=107
2008-07-23 02:38:25 E [afr.c:2382:afr_open] cmt1-afr: self heal
failed, returning EIO
2008-07-23 02:38:25 E [afr.c:2239:afr_selfheal_lock_cbk] cmt1-afr:
(path=/BR_landing/signup/spl02f/index.php child=cmt1Aps37) op_ret=-1
op_errno=107
2008-07-23 02:38:25 E [afr.c:2382:afr_open] cmt1-afr: self heal
failed, returning EIO
2008-07-23 02:38:25 E [afr.c:2239:afr_selfheal_lock_cbk] cmt1-afr:
(path=/BR_landing/signup/spl02f/index.php child=cmt1Aps37) op_ret=-1
op_errno=107
2008-07-23 02:38:25 E [afr.c:2382:afr_open] cmt1-afr: self heal
failed, returning EIO
2008-07-23 02:38:25 E [afr.c:2239:afr_selfheal_lock_cbk] cmt1-afr:
(path=/BR_landing/signup/spl02f/index.php child=cmt1Aps37) op_ret=-1
op_errno=107
2008-07-23 02:38:25 E [afr.c:2382:afr_open] cmt1-afr: self heal
failed, returning EIO
2008-07-23 02:38:25 E [afr.c:2239:afr_selfheal_lock_cbk] cmt1-afr:
(path=/BR_landing/signup/spl02f/index.php child=cmt1Aps37) op_ret=-1
op_errno=107
2008-07-23 02:38:25 E [afr.c:2382:afr_open] cmt1-afr: self heal
failed, returning EIO