[Gluster-devel] gluster rebalance progress

[Emmanuel Dreyfus]

Emmanuel Dreyfus <manu at netbsd.org> wrote:

> Since I am porting glusterfs, I am never fully sure it really works as
> intended. When I try gluster volume rebalance, I get no error reported,
> but the operation seems to take forever.

I found a probable cause for the problem. This is glusterfsd on NetBSD
with a UFS1 filesystem. Extended attributes are stored in backing files
that are located in a .attribute directory on the root of the
filesystems.

Of couse, if glusterfs starts distributing .attribute content among
servers, then chaos occurs, since on a given server, some extended
attributes will disapear during the operation. Logs indeed suggest
something goes wrong with .attribute (see below)

I am not sure of how this should be fixed. We can have a dht.ignodre-dir
xlator option to specify a directory to ignore (or a comma-separated
list?). I can contribute that if this is considered the right way. I
just wonder how the administrator could request such an option to be
added to glusterfs/glusterfsd arguments when glusterd launch them. Any
suggestion?

Another approach, simplier to implement but less general, is to add an
#ifdef __NetBSD__ code section to dht xlator to skip .attribute
treatments. The advantage of this approach is to avoid the problem of
specifying the xlator option from glusterd.

Theses two alternative can also be done in storage/posix xlator, rather
that in cluster/dht, so that .attribute become just absent from any
client view. In fact I think this is better to do it in storage/posix
than in cluster/dht, since the same problem may happen later with other
xlators. 

Annd finaly, if it is considered that it should not be glusterfs' job to
address that, I can add a special case in NetBSD FUSE implementation so
that .attribute just disapear from glusterfs view, but I do not think
this is the right place do that.


 W [afr-common.c:634:afr_lookup_self_heal_check] 0-gfs1-replicate-1:
/.attribute: gfid different on subvolume 
 W [afr-common.c:634:afr_lookup_self_heal_check] 0-gfs1-replicate-1:
/.attribute: gfid different on subvolume 
 W [dht-common.c:177:dht_lookup_dir_cbk] 0-gfs1-dht: /.attribute: gfid
different on gfs1-replicate-1
 W [afr-common.c:634:afr_lookup_self_heal_check] 0-gfs1-replicate-0:
/.attribute: gfid different on subvolume 
 W [dht-common.c:177:dht_lookup_dir_cbk] 0-gfs1-dht: /.attribute: gfid
different on gfs1-replicate-0
 I [client3_1-fops.c:411:client3_1_stat_cbk] 0-gfs1-client-2: remote
operation failed: No such file or directory
 I [client3_1-fops.c:411:client3_1_stat_cbk] 0-gfs1-client-1: remote
operation failed: No such file or directory
 I [client3_1-fops.c:411:client3_1_stat_cbk] 0-gfs1-client-3: remote
operation failed: No such file or directory
 I [client3_1-fops.c:2132:client3_1_opendir_cbk] 0-gfs1-client-1: remote
operation failed: No such file or directory
 W [client3_1-fops.c:5041:client3_1_readdir] 0-gfs1-client-1: (2795520):
failed to get fd ctx. EBADFD
 W [client3_1-fops.c:5106:client3_1_readdir] 0-gfs1-client-1: failed to
send the fop: Bad file descriptor
 I [afr-dir-read.c:120:afr_examine_dir_readdir_cbk] 0-gfs1-replicate-0:
/.attribute: failed to do opendir on gfs1-client-1 
 I [afr-dir-read.c:174:afr_examine_dir_readdir_cbk] 0-gfs1-replicate-0:
entry self-heal triggered. path: /.attribute, reason: checksums of
directory differ, forced merge option set 
 I [client3_1-fops.c:2132:client3_1_opendir_cbk] 0-gfs1-client-2: remote
operation failed: No such file or directory
 I [client3_1-fops.c:1303:client3_1_entrylk_cbk] 0-gfs1-client-1: remote
operation failed: No such file or directory

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu at netbsd.org