[Gluster-users] fix-layout stalls with xattr errors

[Dan Bretherton]

Hello All-
I am having problems with rebalance ... fix-layout in version 3.2.5.  I 
extended a volume with add-brick but the fix-layout stalls after a small 
number of layout fixes and does not make any more progress.  I have 
tried the operation twice on different servers with the same result. 
  The following errors are found in the fuse mount log file on the 
server carrying out the operation.

    [2011-12-28 21:38:14.840013] I
    [afr-common.c:1038:afr_launch_self_heal] 0-nemo2-replicate-4:
    background  data self-heal triggered. path:
    /users/hzu/DATA/ERAINT/ORCA025/2010/snow_ERAINT_2010.nc
    [2011-12-28 21:38:15.93079] E
    [client3_1-fops.c:1498:client3_1_fxattrop_cbk] 0-nemo2-client-8:
    remote operation failed: Operation not supported
    [2011-12-28 21:38:15.93141] E
    [client3_1-fops.c:1498:client3_1_fxattrop_cbk] 0-nemo2-client-9:
    remote operation failed: Operation not supported
    [2011-12-28 21:38:15.93385] I
    [client3_1-fops.c:1187:client3_1_fstat_cbk] 0-nemo2-client-8: remote
    operation failed: Operation not supported
    [2011-12-28 21:38:15.93521] I
    [client3_1-fops.c:1187:client3_1_fstat_cbk] 0-nemo2-client-9: remote
    operation failed: Operation not supported


The file in the error message is a link, and it is not broken as seen 
from the volume mount point or the bricks.

There are some worrying error messages in the brick log files for 
nemo2-client-8 and nemo2-client-9.  Here are some exerpts from the 
nemo2-client-8 log, which is similar to the 0-nemo2-client-9 log.

    [2011-12-28 21:23:05.827877] W [posix.c:3928:do_xattrop]
    0-nemo2-posix: Extended attributes not supported by filesystem
    [2011-12-28 21:23:05.827932] I
    [server3_1-fops.c:1705:server_fxattrop_cbk] 0-nemo2-server: 8438:
    FXATTROP 0 (-2111276040) ==> -1 (Operation not support
    ed)
    [2011-12-28 21:23:05.828848] E [posix.c:4200:posix_fstat]
    0-nemo2-posix: fstat failed on fd=0x2aaaac703804: Operation not
    supported
    [2011-12-28 21:23:05.828879] I
    [server3_1-fops.c:1113:server_fstat_cbk] 0-nemo2-server: 8439: FSTAT
    0 (-2111276040) ==> -1 (Operation not supported)
    [2011-12-28 21:29:29.871213] W
    [socket.c:1494:__socket_proto_state_machine] 0-tcp.nemo2-server:
    reading from socket failed. Error (Transport endpoint i
    s not connected), peer (192.171.166.81:1003)
    [2011-12-28 21:29:29.871305] I
    [server-helpers.c:360:do_lock_table_cleanup] 0-nemo2-server: inodelk
    released on /users/hzu/DATA/ERAINT/ORCA025/2010/sno
    w_ERAINT_2010.nc
    [2011-12-28 21:29:29.871345] I [server-helpers.c:485:do_fd_cleanup]
    0-nemo2-server: fd cleanup on
    /users/hzu/DATA/ERAINT/ORCA025/2010/snow_ERAINT_2010.
    nc

    [2011-12-28 21:34:36.190023] I [server-helpers.c:485:do_fd_cleanup]
    0-nemo2-server: fd cleanup on /
    [2011-12-28 21:34:36.190055] I [server-helpers.c:485:do_fd_cleanup]
    0-nemo2-server: fd cleanup on /users
    [2011-12-28 21:34:36.190086] I [server-helpers.c:485:do_fd_cleanup]
    0-nemo2-server: fd cleanup on /users/hzu
    [2011-12-28 21:34:36.190102] I [server-helpers.c:485:do_fd_cleanup]
    0-nemo2-server: fd cleanup on /users/hzu/DATA
    [2011-12-28 21:34:36.190135] I [server-helpers.c:485:do_fd_cleanup]
    0-nemo2-server: fd cleanup on /users/hzu/DATA/ERAINT
    [2011-12-28 21:34:36.190154] I [server-helpers.c:485:do_fd_cleanup]
    0-nemo2-server: fd cleanup on /users/hzu/DATA/ERAINT/ORCA025
    [2011-12-28 21:34:36.190171] I [server-helpers.c:485:do_fd_cleanup]
    0-nemo2-server: fd cleanup on /users/hzu/DATA/ERAINT/ORCA025/2009

      [2011-12-28 21:38:15.92433] I
    [server3_1-fops.c:1705:server_fxattrop_cbk] 0-nemo2-server: 12228:
    FXATTROP 7 (-2111276040) ==> -1 (Operation not supported)
    [2011-12-28 21:38:15.92743] E [posix.c:4200:posix_fstat]
    0-nemo2-posix: fstat failed on fd=0x2aaaac703804: Operation not
    supported
    [2011-12-28 21:38:15.92775] I
    [server3_1-fops.c:1113:server_fstat_cbk] 0-nemo2-server: 12229:
    FSTAT 7 (-2111276040) ==> -1 (Operation not supported)


The backend filesystems are ext4 and the are mounted with options 
"acl,user_xattr".  I tested extended attribute support (as suggested 
here: 
http://gluster.org/pipermail/gluster-users/2010-December/006257.html) 
and could not find any problems, so I don't understand the "Extended 
attributes not supported by filesystem" error.  The only unusual thing 
about the filesystems is the reduced number of filesystem features 
enabled compared to other bricks.  These are the ext4 features enabled.

has_journal ext_attr resize_inode dir_index filetype needs_recovery 
sparse_super large_file

All the other bricks in the volume have these features plus extent, 
flex_bg, huge_file, uninit_bg, dir_nlink and extra_isize.  I don't know 
if any of these missing ext4 features are part of the problem.  Does 
anybody know what's going on here?

Regards
Dan.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20111229/a5cecfb0/attachment.html>