[Gluster-users] Directory rename prevented by mismatching layouts in 3.1

[Dan Bretherton]

Hello All,
I am having a lot of problems with a distributed, replicated volume in
GlusterFS 3.1.  A user tried to rename a directory using "mv ANALYSIS
ANALYSIS-6HR", but received the error "operation not permitted".  He
then saw that both the original and the newly named directories were
there, but neither contained the data that had been in ANALYSIS.  I was
called in to have a look, and saw another permission error when the user
tried to remame ANALYSIS-6HR back to ANALYSIS, and another one on trying
to rename ANALYSIS-6HR to ANALYSIS-6HR-TT.  Now all three versions are
there but none contains the files that were originaly in ANALYSIS.  The
directories with the first two names are now owned by root, as shown
below in the parent directory listing. 

[root at nemo1 logs]# ls -l /glusterfs/atmos/users/kih/INTERIM
total 244
drwxr-x--- 2 root root 16384 Nov 26 16:57 ANALYSIS
drwxr-xr-x 2 root root 16384 Nov 26 17:04 ANALYSIS-6HR
drwxr-xr-x 2 kih  essc 16384 Nov 26 16:58 ANALYSIS-6HR-TT

The  glusterfs-atmos.log file errors resulting from the above command
are shown at the bottom of this message.  Running a self heal (i.e. ls
-alR) on the volume doesn't solve the problem, only gives more of the
same errors.  The files are actually located in ANALYSIS-6HR-TT, where
they should be after the two rename operations, but only on one of the
replicated bricks.  They do not show up in the mounted GlusterFS volume
(NFS or native GlusterFS client).  Does anybody know how to make the
files visible again and stop this behaviour from occurring?

-Dan;
See log file extract below:

[2010-11-28 22:57:53.57596] I [dht-common.c:415:dht_revalidate_cbk]
atmos-dht: mismatching layouts for /users/kih/INTERIM/ANALYSIS
[2010-11-28 22:57:53.57761] I [dht-layout.c:689:dht_layout_dir_mismatch]
atmos-dht: subvol: atmos-replicate-2; inode layout - 0 - 0; disk
 layout - 0 - 2147483646
[2010-11-28 22:57:53.57781] I [dht-common.c:415:dht_revalidate_cbk]
atmos-dht: mismatching layouts for /users/kih/INTERIM/ANALYSIS
[2010-11-28 22:57:53.57872] I [dht-common.c:415:dht_revalidate_cbk]
atmos-dht: mismatching layouts for /users/kih/INTERIM/ANALYSIS
[2010-11-28 22:57:53.57927] I [dht-layout.c:689:dht_layout_dir_mismatch]
atmos-dht: subvol: atmos-replicate-3; inode layout - 0 - 0; disk layout
- 2147483647 - 4294967295
[2010-11-28 22:57:53.57945] I [dht-common.c:415:dht_revalidate_cbk]
atmos-dht: mismatching layouts for /users/kih/INTERIM/ANALYSIS
[2010-11-28 22:57:53.69267] I [dht-common.c:415:dht_revalidate_cbk]
atmos-dht: mismatching layouts for /users/kih/INTERIM/ANALYSIS
[2010-11-28 22:57:53.69392] I [dht-layout.c:689:dht_layout_dir_mismatch]
atmos-dht: subvol: atmos-replicate-2; inode layout - 0 - 0; disk layout
- 0 - 2147483646
[2010-11-28 22:57:53.69410] I [dht-common.c:415:dht_revalidate_cbk]
atmos-dht: mismatching layouts for /users/kih/INTERIM/ANALYSIS
[2010-11-28 22:57:53.69523] I [dht-layout.c:689:dht_layout_dir_mismatch]
atmos-dht: subvol: atmos-replicate-3; inode layout - 0 - 0; disk layout
- 2147483647 - 4294967295
[2010-11-28 22:57:53.69542] I [dht-common.c:415:dht_revalidate_cbk]
atmos-dht: mismatching layouts for /users/kih/INTERIM/ANALYSIS
[2010-11-28 22:57:53.69581] I [dht-common.c:415:dht_revalidate_cbk]
atmos-dht: mismatching layouts for /users/kih/INTERIM/ANALYSIS
[2010-11-28 22:57:53.163971] I
[dht-layout.c:689:dht_layout_dir_mismatch] atmos-dht: subvol:
atmos-replicate-1; inode layout - 1073741823 - 2147483645; disk layout -
2147483646 - 3221225468
[2010-11-28 22:57:53.164051] I [dht-common.c:415:dht_revalidate_cbk]
atmos-dht: mismatching layouts for /users/kih/INTERIM/ANALYSIS-6HR
[2010-11-28 22:57:53.164098] I
[dht-layout.c:689:dht_layout_dir_mismatch] atmos-dht: subvol:
atmos-replicate-2; inode layout - 2147483646 - 3221225468; disk layout -
3221225469 - 4294967295
[2010-11-28 22:57:53.164116] I [dht-common.c:415:dht_revalidate_cbk]
atmos-dht: mismatching layouts for /users/kih/INTERIM/ANALYSIS-6HR
[2010-11-28 22:57:53.164159] I
[dht-layout.c:689:dht_layout_dir_mismatch] atmos-dht: subvol:
atmos-replicate-3; inode layout - 3221225469 - 4294967295; disk layout -
0 - 1073741822
[2010-11-28 22:57:53.164176] I [dht-common.c:415:dht_revalidate_cbk]
atmos-dht: mismatching layouts for /users/kih/INTERIM/ANALYSIS-6HR
[2010-11-28 22:57:53.164695] I
[dht-layout.c:689:dht_layout_dir_mismatch] atmos-dht: subvol:
atmos-replicate-0; inode layout - 0 - 1073741822; disk layout -
1073741823 - 2147483645
[2010-11-28 22:57:53.164712] I [dht-common.c:415:dht_revalidate_cbk]
atmos-dht: mismatching layouts for /users/kih/INTERIM/ANALYSIS-6HR
[2010-11-28 22:57:53.178847] I
[dht-layout.c:689:dht_layout_dir_mismatch] atmos-dht: subvol:
atmos-replicate-1; inode layout - 1073741823 - 2147483645; disk layout -
2147483646 - 3221225468
[2010-11-28 22:57:53.178884] I [dht-common.c:415:dht_revalidate_cbk]
atmos-dht: mismatching layouts for /users/kih/INTERIM/ANALYSIS-6HR
[2010-11-28 22:57:53.178942] I
[dht-layout.c:689:dht_layout_dir_mismatch] atmos-dht: subvol:
atmos-replicate-2; inode layout - 2147483646 - 3221225468; disk layout -
3221225469 - 4294967295
[2010-11-28 22:57:53.178960] I [dht-common.c:415:dht_revalidate_cbk]
atmos-dht: mismatching layouts for /users/kih/INTERIM/ANALYSIS-6HR
[2010-11-28 22:57:53.179267] I
[dht-layout.c:689:dht_layout_dir_mismatch] atmos-dht: subvol:
atmos-replicate-3; inode layout - 3221225469 - 4294967295; disk layout -
0 - 1073741822
[2010-11-28 22:57:53.179286] I [dht-common.c:415:dht_revalidate_cbk]
atmos-dht: mismatching layouts for /users/kih/INTERIM/ANALYSIS-6HR
[2010-11-28 22:57:53.179582] I
[dht-layout.c:689:dht_layout_dir_mismatch] atmos-dht: subvol:
atmos-replicate-0; inode layout - 0 - 1073741822; disk layout -
1073741823 - 2147483645
[2010-11-28 22:57:53.179600] I [dht-common.c:415:dht_revalidate_cbk]
atmos-dht: mismatching layouts for /users/kih/INTERIM/ANALYSIS-6HR