[Gluster-devel] dht_layout_dir_mismatch with DHT+NUFA

[Leandro Martelli]

Dear All,

I don't know if the previous message had too few information, thus I'm 
trying again.

I have 12 machines running Ubuntu 8.04 with GlusterFS 2.0.0rc4 compiled.
11 machines run glusterfs in NUFA mode, as shown in this snippet:

volume scratch
  type cluster/nufa
  option local-volume-name `hostname`
  subvolumes nc01 nc02 nc03 nc04 nc05 nc06 nc07 nc08 nc09 nc10 nc11
end-volume


The last machine uses the same bricks, but in DHT mode:

volume scratch
  type cluster/dht
  subvolumes nc01 nc02 nc03 nc04 nc05 nc06 nc07 nc08 nc09 nc10 nc11
end-volume


The problem is that nc04 and nc05 where installed after the 'scratch' 
volume has been in use. We umounted on all machines, changed the 
configuration to include them (nc04 and nc05 where not in the subvolumes 
list before) and mounted them all again. We checked that the /brick dir 
s (4 and 5) now have the same directory structure as the other nodes.
Besides, nc10 and nc11 had a power outage and got a bit out of sync, 
being added to the system a few hours later.

I'm now receiving the following errors:

2009-03-27 15:20:46 E [dht-layout.c:550:dht_layout_dir_mismatch] 
scratch: /export/data/astrosoft/lib - disk layout missing
2009-03-27 15:20:46 W [dht-common.c:244:dht_revalidate_cbk] scratch: 
mismatching layouts for /export/data/astrosoft/lib
2009-03-27 15:20:46 W [fuse-bridge.c:301:need_fresh_lookup] fuse-bridge: 
revalidate of /export/data/astrosoft/lib failed (Stale NFS file handle)

Is there any way we could repair this? I tried 'ls -lR' with no luck. 
And it's also not the best solution, as we have plans to grow to a few 
petabytes.

Thanks,

</Martelli>