[Gluster-users] Decomissioned brick, mismatching layouts, EINVAL on file/folder create

[Vincent Caron]

Hello,

  sorry for the strange title, but it might be possible that those
symptoms are related...

  I'm running a Gluster 3.4.2 volume with 8 bricks in distribute mode.

  Easiest problem to diagnose : sometimes a folder goes into a state
where any attempt to create a new file or folder within ends up with
EINVAL (being root or not).

  Those errors may persist or not (sometimes it's fixed by itself a
while later), and it seems to be related to this kind of information in
the client's log :

[2014-05-14 19:17:12.937364] I [dht-common.c:623:dht_revalidate_cbk]
0-xxx-prod-dht: mismatching layouts for /uploads/images/5373
[2014-05-14 19:17:12.938960] I
[dht-layout.c:726:dht_layout_dir_mismatch] 0-xxx-prod-dht:
/uploads/images/5373/9c8b - disk layout missing

  ... where /uploads/images/5373 is a folder which I can list with
readable file, but where creating new files and folder is impossible.


  Meanwhile I historically tried to remove a brick from this volume
which has 8 of them. The rebalance operation took ages and I had to
cancel it because I was suffering from performance problems. Now the 8th
brick is shown in the volume status but marked 'decomissioned' in the
internal metadata, and has this property : while all 8 bricks have the
same inode count (roughly 5 millions), only the 7th first have a
balanced block usage with 800 GB, while the 8th has been stuck 270 GB
(the point where I tried to remove it).

  It looks expected. The 8th brick is still part of the volume but has
only 'historical' data which is still needed. I'd like to re-add this
brick into my volume, is it as simple as issuing a 'volume add-brick'
and then a 'volume rebalance' ?

  In any case if there's a way to avoid those bugs, I'd be interested to
hear about it. Thanks for your experience and your help !