[Gluster-users] Funny behavior of glusterfs after a brick substitution

[Luca Fini]

Please note that I just started to use gluster recently ...

I have a replicated gluster installation: two servers with six bricks
each, and I have a total of 14TB of data under a single "replica 2"
volume.

After replacing an hard disk and doing something wrong in the process,
I decided to delete and recreate the volume with the very same bricks
and this apparently worked so that no data were lost (I followed the
few steps required to rebuild a volume from previously used bricks),
but now I've a strange behaviour:

if I know the full path of a file, I can access it via the gluster
mount point, but neither "ls" or "find" is working. I.e.: (Note:
/ao-data is the glusterfs mount point) I can see with ls a file if I
know the full path:

[root at aostore1 glusterfs]# ls -l /ao-data/lbtdata/adsecdx/aolog/AOARB.R.log
-rw-r--r--. 1 archiver archiver 182678 13 apr  2015
/ao-data/lbtdata/adsecdx/aolog/AOARB.R.log

but if I try to list the directory containing that file, or any
ancestor directory:

[root at aostore1 glusterfs]#  ls -l /ao-data/lbtdata/adsecdx/aolog/
total 0
[root at aostore1 glusterfs]# ls -al /ao-data/lbtdata/adsecdx
total 0
[root at aostore1 glusterfs]# ls -al /ao-data/lbtdata
total 0
[root at aostore1 glusterfs]# ls -al /ao-data
total 124
drwxr-xr-x.  7 root     root     24576 17 ott 14.59 .
drwxr-xr-x. 21 root     root      4096 17 ott 15.05 ..
drwxr-xr-x.  8 root     root     24576 12 ott 13.42 lbtdata
drwxr-xr-x.  3 root     root     24576 16 set 14.10 misc
drwxrwxr-x.  8 archiver archiver 24576 14 ott 03.59 simul_backup
drwxr-xr-x.  3 root     root     24576 17 ott 14.59 .trashcan

As shown by the last command, the three topmost directories are
listed, but that's all. They, too, are apparently empty.

I tried to force healing, to do rebalance, with no success. The only
effect of rebalance was many error lines in file
ao-data-rebalance.log, like the following:

[2015-10-19 22:29:55.134097] E [MSGID: 109010]
[dht-rebalance.c:1950:gf_defrag_get_entry] 0-ao-data-dht:
/simul_backup/data3 gfid not present

This is clearly due to my naive approach, but I'd be grateful for any
hint on how to restore the proper functioning.

 Cheers,
                        l.f.

-- 
Luca Fini.  INAF - Osservatorio Astrofisico di Arcetri
L.go E.Fermi, 5. 50125 Firenze. Italy
Tel: +39 055 2752 307     Fax: +39 055 2752 292
Skype: l.fini
Web: http://www.arcetri.inaf.it/~lfini