[Gluster-users] Self-healing not healing 27k files on GlusterFS 4.1.5 3 nodes replica

[Ravishankar N]

On 11/15/2018 09:17 PM, mabi wrote:
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Thursday, November 15, 2018 1:41 PM, Ravishankar N <ravishankar at redhat.com> wrote:
>
>> Thanks, noted. One more query. Are there files inside each of these
>> directories? Or is it just empty directories?
> You will find below the content of each of these 3 directories taken the brick on node 1:
>
> i)/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10
>
> drwxr-xr-x  4 www-data www-data  4 Nov  5 14:19 .
> drwxr-xr-x 31 www-data www-data 31 Nov  5 14:23 ..
> drwxr-xr-x  3 www-data www-data  3 Nov  5 14:19 dir11
> drwxr-xr-x  3 www-data www-data  3 Nov  5 14:19 another_dir
>
> ii)/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/
> drwxr-xr-x 3 www-data www-data 3 Nov  5 14:19 .
> drwxr-xr-x 4 www-data www-data 4 Nov  5 14:19 ..
> drwxr-xr-x 2 www-data www-data 4 Nov  5 14:19 oc_dir
>
> iii)/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir
>
> drwxr-xr-x 2 www-data www-data   4 Nov  5 14:19 .
> drwxr-xr-x 3 www-data www-data   3 Nov  5 14:19 ..
> -rw-r--r-- 2 www-data www-data  32 Nov  5 14:19 fileKey
> -rw-r--r-- 2 www-data www-data 512 Nov  5 14:19 username.shareKey
>
> so as you see from the output above only the "oc_dir" directory has two files inside.
>
Okay, I'm assuming the list of files+dirs are the same on nodes 2 and 3 
as well. Correct me if that isn't the case.
>> symlinks are only for dirs. For files, they would be hard links to the
>> actual files. So if stat
>> ../brick/.glusterfs/aa/e4/aae4098a-1a71-4155-9cc9-e564b89957cf gives you
>> a file, then you can use find -samefile to get the other hardlinks like so:
>> #cd /brick/.glusterfs/aa/e4/aae4098a-1a71-4155-9cc9-e564b89957cf
>> #find /brick -samefile aae4098a-1a71-4155-9cc9-e564b89957cf
>>
>> If it is a hardlink, then you can do a getfattr on
>> /brick/.glusterfs/aa/e4/aae4098a-1a71-4155-9cc9-e564b89957cf itself.
>> -Ravi
> Thank you for explaining this important part. So yes with your help I could find the filenames associated to these 2 GFIDs and guess what? they are the 2 files which listed in the output above of the "oc_dir" directory. Have a look at this:
>
> # find /data/myvol-pro/brick -samefile aae4098a-1a71-4155-9cc9-e564b89957cf
> /data/myvol-pro/brick/.glusterfs/aa/e4/aae4098a-1a71-4155-9cc9-e564b89957cf
> /data/myvol-pro/brick/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir/fileKey
>
> # find /data/myvol-pro/brick -samefile 3c92459b-8fa1-4669-9a3d-b38b8d41c360
> /data/myvol-pro/brick/.glusterfs/3c/92/3c92459b-8fa1-4669-9a3d-b38b8d41c360
> /data/myvol-pro/brick/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir/username.shareKey
Okay, as asked in the previous mail, please share the getfattr output 
from all bricks for these 2 files. I think once we have this, we can try 
either 'adjusting' the the gfid and symlinks on node 2 for dir11 and 
oc_dir or see if we can set afr xattrs on dir10 for self-heal to purge 
everything under it on node 2 and recreate it using the other 2 nodes.
-Ravi
>
> I hope that helps the debug further else let me know if you need anything else.