[Gluster-users] lingering <gfid:*> entries in volume heal, gluster 3.6.3

[Kingsley]

On Fri, 2016-07-15 at 20:31 +0530, Ravishankar N wrote:
> On 07/15/2016 06:49 PM, Kingsley wrote:
> > On Fri, 2016-07-15 at 18:38 +0530, Ravishankar N wrote:
> >> On 07/15/2016 06:05 PM, Kingsley wrote:
> >>>     chomp (my @output=`getfattr $path`);
> >>
> >> Could you try with `getfattr -d -m. -e hex $path` ?
> > Sure. I'm not really sure what I should be seeing, so I've uploaded the
> > full output here as it's rather large for a group mailing list IMO:
> >
> > http://gluster.dogwind.com/files/gfid-getfattr.txt
> >
> > The output is the same for each of the stayed-up bricks.
> If it is the same in all 3 bricks, the the xattrs of the file indicate  
> heal is pending on the 3rd brick (trusted.afr.callrec-client-2) which I 
> guess was the one that was down. Selfheal daemon should heal them 
> ideally. But you're saying there are no messages in the shd logs 
> corresponding to these gfids.
> Could you check the hardlink count (star -c%h) of these gfids (the ones 
> corresponding to files, not directories)? It must be > 1.

Hi,

I don't have star installed so I used ls, but yes they all have 2 links
to them (see below).

BTW, I noticed that the entries my script said didn't exist are actually
symlinks to other gfid entries that are directories. Most of these
target directories have 2 links, but one has 3 and one has 64. Anyway,
the files:

-rw-r--r--. 2 root root     850 Jun  1 01:13 /data/brick/callrec/.glusterfs/09/87/09872041-bc68-4eee-9c4d-ef11e4137f66
-rw-r--r--. 2 root root 2361324 May 31 19:11 /data/brick/callrec/.glusterfs/0d/16/0d1671b4-e31f-4c81-8600-fe63ffc84272
-rw-r--r--. 2 root root   35564 May 31 21:01 /data/brick/callrec/.glusterfs/0d/44/0d4431f1-10b1-4339-96a9-d8370bbd77a7
-rw-r--r--. 2 root root  104684 May 31 19:10 /data/brick/callrec/.glusterfs/16/fd/16fd1032-3bd5-474a-b30e-85971d69aaa9
-rw-r--r--. 2 root root  356204 May 31 19:11 /data/brick/callrec/.glusterfs/18/e2/18e2f17d-801f-428d-98c1-c35f7ac8a68d
-rw-r--r--. 2 root root  947884 May 31 19:11 /data/brick/callrec/.glusterfs/19/29/19291239-e44c-4025-9f74-af7431aac6b9
-rw-r--r--. 2 root root   54444 May 31 20:57 /data/brick/callrec/.glusterfs/1b/a8/1ba8d90a-5f0c-4387-a79a-592427f3d1c5
-rw-r--r--. 2 root root   35564 May 31 21:38 /data/brick/callrec/.glusterfs/23/e7/23e75764-9794-4efd-b319-6eac717e6f28
-rw-r--r--. 2 root root  778284 May 31 19:11 /data/brick/callrec/.glusterfs/25/af/25af9c71-aab5-4f39-bd4a-bb4f0dab9342
-rw-r--r--. 2 root root  576364 May 31 19:11 /data/brick/callrec/.glusterfs/2b/bb/2bbb8529-4874-440d-959b-6a0745fdfda9
-rw-r--r--. 2 root root    1232 Jun  1 01:15 /data/brick/callrec/.glusterfs/33/8b/338baad1-6191-47f3-9737-dca2daf79fd8
-rw-r--r--. 2 root root   74924 May 31 19:09 /data/brick/callrec/.glusterfs/3a/50/3a50792e-0ce9-49a2-99d5-d59d1fd7e1a9
-rw-r--r--. 2 root root    6124 May 31 19:09 /data/brick/callrec/.glusterfs/3b/c9/3bc95362-be4f-4561-b95f-4cd1260c1781
-rw-r--r--. 2 root root 1113644 May 31 19:11 /data/brick/callrec/.glusterfs/3d/f8/3df80623-6bf4-47e3-a379-4e5605d0eda6
-rw-r--r--. 2 root root  451564 May 31 19:10 /data/brick/callrec/.glusterfs/3f/9f/3f9fcead-ffbe-4dd0-9dcf-8e815dfdc5b4
-rw-r--r--. 2 root root  199724 May 31 19:10 /data/brick/callrec/.glusterfs/40/3c/403c0233-432f-42d8-8ed0-eeca68c4b3f1
-rw-r--r--. 2 root root  550124 May 31 19:09 /data/brick/callrec/.glusterfs/41/97/4197cd59-1cec-452e-8c3c-c212ea9f17f6
-rw-r--r--. 2 root root   35564 May 31 20:48 /data/brick/callrec/.glusterfs/45/5c/455c2911-7a9d-401f-9ded-4380c6cec405
-rw-r--r--. 2 root root  203564 May 31 19:10 /data/brick/callrec/.glusterfs/45/93/459313b5-7ac8-43c6-a3b8-3a8e80afcebd
-rw-r--r--. 2 root root   80044 May 31 21:01 /data/brick/callrec/.glusterfs/45/ae/45aed765-2ae2-4fc8-9982-8e1b5d6c19d4
-rw-r--r--. 2 root root   80044 May 31 20:57 /data/brick/callrec/.glusterfs/4e/86/4e86088b-f975-4228-ad5a-8d08fbd456fe
-rw-r--r--. 2 root root   82284 May 31 19:11 /data/brick/callrec/.glusterfs/50/78/50787eb7-90ad-41fb-bd5e-3ba058b69c32
-rw-r--r--. 2 root root  663084 May 31 19:11 /data/brick/callrec/.glusterfs/53/75/5375e1e8-287f-4fc5-8b01-ec2f856eebf7
-rw-r--r--. 2 root root   35564 May 31 20:57 /data/brick/callrec/.glusterfs/5e/e3/5ee303e4-44fe-44a1-83b5-5cd6a91bc76a
-rw-r--r--. 2 root root     291 May 31 21:01 /data/brick/callrec/.glusterfs/61/bf/61bf5407-c250-4448-a68f-d3bd82821260
-rw-r--r--. 2 root root  388204 May 31 19:10 /data/brick/callrec/.glusterfs/65/39/6539988f-2fe6-4500-aa7c-a7cd7cfcc5df
-rw-r--r--. 2 root root   16684 May 31 19:10 /data/brick/callrec/.glusterfs/66/4f/664f979e-5acc-42f1-93be-9a46d783d430
-rw-r--r--. 2 root root   57644 May 31 19:09 /data/brick/callrec/.glusterfs/66/a4/66a4001b-9456-44e2-b547-5036383500d8
-rw-r--r--. 2 root root   80044 May 31 21:39 /data/brick/callrec/.glusterfs/68/12/68127ab7-807d-4a99-a609-6569d366d3aa
-rw-r--r--. 2 root root  868844 May 31 19:10 /data/brick/callrec/.glusterfs/70/0e/700e2b0a-e6c0-4888-a5ea-f52cac55e770
-rw-r--r--. 2 root root   14764 May 31 21:01 /data/brick/callrec/.glusterfs/70/b7/70b7db21-b70f-4ae0-8660-921c3194f209
-rw-r--r--. 2 root root   59884 May 31 19:11 /data/brick/callrec/.glusterfs/70/ea/70eab476-d53d-4f2b-b56f-13cf12821a24
-rw-r--r--. 2 root root  915564 May 31 19:10 /data/brick/callrec/.glusterfs/75/c6/75c6e98b-7700-46f0-8be4-e897f969a5df
-rw-r--r--. 2 root root  956204 May 31 19:10 /data/brick/callrec/.glusterfs/79/3a/793a81de-73e8-4d84-8ac1-be03c5ac2c47
-rw-r--r--. 2 root root   55084 May 31 19:10 /data/brick/callrec/.glusterfs/7a/d4/7ad41386-9d57-4a93-9fc2-3354e40d9927
-rw-r--r--. 2 root root  486764 May 31 19:09 /data/brick/callrec/.glusterfs/7c/e8/7ce8062a-37fd-4388-aa71-42e3f1125d20
-rw-r--r--. 2 root root  713324 May 31 19:11 /data/brick/callrec/.glusterfs/84/d3/84d3674c-6ec4-43b7-b590-8a125cbfbe43
-rw-r--r--. 2 root root   13484 May 31 19:09 /data/brick/callrec/.glusterfs/87/81/8781eb74-7409-4b1b-aa69-e30bf9c2387d
-rw-r--r--. 2 root root  390444 May 31 19:09 /data/brick/callrec/.glusterfs/88/0c/880cd05b-46e5-4cf4-b042-bdb729acdd56
-rw-r--r--. 2 root root  628844 May 31 19:10 /data/brick/callrec/.glusterfs/8d/f8/8df8af4d-38cf-4cbd-b68e-4a6f2d41b308
-rw-r--r--. 2 root root  388524 May 31 19:11 /data/brick/callrec/.glusterfs/8f/3a/8f3a7de0-173a-4507-9b46-bc9db0a6bc41
-rw-r--r--. 2 root root  344684 May 31 19:11 /data/brick/callrec/.glusterfs/94/b5/94b5924f-65b9-45d6-8a03-b7084f8c4bdb
-rw-r--r--. 2 root root  179564 May 31 19:11 /data/brick/callrec/.glusterfs/9c/62/9c6254df-e8dd-40a3-a7d1-760f6c19027a
-rw-r--r--. 2 root root  276844 May 31 19:10 /data/brick/callrec/.glusterfs/9c/f4/9cf45591-c304-4348-b6ce-9dc9f8f335b1
-rw-r--r--. 2 root root  657324 May 31 19:09 /data/brick/callrec/.glusterfs/9d/80/9d80ae3c-833d-4a7c-81fc-b4ba9e645d5f
-rw-r--r--. 2 root root 2367084 May 31 19:11 /data/brick/callrec/.glusterfs/9e/a0/9ea01327-d5ed-48bd-8049-cc552555b774
-rw-r--r--. 2 root root   40684 May 31 19:09 /data/brick/callrec/.glusterfs/a1/94/a194006c-b0c6-49e9-a2ec-3613403f869f
-rw-r--r--. 2 root root 2217324 May 31 19:10 /data/brick/callrec/.glusterfs/a2/9c/a29c32b6-e9aa-4e51-8b07-311bb7512d89
-rw-r--r--. 2 root root 1308524 May 31 19:11 /data/brick/callrec/.glusterfs/a4/74/a4745c5f-a3ad-4bed-a504-5ae5b31200a3
-rw-r--r--. 2 root root   80044 May 31 21:39 /data/brick/callrec/.glusterfs/a6/cb/a6cb8d18-f2b3-48d4-ac37-c022603b8f8e
-rw-r--r--. 2 root root   23724 May 31 20:49 /data/brick/callrec/.glusterfs/a9/e1/a9e1cad9-cd31-48f0-b9f8-322e1f602401
-rw-r--r--. 2 root root  626924 May 31 19:10 /data/brick/callrec/.glusterfs/ac/75/ac75a8e5-0906-40ea-a2ab-145f3cfcce2e
-rw-r--r--. 2 root root  145004 May 31 19:10 /data/brick/callrec/.glusterfs/ae/44/ae447d7b-04b9-414e-ada6-d1880a5c6555
-rw-r--r--. 2 root root     293 May 31 21:38 /data/brick/callrec/.glusterfs/b0/f4/b0f4e951-63b3-4152-b0b0-7aad8e6ed729
-rw-r--r--. 2 root root   40684 May 31 19:11 /data/brick/callrec/.glusterfs/b1/13/b1138fe2-34cf-4d0d-af99-3b9d9aec0317
-rw-r--r--. 2 root root   65644 May 31 19:10 /data/brick/callrec/.glusterfs/b1/db/b1db87aa-c1fa-409b-aa10-429860c20dbe
-rw-r--r--. 2 root root  117164 May 31 19:09 /data/brick/callrec/.glusterfs/b2/3a/b23a64dd-4522-408c-8465-4096c2977da4
-rw-r--r--. 2 root root 1595244 May 31 19:11 /data/brick/callrec/.glusterfs/c3/41/c3419ad6-3e32-4fc5-93da-2b04a6090cfa
-rw-r--r--. 2 root root    4725 Jun  1 01:15 /data/brick/callrec/.glusterfs/c4/a2/c4a200b9-bb31-46cb-92c6-551fd6ad9ec3
-rw-r--r--. 2 root root   41004 May 31 21:39 /data/brick/callrec/.glusterfs/da/d2/dad27361-d38f-46b3-a77c-b6ac5df054d9
-rw-r--r--. 2 root root  987244 May 31 19:10 /data/brick/callrec/.glusterfs/de/26/de268a4d-06fd-479d-8efc-954002ff7222
-rw-r--r--. 2 root root   80044 May 31 21:39 /data/brick/callrec/.glusterfs/e3/56/e356342a-3dab-4049-b4bc-ea0de4a8ee87
-rw-r--r--. 2 root root   80684 May 31 19:09 /data/brick/callrec/.glusterfs/e9/7c/e97cb502-924d-4ea8-9730-f84eda0b69fd
-rw-r--r--. 2 root root  147244 May 31 19:10 /data/brick/callrec/.glusterfs/eb/87/eb87548b-a90a-458d-b215-939ad59f5ec0
-rw-r--r--. 2 root root  510124 May 31 19:11 /data/brick/callrec/.glusterfs/ef/58/ef583ff4-72b3-408d-bf34-88ca5534c71e
-rw-r--r--. 2 root root   80044 May 31 20:48 /data/brick/callrec/.glusterfs/ef/bd/efbd6028-b9f8-45de-a653-b27f75570b81
-rw-r--r--. 2 root root 1103084 May 31 19:10 /data/brick/callrec/.glusterfs/ef/bf/efbfc625-c42c-4d5e-a22a-173f84362f24
-rw-r--r--. 2 root root     292 May 31 20:48 /data/brick/callrec/.glusterfs/f1/2b/f12b4d8e-b5e6-4f71-a06c-0c0d067a7eb9
-rw-r--r--. 2 root root 1084204 May 31 19:11 /data/brick/callrec/.glusterfs/f3/4d/f34d746e-1555-4b54-9ebd-2bfe7c96b0ef
-rw-r--r--. 2 root root   67564 May 31 19:10 /data/brick/callrec/.glusterfs/f4/8b/f48beb09-975f-4ff6-843e-d8906d3b21b3
-rw-r--r--. 2 root root 1148524 May 31 19:09 /data/brick/callrec/.glusterfs/f6/c7/f6c76172-c73b-4cf0-ad8a-00a6f9c6d7d2
-rw-r--r--. 2 root root   80044 May 31 20:49 /data/brick/callrec/.glusterfs/f7/0b/f70b50fb-3061-4eb9-94f0-85fb2d789d27
-rw-r--r--. 2 root root  122604 May 31 19:09 /data/brick/callrec/.glusterfs/f7/d3/f7d3ef0f-3c01-412e-93fa-704171267d9e

Cheers,
Kingsley.



> -Ravi
> 
> 
> 
> >
> > FWIW, many of the entries have the same values:
> >
> > [root at gluster2b-1 ~]# sort gfids | perl gfid-to-fattr.pl 2> /dev/null | grep trusted.afr | sort | uniq -c
> >       73 trusted.afr.callrec-client-0=0x000000000000000000000000
> >       73 trusted.afr.callrec-client-1=0x000000000000000000000000
> >       25 trusted.afr.callrec-client-2=0x000000020000000300000000
> >        5 trusted.afr.callrec-client-2=0x000000040000000300000000
> >       33 trusted.afr.callrec-client-2=0x000000050000000300000000
> >        1 trusted.afr.callrec-client-2=0x00000005000000ea00000000
> >        1 trusted.afr.callrec-client-2=0x00000005000001f900000000
> >        1 trusted.afr.callrec-client-2=0x000000050000021400000000
> >        3 trusted.afr.callrec-client-2=0x000000060000000300000000
> >        2 trusted.afr.callrec-client-2=0x000000080000000300000000
> >        1 trusted.afr.callrec-client-2=0x000000090000000300000000
> >        1 trusted.afr.callrec-client-2=0x0000000e0000000300000000
> >       73 trusted.afr.callrec-client-3=0x000000000000000000000000
> >       73 trusted.afr.dirty=0x000000000000000000000000
> >
> > Cheers,
> > Kingsley.
> >
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>