[Gluster-users] Self-healing not healing 27k files on GlusterFS 4.1.5 3 nodes replica

[Ravishankar N]

On 11/08/2018 06:09 PM, mabi wrote:
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Thursday, November 8, 2018 11:05 AM, Ravishankar N <ravishankar at redhat.com> wrote:
>
>> It is not a split-brain. Nodes 1 and 3 have xattrs indicating a pending
>> entry heal on node2 , so heal must have happened ideally. Can you check
>> a few things?
>> -   Is there any disconnects between each of the shds and the brick
>>      processes (check via statedump or look for disconnect messages in
>>      glustershd.log). Does restarting shd via a `volume start force` solve
>>      the problem?
> Yes there is one disconnect at 14:21 (UTC 13:21) because node2 ran out of memory (although it has 32 GB of RAM) and I had to reboot it. Here are the relevant log entries taken from glustershd.log on node1:
>
> [2018-11-05 13:21:16.284239] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-myvol-pro-client-1: server 192.168.10.33:49154 has not responded in the last 42 seconds, disconnecting.
> [2018-11-05 13:21:16.284385] I [MSGID: 114018] [client.c:2254:client_rpc_notify] 0-myvol-pro-client-1: disconnected from myvol-pro-client-1. Client process will keep trying to connect to glusterd until brick's port is available
> [2018-11-05 13:21:16.284889] W [rpc-clnt-ping.c:222:rpc_clnt_ping_cbk] 0-myvol-pro-client-1: socket disconnected
>
> I also just ran a "volume start force" and saw that the glustershd processes got restarted on all 3 nodes but that did not trigger any healing. There are still the same amount of files/dirs pending heal...
>
>> -   Is the symlink pointing to oc_dir present inside .glusterfs/25/e2 in
>>      all 3 bricks?
> They are yes for node1 and node3 but node2 there is no such symlink...
>
> I hope that helps to debug the issue further, else please let me know if you need more info
Please re-create the symlink on node 2 to match how it is in the other 
nodes and launch heal again. Check if this is the case for other entries 
too.
-Ravi