[Gluster-users] GlusterFS Split Brain issue

Tue Sep 22 11:24:22 UTC 2015

Hi Ankur, 

It looks like some of the files/directories are in gfid split-brain. 
>From the logs that you attached, here is the list of gfids of directories in gfid split-brain, based on the message id for gfid split-brain log message (108008): 

[kdhananjay at dhcp35-215 logs]$ grep -iarT '108008' * | awk '{print $13}' | cut -f1 -d'/' | sort | uniq 
<16d8005d-3ae2-4c72-9097-2aedd458b5e0 
<3539c175-d694-409d-949f-f9a3e18df17b 
<3fd13508-b29e-4d52-8c9c-14ccd2f24b9f 
<6b1e5a5a-bb65-46c1-a7c3-0526847beece 
<971b5249-92fb-4166-b1a0-33b7efcc39a8 
<b582f326-c8ee-4b04-aba0-d37cb0a6f89a 
<cc9d0e49-c9ab-4dab-bca4-1c06c8a7a4e3 

There are 7 such directories. 

Also, there are 457 entries in gfid split-brain: 
[kdhananjay at dhcp35-215 logs]$ grep -iarT '108008' glustershd.log | awk '{print $13}' | sort | uniq | wc -l 
457 

You will need to do the following to get things back to the normal state: 

1) For each gfid in the list of the 7 directories in split-brain, get the list of files in split-brain. 
For example, for <16d8005d-3ae2-4c72-9097-2aedd458b5e0 , the command would be `grep -iarT '108008' * | grep 16d8005d-3ae2-4c72-9097-2aedd458b5e0` 
You will need to omit the repeating messages of course. 
You would get messages of the following kind: 
glustershd.log :[2015-09-10 01:44:05.512589] E [MSGID: 108008] [afr-self-heal-entry.c:253:afr_selfheal_detect_gfid_and_type_mismatch] 0-repl-vol-replicate-0: Gfid mismatch detected for <16d8005d-3ae2-4c72-9097-2aedd458b5e0/100000075944.jpg>, d9f15b28-9c9c-4f31-ba3c-543a5331cb9d on repl-vol-client-1 and 583295f0-1ec4-4783-9b35-1e18b8b4f92c on repl-vol-client-0. Skipping conservative merge on the file. 

2) Examine the two copies (one per replica) of each such file, choose one copy and delete the copy from the other replica. 
In the above example, the parent is 16d8005d-3ae2-4c72-9097-2aedd458b5e0 and the entry is '100000075944.jpg'. 
So you can examine the two different copies at <brick-path>/.glusterfs/16/d8/16d8005d-3ae2-4c72-9097-2aedd458b5e0/100000075944.jpg to decide which one you want to keep. 
Once you have decided on the copy you choose to keep, you need to delete the bad copy and its hard link. This is assuming all of the entries in gfid split-brain are regular files. At least that is what I gathered from the logs since they were all .jpg files. 
You can get the absolute path of the entry by noting down inode number of the gfid link on the bad brick and then grepping for the corresponding number under the same brick. 
In this example, the gfid link would be <bad-brick-path>/.glusterfs/16/d8/16d8005d-3ae2-4c72-9097-2aedd458b5e0/100000075944.jpg. 
So you would need to get its inode number (by doing stat on it) and do a 'find <bad-brick-path> -inum <inodenumber of gfid link> to get its absolute path. 
Once you have both, unlink them both. If hard links exist, delete them as well on the bad brick. 

There are about 457 files where you need to repeat this exercise. 

Once you are done, you could execute 'gluster volume heal <VOL>'. This would take care of healing the good copies to the bricks where the file was deleted from. 
After the heal is complete, heal info split-brain should not be showing any entries. 

As for the performance problem, it is possible that it was due to self-heal daemon periodically trying to heal the files in gfid-split-brain in vain, and should most likely go away once the split-brain is resolved. 

As an aside, it is not clear why so many files ran into gfid split-brain. You might want to check if the network link between the clients and the servers was fine. 

Hope that helps. Let me know if you need more clarification. 
-Krutika 
----- Original Message -----

> From: "Ankur Pandey" <ankur.p at ambab.com>
> To: gluster-users at gluster.org, pkarampu at redhat.com
> Cc: "Dhaval Kamani" <dhaval at ambab.com>
> Sent: Saturday, September 12, 2015 12:51:31 PM
> Subject: [Gluster-users] GlusterFS Split Brain issue

> HI Team GlusterFS,

> With reference to Question on server fault.

> http://serverfault.com/questions/721067/glusterfs-split-brain-issue

> On request of Pranith I am sending you logs. Please tell me if you need
> anything else.

> Attaching logs for 2 master servers.

> Regards
> Ankur Pandey
> +91 9702 831 855

> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150922/09845b18/attachment.html>