<div dir="ltr"><div><div><div><div>Hi,<br><br></div>To resolve the gfid split-brain you can follow the steps at [1].<br></div><div>Since we don't have the pending markers set on the files, it is not showing in the heal info.<br>To debug this issue, need some more data from you. Could you provide these things?<br></div><div>1. volume info<br></div><div>2. mount log<br></div><div>3. brick logs<br></div><div>4. shd log<br></div><div><br></div>May I also know which version of gluster you are running. From the info you have provided it looks like an old version.<br></div>If it is, then it would be great if you can upgarde to one of the latest supported release.<br></div><div><div><div><div><div><br>[1] <a href="http://docs.gluster.org/en/latest/Troubleshooting/split-brain/#fixing-directory-entry-split-brain">http://docs.gluster.org/en/latest/Troubleshooting/split-brain/#fixing-directory-entry-split-brain</a><br><div class="gmail_extra"><br></div><div class="gmail_extra">Thanks & Regards,<br></div><div class="gmail_extra">Karthik<br><br></div><div class="gmail_extra"><div class="gmail_quote">On Wed, Sep 27, 2017 at 9:42 AM, Zhou, Cynthia (NSB - CN/Hangzhou) <span dir="ltr"><<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<font size="2" face="Times New Roman"><span style="font-size:10.5pt">
<div style="text-align:justify" align="left"> </div>
<div style="text-align:justify" align="left"><font face="Calibri">HI gluster experts,</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"> </font></div>
<div style="text-align:justify" align="left"><font face="Calibri">I meet a tough problem about “split-brain” issue. Sometimes, after hard reboot, we will find some files in split-brain, however its parent directory or anything could be shown in command “gluster
volume heal <volume-name> info”, also, no entry in .glusterfs/indices/xattrop directory, can you help to shed some lights on this issue? Thanks!</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"> </font></div>
<div style="text-align:justify" align="left"><font face="Calibri"> </font></div>
<div style="text-align:justify" align="left"><font face="Calibri"> </font></div>
<div style="text-align:justify" align="left"><font face="Calibri">Following is some info from our env,</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"> </font></div>
<div style="text-align:justify" align="left"><font face="Calibri"><b>Checking from sn-0 cliet, nothing is shown in-split-brain!</b></font></div>
<div style="text-align:justify" align="left"> </div>
<div style="text-align:justify" align="left"><font face="Calibri">[root@sn-0:/mnt/bricks/<wbr>services/brick/netserv/ethip]</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"># gluster v heal services info</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">Brick sn-0:/mnt/bricks/services/<wbr>brick/</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">Number of entries: 0</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"> </font></div>
<div style="text-align:justify" align="left"><font face="Calibri">Brick sn-1:/mnt/bricks/services/<wbr>brick/</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">Number of entries: 0</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"> </font></div>
<div style="text-align:justify" align="left"><font face="Calibri">[root@sn-0:/mnt/bricks/<wbr>services/brick/netserv/ethip]</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">[root@sn-0:/mnt/bricks/<wbr>services/brick/netserv/ethip]</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"># gluster v heal services info split-brain</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">Gathering list of split brain entries on volume services has been successful </font></div>
<div style="text-align:justify" align="left"><font face="Calibri"> </font></div>
<div style="text-align:justify" align="left"><font face="Calibri">Brick sn-0.local:/mnt/bricks/<wbr>services/brick</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">Number of entries: 0</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"> </font></div>
<div style="text-align:justify" align="left"><font face="Calibri">Brick sn-1.local:/mnt/bricks/<wbr>services/brick</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">Number of entries: 0</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"> </font></div>
<div style="text-align:justify" align="left"><font face="Calibri">[root@sn-0:/mnt/bricks/<wbr>services/brick/netserv/ethip]</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"># ls -l /mnt/services/netserv/ethip/</font></div>
<div style="text-align:justify" align="left"><font color="red" face="Calibri">ls: cannot access '/mnt/services/netserv/ethip/<wbr>sn-2': Input/output error</font></div>
<div style="text-align:justify" align="left"><font color="red" face="Calibri">ls: cannot access '/mnt/services/netserv/ethip/<wbr>mn-1': Input/output error</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">total 3</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">-rw-r--r-- 1 root root 144 Sep 26 20:35 as-0</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">-rw-r--r-- 1 root root 144 Sep 26 20:35 as-1</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">-rw-r--r-- 1 root root 145 Sep 26 20:35 as-2</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">-rw-r--r-- 1 root root 237 Sep 26 20:36 mn-0</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">-????????? ? ? ? ? ? mn-1</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">-rw-r--r-- 1 root root 73 Sep 26 20:35 sn-0</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">-rw-r--r-- 1 root root 73 Sep 26 20:35 sn-1</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">-????????? ? ? ? ? ? sn-2</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">[root@sn-0:/mnt/bricks/<wbr>services/brick/netserv/ethip]</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"> </font></div>
<div style="text-align:justify" align="left"><font face="Calibri"><b>Checking from glusterfs server side, the gfid of mn-1 on sn-0 and sn-1 is different</b></font></div>
<div style="text-align:justify" align="left"><font face="Calibri"> </font></div>
<div style="text-align:justify" align="left"><font face="Calibri"><b>[SN-0]</b></font></div>
<div style="text-align:justify" align="left"><font face="Calibri">[root@sn-0:/mnt/bricks/<wbr>services/brick/.glusterfs/53/<wbr>a3]</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"># getfattr -m . -d -e hex /mnt/bricks/services/brick/<wbr>netserv/ethip</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">getfattr: Removing leading '/' from absolute path names</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"># file: mnt/bricks/services/brick/<wbr>netserv/ethip</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">trusted.gfid=<wbr>0xee71d19ac0f84f60b11eb42a0836<wbr>44e4</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">trusted.glusterfs.dht=<wbr>0x000000010000000000000000ffff<wbr>ffff</font></div>
<div style="text-align:justify" align="left"> </div>
<div style="text-align:justify" align="left"><font face="Calibri">[root@sn-0:/mnt/bricks/<wbr>services/brick/netserv/ethip]</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"># getfattr -m . -d -e hex mn-1</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"># file: mn-1</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">trusted.afr.dirty=<wbr>0x000000000000000000000000</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">trusted.afr.services-client-0=<wbr>0x000000000000000000000000</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">trusted.afr.services-client-1=<wbr>0x000000000000000000000000</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">trusted.gfid=<font color="red">0x53a33f437464475<wbr>486f31c4e44d83afd</font></font></div>
<div style="text-align:justify" align="left"><font face="Calibri">[root@sn-0:/mnt/bricks/<wbr>services/brick/netserv/ethip]</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"># stat mn-1</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"> File: mn-1</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"> Size: 237 Blocks: 16 IO Block: 4096 regular file</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">Device: fd51h/64849d Inode: 2536 Links: 2</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">Access: 2017-09-26 20:30:25.679000000 +0300</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">Modify: 2017-09-26 20:30:24.604000000 +0300</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">Change: 2017-09-26 20:30:24.610000000 +0300</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"> Birth: -</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">[root@sn-0:/mnt/bricks/<wbr>services/brick/.glusterfs/<wbr>indices/xattrop]</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"># ls</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">xattrop-63f8bbcb-7fa6-4fc8-<wbr>b721-675a05de0ab3</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">[root@sn-0:/mnt/bricks/<wbr>services/brick/.glusterfs/<wbr>indices/xattrop]</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"> </font></div>
<div style="text-align:justify" align="left"><font face="Calibri">[root@sn-0:/mnt/bricks/<wbr>services/brick/.glusterfs/53/<wbr>a3]</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"># ls</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">53a33f43-7464-4754-86f3-<wbr>1c4e44d83afd</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">[root@sn-0:/mnt/bricks/<wbr>services/brick/.glusterfs/53/<wbr>a3]</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"># stat 53a33f43-7464-4754-86f3-<wbr>1c4e44d83afd </font></div>
<div style="text-align:justify" align="left"><font face="Calibri"> File: 53a33f43-7464-4754-86f3-<wbr>1c4e44d83afd</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"> Size: 237 Blocks: 16 IO Block: 4096 regular file</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">Device: fd51h/64849d Inode: 2536 Links: 2</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">Access: 2017-09-26 20:30:25.679000000 +0300</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">Modify: 2017-09-26 20:30:24.604000000 +0300</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">Change: 2017-09-26 20:30:24.610000000 +0300</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"> Birth: -</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"> </font></div>
<div style="text-align:justify" align="left"><font face="Calibri">#</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"><b>[SN-1]</b></font></div>
<div style="text-align:justify" align="left"> </div>
<div style="text-align:justify" align="left"><font face="Calibri">[root@sn-1:/mnt/bricks/<wbr>services/brick/.glusterfs/f7/<wbr>f1]</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"># getfattr -m . -d -e hex /mnt/bricks/services/brick/<wbr>netserv/ethip</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">getfattr: Removing leading '/' from absolute path names</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"># file: mnt/bricks/services/brick/<wbr>netserv/ethip</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">trusted.gfid=<wbr>0xee71d19ac0f84f60b11eb42a0836<wbr>44e4</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">trusted.glusterfs.dht=<wbr>0x000000010000000000000000ffff<wbr>ffff</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"> </font></div>
<div style="text-align:justify" align="left"><font face="Calibri">[root@sn-1:/mnt/bricks/<wbr>services/brick/.glusterfs/f7/<wbr>f1]</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"><b>#</b></font></div>
<div style="text-align:justify" align="left"><font face="Calibri">[root@sn-1:/mnt/bricks/<wbr>services/brick/netserv/ethip]</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"># getfattr -m . -d -e hex mn-1</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"># file: mn-1</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">trusted.afr.dirty=<wbr>0x000000000000000000000000</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">trusted.afr.services-client-0=<wbr>0x000000000000000000000000</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">trusted.afr.services-client-1=<wbr>0x000000000000000000000000</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">trusted.gfid=<font color="red">0xf7f10f980acc404<wbr>1a015e48018571d4a</font></font></div>
<div style="text-align:justify" align="left"> </div>
<div style="text-align:justify" align="left"><font face="Calibri">[root@sn-1:/mnt/bricks/<wbr>services/brick/netserv/ethip]</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"># stat mn-1</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"> File: mn-1</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"> Size: 237 Blocks: 16 IO Block: 4096 regular file</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">Device: fd41h/64833d Inode: 2608 Links: 2</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">Access: 2017-09-26 20:31:48.231000000 +0300</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">Modify: 2017-09-26 20:31:46.872000000 +0300</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">Change: 2017-09-26 20:31:46.875000000 +0300</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"> Birth: -</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">[root@sn-1:/mnt/bricks/<wbr>services/brick/.glusterfs/<wbr>indices/xattrop]</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"># ls</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">xattrop-240713ea-eda3-4914-<wbr>a55d-7dd4aed724ed</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">[root@sn-1:/mnt/bricks/<wbr>services/brick/.glusterfs/<wbr>indices/xattrop]</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"> </font></div>
<div style="text-align:justify" align="left"><font face="Calibri">[root@sn-1:/mnt/bricks/<wbr>services/brick/.glusterfs/f7/<wbr>f1]</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"># stat f7f10f98-0acc-4041-a015-<wbr>e48018571d4a </font></div>
<div style="text-align:justify" align="left"><font face="Calibri"> File: f7f10f98-0acc-4041-a015-<wbr>e48018571d4a</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"> Size: 237 Blocks: 16 IO Block: 4096 regular file</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">Device: fd41h/64833d Inode: 2608 Links: 2</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">Access: 2017-09-26 20:31:48.231000000 +0300</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">Modify: 2017-09-26 20:31:46.872000000 +0300</font></div>
<div style="text-align:justify" align="left"><font face="Calibri">Change: 2017-09-26 20:31:46.875000000 +0300</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"> Birth: -</font></div>
<div style="text-align:justify" align="left"><font face="Calibri"> </font></div>
<div style="text-align:justify" align="left"><font face="Calibri"> </font></div>
<div style="text-align:justify" align="left"><font face="Arial">Best regards,<br>
<b>Cynthia </b><font face="宋体"><b>（周琳）</b></font></font></div>
<div style="text-align:justify" align="left"><font face="Arial">MBB SM HETRAN SW3 MATRIX </font></div>
<div style="text-align:justify" align="left"><font face="Arial">Storage <br>
Mobile: +86 (0)18657188311</font></div>
<div style="text-align:justify" align="left"> </div>
<div style="text-align:justify" align="left"> </div>
<div style="text-align:justify" align="left"> </div>
<div style="text-align:justify" align="left"><font face="Arial">Best regards,<br>
<b>Cynthia </b><font face="宋体"><b>（周琳）</b></font></font></div>
<div style="text-align:justify" align="left"><font face="Arial">MBB SM HETRAN SW3 MATRIX </font></div>
<div style="text-align:justify" align="left"><font face="Arial">Storage <br>
Mobile: +86 (0)18657188311</font></div>
<div style="text-align:justify" align="left"> </div>
<div style="text-align:justify" align="left"> </div>
<div style="text-align:justify" align="left"> </div>
</span></font>
</div>
<br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></blockquote></div><br></div></div></div></div></div></div></div>