<div dir="auto"><a href="https://docs.gluster.org/en/v3/Troubleshooting/resolving-splitbrain/">https://docs.gluster.org/en/v3/Troubleshooting/resolving-splitbrain/</a><div dir="auto"><br></div><div dir="auto">Hopefully the link above will help you fix it.</div><div dir="auto"><br></div><div dir="auto">Diego</div></div><br><div class="gmail_quote"><div dir="ltr">On Wed, May 9, 2018, 21:53 Thing <<a href="mailto:thing.thing@gmail.com">thing.thing@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>[trying to read,</div><div><br></div><div><br></div><div>I cant understand what is wrong?</div><div><br></div><div>root@glusterp1 gv0]# gluster volume heal gv0 info</div><div>Brick glusterp1:/bricks/brick1/gv0</div><div><gfid:eafb8799-4e7a-4264-9213-26997c5a4693> - Is in split-brain</div><div><br></div><div>Status: Connected</div><div>Number of entries: 1</div><div><br></div><div>Brick glusterp2:/bricks/brick1/gv0</div><div><gfid:eafb8799-4e7a-4264-9213-26997c5a4693> - Is in split-brain</div><div><br></div><div>Status: Connected</div><div>Number of entries: 1</div><div><br></div><div>Brick glusterp3:/bricks/brick1/gv0</div><div><gfid:eafb8799-4e7a-4264-9213-26997c5a4693> - Is in split-brain</div><div><br></div><div>Status: Connected</div><div>Number of entries: 1</div><div><br></div><div>[root@glusterp1 gv0]# getfattr -d -m . -e hex /bricks/brick1/gv0</div><div>getfattr: Removing leading '/' from absolute path names</div><div># file: bricks/brick1/gv0</div><div>security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000</div><div>trusted.gfid=0x00000000000000000000000000000001</div><div>trusted.glusterfs.dht=0x000000010000000000000000ffffffff</div><div>trusted.glusterfs.volume-id=0xcfceb3535f0e4cf18b533ccfb1f091d3</div><div><br></div><div>[root@glusterp1 gv0]# gluster volume info vol</div><div>Volume vol does not exist</div><div>[root@glusterp1 gv0]# gluster volume info gv0</div><div> </div><div>Volume Name: gv0</div><div>Type: Replicate</div><div>Volume ID: cfceb353-5f0e-4cf1-8b53-3ccfb1f091d3</div><div>Status: Started</div><div>Snapshot Count: 0</div><div>Number of Bricks: 1 x 3 = 3</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: glusterp1:/bricks/brick1/gv0</div><div>Brick2: glusterp2:/bricks/brick1/gv0</div><div>Brick3: glusterp3:/bricks/brick1/gv0</div><div>Options Reconfigured:</div><div>performance.client-io-threads: off</div><div>nfs.disable: on</div><div>transport.address-family: inet</div><div>[root@glusterp1 gv0]# </div><div><br></div><div><br></div><div>================</div><div><br></div><div><div>[root@glusterp2 gv0]# getfattr -d -m . -e hex /bricks/brick1/gv0</div><div>getfattr: Removing leading '/' from absolute path names</div><div># file: bricks/brick1/gv0</div><div>security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000</div><div>trusted.gfid=0x00000000000000000000000000000001</div><div>trusted.glusterfs.dht=0x000000010000000000000000ffffffff</div><div>trusted.glusterfs.volume-id=0xcfceb3535f0e4cf18b533ccfb1f091d3</div><div><br></div><div>[root@glusterp2 gv0]#</div></div><div><br></div><div>================</div><div><br></div><div><div>[root@glusterp3 isos]# getfattr -d -m . -e hex /bricks/brick1/gv0</div><div>getfattr: Removing leading '/' from absolute path names</div><div># file: bricks/brick1/gv0</div><div>security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000</div><div>trusted.gfid=0x00000000000000000000000000000001</div><div>trusted.glusterfs.dht=0x000000010000000000000000ffffffff</div><div>trusted.glusterfs.volume-id=0xcfceb3535f0e4cf18b533ccfb1f091d3</div><div><br></div><div>[root@glusterp3 isos]# </div></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 10 May 2018 at 13:22, Thing <span dir="ltr"><<a href="mailto:thing.thing@gmail.com" target="_blank" rel="noreferrer">thing.thing@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Whatever repair happened has now finished but I still have this,</div><div><br></div><div>I cant find anything so far telling me how to fix it. Looking at </div><div><br></div><div><a href="http://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Features/heal-info-and-split-brain-resolution/" target="_blank" rel="noreferrer">http://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Features/heal-info-and-split-brain-resolution/</a><br></div><div><br></div><div>I cant determine what file? dir gvo? is actually the issue.</div><div><br></div><div>[root@glusterp1 gv0]# gluster volume heal gv0 info split-brain</div><span><div>Brick glusterp1:/bricks/brick1/gv0</div><div><gfid:eafb8799-4e7a-4264-9213-26997c5a4693></div></span><div>Status: Connected</div><div>Number of entries in split-brain: 1</div><span><div><br></div><div>Brick glusterp2:/bricks/brick1/gv0</div><div><gfid:eafb8799-4e7a-4264-9213-26997c5a4693></div><div>Status: Connected</div></span><div>Number of entries in split-brain: 1</div><span><div><br></div><div>Brick glusterp3:/bricks/brick1/gv0</div><div><gfid:eafb8799-4e7a-4264-9213-26997c5a4693></div></span><div>Status: Connected</div><div>Number of entries in split-brain: 1</div><div><br></div><div>[root@glusterp1 gv0]# </div><div><div class="m_-6379197989187999239h5"><div><br></div><div class="gmail_extra"><br><div class="gmail_quote">On 10 May 2018 at 12:22, Thing <span dir="ltr"><<a href="mailto:thing.thing@gmail.com" target="_blank" rel="noreferrer">thing.thing@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">also I have this "split brain"?<div><br></div><div><div>[root@glusterp1 gv0]# gluster volume heal gv0 info</div><span><div>Brick glusterp1:/bricks/brick1/gv0</div><div><gfid:eafb8799-4e7a-4264-9213-26997c5a4693> - Is in split-brain</div><div><br></div><div>Status: Connected</div><div>Number of entries: 1</div><div><br></div><div>Brick glusterp2:/bricks/brick1/gv0</div><div><gfid:eafb8799-4e7a-4264-9213-26997c5a4693> - Is in split-brain</div><div><br></div></span><span><div>/glusterp1/images/centos-server-001.qcow2 </div><div>/glusterp1/images/kubernetes-template.qcow2 </div><div>/glusterp1/images/kworker01.qcow2 </div><div>/glusterp1/images/kworker02.qcow2 </div><div>Status: Connected</div></span><div>Number of entries: 5</div><span><div><br></div><div>Brick glusterp3:/bricks/brick1/gv0</div><div><gfid:eafb8799-4e7a-4264-9213-26997c5a4693> - Is in split-brain</div><div><br></div><div>/glusterp1/images/centos-server-001.qcow2 </div><div>/glusterp1/images/kubernetes-template.qcow2 </div><div>/glusterp1/images/kworker01.qcow2 </div><div>/glusterp1/images/kworker02.qcow2 </div><div>Status: Connected</div></span><div>Number of entries: 5</div><div><br></div><div>[root@glusterp1 gv0]# </div></div></div><div class="m_-6379197989187999239m_-5261765341362211262gmail-m_-391425691660754983HOEnZb"><div class="m_-6379197989187999239m_-5261765341362211262gmail-m_-391425691660754983h5"><div class="gmail_extra"><br><div class="gmail_quote">On 10 May 2018 at 12:20, Thing <span dir="ltr"><<a href="mailto:thing.thing@gmail.com" target="_blank" rel="noreferrer">thing.thing@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>[root@glusterp1 gv0]# !737</div><span><div>gluster v status</div><div>Status of volume: gv0</div><div>Gluster process TCP Port RDMA Port Online Pid</div><div>------------------------------------------------------------------------------</div><div>Brick glusterp1:/bricks/brick1/gv0 49152 0 Y 5229 </div><div>Brick glusterp2:/bricks/brick1/gv0 49152 0 Y 2054 </div><div>Brick glusterp3:/bricks/brick1/gv0 49152 0 Y 2110 </div><div>Self-heal Daemon on localhost N/A N/A Y 5219 </div><div>Self-heal Daemon on glusterp2 N/A N/A Y 1943 </div><div>Self-heal Daemon on glusterp3 N/A N/A Y 2067 </div><div> </div><div>Task Status of Volume gv0</div><div>------------------------------------------------------------------------------</div><div>There are no active volume tasks</div><div> </div></span><div>[root@glusterp1 gv0]# ls -l glusterp1/images/</div><div>total 2877064</div><div>-rw-------. 2 root root 107390828544 May 10 12:18 centos-server-001.qcow2</div><div>-rw-r--r--. 2 root root 0 May 8 14:32 file1</div><div>-rw-r--r--. 2 root root 0 May 9 14:41 file1-1</div><div>-rw-------. 2 root root 85912715264 May 10 12:18 kubernetes-template.qcow2</div><div>-rw-------. 2 root root 0 May 10 12:08 kworker01.qcow2</div><div>-rw-------. 2 root root 0 May 10 12:08 kworker02.qcow2</div><div>[root@glusterp1 gv0]# </div><div><br></div><div><br></div><div>while,</div><div><br></div><div><div>[root@glusterp2 gv0]# ls -l glusterp1/images/</div><div>total 11209084</div><div>-rw-------. 2 root root 107390828544 May 9 14:45 centos-server-001.qcow2</div><div>-rw-r--r--. 2 root root 0 May 8 14:32 file1</div><div>-rw-r--r--. 2 root root 0 May 9 14:41 file1-1</div><div>-rw-------. 2 root root 85912715264 May 9 15:59 kubernetes-template.qcow2</div><div>-rw-------. 2 root root 3792371712 May 9 16:15 kworker01.qcow2</div><div>-rw-------. 2 root root 3792371712 May 10 11:20 kworker02.qcow2</div><div>[root@glusterp2 gv0]#</div></div><div><br></div><div>So some files have re-synced but not the kworker machines network activity has stopped.</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote"><span>On 10 May 2018 at 12:05, Diego Remolina <span dir="ltr"><<a href="mailto:dijuremo@gmail.com" target="_blank" rel="noreferrer">dijuremo@gmail.com</a>></span> wrote:<br></span><div><div class="m_-6379197989187999239m_-5261765341362211262gmail-m_-391425691660754983m_-963754921227385593h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div>Show us output from: gluster v status<div dir="auto"><br></div><div dir="auto">It should be easy to fix. Stop gluster daemon on that node, mount the brick, start gluster daemon again.</div><div dir="auto"><br></div><div dir="auto">Check: gluster v status</div><div dir="auto"><br></div><div dir="auto">Does it show the brick up?</div><div dir="auto"><br></div><div dir="auto">HTH,</div><div dir="auto"><br></div><div dir="auto">Diego</div><br><br><div class="gmail_quote"><div><div class="m_-6379197989187999239m_-5261765341362211262gmail-m_-391425691660754983m_-963754921227385593m_-3860595733488611571h5"><div dir="ltr">On Wed, May 9, 2018, 20:01 Thing <<a href="mailto:thing.thing@gmail.com" target="_blank" rel="noreferrer">thing.thing@gmail.com</a>> wrote:<br></div></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="m_-6379197989187999239m_-5261765341362211262gmail-m_-391425691660754983m_-963754921227385593m_-3860595733488611571h5"><div dir="ltr">Hi,<div><br></div><div>I have 3 Centos7.4 machines setup as a 3 way raid 1.</div><div><br></div><div>Due to an oopsie on my part for glusterp1 /bricks/brick1/gv0 didnt mount on boot and as a result its empty.</div><div><br></div><div>Meanwhile I have data on <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">glusterp2 /bricks/brick1/gv0 and <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">glusterp3 /bricks/brick1/gv0 as expected.</span></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Is there a way to get glusterp1's gv0 to sync off the other 2? there must be but, </span></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">I have looked at the gluster docs and I cant find anything about repairing resyncing?</span></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Where am I meant to look for such info?</span></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">thanks</span></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></span></div></div></div></div>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" rel="noreferrer noreferrer" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer noreferrer noreferrer" target="_blank">http://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div></div></div>
<br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank" rel="noreferrer">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer noreferrer" target="_blank">http://lists.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div></div></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div></div></div>
</blockquote></div><br></div>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank" rel="noreferrer">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer noreferrer" target="_blank">http://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div>