<div dir="ltr">Hi, I did a maintenance on the 2 bricks that we have. I added RAM. One of the brick was down for about 30 minutes and the other one for about 10 minutes. In between the shutdown, I only gave a few minutes to gluster to heal. I know that many files were still not in synch when I have shutdown the second brick.<div><br></div><div>The rest is some assumption. I know that one of the user was trying to share the zsh history file between multiple dockers. He tried to use the same file and also tried to use a directory to have multiple history files.</div><div><br></div><div>My guess is that when I shutdown the first node, he created the directory. When I rebooted the first brick and shutdown the second one, I most likely have not give enough time to heal the 2 bricks. Then, he created the file on the second node. When I rebooted the second brick, Gluster was not able to recover.</div><div><br></div><div>Would a third brick have solved this situation? I am not entirely sure.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jun 15, 2017 at 1:43 AM, Mohammed Rafi K C <span dir="ltr"><<a href="mailto:rkavunga@redhat.com" target="_blank">rkavunga@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>Can you please explain How we ended up in this scenario. I think
that will help to understand more about this scenarios and why
gluster recommend replica 3 or arbiter volume.</p>
<p>Regards</p>
<p>Rafi KC<br>
</p><div><div class="h5">
<br>
<div class="m_7731477008923557787moz-cite-prefix">On 06/15/2017 10:46 AM, Karthik
Subrahmanya wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>Hi Ludwig,<br>
<br>
</div>
There is no way to resolve gfid split-brains with type
mismatch. You have to do it manually by following the steps in
[1].<br>
</div>
<div>In case of type mismatch it is recommended to resolve it
manually. But for only gfid mismatch in 3.11 we have a way to<br>
</div>
<div>resolve it by using the *favorite-child-policy*.<br>
</div>
<div>Since the file is not important, you can go with deleting
that.<br>
</div>
<div><br>
[1] <a href="https://gluster.readthedocs.io/en/latest/Troubleshooting/split-brain/#fixing-directory-entry-split-brain" target="_blank">https://gluster.readthedocs.<wbr>io/en/latest/Troubleshooting/<wbr>split-brain/#fixing-directory-<wbr>entry-split-brain</a><br>
<br>
</div>
<div>HTH,<br>
</div>
<div>Karthik<br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Jun 15, 2017 at 8:23 AM, Ludwig
Gamache <span dir="ltr"><<a href="mailto:ludwig@elementai.com" target="_blank">ludwig@elementai.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">I am new to gluster but already like it. I
did a maintenance last week where I shutdown both nodes
(one after each others). I had many files that needed to
be healed after that. Everything worked well, except for 1
file. It is in split-brain, with 2 different GFID. I read
the documentation but it only covers the cases where the
GFID is the same on both bricks. BTW, I am running Gluster
3.10.
<div><br>
</div>
<div>Here are some details...</div>
<div>
<p><span style="font-variant-ligatures:no-common-ligatures">[root@NAS-01
.glusterfs]# gluster volume heal data01 info</span></p>
<p><span style="font-variant-ligatures:no-common-ligatures">Brick
192.168.186.11:/mnt/DATA/data</span></p>
<p><span style="font-variant-ligatures:no-common-ligatures">/abc/.zsh_history </span></p>
<p><span style="font-variant-ligatures:no-common-ligatures">/abc
- Is in split-brain</span></p>
<p><span style="font-variant-ligatures:no-common-ligatures"></span><br>
</p>
<p><span style="font-variant-ligatures:no-common-ligatures">Status:
Connected</span></p>
<p><span style="font-variant-ligatures:no-common-ligatures">Number
of entries: 2</span></p>
<p><span style="font-variant-ligatures:no-common-ligatures"></span><br>
</p>
<p><span style="font-variant-ligatures:no-common-ligatures">Brick
192.168.186.12:/mnt/DATA/data</span></p>
<p><span style="font-variant-ligatures:no-common-ligatures">/abc
- Is in split-brain</span></p>
<p><span style="font-variant-ligatures:no-common-ligatures"></span><br>
</p>
<p><span style="font-variant-ligatures:no-common-ligatures">/abc/.zsh_history </span></p>
<p><span style="font-variant-ligatures:no-common-ligatures">Status:
Connected</span></p>
<p><span style="font-variant-ligatures:no-common-ligatures">Number
of entries: 2</span></p>
<div><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></div>
<div>On brick 1:</div>
<div>
<p><span style="font-variant-ligatures:no-common-ligatures">[root@NAS-01
abc]# ls -lart</span></p>
<p><span style="font-variant-ligatures:no-common-ligatures">total
75</span></p>
<p><span style="font-variant-ligatures:no-common-ligatures">dr</span><span style="font-variant-ligatures:no-common-ligatures">wxr-xr-x.
2 root root 2 Jun 8 13:26 </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(76,122,255)">.zsh_history</span></p>
<p><span style="font-variant-ligatures:no-common-ligatures">drwxr-xr-x.
3 12078 root 3 Jun 12 11:36 </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(76,122,255)">.</span></p>
<p><span style="font-variant-ligatures:no-common-ligatures">drwxrwxrwt.
17 root root 17 Jun 12 12:20 </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0);background-color:rgb(0,217,0)">..</span></p>
</div>
<div><br>
</div>
<div>On brick 2:</div>
<div>
<p><span style="font-variant-ligatures:no-common-ligatures">[root@DC-MTL-NAS-02
abc]# ls -lart</span></p>
<p><span style="font-variant-ligatures:no-common-ligatures">total
66</span></p>
<p><span style="font-variant-ligatures:no-common-ligatures">-rw-rw-r--.
2 12078 12078 1085 Jun 12 04:42 </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(230,230,230);background-color:rgb(0,0,179)">.zsh_history</span></p>
<p><span style="font-variant-ligatures:no-common-ligatures">drwxr-xr-x.
2 12078 root 3 Jun 12 10:36 </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(76,122,255)">.</span></p>
<p><span style="font-variant-ligatures:no-common-ligatures">drwxrwxrwt.
17 root root 17 Jun 12 11:20 </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0);background-color:rgb(0,217,0)">..</span></p>
</div>
<div><br>
</div>
<div>Notice that on one brick, it is a file and on the
other one it is a directory.</div>
<div><br>
</div>
<div>On brick 1:</div>
<div>
<p><span style="font-variant-ligatures:no-common-ligatures">[root@NAS-01
abc]# getfattr -d -m . -e hex
/mnt/DATA/data/abc/.zsh_histor<wbr>y</span></p>
<p><span style="font-variant-ligatures:no-common-ligatures">getfattr:
Removing leading '/' from absolute path names</span></p>
<p><span style="font-variant-ligatures:no-common-ligatures">#
file: mnt/DATA/data/abc/.zsh_history</span></p>
<p><span style="font-variant-ligatures:no-common-ligatures">security.selinux=0x73797374656<wbr>d5f753a6f626a6563745f723a756e6<wbr>c6162656c65645f743a733000</span></p>
<p><span style="font-variant-ligatures:no-common-ligatures">trusted.afr.data01-client-0=0x<wbr>000000000000000000000000</span></p>
<p><span style="font-variant-ligatures:no-common-ligatures">trusted.afr.data01-client-1=0x<wbr>000000000000000200000000</span></p>
<p><span style="font-variant-ligatures:no-common-ligatures">trusted.gfid=0xdee43407139d41f<wbr>091d13e106a51f262</span></p>
<p><span style="font-variant-ligatures:no-common-ligatures">trusted.glusterfs.dht=0x000000<wbr>010000000000000000ffffffff</span></p>
</div>
<div><br>
</div>
<div>On brick 2:</div>
<div>
<p><span style="font-variant-ligatures:no-common-ligatures">root@NAS-02
abc]# getfattr -d -m . -e hex
/mnt/DATA/data/abc/.zsh_histor<wbr>y </span></p>
<p><span style="font-variant-ligatures:no-common-ligatures">getfattr:
Removing leading '/' from absolute path names</span></p>
<p><span style="font-variant-ligatures:no-common-ligatures">#
file: mnt/DATA/data/abc/.zsh_history</span></p>
<p><span style="font-variant-ligatures:no-common-ligatures">security.selinux=0x73797374656<wbr>d5f753a6f626a6563745f723a756e6<wbr>c6162656c65645f743a733000</span></p>
<p><span style="font-variant-ligatures:no-common-ligatures">trusted.afr.data01-client-0=0x<wbr>000000170000000200000000</span></p>
<p><span style="font-variant-ligatures:no-common-ligatures">trusted.afr.data01-client-1=0x<wbr>000000000000000000000000</span></p>
<p><span style="font-variant-ligatures:no-common-ligatures">trusted.bit-rot.version=0x0600<wbr>00000000000059397acd0005dadd</span></p>
<p><span style="font-variant-ligatures:no-common-ligatures">trusted.gfid=0xa70ae9af887a4a3<wbr>7875f5c7c81ebc803</span></p>
</div>
<div><br>
</div>
<div>Any recommendation on how to recover from that?
BTW, the file is not important and I could easily get
rid of it without impact. So, if this is an easy
solution...</div>
<div><br>
</div>
<div>Regards,</div>
<span class="m_7731477008923557787HOEnZb"><font color="#888888">
<div><br>
</div>
-- <br>
<div class="m_7731477008923557787m_-1290886068418900218gmail_signature">
<div dir="ltr">
<div dir="ltr"><span style="color:rgb(136,136,136)">Ludwig
Gamache</span><br>
<div style="color:rgb(136,136,136)">
<div dir="ltr">
<div dir="ltr"><br>
</div>
</div>
</div>
</div>
</div>
</div>
</font></span></div>
</div>
<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="m_7731477008923557787mimeAttachmentHeader"></fieldset>
<br>
<pre>______________________________<wbr>_________________
Gluster-users mailing list
<a class="m_7731477008923557787moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a class="m_7731477008923557787moz-txt-link-freetext" href="http://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><span style="color:rgb(136,136,136)">Ludwig Gamache</span><br><div style="color:rgb(136,136,136)"><div dir="ltr"><div dir="ltr"><div>IT Director - Element AI</div><div>4200 St-Laurent, suite 1200</div><div>514-704-0564</div></div></div></div></div></div></div></div>
</div>