<div dir="ltr">Hi, I did a maintenance on the 2 bricks that we have. I added RAM. One of the brick was down for about 30 minutes and the other one for about 10 minutes. In between the shutdown, I only gave a few minutes to gluster to heal. I know that many files were still not in synch when I have shutdown the second brick.<div><br></div><div>The rest is some assumption. I know that one of the user was trying to share the zsh history file between multiple dockers. He tried to use the same file and also tried to use a directory to have multiple history files.</div><div><br></div><div>My guess is that when I shutdown the first node, he created the directory. When I rebooted the first brick and shutdown the second one, I most likely have not give enough time to heal the 2 bricks. Then, he created the file on the second node. When I rebooted the second brick, Gluster was not able to recover.</div><div><br></div><div>Would a third brick have solved this situation? I am not entirely sure.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jun 15, 2017 at 1:43 AM, Mohammed Rafi K C <span dir="ltr">&lt;<a href="mailto:rkavunga@redhat.com" target="_blank">rkavunga@redhat.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div bgcolor="#FFFFFF" text="#000000">
    <p>Can you please explain How we ended up in this scenario. I think
      that will help to understand more about this scenarios and why
      gluster recommend replica 3 or arbiter volume.</p>
    <p>Regards</p>
    <p>Rafi KC<br>
    </p><div><div class="h5">
    <br>
    <div class="m_7731477008923557787moz-cite-prefix">On 06/15/2017 10:46 AM, Karthik
      Subrahmanya wrote:<br>
    </div>
    <blockquote type="cite">
      <div dir="ltr">
        <div>
          <div>Hi Ludwig,<br>
            <br>
          </div>
          There is no way to resolve gfid split-brains with type
          mismatch. You have to do it manually by following the steps in
          [1].<br>
        </div>
        <div>In case of type mismatch it is recommended to resolve it
          manually. But for only gfid mismatch in 3.11 we have a way to<br>
        </div>
        <div>resolve it by using the *favorite-child-policy*.<br>
        </div>
        <div>Since the file is not important, you can go with deleting
          that.<br>
        </div>
        <div><br>
          [1] <a href="https://gluster.readthedocs.io/en/latest/Troubleshooting/split-brain/#fixing-directory-entry-split-brain" target="_blank">https://gluster.readthedocs.<wbr>io/en/latest/Troubleshooting/<wbr>split-brain/#fixing-directory-<wbr>entry-split-brain</a><br>
          <br>
        </div>
        <div>HTH,<br>
        </div>
        <div>Karthik<br>
        </div>
      </div>
      <div class="gmail_extra"><br>
        <div class="gmail_quote">On Thu, Jun 15, 2017 at 8:23 AM, Ludwig
          Gamache <span dir="ltr">&lt;<a href="mailto:ludwig@elementai.com" target="_blank">ludwig@elementai.com</a>&gt;</span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div dir="ltr">I am new to gluster but already like it. I
              did a maintenance last week where I shutdown both nodes
              (one after each others). I had many files that needed to
              be healed after that. Everything worked well, except for 1
              file. It is in split-brain, with 2 different GFID. I read
              the documentation but it only covers the cases where the
              GFID is the same on both bricks. BTW, I am running Gluster
              3.10.
              <div><br>
              </div>
              <div>Here are some details...</div>
              <div>
                <p><span style="font-variant-ligatures:no-common-ligatures">[root@NAS-01
                    .glusterfs]# gluster volume heal data01 info</span></p>
                <p><span style="font-variant-ligatures:no-common-ligatures">Brick
                    192.168.186.11:/mnt/DATA/data</span></p>
                <p><span style="font-variant-ligatures:no-common-ligatures">/abc/.zsh_history </span></p>
                <p><span style="font-variant-ligatures:no-common-ligatures">/abc
                    - Is in split-brain</span></p>
                <p><span style="font-variant-ligatures:no-common-ligatures"></span><br>
                </p>
                <p><span style="font-variant-ligatures:no-common-ligatures">Status:
                    Connected</span></p>
                <p><span style="font-variant-ligatures:no-common-ligatures">Number
                    of entries: 2</span></p>
                <p><span style="font-variant-ligatures:no-common-ligatures"></span><br>
                </p>
                <p><span style="font-variant-ligatures:no-common-ligatures">Brick
                    192.168.186.12:/mnt/DATA/data</span></p>
                <p><span style="font-variant-ligatures:no-common-ligatures">/abc
                    - Is in split-brain</span></p>
                <p><span style="font-variant-ligatures:no-common-ligatures"></span><br>
                </p>
                <p><span style="font-variant-ligatures:no-common-ligatures">/abc/.zsh_history </span></p>
                <p><span style="font-variant-ligatures:no-common-ligatures">Status:
                    Connected</span></p>
                <p><span style="font-variant-ligatures:no-common-ligatures">Number
                    of entries: 2</span></p>
                <div><span style="font-variant-ligatures:no-common-ligatures"><br>
                  </span></div>
                <div>On brick 1:</div>
                <div>
                  <p><span style="font-variant-ligatures:no-common-ligatures">[root@NAS-01
                      abc]# ls -lart</span></p>
                  <p><span style="font-variant-ligatures:no-common-ligatures">total
                      75</span></p>
                  <p><span style="font-variant-ligatures:no-common-ligatures">dr</span><span style="font-variant-ligatures:no-common-ligatures">wxr-xr-x. 
                      2 root  root  2 Jun  8 13:26 </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(76,122,255)">.zsh_history</span></p>
                  <p><span style="font-variant-ligatures:no-common-ligatures">drwxr-xr-x. 
                      3 12078 root  3 Jun 12 11:36 </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(76,122,255)">.</span></p>
                  <p><span style="font-variant-ligatures:no-common-ligatures">drwxrwxrwt.
                      17 root  root 17 Jun 12 12:20 </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0);background-color:rgb(0,217,0)">..</span></p>
                </div>
                <div><br>
                </div>
                <div>On brick 2:</div>
                <div>
                  <p><span style="font-variant-ligatures:no-common-ligatures">[root@DC-MTL-NAS-02
                      abc]# ls -lart</span></p>
                  <p><span style="font-variant-ligatures:no-common-ligatures">total
                      66</span></p>
                  <p><span style="font-variant-ligatures:no-common-ligatures">-rw-rw-r--. 
                      2 12078 12078 1085 Jun 12 04:42 </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(230,230,230);background-color:rgb(0,0,179)">.zsh_history</span></p>
                  <p><span style="font-variant-ligatures:no-common-ligatures">drwxr-xr-x. 
                      2 12078 root     3 Jun 12 10:36 </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(76,122,255)">.</span></p>
                  <p><span style="font-variant-ligatures:no-common-ligatures">drwxrwxrwt.
                      17 root  root    17 Jun 12 11:20 </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0);background-color:rgb(0,217,0)">..</span></p>
                </div>
                <div><br>
                </div>
                <div>Notice that on one brick, it is a file and on the
                  other one it is a directory.</div>
                <div><br>
                </div>
                <div>On brick 1:</div>
                <div>
                  <p><span style="font-variant-ligatures:no-common-ligatures">[root@NAS-01
                      abc]# getfattr -d -m . -e hex
                      /mnt/DATA/data/abc/.zsh_histor<wbr>y</span></p>
                  <p><span style="font-variant-ligatures:no-common-ligatures">getfattr:
                      Removing leading &#39;/&#39; from absolute path names</span></p>
                  <p><span style="font-variant-ligatures:no-common-ligatures">#
                      file: mnt/DATA/data/abc/.zsh_history</span></p>
                  <p><span style="font-variant-ligatures:no-common-ligatures">security.selinux=0x73797374656<wbr>d5f753a6f626a6563745f723a756e6<wbr>c6162656c65645f743a733000</span></p>
                  <p><span style="font-variant-ligatures:no-common-ligatures">trusted.afr.data01-client-0=0x<wbr>000000000000000000000000</span></p>
                  <p><span style="font-variant-ligatures:no-common-ligatures">trusted.afr.data01-client-1=0x<wbr>000000000000000200000000</span></p>
                  <p><span style="font-variant-ligatures:no-common-ligatures">trusted.gfid=0xdee43407139d41f<wbr>091d13e106a51f262</span></p>
                  <p><span style="font-variant-ligatures:no-common-ligatures">trusted.glusterfs.dht=0x000000<wbr>010000000000000000ffffffff</span></p>
                </div>
                <div><br>
                </div>
                <div>On brick 2:</div>
                <div>
                  <p><span style="font-variant-ligatures:no-common-ligatures">root@NAS-02
                      abc]# getfattr -d -m . -e hex
                      /mnt/DATA/data/abc/.zsh_histor<wbr>y </span></p>
                  <p><span style="font-variant-ligatures:no-common-ligatures">getfattr:
                      Removing leading &#39;/&#39; from absolute path names</span></p>
                  <p><span style="font-variant-ligatures:no-common-ligatures">#
                      file: mnt/DATA/data/abc/.zsh_history</span></p>
                  <p><span style="font-variant-ligatures:no-common-ligatures">security.selinux=0x73797374656<wbr>d5f753a6f626a6563745f723a756e6<wbr>c6162656c65645f743a733000</span></p>
                  <p><span style="font-variant-ligatures:no-common-ligatures">trusted.afr.data01-client-0=0x<wbr>000000170000000200000000</span></p>
                  <p><span style="font-variant-ligatures:no-common-ligatures">trusted.afr.data01-client-1=0x<wbr>000000000000000000000000</span></p>
                  <p><span style="font-variant-ligatures:no-common-ligatures">trusted.bit-rot.version=0x0600<wbr>00000000000059397acd0005dadd</span></p>
                  <p><span style="font-variant-ligatures:no-common-ligatures">trusted.gfid=0xa70ae9af887a4a3<wbr>7875f5c7c81ebc803</span></p>
                </div>
                <div><br>
                </div>
                <div>Any recommendation on how to recover from that?
                  BTW, the file is not important and I could easily get
                  rid of it without impact. So, if this is an easy
                  solution...</div>
                <div><br>
                </div>
                <div>Regards,</div>
                <span class="m_7731477008923557787HOEnZb"><font color="#888888">
                    <div><br>
                    </div>
                    -- <br>
                    <div class="m_7731477008923557787m_-1290886068418900218gmail_signature">
                      <div dir="ltr">
                        <div dir="ltr"><span style="color:rgb(136,136,136)">Ludwig
                            Gamache</span><br>
                          <div style="color:rgb(136,136,136)">
                            <div dir="ltr">
                              <div dir="ltr"><br>
                              </div>
                            </div>
                          </div>
                        </div>
                      </div>
                    </div>
                  </font></span></div>
            </div>
            <br>
            ______________________________<wbr>_________________<br>
            Gluster-users mailing list<br>
            <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
            <a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
          </blockquote>
        </div>
        <br>
      </div>
      <br>
      <fieldset class="m_7731477008923557787mimeAttachmentHeader"></fieldset>
      <br>
      <pre>______________________________<wbr>_________________
Gluster-users mailing list
<a class="m_7731477008923557787moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a class="m_7731477008923557787moz-txt-link-freetext" href="http://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a></pre>
    </blockquote>
    <br>
  </div></div></div>

</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><span style="color:rgb(136,136,136)">Ludwig Gamache</span><br><div style="color:rgb(136,136,136)"><div dir="ltr"><div dir="ltr"><div>IT Director - Element AI</div><div>4200 St-Laurent, suite 1200</div><div>514-704-0564</div></div></div></div></div></div></div></div>
</div>