<div dir="ltr">Hi,<div><br></div><div>I seem to have a split brain issue, but I cannot figure out where this is and what it is, can someone help me pls,  I cant find what to fix here.</div><div><br></div><div>==========</div><div><div>root@salt-001:~# salt gluster* cmd.run &#39;df -h&#39;<br></div><div><a href="http://glusterp2.graywitch.co.nz">glusterp2.graywitch.co.nz</a>:</div><div>    Filesystem                                               Size  Used Avail Use% Mounted on</div><div>    /dev/mapper/centos-root                                   19G  3.4G   16G  19% /</div><div>    devtmpfs                                                 3.8G     0  3.8G   0% /dev</div><div>    tmpfs                                                    3.8G   12K  3.8G   1% /dev/shm</div><div>    tmpfs                                                    3.8G  9.1M  3.8G   1% /run</div><div>    tmpfs                                                    3.8G     0  3.8G   0% /sys/fs/cgroup</div><div>    /dev/mapper/centos-tmp                                   3.8G   33M  3.7G   1% /tmp</div><div>    /dev/mapper/centos-var                                    19G  213M   19G   2% /var</div><div>    /dev/mapper/centos-home                                   47G   38M   47G   1% /home</div><div>    /dev/mapper/centos-data1                                 112G   33M  112G   1% /data1</div><div>    /dev/mapper/centos-var_lib                               9.4G  178M  9.2G   2% /var/lib</div><div>    /dev/mapper/vg--gluster--prod--1--2-gluster--prod--1--2  932G  264G  668G  29% /bricks/brick1</div><div>    /dev/sda1                                                950M  235M  715M  25% /boot</div><div>    tmpfs                                                    771M   12K  771M   1% /run/user/42</div><div>    glusterp2:gv0/glusterp2/images                           932G  273G  659G  30% /var/lib/libvirt/images</div><div>    glusterp2:gv0                                            932G  273G  659G  30% /isos</div><div>    tmpfs                                                    771M   48K  771M   1% /run/user/1000</div><div>    tmpfs                                                    771M     0  771M   0% /run/user/0</div><div><a href="http://glusterp1.graywitch.co.nz">glusterp1.graywitch.co.nz</a>:</div><div>    Filesystem                                     Size  Used Avail Use% Mounted on</div><div>    /dev/mapper/centos-root                         20G  3.5G   17G  18% /</div><div>    devtmpfs                                       3.8G     0  3.8G   0% /dev</div><div>    tmpfs                                          3.8G   12K  3.8G   1% /dev/shm</div><div>    tmpfs                                          3.8G  9.0M  3.8G   1% /run</div><div>    tmpfs                                          3.8G     0  3.8G   0% /sys/fs/cgroup</div><div>    /dev/sda1                                      969M  206M  713M  23% /boot</div><div>    /dev/mapper/centos-home                         50G  4.3G   46G   9% /home</div><div>    /dev/mapper/centos-tmp                         3.9G   33M  3.9G   1% /tmp</div><div>    /dev/mapper/centos-data1                       120G   36M  120G   1% /data1</div><div>    /dev/mapper/vg--gluster--prod1-gluster--prod1  932G  260G  673G  28% /bricks/brick1</div><div>    /dev/mapper/centos-var                          20G  413M   20G   3% /var</div><div>    /dev/mapper/centos00-var_lib                   9.4G  179M  9.2G   2% /var/lib</div><div>    tmpfs                                          771M  8.0K  771M   1% /run/user/42</div><div>    glusterp1:gv0                                  932G  273G  659G  30% /isos</div><div>    glusterp1:gv0/glusterp1/images                 932G  273G  659G  30% /var/lib/libvirt/images</div><div><a href="http://glusterp3.graywitch.co.nz">glusterp3.graywitch.co.nz</a>:</div><div>    Filesystem                                               Size  Used Avail Use% Mounted on</div><div>    /dev/mapper/centos-root                                   20G  3.5G   17G  18% /</div><div>    devtmpfs                                                 3.8G     0  3.8G   0% /dev</div><div>    tmpfs                                                    3.8G   12K  3.8G   1% /dev/shm</div><div>    tmpfs                                                    3.8G  9.0M  3.8G   1% /run</div><div>    tmpfs                                                    3.8G     0  3.8G   0% /sys/fs/cgroup</div><div>    /dev/sda1                                                969M  206M  713M  23% /boot</div><div>    /dev/mapper/centos-var                                    20G  206M   20G   2% /var</div><div>    /dev/mapper/centos-tmp                                   3.9G   33M  3.9G   1% /tmp</div><div>    /dev/mapper/centos-home                                   50G   37M   50G   1% /home</div><div>    /dev/mapper/centos-data01                                120G   33M  120G   1% /data1</div><div>    /dev/mapper/centos00-var_lib                             9.4G  180M  9.2G   2% /var/lib</div><div>    /dev/mapper/vg--gluster--prod--1--3-gluster--prod--1--3  932G  262G  670G  29% /bricks/brick1</div><div>    glusterp3:gv0                                            932G  273G  659G  30% /isos</div><div>    glusterp3:gv0/glusterp3/images                           932G  273G  659G  30% /var/lib/libvirt/images</div><div>    tmpfs                                                    771M   12K  771M   1% /run/user/42</div><div>root@salt-001:~# salt gluster* cmd.run &#39;getfattr -d -m . -e hex /bricks/brick1/gv0&#39;</div><div><a href="http://glusterp2.graywitch.co.nz">glusterp2.graywitch.co.nz</a>:</div><div>    getfattr: Removing leading &#39;/&#39; from absolute path names</div><div>    # file: bricks/brick1/gv0</div><div>    security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000</div><div>    trusted.gfid=0x00000000000000000000000000000001</div><div>    trusted.glusterfs.dht=0x000000010000000000000000ffffffff</div><div>    trusted.glusterfs.volume-id=0xcfceb3535f0e4cf18b533ccfb1f091d3</div><div><a href="http://glusterp3.graywitch.co.nz">glusterp3.graywitch.co.nz</a>:</div><div>    getfattr: Removing leading &#39;/&#39; from absolute path names</div><div>    # file: bricks/brick1/gv0</div><div>    security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000</div><div>    trusted.gfid=0x00000000000000000000000000000001</div><div>    trusted.glusterfs.dht=0x000000010000000000000000ffffffff</div><div>    trusted.glusterfs.volume-id=0xcfceb3535f0e4cf18b533ccfb1f091d3</div><div><a href="http://glusterp1.graywitch.co.nz">glusterp1.graywitch.co.nz</a>:</div><div>    getfattr: Removing leading &#39;/&#39; from absolute path names</div><div>    # file: bricks/brick1/gv0</div><div>    security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000</div><div>    trusted.gfid=0x00000000000000000000000000000001</div><div>    trusted.glusterfs.dht=0x000000010000000000000000ffffffff</div><div>    trusted.glusterfs.volume-id=0xcfceb3535f0e4cf18b533ccfb1f091d3</div><div>root@salt-001:~# salt gluster* cmd.run &#39;gluster volume heal gv0 info&#39;</div><div><a href="http://glusterp2.graywitch.co.nz">glusterp2.graywitch.co.nz</a>:</div><div>    Brick glusterp1:/bricks/brick1/gv0</div><div>    &lt;gfid:eafb8799-4e7a-4264-9213-26997c5a4693&gt; - Is in split-brain</div><div>    </div><div>    Status: Connected</div><div>    Number of entries: 1</div><div>    </div><div>    Brick glusterp2:/bricks/brick1/gv0</div><div>    &lt;gfid:eafb8799-4e7a-4264-9213-26997c5a4693&gt; - Is in split-brain</div><div>    </div><div>    Status: Connected</div><div>    Number of entries: 1</div><div>    </div><div>    Brick glusterp3:/bricks/brick1/gv0</div><div>    &lt;gfid:eafb8799-4e7a-4264-9213-26997c5a4693&gt; - Is in split-brain</div><div>    </div><div>    Status: Connected</div><div>    Number of entries: 1</div><div><a href="http://glusterp3.graywitch.co.nz">glusterp3.graywitch.co.nz</a>:</div><div>    Brick glusterp1:/bricks/brick1/gv0</div><div>    &lt;gfid:eafb8799-4e7a-4264-9213-26997c5a4693&gt; - Is in split-brain</div><div>    </div><div>    Status: Connected</div><div>    Number of entries: 1</div><div>    </div><div>    Brick glusterp2:/bricks/brick1/gv0</div><div>    &lt;gfid:eafb8799-4e7a-4264-9213-26997c5a4693&gt; - Is in split-brain</div><div>    </div><div>    Status: Connected</div><div>    Number of entries: 1</div><div>    </div><div>    Brick glusterp3:/bricks/brick1/gv0</div><div>    &lt;gfid:eafb8799-4e7a-4264-9213-26997c5a4693&gt; - Is in split-brain</div><div>    </div><div>    Status: Connected</div><div>    Number of entries: 1</div><div><a href="http://glusterp1.graywitch.co.nz">glusterp1.graywitch.co.nz</a>:</div><div>    Brick glusterp1:/bricks/brick1/gv0</div><div>    &lt;gfid:eafb8799-4e7a-4264-9213-26997c5a4693&gt; - Is in split-brain</div><div>    </div><div>    Status: Connected</div><div>    Number of entries: 1</div><div>    </div><div>    Brick glusterp2:/bricks/brick1/gv0</div><div>    &lt;gfid:eafb8799-4e7a-4264-9213-26997c5a4693&gt; - Is in split-brain</div><div>    </div><div>    Status: Connected</div><div>    Number of entries: 1</div><div>    </div><div>    Brick glusterp3:/bricks/brick1/gv0</div><div>    &lt;gfid:eafb8799-4e7a-4264-9213-26997c5a4693&gt; - Is in split-brain</div><div>    </div><div>    Status: Connected</div><div>    Number of entries: 1</div><div>root@salt-001:~# salt gluster* cmd.run &#39;getfattr -d -m . -e hex /bricks/brick1/&#39;</div><div><a href="http://glusterp2.graywitch.co.nz">glusterp2.graywitch.co.nz</a>:</div><div><a href="http://glusterp3.graywitch.co.nz">glusterp3.graywitch.co.nz</a>:</div><div><a href="http://glusterp1.graywitch.co.nz">glusterp1.graywitch.co.nz</a>:</div><div>root@salt-001:~# salt gluster* cmd.run &#39;gluster volume info gv0&#39;</div><div><a href="http://glusterp2.graywitch.co.nz">glusterp2.graywitch.co.nz</a>:</div><div>     </div><div>    Volume Name: gv0</div><div>    Type: Replicate</div><div>    Volume ID: cfceb353-5f0e-4cf1-8b53-3ccfb1f091d3</div><div>    Status: Started</div><div>    Snapshot Count: 0</div><div>    Number of Bricks: 1 x 3 = 3</div><div>    Transport-type: tcp</div><div>    Bricks:</div><div>    Brick1: glusterp1:/bricks/brick1/gv0</div><div>    Brick2: glusterp2:/bricks/brick1/gv0</div><div>    Brick3: glusterp3:/bricks/brick1/gv0</div><div>    Options Reconfigured:</div><div>    transport.address-family: inet</div><div>    nfs.disable: on</div><div>    performance.client-io-threads: off</div><div><a href="http://glusterp1.graywitch.co.nz">glusterp1.graywitch.co.nz</a>:</div><div>     </div><div>    Volume Name: gv0</div><div>    Type: Replicate</div><div>    Volume ID: cfceb353-5f0e-4cf1-8b53-3ccfb1f091d3</div><div>    Status: Started</div><div>    Snapshot Count: 0</div><div>    Number of Bricks: 1 x 3 = 3</div><div>    Transport-type: tcp</div><div>    Bricks:</div><div>    Brick1: glusterp1:/bricks/brick1/gv0</div><div>    Brick2: glusterp2:/bricks/brick1/gv0</div><div>    Brick3: glusterp3:/bricks/brick1/gv0</div><div>    Options Reconfigured:</div><div>    performance.client-io-threads: off</div><div>    nfs.disable: on</div><div>    transport.address-family: inet</div><div><a href="http://glusterp3.graywitch.co.nz">glusterp3.graywitch.co.nz</a>:</div><div>     </div><div>    Volume Name: gv0</div><div>    Type: Replicate</div><div>    Volume ID: cfceb353-5f0e-4cf1-8b53-3ccfb1f091d3</div><div>    Status: Started</div><div>    Snapshot Count: 0</div><div>    Number of Bricks: 1 x 3 = 3</div><div>    Transport-type: tcp</div><div>    Bricks:</div><div>    Brick1: glusterp1:/bricks/brick1/gv0</div><div>    Brick2: glusterp2:/bricks/brick1/gv0</div><div>    Brick3: glusterp3:/bricks/brick1/gv0</div><div>    Options Reconfigured:</div><div>    performance.client-io-threads: off</div><div>    nfs.disable: on</div><div>    transport.address-family: inet</div><div>root@salt-001:~# </div></div><div><br></div><div>===========</div><div><br></div><div><br></div><div>How do I find what  &quot;<span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">eafb8799-4e7a-4264-9213-26997c5a4693&quot;  is?</span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div></div>