[Gluster-users] split brain? but where?

Thing thing.thing at gmail.com
Tue May 22 20:35:58 UTC 2018


I tried looking for a file of the same size and the gfid doesnt show up,

8><---
[root at glusterp2 fb]# pwd
/bricks/brick1/gv0/.glusterfs/ea/fb
[root at glusterp2 fb]# ls -al
total 3130892
drwx------. 2 root root         64 May 22 13:01 .
drwx------. 4 root root         24 May  8 14:27 ..
-rw-------. 1 root root 3294887936 May  4 11:07
eafb8799-4e7a-4264-9213-26997c5a4693
-rw-r--r--. 1 root root       1396 May 22 13:01 gfid.run

so the gfid seems large....but du cant see it...

[root at glusterp2 fb]# du -a /bricks/brick1/gv0 | sort -n -r | head -n 10
275411712 /bricks/brick1/gv0
275411696 /bricks/brick1/gv0/.glusterfs
22484988 /bricks/brick1/gv0/.glusterfs/57
20974980 /bricks/brick1/gv0/.glusterfs/a5
20974976
/bricks/brick1/gv0/.glusterfs/d5/99/d5991797-848d-4d60-b9dc-d31174f63f72
20974976 /bricks/brick1/gv0/.glusterfs/d5/99
20974976 /bricks/brick1/gv0/.glusterfs/d5
20974976
/bricks/brick1/gv0/.glusterfs/a5/27/a5279083-4d24-4dc8-be2d-4f507c5372cf
20974976 /bricks/brick1/gv0/.glusterfs/a5/27
20974976
/bricks/brick1/gv0/.glusterfs/74/75/7475bd15-05a6-45c2-b395-bc9fd3d1763f
[root at glusterp2 fb]#

8><---



On 23 May 2018 at 08:29, Thing <thing.thing at gmail.com> wrote:

> I tried this already.
>
> 8><---
> [root at glusterp2 fb]# find /bricks/brick1/gv0 -samefile
> /bricks/brick1/gv0/.glusterfs/ea/fb/eafb8799-4e7a-4264-9213-26997c5a4693
> /bricks/brick1/gv0/.glusterfs/ea/fb/eafb8799-4e7a-4264-9213-26997c5a4693
> [root at glusterp2 fb]#
> 8><---
>
> gluster 4
> Centos 7.4
>
> 8><---
> df -h
> [root at glusterp2 fb]# df -h
> Filesystem                                               Size  Used Avail
> Use% Mounted on
> /dev/mapper/centos-root                                   19G  3.4G   16G
> 19% /
> devtmpfs                                                 3.8G     0  3.8G
>  0% /dev
> tmpfs                                                    3.8G   12K  3.8G
>  1% /dev/shm
> tmpfs                                                    3.8G  9.0M  3.8G
>  1% /run
> tmpfs                                                    3.8G     0  3.8G
>  0% /sys/fs/cgroup
> /dev/mapper/centos-data1                                 112G   33M  112G
>  1% /data1
> /dev/mapper/centos-var                                    19G  219M   19G
>  2% /var
> /dev/mapper/centos-home                                   47G   38M   47G
>  1% /home
> /dev/mapper/centos-var_lib                               9.4G  178M  9.2G
>  2% /var/lib
> /dev/mapper/vg--gluster--prod--1--2-gluster--prod--1--2  932G  263G
> 669G  29% /bricks/brick1
> /dev/sda1                                                950M  235M  715M
> 25% /boot
> 8><---
>
>
>
> So the output isnt helping..........
>
>
>
>
>
>
>
> On 23 May 2018 at 00:29, Karthik Subrahmanya <ksubrahm at redhat.com> wrote:
>
>> Hi,
>>
>> Which version of gluster you are using?
>>
>> You can find which file is that using the following command
>> find <brickpath> -samefile <brickpath/.glusterfs/<first two bits of
>> gfid>/<next 2 bits of gfid>/<full gfid>
>>
>> Please provide the getfatr output of the file which is in split brain.
>> The steps to recover from split-brain can be found here,
>> http://gluster.readthedocs.io/en/latest/Troubleshooting/reso
>> lving-splitbrain/
>>
>> HTH,
>> Karthik
>>
>> On Tue, May 22, 2018 at 4:03 AM, Joe Julian <joe at julianfamily.org> wrote:
>>
>>> How do I find what  "eafb8799-4e7a-4264-9213-26997c5a4693"  is?
>>>
>>> https://docs.gluster.org/en/v3/Troubleshooting/gfid-to-path/
>>>
>>>
>>> On May 21, 2018 3:22:01 PM PDT, Thing <thing.thing at gmail.com> wrote:
>>> >Hi,
>>> >
>>> >I seem to have a split brain issue, but I cannot figure out where this
>>> >is
>>> >and what it is, can someone help me pls,  I cant find what to fix here.
>>> >
>>> >==========
>>> >root at salt-001:~# salt gluster* cmd.run 'df -h'
>>> >glusterp2.graywitch.co.nz:
>>> >    Filesystem                                               Size  Used
>>> >Avail Use% Mounted on
>>> >    /dev/mapper/centos-root                                   19G  3.4G
>>> > 16G  19% /
>>> >    devtmpfs                                                 3.8G     0
>>> >3.8G   0% /dev
>>> >    tmpfs                                                    3.8G   12K
>>> >3.8G   1% /dev/shm
>>> >    tmpfs                                                    3.8G  9.1M
>>> >3.8G   1% /run
>>> >    tmpfs                                                    3.8G     0
>>> >3.8G   0% /sys/fs/cgroup
>>> >    /dev/mapper/centos-tmp                                   3.8G   33M
>>> >3.7G   1% /tmp
>>> >    /dev/mapper/centos-var                                    19G  213M
>>> > 19G   2% /var
>>> >    /dev/mapper/centos-home                                   47G   38M
>>> > 47G   1% /home
>>> >    /dev/mapper/centos-data1                                 112G   33M
>>> >112G   1% /data1
>>> >    /dev/mapper/centos-var_lib                               9.4G  178M
>>> >9.2G   2% /var/lib
>>> >    /dev/mapper/vg--gluster--prod--1--2-gluster--prod--1--2  932G  264G
>>> >668G  29% /bricks/brick1
>>> >    /dev/sda1                                                950M  235M
>>> >715M  25% /boot
>>> >    tmpfs                                                    771M   12K
>>> >771M   1% /run/user/42
>>> >    glusterp2:gv0/glusterp2/images                           932G  273G
>>> >659G  30% /var/lib/libvirt/images
>>> >    glusterp2:gv0                                            932G  273G
>>> >659G  30% /isos
>>> >    tmpfs                                                    771M   48K
>>> >771M   1% /run/user/1000
>>> >    tmpfs                                                    771M     0
>>> >771M   0% /run/user/0
>>> >glusterp1.graywitch.co.nz:
>>> >   Filesystem                                     Size  Used Avail Use%
>>> >Mounted on
>>> > /dev/mapper/centos-root                         20G  3.5G   17G  18% /
>>> >   devtmpfs                                       3.8G     0  3.8G   0%
>>> >/dev
>>> >   tmpfs                                          3.8G   12K  3.8G   1%
>>> >/dev/shm
>>> >   tmpfs                                          3.8G  9.0M  3.8G   1%
>>> >/run
>>> >   tmpfs                                          3.8G     0  3.8G   0%
>>> >/sys/fs/cgroup
>>> >   /dev/sda1                                      969M  206M  713M  23%
>>> >/boot
>>> >   /dev/mapper/centos-home                         50G  4.3G   46G   9%
>>> >/home
>>> >   /dev/mapper/centos-tmp                         3.9G   33M  3.9G   1%
>>> >/tmp
>>> >   /dev/mapper/centos-data1                       120G   36M  120G   1%
>>> >/data1
>>> >   /dev/mapper/vg--gluster--prod1-gluster--prod1  932G  260G  673G  28%
>>> >/bricks/brick1
>>> >   /dev/mapper/centos-var                          20G  413M   20G   3%
>>> >/var
>>> >   /dev/mapper/centos00-var_lib                   9.4G  179M  9.2G   2%
>>> >/var/lib
>>> >   tmpfs                                          771M  8.0K  771M   1%
>>> >/run/user/42
>>> >   glusterp1:gv0                                  932G  273G  659G  30%
>>> >/isos
>>> >   glusterp1:gv0/glusterp1/images                 932G  273G  659G  30%
>>> >/var/lib/libvirt/images
>>> >glusterp3.graywitch.co.nz:
>>> >    Filesystem                                               Size  Used
>>> >Avail Use% Mounted on
>>> >    /dev/mapper/centos-root                                   20G  3.5G
>>> > 17G  18% /
>>> >    devtmpfs                                                 3.8G     0
>>> >3.8G   0% /dev
>>> >    tmpfs                                                    3.8G   12K
>>> >3.8G   1% /dev/shm
>>> >    tmpfs                                                    3.8G  9.0M
>>> >3.8G   1% /run
>>> >    tmpfs                                                    3.8G     0
>>> >3.8G   0% /sys/fs/cgroup
>>> >    /dev/sda1                                                969M  206M
>>> >713M  23% /boot
>>> >    /dev/mapper/centos-var                                    20G  206M
>>> > 20G   2% /var
>>> >    /dev/mapper/centos-tmp                                   3.9G   33M
>>> >3.9G   1% /tmp
>>> >    /dev/mapper/centos-home                                   50G   37M
>>> > 50G   1% /home
>>> >    /dev/mapper/centos-data01                                120G   33M
>>> >120G   1% /data1
>>> >    /dev/mapper/centos00-var_lib                             9.4G  180M
>>> >9.2G   2% /var/lib
>>> >    /dev/mapper/vg--gluster--prod--1--3-gluster--prod--1--3  932G  262G
>>> >670G  29% /bricks/brick1
>>> >    glusterp3:gv0                                            932G  273G
>>> >659G  30% /isos
>>> >    glusterp3:gv0/glusterp3/images                           932G  273G
>>> >659G  30% /var/lib/libvirt/images
>>> >    tmpfs                                                    771M   12K
>>> >771M   1% /run/user/42
>>> >root at salt-001:~# salt gluster* cmd.run 'getfattr -d -m . -e hex
>>> >/bricks/brick1/gv0'
>>> >glusterp2.graywitch.co.nz:
>>> >    getfattr: Removing leading '/' from absolute path names
>>> >    # file: bricks/brick1/gv0
>>> >
>>> >security.selinux=0x73797374656d5f753a6f626a6563745f723a756e
>>> 6c6162656c65645f743a733000
>>> >    trusted.gfid=0x00000000000000000000000000000001
>>> >    trusted.glusterfs.dht=0x000000010000000000000000ffffffff
>>> >    trusted.glusterfs.volume-id=0xcfceb3535f0e4cf18b533ccfb1f091d3
>>> >glusterp3.graywitch.co.nz:
>>> >    getfattr: Removing leading '/' from absolute path names
>>> >    # file: bricks/brick1/gv0
>>> >
>>> >security.selinux=0x73797374656d5f753a6f626a6563745f723a756e
>>> 6c6162656c65645f743a733000
>>> >    trusted.gfid=0x00000000000000000000000000000001
>>> >    trusted.glusterfs.dht=0x000000010000000000000000ffffffff
>>> >    trusted.glusterfs.volume-id=0xcfceb3535f0e4cf18b533ccfb1f091d3
>>> >glusterp1.graywitch.co.nz:
>>> >    getfattr: Removing leading '/' from absolute path names
>>> >    # file: bricks/brick1/gv0
>>> >
>>> >security.selinux=0x73797374656d5f753a6f626a6563745f723a756e
>>> 6c6162656c65645f743a733000
>>> >    trusted.gfid=0x00000000000000000000000000000001
>>> >    trusted.glusterfs.dht=0x000000010000000000000000ffffffff
>>> >    trusted.glusterfs.volume-id=0xcfceb3535f0e4cf18b533ccfb1f091d3
>>> >root at salt-001:~# salt gluster* cmd.run 'gluster volume heal gv0 info'
>>> >glusterp2.graywitch.co.nz:
>>> >    Brick glusterp1:/bricks/brick1/gv0
>>> >    <gfid:eafb8799-4e7a-4264-9213-26997c5a4693> - Is in split-brain
>>> >
>>> >    Status: Connected
>>> >    Number of entries: 1
>>> >
>>> >    Brick glusterp2:/bricks/brick1/gv0
>>> >    <gfid:eafb8799-4e7a-4264-9213-26997c5a4693> - Is in split-brain
>>> >
>>> >    Status: Connected
>>> >    Number of entries: 1
>>> >
>>> >    Brick glusterp3:/bricks/brick1/gv0
>>> >    <gfid:eafb8799-4e7a-4264-9213-26997c5a4693> - Is in split-brain
>>> >
>>> >    Status: Connected
>>> >    Number of entries: 1
>>> >glusterp3.graywitch.co.nz:
>>> >    Brick glusterp1:/bricks/brick1/gv0
>>> >    <gfid:eafb8799-4e7a-4264-9213-26997c5a4693> - Is in split-brain
>>> >
>>> >    Status: Connected
>>> >    Number of entries: 1
>>> >
>>> >    Brick glusterp2:/bricks/brick1/gv0
>>> >    <gfid:eafb8799-4e7a-4264-9213-26997c5a4693> - Is in split-brain
>>> >
>>> >    Status: Connected
>>> >    Number of entries: 1
>>> >
>>> >    Brick glusterp3:/bricks/brick1/gv0
>>> >    <gfid:eafb8799-4e7a-4264-9213-26997c5a4693> - Is in split-brain
>>> >
>>> >    Status: Connected
>>> >    Number of entries: 1
>>> >glusterp1.graywitch.co.nz:
>>> >    Brick glusterp1:/bricks/brick1/gv0
>>> >    <gfid:eafb8799-4e7a-4264-9213-26997c5a4693> - Is in split-brain
>>> >
>>> >    Status: Connected
>>> >    Number of entries: 1
>>> >
>>> >    Brick glusterp2:/bricks/brick1/gv0
>>> >    <gfid:eafb8799-4e7a-4264-9213-26997c5a4693> - Is in split-brain
>>> >
>>> >    Status: Connected
>>> >    Number of entries: 1
>>> >
>>> >    Brick glusterp3:/bricks/brick1/gv0
>>> >    <gfid:eafb8799-4e7a-4264-9213-26997c5a4693> - Is in split-brain
>>> >
>>> >    Status: Connected
>>> >    Number of entries: 1
>>> >root at salt-001:~# salt gluster* cmd.run 'getfattr -d -m . -e hex
>>> >/bricks/brick1/'
>>> >glusterp2.graywitch.co.nz:
>>> >glusterp3.graywitch.co.nz:
>>> >glusterp1.graywitch.co.nz:
>>> >root at salt-001:~# salt gluster* cmd.run 'gluster volume info gv0'
>>> >glusterp2.graywitch.co.nz:
>>> >
>>> >    Volume Name: gv0
>>> >    Type: Replicate
>>> >    Volume ID: cfceb353-5f0e-4cf1-8b53-3ccfb1f091d3
>>> >    Status: Started
>>> >    Snapshot Count: 0
>>> >    Number of Bricks: 1 x 3 = 3
>>> >    Transport-type: tcp
>>> >    Bricks:
>>> >    Brick1: glusterp1:/bricks/brick1/gv0
>>> >    Brick2: glusterp2:/bricks/brick1/gv0
>>> >    Brick3: glusterp3:/bricks/brick1/gv0
>>> >    Options Reconfigured:
>>> >    transport.address-family: inet
>>> >    nfs.disable: on
>>> >    performance.client-io-threads: off
>>> >glusterp1.graywitch.co.nz:
>>> >
>>> >    Volume Name: gv0
>>> >    Type: Replicate
>>> >    Volume ID: cfceb353-5f0e-4cf1-8b53-3ccfb1f091d3
>>> >    Status: Started
>>> >    Snapshot Count: 0
>>> >    Number of Bricks: 1 x 3 = 3
>>> >    Transport-type: tcp
>>> >    Bricks:
>>> >    Brick1: glusterp1:/bricks/brick1/gv0
>>> >    Brick2: glusterp2:/bricks/brick1/gv0
>>> >    Brick3: glusterp3:/bricks/brick1/gv0
>>> >    Options Reconfigured:
>>> >    performance.client-io-threads: off
>>> >    nfs.disable: on
>>> >    transport.address-family: inet
>>> >glusterp3.graywitch.co.nz:
>>> >
>>> >    Volume Name: gv0
>>> >    Type: Replicate
>>> >    Volume ID: cfceb353-5f0e-4cf1-8b53-3ccfb1f091d3
>>> >    Status: Started
>>> >    Snapshot Count: 0
>>> >    Number of Bricks: 1 x 3 = 3
>>> >    Transport-type: tcp
>>> >    Bricks:
>>> >    Brick1: glusterp1:/bricks/brick1/gv0
>>> >    Brick2: glusterp2:/bricks/brick1/gv0
>>> >    Brick3: glusterp3:/bricks/brick1/gv0
>>> >    Options Reconfigured:
>>> >    performance.client-io-threads: off
>>> >    nfs.disable: on
>>> >    transport.address-family: inet
>>> >root at salt-001:~#
>>> >
>>> >===========
>>> >
>>> >
>>> >How do I find what  "eafb8799-4e7a-4264-9213-26997c5a4693"  is?
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180523/091056d7/attachment.html>


More information about the Gluster-users mailing list