<div dir="ltr">Next I ran a test and your find works....I am wondering if I can simply delete this GFID?<div><br></div><div>8><-----</div><div><div>[root@glusterp2 fb]# find /bricks/brick1/gv0/ -samefile /bricks/brick1/gv0/.glusterfs/74/75/7475bd15-05a6-45c2-b395-bc9fd3d1763f</div><div>/bricks/brick1/gv0/.glusterfs/74/75/7475bd15-05a6-45c2-b395-bc9fd3d1763f</div><div>/bricks/brick1/gv0/dell6430-001/virtual-machines-backups/21-3-2018/images/debian9-002.qcow2</div><div>[root@glusterp2 fb]#</div></div><div>8><-----</div><div><br></div><div>On another node also no return,</div><div><br></div><div>8><----</div><div><div>[root@glusterp1 ~]# find /bricks/brick1/gv0 -samefile /bricks/brick1/gv0/.glusterfs/ea/fb/eafb8799-4e7a-4264-9213-26997c5a4693</div><div>/bricks/brick1/gv0/.glusterfs/ea/fb/eafb8799-4e7a-4264-9213-26997c5a4693</div><div>[root@glusterp1 ~]# </div></div><div>8><----</div><div><br></div><div>and on the final node,</div><div>8><---</div><div><div>[root@glusterp3 ~]# find /bricks/brick1/gv0 -samefile /bricks/brick1/gv0/.glusterfs/ea/fb/eafb8799-4e7a-4264-9213-26997c5a4693</div><div>/bricks/brick1/gv0/.glusterfs/ea/fb/eafb8799-4e7a-4264-9213-26997c5a4693</div><div>[root@glusterp3 ~]# </div></div><div>8><---</div><div><br></div><div>So none of the 3 nodes has an actual file?</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 23 May 2018 at 08:35, Thing <span dir="ltr"><<a href="mailto:thing.thing@gmail.com" target="_blank">thing.thing@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I tried looking for a file of the same size and the gfid doesnt show up,<div><br></div><div>8><---</div><div><div>[root@glusterp2 fb]# pwd</div><div>/bricks/brick1/gv0/.glusterfs/<wbr>ea/fb</div><div>[root@glusterp2 fb]# ls -al</div><div>total 3130892</div><div>drwx------. 2 root root 64 May 22 13:01 .</div><div>drwx------. 4 root root 24 May 8 14:27 ..</div><div>-rw-------. 1 root root 3294887936 May 4 11:07 eafb8799-4e7a-4264-9213-<wbr>26997c5a4693</div><div>-rw-r--r--. 1 root root 1396 May 22 13:01 gfid.run</div><div><br></div><div>so the gfid seems large....but du cant see it...</div><div><br></div><div>[root@glusterp2 fb]# du -a /bricks/brick1/gv0 | sort -n -r | head -n 10</div><div>275411712<span style="white-space:pre-wrap"> </span>/bricks/brick1/gv0</div><div>275411696<span style="white-space:pre-wrap"> </span>/bricks/brick1/gv0/.glusterfs</div><div>22484988<span style="white-space:pre-wrap"> </span>/bricks/brick1/gv0/.glusterfs/<wbr>57</div><div>20974980<span style="white-space:pre-wrap"> </span>/bricks/brick1/gv0/.glusterfs/<wbr>a5</div><div>20974976<span style="white-space:pre-wrap"> </span>/bricks/brick1/gv0/.glusterfs/<wbr>d5/99/d5991797-848d-4d60-b9dc-<wbr>d31174f63f72</div><div>20974976<span style="white-space:pre-wrap"> </span>/bricks/brick1/gv0/.glusterfs/<wbr>d5/99</div><div>20974976<span style="white-space:pre-wrap"> </span>/bricks/brick1/gv0/.glusterfs/<wbr>d5</div><div>20974976<span style="white-space:pre-wrap"> </span>/bricks/brick1/gv0/.glusterfs/<wbr>a5/27/a5279083-4d24-4dc8-be2d-<wbr>4f507c5372cf</div><div>20974976<span style="white-space:pre-wrap"> </span>/bricks/brick1/gv0/.glusterfs/<wbr>a5/27</div><div>20974976<span style="white-space:pre-wrap"> </span>/bricks/brick1/gv0/.glusterfs/<wbr>74/75/7475bd15-05a6-45c2-b395-<wbr>bc9fd3d1763f</div><div>[root@glusterp2 fb]# </div></div><div><br></div><div>8><---</div><div><br></div><div><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On 23 May 2018 at 08:29, Thing <span dir="ltr"><<a href="mailto:thing.thing@gmail.com" target="_blank">thing.thing@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>I tried this already.</div><div><br></div><div>8><---</div><div>[root@glusterp2 fb]# find /bricks/brick1/gv0 -samefile /bricks/brick1/gv0/.glusterfs/<wbr>ea/fb/eafb8799-4e7a-4264-9213-<wbr>26997c5a4693</div><div>/bricks/brick1/gv0/.glusterfs/<wbr>ea/fb/eafb8799-4e7a-4264-9213-<wbr>26997c5a4693</div><div>[root@glusterp2 fb]#</div><div>8><---</div><div><br></div><div>gluster 4</div><div>Centos 7.4</div><div><br></div><div>8><---</div><div>df -h</div><div><div>[root@glusterp2 fb]# df -h</div><span><div>Filesystem Size Used Avail Use% Mounted on</div><div>/dev/mapper/centos-root 19G 3.4G 16G 19% /</div><div>devtmpfs 3.8G 0 3.8G 0% /dev</div><div>tmpfs 3.8G 12K 3.8G 1% /dev/shm</div></span><span><div>tmpfs 3.8G 9.0M 3.8G 1% /run</div><div>tmpfs 3.8G 0 3.8G 0% /sys/fs/cgroup</div></span><span><div>/dev/mapper/centos-data1 112G 33M 112G 1% /data1</div></span><div>/dev/mapper/centos-var 19G 219M 19G 2% /var</div><span><div>/dev/mapper/centos-home 47G 38M 47G 1% /home</div></span><span><div>/dev/mapper/centos-var_lib 9.4G 178M 9.2G 2% /var/lib</div></span><div>/dev/mapper/vg--gluster--prod-<wbr>-1--2-gluster--prod--1--2 932G 263G 669G 29% /bricks/brick1</div><span><div>/dev/sda1 950M 235M 715M 25% /boot</div></span></div><div>8><---</div><div><br></div><div><br></div><div><br></div><div>So the output isnt helping..........</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote"><span>On 23 May 2018 at 00:29, Karthik Subrahmanya <span dir="ltr"><<a href="mailto:ksubrahm@redhat.com" target="_blank">ksubrahm@redhat.com</a>></span> wrote:<br></span><div><div class="m_7860373731795946834h5"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>Which version of gluster you are using?</div><div><br></div><div>You can find which file is that using the following command</div><div>find <brickpath> -samefile <brickpath/.glusterfs/<first two bits of gfid>/<next 2 bits of gfid>/<full gfid><br></div><div><br></div><div>Please provide the getfatr output of the file which is in split brain.</div><div>The steps to recover from split-brain can be found here,</div><div><a href="http://gluster.readthedocs.io/en/latest/Troubleshooting/resolving-splitbrain/" target="_blank">http://gluster.readthedocs.io/<wbr>en/latest/Troubleshooting/reso<wbr>lving-splitbrain/</a><br></div><div><br></div><div>HTH,</div><div>Karthik</div></div><div class="gmail_extra"><br><div class="gmail_quote"><span>On Tue, May 22, 2018 at 4:03 AM, Joe Julian <span dir="ltr"><<a href="mailto:joe@julianfamily.org" target="_blank">joe@julianfamily.org</a>></span> wrote:<br></span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span><span>How do I find what "eafb8799-4e7a-4264-9213-26997<wbr>c5a4693" is?<br>
<br>
</span></span><a href="https://docs.gluster.org/en/v3/Troubleshooting/gfid-to-path/" rel="noreferrer" target="_blank">https://docs.gluster.org/en/v3<wbr>/Troubleshooting/gfid-to-path/</a><div><div class="m_7860373731795946834m_-3767022637443224689h5"><br>
<div><div class="m_7860373731795946834m_-3767022637443224689m_3152635802185151233h5"><br>
On May 21, 2018 3:22:01 PM PDT, Thing <<a href="mailto:thing.thing@gmail.com" target="_blank">thing.thing@gmail.com</a>> wrote:<br>
>Hi,<br>
><br>
>I seem to have a split brain issue, but I cannot figure out where this<br>
>is<br>
>and what it is, can someone help me pls, I cant find what to fix here.<br>
><br>
>==========<br>
>root@salt-001:~# salt gluster* cmd.run 'df -h'<br>
><a href="http://glusterp2.graywitch.co.nz" rel="noreferrer" target="_blank">glusterp2.graywitch.co.nz</a>:<br>
> Filesystem Size Used<br>
>Avail Use% Mounted on<br>
> /dev/mapper/centos-root 19G 3.4G<br>
> 16G 19% /<br>
> devtmpfs 3.8G 0<br>
>3.8G 0% /dev<br>
> tmpfs 3.8G 12K<br>
>3.8G 1% /dev/shm<br>
> tmpfs 3.8G 9.1M<br>
>3.8G 1% /run<br>
> tmpfs 3.8G 0<br>
>3.8G 0% /sys/fs/cgroup<br>
> /dev/mapper/centos-tmp 3.8G 33M<br>
>3.7G 1% /tmp<br>
> /dev/mapper/centos-var 19G 213M<br>
> 19G 2% /var<br>
> /dev/mapper/centos-home 47G 38M<br>
> 47G 1% /home<br>
> /dev/mapper/centos-data1 112G 33M<br>
>112G 1% /data1<br>
> /dev/mapper/centos-var_lib 9.4G 178M<br>
>9.2G 2% /var/lib<br>
> /dev/mapper/vg--gluster--prod-<wbr>-1--2-gluster--prod--1--2 932G 264G<br>
>668G 29% /bricks/brick1<br>
> /dev/sda1 950M 235M<br>
>715M 25% /boot<br>
> tmpfs 771M 12K<br>
>771M 1% /run/user/42<br>
> glusterp2:gv0/glusterp2/images<wbr> 932G 273G<br>
>659G 30% /var/lib/libvirt/images<br>
> glusterp2:gv0 932G 273G<br>
>659G 30% /isos<br>
> tmpfs 771M 48K<br>
>771M 1% /run/user/1000<br>
> tmpfs 771M 0<br>
>771M 0% /run/user/0<br>
><a href="http://glusterp1.graywitch.co.nz" rel="noreferrer" target="_blank">glusterp1.graywitch.co.nz</a>:<br>
> Filesystem Size Used Avail Use%<br>
>Mounted on<br>
> /dev/mapper/centos-root 20G 3.5G 17G 18% /<br>
> devtmpfs 3.8G 0 3.8G 0%<br>
>/dev<br>
> tmpfs 3.8G 12K 3.8G 1%<br>
>/dev/shm<br>
> tmpfs 3.8G 9.0M 3.8G 1%<br>
>/run<br>
> tmpfs 3.8G 0 3.8G 0%<br>
>/sys/fs/cgroup<br>
> /dev/sda1 969M 206M 713M 23%<br>
>/boot<br>
> /dev/mapper/centos-home 50G 4.3G 46G 9%<br>
>/home<br>
> /dev/mapper/centos-tmp 3.9G 33M 3.9G 1%<br>
>/tmp<br>
> /dev/mapper/centos-data1 120G 36M 120G 1%<br>
>/data1<br>
> /dev/mapper/vg--gluster--prod<wbr>1-gluster--prod1 932G 260G 673G 28%<br>
>/bricks/brick1<br>
> /dev/mapper/centos-var 20G 413M 20G 3%<br>
>/var<br>
> /dev/mapper/centos00-var_lib 9.4G 179M 9.2G 2%<br>
>/var/lib<br>
> tmpfs 771M 8.0K 771M 1%<br>
>/run/user/42<br>
> glusterp1:gv0 932G 273G 659G 30%<br>
>/isos<br>
> glusterp1:gv0/glusterp1/image<wbr>s 932G 273G 659G 30%<br>
>/var/lib/libvirt/images<br>
><a href="http://glusterp3.graywitch.co.nz" rel="noreferrer" target="_blank">glusterp3.graywitch.co.nz</a>:<br>
> Filesystem Size Used<br>
>Avail Use% Mounted on<br>
> /dev/mapper/centos-root 20G 3.5G<br>
> 17G 18% /<br>
> devtmpfs 3.8G 0<br>
>3.8G 0% /dev<br>
> tmpfs 3.8G 12K<br>
>3.8G 1% /dev/shm<br>
> tmpfs 3.8G 9.0M<br>
>3.8G 1% /run<br>
> tmpfs 3.8G 0<br>
>3.8G 0% /sys/fs/cgroup<br>
> /dev/sda1 969M 206M<br>
>713M 23% /boot<br>
> /dev/mapper/centos-var 20G 206M<br>
> 20G 2% /var<br>
> /dev/mapper/centos-tmp 3.9G 33M<br>
>3.9G 1% /tmp<br>
> /dev/mapper/centos-home 50G 37M<br>
> 50G 1% /home<br>
> /dev/mapper/centos-data01 120G 33M<br>
>120G 1% /data1<br>
> /dev/mapper/centos00-var_lib 9.4G 180M<br>
>9.2G 2% /var/lib<br>
> /dev/mapper/vg--gluster--prod-<wbr>-1--3-gluster--prod--1--3 932G 262G<br>
>670G 29% /bricks/brick1<br>
> glusterp3:gv0 932G 273G<br>
>659G 30% /isos<br>
> glusterp3:gv0/glusterp3/images<wbr> 932G 273G<br>
>659G 30% /var/lib/libvirt/images<br>
> tmpfs 771M 12K<br>
>771M 1% /run/user/42<br>
>root@salt-001:~# salt gluster* cmd.run 'getfattr -d -m . -e hex<br>
>/bricks/brick1/gv0'<br>
><a href="http://glusterp2.graywitch.co.nz" rel="noreferrer" target="_blank">glusterp2.graywitch.co.nz</a>:<br>
> getfattr: Removing leading '/' from absolute path names<br>
> # file: bricks/brick1/gv0<br>
><br>
>security.selinux=0x7379737465<wbr>6d5f753a6f626a6563745f723a756e<wbr>6c6162656c65645f743a733000<br>
> trusted.gfid=0x000000000000000<wbr>00000000000000001<br>
> trusted.glusterfs.dht=0x000000<wbr>010000000000000000ffffffff<br>
> trusted.glusterfs.volume-id=0x<wbr>cfceb3535f0e4cf18b533ccfb1f091<wbr>d3<br>
><a href="http://glusterp3.graywitch.co.nz" rel="noreferrer" target="_blank">glusterp3.graywitch.co.nz</a>:<br>
> getfattr: Removing leading '/' from absolute path names<br>
> # file: bricks/brick1/gv0<br>
><br>
>security.selinux=0x7379737465<wbr>6d5f753a6f626a6563745f723a756e<wbr>6c6162656c65645f743a733000<br>
> trusted.gfid=0x000000000000000<wbr>00000000000000001<br>
> trusted.glusterfs.dht=0x000000<wbr>010000000000000000ffffffff<br>
> trusted.glusterfs.volume-id=0x<wbr>cfceb3535f0e4cf18b533ccfb1f091<wbr>d3<br>
><a href="http://glusterp1.graywitch.co.nz" rel="noreferrer" target="_blank">glusterp1.graywitch.co.nz</a>:<br>
> getfattr: Removing leading '/' from absolute path names<br>
> # file: bricks/brick1/gv0<br>
><br>
>security.selinux=0x7379737465<wbr>6d5f753a6f626a6563745f723a756e<wbr>6c6162656c65645f743a733000<br>
> trusted.gfid=0x000000000000000<wbr>00000000000000001<br>
> trusted.glusterfs.dht=0x000000<wbr>010000000000000000ffffffff<br>
> trusted.glusterfs.volume-id=0x<wbr>cfceb3535f0e4cf18b533ccfb1f091<wbr>d3<br>
>root@salt-001:~# salt gluster* cmd.run 'gluster volume heal gv0 info'<br>
><a href="http://glusterp2.graywitch.co.nz" rel="noreferrer" target="_blank">glusterp2.graywitch.co.nz</a>:<br>
> Brick glusterp1:/bricks/brick1/gv0<br>
> <gfid:eafb8799-4e7a-4264-9213-<wbr>26997c5a4693> - Is in split-brain<br>
><br>
> Status: Connected<br>
> Number of entries: 1<br>
><br>
> Brick glusterp2:/bricks/brick1/gv0<br>
> <gfid:eafb8799-4e7a-4264-9213-<wbr>26997c5a4693> - Is in split-brain<br>
><br>
> Status: Connected<br>
> Number of entries: 1<br>
><br>
> Brick glusterp3:/bricks/brick1/gv0<br>
> <gfid:eafb8799-4e7a-4264-9213-<wbr>26997c5a4693> - Is in split-brain<br>
><br>
> Status: Connected<br>
> Number of entries: 1<br>
><a href="http://glusterp3.graywitch.co.nz" rel="noreferrer" target="_blank">glusterp3.graywitch.co.nz</a>:<br>
> Brick glusterp1:/bricks/brick1/gv0<br>
> <gfid:eafb8799-4e7a-4264-9213-<wbr>26997c5a4693> - Is in split-brain<br>
><br>
> Status: Connected<br>
> Number of entries: 1<br>
><br>
> Brick glusterp2:/bricks/brick1/gv0<br>
> <gfid:eafb8799-4e7a-4264-9213-<wbr>26997c5a4693> - Is in split-brain<br>
><br>
> Status: Connected<br>
> Number of entries: 1<br>
><br>
> Brick glusterp3:/bricks/brick1/gv0<br>
> <gfid:eafb8799-4e7a-4264-9213-<wbr>26997c5a4693> - Is in split-brain<br>
><br>
> Status: Connected<br>
> Number of entries: 1<br>
><a href="http://glusterp1.graywitch.co.nz" rel="noreferrer" target="_blank">glusterp1.graywitch.co.nz</a>:<br>
> Brick glusterp1:/bricks/brick1/gv0<br>
> <gfid:eafb8799-4e7a-4264-9213-<wbr>26997c5a4693> - Is in split-brain<br>
><br>
> Status: Connected<br>
> Number of entries: 1<br>
><br>
> Brick glusterp2:/bricks/brick1/gv0<br>
> <gfid:eafb8799-4e7a-4264-9213-<wbr>26997c5a4693> - Is in split-brain<br>
><br>
> Status: Connected<br>
> Number of entries: 1<br>
><br>
> Brick glusterp3:/bricks/brick1/gv0<br>
> <gfid:eafb8799-4e7a-4264-9213-<wbr>26997c5a4693> - Is in split-brain<br>
><br>
> Status: Connected<br>
> Number of entries: 1<br>
>root@salt-001:~# salt gluster* cmd.run 'getfattr -d -m . -e hex<br>
>/bricks/brick1/'<br>
><a href="http://glusterp2.graywitch.co.nz" rel="noreferrer" target="_blank">glusterp2.graywitch.co.nz</a>:<br>
><a href="http://glusterp3.graywitch.co.nz" rel="noreferrer" target="_blank">glusterp3.graywitch.co.nz</a>:<br>
><a href="http://glusterp1.graywitch.co.nz" rel="noreferrer" target="_blank">glusterp1.graywitch.co.nz</a>:<br>
>root@salt-001:~# salt gluster* cmd.run 'gluster volume info gv0'<br>
><a href="http://glusterp2.graywitch.co.nz" rel="noreferrer" target="_blank">glusterp2.graywitch.co.nz</a>:<br>
><br>
> Volume Name: gv0<br>
> Type: Replicate<br>
> Volume ID: cfceb353-5f0e-4cf1-8b53-3ccfb1<wbr>f091d3<br>
> Status: Started<br>
> Snapshot Count: 0<br>
> Number of Bricks: 1 x 3 = 3<br>
> Transport-type: tcp<br>
> Bricks:<br>
> Brick1: glusterp1:/bricks/brick1/gv0<br>
> Brick2: glusterp2:/bricks/brick1/gv0<br>
> Brick3: glusterp3:/bricks/brick1/gv0<br>
> Options Reconfigured:<br>
> transport.address-family: inet<br>
> nfs.disable: on<br>
> performance.client-io-threads: off<br>
><a href="http://glusterp1.graywitch.co.nz" rel="noreferrer" target="_blank">glusterp1.graywitch.co.nz</a>:<br>
><br>
> Volume Name: gv0<br>
> Type: Replicate<br>
> Volume ID: cfceb353-5f0e-4cf1-8b53-3ccfb1<wbr>f091d3<br>
> Status: Started<br>
> Snapshot Count: 0<br>
> Number of Bricks: 1 x 3 = 3<br>
> Transport-type: tcp<br>
> Bricks:<br>
> Brick1: glusterp1:/bricks/brick1/gv0<br>
> Brick2: glusterp2:/bricks/brick1/gv0<br>
> Brick3: glusterp3:/bricks/brick1/gv0<br>
> Options Reconfigured:<br>
> performance.client-io-threads: off<br>
> nfs.disable: on<br>
> transport.address-family: inet<br>
><a href="http://glusterp3.graywitch.co.nz" rel="noreferrer" target="_blank">glusterp3.graywitch.co.nz</a>:<br>
><br>
> Volume Name: gv0<br>
> Type: Replicate<br>
> Volume ID: cfceb353-5f0e-4cf1-8b53-3ccfb1<wbr>f091d3<br>
> Status: Started<br>
> Snapshot Count: 0<br>
> Number of Bricks: 1 x 3 = 3<br>
> Transport-type: tcp<br>
> Bricks:<br>
> Brick1: glusterp1:/bricks/brick1/gv0<br>
> Brick2: glusterp2:/bricks/brick1/gv0<br>
> Brick3: glusterp3:/bricks/brick1/gv0<br>
> Options Reconfigured:<br>
> performance.client-io-threads: off<br>
> nfs.disable: on<br>
> transport.address-family: inet<br>
>root@salt-001:~#<br>
><br>
>===========<br>
><br>
><br>
>How do I find what "eafb8799-4e7a-4264-9213-26997<wbr>c5a4693" is?<br>
</div></div></div></div>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
</blockquote></div><br></div>
</blockquote></div></div></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>