<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p><br>
</p>
<div class="moz-cite-prefix">On 22/05/19 1:29 PM, Hu Bert wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAAV-98_LBRFEMV0OeN0=htuDNOKoLoKQHUZ0wuo9Sd792=fHnQ@mail.gmail.com">
<pre class="moz-quote-pre" wrap="">Hi Ravi,
mount path of the volume is /shared/public, so complete paths are
/shared/public/staticmap/120/710/ and
/shared/public/staticmap/120/710/120710351/ .
getfattr -n glusterfs.gfid.string /shared/public/staticmap/120/710/
getfattr: Removing leading '/' from absolute path names
# file: shared/public/staticmap/120/710/
glusterfs.gfid.string="751233b0-7789-4550-bd95-4dd9c8f57c19"
getfattr -n glusterfs.gfid.string /shared/public/staticmap/120/710/120710351/
getfattr: Removing leading '/' from absolute path names
# file: shared/public/staticmap/120/710/120710351/
glusterfs.gfid.string="eaf2f31e-b4a7-4fa8-b710-d6ff9cd4eace"
So that fits. It somehow took a couple of attempts to resolve this,
and none of the commands seem to have "officially" succeeded:
gluster3 (host with the "fail"):
gluster volume heal workdata split-brain source-brick
gluster1:/gluster/md4/workdata
/shared/public/staticmap/120/710/120710351/
Lookup failed on /shared/public/staticmap/120/710:No such file or directory</pre>
</blockquote>
<p>The file path given to this command must be the absolute path <i>as
seen from the root of the volume</i>. So the location where it
is mounted (/shared/public) must be omitted. Only
/staticmap/120/710/120710351/ is required.</p>
<p>HTH,</p>
<p>Ravi<br>
</p>
<blockquote type="cite"
cite="mid:CAAV-98_LBRFEMV0OeN0=htuDNOKoLoKQHUZ0wuo9Sd792=fHnQ@mail.gmail.com">
<pre class="moz-quote-pre" wrap="">
Volume heal failed.
gluster1 ("good" host):
gluster volume heal workdata split-brain source-brick
gluster1:/gluster/md4/workdata
/shared/public/staticmap/120/710/120710351/
Lookup failed on /shared/public/staticmap/120/710:No such file or directory
Volume heal failed.
Only in the logs i see:
[2019-05-22 07:42:22.004182] I [MSGID: 108026]
[afr-self-heal-metadata.c:52:__afr_selfheal_metadata_do]
0-workdata-replicate-0: performing metadata selfheal on
eaf2f31e-b4a7-4fa8-b710-d6ff9cd4eace
[2019-05-22 07:42:22.008502] I [MSGID: 108026]
[afr-self-heal-common.c:1729:afr_log_selfheal] 0-workdata-replicate-0:
Completed metadata selfheal on eaf2f31e-b4a7-4fa8-b710-d6ff9cd4eace.
sources=0 [1] sinks=2
And via "gluster volume heal workdata statistics heal-count" there are
0 entries left. Files/directories are there. Happened the first time
with this setup, but everything ok now.
Thx for your fast help :-)
Hubert
Am Mi., 22. Mai 2019 um 09:32 Uhr schrieb Ravishankar N
<a class="moz-txt-link-rfc2396E" href="mailto:ravishankar@redhat.com"><ravishankar@redhat.com></a>:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">
On 22/05/19 12:39 PM, Hu Bert wrote:
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">Hi @ll,
today i updated and rebooted the 3 servers of my replicate 3 setup;
after the 3rd one came up again i noticed this error:
[2019-05-22 06:41:26.781165] E [MSGID: 108008]
[afr-self-heal-common.c:392:afr_gfid_split_brain_source]
0-workdata-replicate-0: Gfid mismatch detected for
<gfid:751233b0-7789-4550-bd95-4dd9c8f57c19>/120710351>,
82025ab3-8034-4257-9628-d8ebde909629 on workdata-client-2 and
eaf2f31e-b4a7-4fa8-b710-d6ff9cd4eace on workdata-client-1.
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">
120710351 seems to be the entry that is in split-brain. Is
/staticmap/120/710/120710351 the complete path to that entry? (check if
gfid:751233b0-7789-4550-bd95-4dd9c8f57c19 corresponds to the gfid of 710).
You can then try "gluster volume heal workdata split-brain source-brick
gluster1:/gluster/md4/workdata /staticmap/120/710/120710351"
-Ravi
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">[2019-05-22 06:41:27.069969] W [MSGID: 108027]
[afr-common.c:2270:afr_attempt_readsubvol_set] 0-workdata-replicate-0:
no read subvols for /staticmap/120/710/120710351
[2019-05-22 06:41:27.808532] W [fuse-bridge.c:582:fuse_entry_cbk]
0-glusterfs-fuse: 1834335: LOOKUP() /staticmap/120/710/120710351 => -1
(Transport endpoint is not connected)
A simple 'gluster volume heal workdata' didn't help; 'gluster volume
heal workdata info' says:
Brick gluster1:/gluster/md4/workdata
/staticmap/120/710
/staticmap/120/710/120710351
<gfid:fe7fdbe8-9a39-4793-8d38-6dfdd3d5089b>
Status: Connected
Number of entries: 3
Brick gluster2:/gluster/md4/workdata
/staticmap/120/710
/staticmap/120/710/120710351
<gfid:fe7fdbe8-9a39-4793-8d38-6dfdd3d5089b>
Status: Connected
Number of entries: 3
Brick gluster3:/gluster/md4/workdata
/staticmap/120/710/120710351
Status: Connected
Number of entries: 1
There's a mismatch in one directory; I tried to follow these instructions:
<a class="moz-txt-link-freetext" href="https://docs.gluster.org/en/latest/Troubleshooting/resolving-splitbrain/">https://docs.gluster.org/en/latest/Troubleshooting/resolving-splitbrain/</a>
gluster volume heal workdata split-brain source-brick
gluster1:/gluster/md4/workdata
gfid:fe7fdbe8-9a39-4793-8d38-6dfdd3d5089b
Healing gfid:fe7fdbe8-9a39-4793-8d38-6dfdd3d5089b failed: File not in
split-brain.
Volume heal failed.
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">
Is there any other documentation for gfid mismatch and how to resolve this?
Thx,
Hubert
_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="https://lists.gluster.org/mailman/listinfo/gluster-users">https://lists.gluster.org/mailman/listinfo/gluster-users</a>
</pre>
</blockquote>
</blockquote>
</blockquote>
</body>
</html>