<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hi Guys,<br></div> I've got a distributed replica 2+1 (rep 3 arbiter 1) cluster, and it appears a shard has been assigned different GFIDs on each replica set.</div><div dir="ltr"><br></div><div dir="ltr">===============<br>[2018-11-29 10:05:12.035422] W [MSGID: 109009] [dht-common.c:2148:dht_lookup_linkfile_cbk] 0-data-novo-dht: /.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846: gfid different on data file on data-novo-replicate-1, gfid local = 00000000-0000-0000-0000-000000000000, gfid node = 492d52d6-e3d1-4ed4-918d-9cdab7a135e0<br>[2018-11-29 10:05:12.036120] W [MSGID: 109009] [dht-common.c:1887:dht_lookup_everywhere_cbk] 0-data-novo-dht: /.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846: gfid differs on subvolume data-novo-replicate-1, gfid local = c7f6cc63-ae40-4d1a-aa6f-fe97f7912036, gfid node = 492d52d6-e3d1-4ed4-918d-9cdab7a135e0<br>[2018-11-29 10:05:12.036159] E [MSGID: 133010] [shard.c:1582:shard_common_lookup_shards_cbk] 0-data-novo-shard: Lookup on shard 1846 failed. Base file gfid = a46fd27c-5aa6-4fc8-b8e1-c097065e7096 [Stale file handle]<br>[2018-11-29 10:05:12.036184] W [fuse-bridge.c:2228:fuse_readv_cbk] 0-glusterfs-fuse: 6916126: READ => -1 gfid=a46fd27c-5aa6-4fc8-b8e1-c097065e7096 fd=0x7f12fa57f06c (Stale file handle)<br>===============</div><div dir="ltr"><br></div><div>==== FIRST REPLICA SET (v0-v2) ====</div><div><br></div><div>v0:~$ ls -l /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846<br></div><div>---------T. 2 root root 0 Oct 17 11:28 /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846</div><div><br></div><div>v0:~$ getfattr -d -m . -e hex /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846</div><div># file: gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846<br>security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000<br>trusted.gfid=0xc7f6cc63ae404d1aaa6ffe97f7912036<br>trusted.glusterfs.dht.linkto=0x646174612d6e6f766f2d7265706c69636174652d3100<br></div><div><br></div><div>v1:~$
ls -l /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846
</div><div>---------T. 2 root root 0 Oct 17 11:28 /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846<br></div><div><br></div><div>v1:~$ getfattr -d -m . -e hex /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846<br></div><div># file: gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846<br>security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000<br>trusted.gfid=0xc7f6cc63ae404d1aaa6ffe97f7912036<br>trusted.glusterfs.dht.linkto=0x646174612d6e6f766f2d7265706c69636174652d3100<br></div><div><br></div><div>
v2:~$
ls -l /gluster/data-novo/arbiter/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846 <br></div><div>---------T. 2 root root 0 Oct 17 11:28 /gluster/data-novo/arbiter/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846<br><br></div><div>v2:~$ getfattr -d -m . -e hex /gluster/data-novo/arbiter/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846<br></div><div># file: gluster/data-novo/arbiter/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846<br>security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000<br>trusted.gfid=0xc7f6cc63ae404d1aaa6ffe97f7912036<br>trusted.glusterfs.dht.linkto=0x646174612d6e6f766f2d7265706c69636174652d3100<br></div><div><br></div><div></div><div>==== SECOND REPLICA SET (v0-v2) ====</div><div><br></div><div>
<div>
v2:~$
ls -l /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846 <br></div><div>-rw-rw----. 2 root root 536870912 Nov 27 14:15 /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846<br><br></div><div>v2:~$ getfattr -d -m . -e hex /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846</div>
</div><div># file: gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846<br>security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000<br>trusted.afr.dirty=0x000000000000000000000000<br>trusted.bit-rot.version=0x02000000000000005b4a3790000750d5<br>trusted.gfid=0x492d52d6e3d14ed4918d9cdab7a135e0<br><br></div><div>v3:~$
ls -l /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846 <br></div><div>-rw-rw----. 2 root root 536870912 Nov 27 14:15 /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846<br><br></div><div>
v3:~$ getfattr -d -m . -e hex /gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846</div><div dir="ltr"># file: gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846<br>security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000<br>trusted.afr.dirty=0x000000000000000000000000<br>trusted.bit-rot.version=0x04000000000000005bda0242000c7cdd<br>trusted.gfid=0x492d52d6e3d14ed4918d9cdab7a135e0<br><br></div><div dir="ltr">v0:~$
ls -l /gluster/data-novo/arbiter/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846 <br></div><div dir="ltr">-rw-rw----. 2 root root 0 Oct 17 11:28 /gluster/data-novo/arbiter/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846<br><br></div><div dir="ltr">v0:~$ getfattr -d -m . -e hex /gluster/data-novo/arbiter/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846</div><div dir="ltr"># file: gluster/data-novo/arbiter/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846<br>security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000<br>trusted.afr.dirty=0x000000000000000000000000<br>trusted.bit-rot.version=0x03000000000000005bda02410001b61a<br>trusted.gfid=0x492d52d6e3d14ed4918d9cdab7a135e0<br></div><div dir="ltr"><br></div><div dir="ltr">=====================================</div><div dir="ltr"><br></div><div>So the shard has GFID "
0xc7f6cc63ae404d1aaa6ffe97f7912036" on the first replica set, and GFID "0x492d52d6e3d14ed4918d9cdab7a135e0" on the second replica set.</div><div><br></div><div>How do I go about correcting this?</div><div><br></div><div>Many thanks,</div><div> Doug<br></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>