[Gluster-users] "gfid differs on subvolume"

[Gambit15]

Hi Guys,
 I've got a distributed replica 2+1 (rep 3 arbiter 1) cluster, and it
appears a shard has been assigned different GFIDs on each replica set.

===============
[2018-11-29 10:05:12.035422] W [MSGID: 109009]
[dht-common.c:2148:dht_lookup_linkfile_cbk] 0-data-novo-dht:
/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846: gfid different on data
file on data-novo-replicate-1, gfid local =
00000000-0000-0000-0000-000000000000, gfid node =
492d52d6-e3d1-4ed4-918d-9cdab7a135e0
[2018-11-29 10:05:12.036120] W [MSGID: 109009]
[dht-common.c:1887:dht_lookup_everywhere_cbk] 0-data-novo-dht:
/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846: gfid differs on
subvolume data-novo-replicate-1, gfid local =
c7f6cc63-ae40-4d1a-aa6f-fe97f7912036, gfid node =
492d52d6-e3d1-4ed4-918d-9cdab7a135e0
[2018-11-29 10:05:12.036159] E [MSGID: 133010]
[shard.c:1582:shard_common_lookup_shards_cbk] 0-data-novo-shard: Lookup on
shard 1846 failed. Base file gfid = a46fd27c-5aa6-4fc8-b8e1-c097065e7096
[Stale file handle]
[2018-11-29 10:05:12.036184] W [fuse-bridge.c:2228:fuse_readv_cbk]
0-glusterfs-fuse: 6916126: READ => -1
gfid=a46fd27c-5aa6-4fc8-b8e1-c097065e7096 fd=0x7f12fa57f06c (Stale file
handle)
===============

==== FIRST REPLICA SET (v0-v2) ====

v0:~$ ls -l
/gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846
---------T. 2 root root 0 Oct 17 11:28
/gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846

v0:~$ getfattr -d -m . -e hex
/gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846
# file:
gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.gfid=0xc7f6cc63ae404d1aaa6ffe97f7912036
trusted.glusterfs.dht.linkto=0x646174612d6e6f766f2d7265706c69636174652d3100

v1:~$ ls -l
/gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846
---------T. 2 root root 0 Oct 17 11:28
/gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846

v1:~$ getfattr -d -m . -e hex
/gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846
# file:
gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.gfid=0xc7f6cc63ae404d1aaa6ffe97f7912036
trusted.glusterfs.dht.linkto=0x646174612d6e6f766f2d7265706c69636174652d3100

v2:~$ ls -l
/gluster/data-novo/arbiter/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846
---------T. 2 root root         0 Oct 17 11:28
/gluster/data-novo/arbiter/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846

v2:~$ getfattr -d -m . -e hex
/gluster/data-novo/arbiter/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846
# file:
gluster/data-novo/arbiter/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.gfid=0xc7f6cc63ae404d1aaa6ffe97f7912036
trusted.glusterfs.dht.linkto=0x646174612d6e6f766f2d7265706c69636174652d3100

==== SECOND REPLICA SET (v0-v2) ====

v2:~$ ls -l
/gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846
-rw-rw----. 2 root root 536870912 Nov 27 14:15
/gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846

v2:~$ getfattr -d -m . -e hex
/gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846
# file:
gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.bit-rot.version=0x02000000000000005b4a3790000750d5
trusted.gfid=0x492d52d6e3d14ed4918d9cdab7a135e0

v3:~$ ls -l
/gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846
-rw-rw----. 2 root root 536870912 Nov 27 14:15
/gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846

v3:~$ getfattr -d -m . -e hex
/gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846
# file:
gluster/data-novo/brick/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.bit-rot.version=0x04000000000000005bda0242000c7cdd
trusted.gfid=0x492d52d6e3d14ed4918d9cdab7a135e0

v0:~$ ls -l
/gluster/data-novo/arbiter/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846
-rw-rw----. 2 root root 0 Oct 17 11:28
/gluster/data-novo/arbiter/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846

v0:~$ getfattr -d -m . -e hex
/gluster/data-novo/arbiter/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846
# file:
gluster/data-novo/arbiter/.shard/a46fd27c-5aa6-4fc8-b8e1-c097065e7096.1846
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.bit-rot.version=0x03000000000000005bda02410001b61a
trusted.gfid=0x492d52d6e3d14ed4918d9cdab7a135e0

=====================================

So the shard has GFID " 0xc7f6cc63ae404d1aaa6ffe97f7912036" on the first
replica set, and GFID "0x492d52d6e3d14ed4918d9cdab7a135e0" on the second
replica set.

How do I go about correcting this?

Many thanks,
 Doug
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20181204/72542b8e/attachment.html>