[Gluster-users] same gfid for two directories and dht_layout_mismatch and failed to combine inode error msgs

Mon Aug 26 10:12:14 UTC 2019

Hi,

I am facing an issue in my gluster setup.

glusterfs version 3.10.1
distributed disperse volume
glusterfs-fuse client.

One of node SAS BackPlane had some problem so my drives are ejected from
service, this caused the brick process to terminate but glusterd daemon was
still running.
gluster peer status showing faulty node as online but bricks in the faulty
node is offline. In the meantime, client upload process was going on, cpu
load and memory usage on other nodes keep on increasing.

so, I restarted the problem node and started glusterd service after
mounting all disks. now gluster status showing all brick is online.

now the problem I am facing here is folders which are uploaded or moved
between one path to another path when bringing back problem node to live is
not listing
files originally they had and I am not able to delete the folder even
though I have permission, throwing error "permission denied" and "folder
not empty"
but doing "ls" of the showing empty folder.

error msg in fuse mount log.

[2019-08-26 07:33:13.005165] W [MSGID: 122006]
[ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to
combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1,
uid: 0-0, gid: 0-0, rdev: 0-0, size: 0-512, mode: 100644-100644)
[2019-08-26 07:33:13.005211] N [MSGID: 122029]
[ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching
iatt in answers of 'GF_FOP_LOOKUP'
[2019-08-26 07:33:21.020529] W [MSGID: 122006]
[ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to
combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1,
uid: 0-0, gid: 0-0, rdev: 0-0, size: 512-0, mode: 100644-100644)
[2019-08-26 07:34:59.005318] W [MSGID: 122006]
[ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to
combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1,
uid: 0-0, gid: 0-0, rdev: 0-0, size: 512-0, mode: 100644-100644)
The message "W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine]
0-glusterdr-disperse-91: Failed to combine iatt (inode:
10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0,
rdev: 0-0, size: 0-512, mode: 100644-100644)" repeated 23 times between
[2019-08-26 07:33:13.005165] and [2019-08-26 07:35:11.005147]
The message "N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup]
0-glusterdr-disperse-91: Mismatching iatt in answers of 'GF_FOP_LOOKUP'"
repeated 25 times between [2019-08-26 07:33:13.005211] and [2019-08-26
07:35:11.005149]
[2019-08-26 07:35:13.006967] W [MSGID: 122006]
[ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to
combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1,
uid: 0-0, gid: 0-0, rdev: 0-0, size: 0-512, mode: 100644-100644)
[2019-08-26 07:35:13.007028] N [MSGID: 122029]
[ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching
iatt in answers of 'GF_FOP_LOOKUP'
[2019-08-26 07:35:37.005121] W [MSGID: 122006]
[ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to
combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1,
uid: 0-0, gid: 0-0, rdev: 0-0, size: 512-0, mode: 100644-100644)
The message "W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine]
0-glusterdr-disperse-91: Failed to combine iatt (inode:
10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0,
rdev: 0-0, size: 512-0, mode: 100644-100644)" repeated 2 times between
[2019-08-26 07:35:37.005121] and [2019-08-26 07:36:21.012016]
The message "W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine]
0-glusterdr-disperse-91: Failed to combine iatt (inode:
10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0,
rdev: 0-0, size: 0-512, mode: 100644-100644)" repeated 21 times between
[2019-08-26 07:35:13.006967] and [2019-08-26 07:37:11.005302]
The message "N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup]
0-glusterdr-disperse-91: Mismatching iatt in answers of 'GF_FOP_LOOKUP'"
repeated 24 times between [2019-08-26 07:35:13.007028] and [2019-08-26
07:37:11.005305]
[2019-08-26 07:37:13.006618] W [MSGID: 122006]
[ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to
combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1,
uid: 0-0, gid: 0-0, rdev: 0-0, size: 512-0, mode: 100644-100644)
[2019-08-26 07:37:13.006721] N [MSGID: 122029]
[ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching
iatt in answers of 'GF_FOP_LOOKUP'
[2019-08-26 07:37:21.024033] W [MSGID: 122006]
[ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to
combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1,
uid: 0-0, gid: 0-0, rdev: 0-0, size: 0-512, mode: 100644-100644)
[2019-08-26 07:38:22.026108] W [MSGID: 122006]
[ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-139: Failed to
combine iatt (inode: 11541463450251411113-11541463450251411113, links: 1-1,
uid: 1000-1000, gid: 1000-1000, rdev: 0-0, size: 517816320-517799936, mode:
100600-100600)
[2019-08-26 07:38:22.026144] N [MSGID: 122029]
[ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-139: Mismatching
iatt in answers of 'GF_FOP_LOOKUP'
The message "W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine]
0-glusterdr-disperse-91: Failed to combine iatt (inode:
10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0,
rdev: 0-0, size: 0-512, mode: 100644-100644)" repeated 19 times between
[2019-08-26 07:37:21.024033] and [2019-08-26 07:39:10.005761]
The message "W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine]
0-glusterdr-disperse-91: Failed to combine iatt (inode:
10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0,
rdev: 0-0, size: 512-0, mode: 100644-100644)" repeated 3 times between
[2019-08-26 07:37:13.006618] and [2019-08-26 07:39:11.004587]
The message "N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup]
0-glusterdr-disperse-91: Mismatching iatt in answers of 'GF_FOP_LOOKUP'"
repeated 23 times between [2019-08-26 07:37:13.006721] and [2019-08-26
07:39:11.004591]
[2019-08-26 07:39:13.007257] W [MSGID: 122006]
[ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to
combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1,
uid: 0-0, gid: 0-0, rdev: 0-0, size: 0-512, mode: 100644-100644)
[2019-08-26 07:39:13.007335] N [MSGID: 122029]
[ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching
iatt in answers of 'GF_FOP_LOOKUP'
[2019-08-26 07:39:39.004926] W [MSGID: 122006]
[ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to
combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1,
uid: 0-0, gid: 0-0, rdev: 0-0, size: 512-0, mode: 100644-100644)
The message "W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine]
0-glusterdr-disperse-91: Failed to combine iatt (inode:
10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0,
rdev: 0-0, size: 512-0, mode: 100644-100644)" repeated 2 times between
[2019-08-26 07:39:39.004926] and [2019-08-26 07:40:11.004852]
The message "W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine]
0-glusterdr-disperse-91: Failed to combine iatt (inode:
10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0,
rdev: 0-0, size: 0-512, mode: 100644-100644)" repeated 23 times between
[2019-08-26 07:39:13.007257] and [2019-08-26 07:41:13.006759]
The message "N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup]
0-glusterdr-disperse-91: Mismatching iatt in answers of 'GF_FOP_LOOKUP'"
repeated 26 times between [2019-08-26 07:39:13.007335] and [2019-08-26
07:41:13.006761]
[2019-08-26 07:41:21.010830] W [MSGID: 122006]
[ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to
combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1,
uid: 0-0, gid: 0-0, rdev: 0-0, size: 512-0, mode: 100644-100644)
[2019-08-26 07:41:21.010861] N [MSGID: 122029]
[ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching
iatt in answers of 'GF_FOP_LOOKUP'

The message "N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup]
0-glusterdr-disperse-91: Mismatching iatt in answers of 'GF_FOP_LOOKUP'"
repeated 4 times between [2019-08-26 07:18:22.003323] and [2019-08-26
07:18:42.019306]
[2019-08-26 07:18:47.003349] W [MSGID: 122006]
[ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to
combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1,
uid: 0-0, gid: 0-0, rdev: 0-0, size: 512-0, mode: 100644-100644)
[2019-08-26 07:18:47.003392] N [MSGID: 122029]
[ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching
iatt in answers of 'GF_FOP_LOOKUP'
The message "W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine]
0-glusterdr-disperse-137: Failed to combine iatt (inode:
12704371770406674620-12704371770406674620, links: 1-1, uid: 1000-1000, gid:
1000-1000, rdev: 0-0, size: 696369152-696352768, mode: 100600-100600)"
repeated 2 times between [2019-08-26 07:17:11.019084] and [2019-08-26
07:17:11.019201]
The message "N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup]
0-glusterdr-disperse-137: Mismatching iatt in answers of 'GF_FOP_LOOKUP'"
repeated 2 times between [2019-08-26 07:17:11.019111] and [2019-08-26
07:17:11.019202]
The message "W [MSGID: 122006] [ec-combine.c:191:ec_iatt_combine]
0-glusterdr-disperse-91: Failed to combine iatt (inode:
10327606865298950809-10327606865298950809, links: 1-1, uid: 0-0, gid: 0-0,
rdev: 0-0, size: 0-512, mode: 100644-100644)" repeated 19 times between
[2019-08-26 07:17:08.003143] and [2019-08-26 07:18:50.003418]
The message "N [MSGID: 122029] [ec-generic.c:684:ec_combine_lookup]
0-glusterdr-disperse-91: Mismatching iatt in answers of 'GF_FOP_LOOKUP'"
repeated 4 times between [2019-08-26 07:18:47.003392] and [2019-08-26
07:18:50.003420]
[2019-08-26 07:19:08.002986] W [MSGID: 122006]
[ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to
combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1,
uid: 0-0, gid: 0-0, rdev: 0-0, size: 0-512, mode: 100644-100644)
[2019-08-26 07:19:08.003012] N [MSGID: 122029]
[ec-generic.c:684:ec_combine_lookup] 0-glusterdr-disperse-91: Mismatching
iatt in answers of 'GF_FOP_LOOKUP'
[2019-08-26 07:19:39.011760] W [MSGID: 122006]
[ec-combine.c:191:ec_iatt_combine] 0-glusterdr-disperse-91: Failed to
combine iatt (inode: 10327606865298950809-10327606865298950809, links: 1-1,
uid: 0-0, gid: 0-0, rdev: 0-0, size: 512-0, mode: 100644-100644)
[2019-08-26 09:21:53.015705] W [MSGID: 122006]
[ec-combine.c:191:ec_iatt_combine] 0-qubevaultdr-disperse-139: Failed to
combine iatt (inode: 11196349995399397487-11196349995399397487, links: 1-1,
uid: 1000-1000, gid: 1000-1000, rdev: 0-0, size: 1177141248-1177124864,
mode: 100600-100600)
[2019-08-26 09:21:53.015747] N [MSGID: 122029]
[ec-generic.c:684:ec_combine_lookup] 0-qubevaultdr-disperse-139:
Mismatching iatt in answers of 'GF_FOP_LOOKUP'

[2019-08-26 04:49:11.580839] I [MSGID: 109064]
[dht-layout.c:837:dht_layout_dir_mismatch] 0-glusterdr-dht: subvol:
glusterdr-disperse-135; inode layout - 3251903722 - 3282582058 - 1; disk
layout - 0 - 0 - 0
[2019-08-26 04:49:11.580850] I [MSGID: 109018]
[dht-common.c:1096:dht_revalidate_cbk] 0-glusterdr-dht: Mismatching layouts
for /Incoming/client_folder, gfid = 00000000-0000-0000-0000-000000000000
[2019-08-26 04:49:11.580986] I [MSGID: 109064]
[dht-layout.c:837:dht_layout_dir_mismatch] 0-glusterdr-dht: subvol:
glusterdr-disperse-136; inode layout - 3282582059 - 3313260395 - 1; disk
layout - 0 - 1073741822 - 1
[2019-08-26 04:49:11.580997] I [MSGID: 109018]
[dht-common.c:1096:dht_revalidate_cbk] 0-glusterdr-dht: Mismatching layouts
for /Incoming/client_folder, gfid = 00000000-0000-0000-0000-000000000000
[2019-08-26 04:49:11.581026] I [MSGID: 109064]
[dht-layout.c:837:dht_layout_dir_mismatch] 0-glusterdr-dht: subvol:
glusterdr-disperse-137; inode layout - 3313260396 - 3343938732 - 1; disk
layout - 1073741823 - 2147483645 - 1
[2019-08-26 04:49:11.581038] I [MSGID: 109018]
[dht-common.c:1096:dht_revalidate_cbk] 0-glusterdr-dht: Mismatching layouts
for /Incoming/client_folder, gfid = 00000000-0000-0000-0000-000000000000
[2019-08-26 04:49:11.581232] I [MSGID: 109064]
[dht-layout.c:837:dht_layout_dir_mismatch] 0-glusterdr-dht: subvol:
glusterdr-disperse-138; inode layout - 3343938733 - 3374617069 - 1; disk
layout - 2147483646 - 3221225468 - 1
[2019-08-26 04:49:11.581243] I [MSGID: 109018]
[dht-common.c:1096:dht_revalidate_cbk] 0-glusterdr-dht: Mismatching layouts
for /Incoming/client_folder, gfid = 00000000-0000-0000-0000-000000000000
[2019-08-26 04:49:11.581333] I [MSGID: 109064]
[dht-layout.c:837:dht_layout_dir_mismatch] 0-glusterdr-dht: subvol:
glusterdr-disperse-139; inode layout - 3374617070 - 3405295406 - 1; disk
layout - 3221225469 - 4294967295 - 1
[2019-08-26 04:49:11.581350] I [MSGID: 109018]
[dht-common.c:1096:dht_revalidate_cbk] 0-glusterdr-dht: Mismatching layouts
for /Incoming/client_folder, gfid = 00000000-0000-0000-0000-000000000000

error message in brick log.

[2019-08-24 13:12:54.041605] W [MSGID: 113026] [posix.c:1542:posix_mkdir]
0-glusterdr-posix: mkdir (/Incoming/TempUpload/Weekly
UPload/23-.8-19/1-2-19/client_folder2): gfid
(b2bf1c5b-2be4-4074-be37-8118f1279d04) is already associated with directory
(/media/disk13/brick13/.glusterfs/61/9a/619a5949-9001-4457-94c1-8f02c45996ba/client_folder2).
Hence, both directories will share same gfid and this can lead to
inconsistencies.

regards
Amudhan P
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190826/a28a6926/attachment.html>