[Gluster-users] "Mismatching layouts" in glusterfs client logs after new brick addition and rebalance
Shreyansh Shah
shreyansh.shah at alpha-grep.com
Thu Jul 2 14:39:25 UTC 2020
Hi All,
*We are facing "Mismatching layouts for <directory>,gfid = <id>" errors.*
We have a distributed glusterfs 5.10, no replication, 2 bricks (4TB each)
on each node, 7 nodes in total. We added new bricks yesterday to the
existing setup.
Post that we did a rebalance fix-layout and then a rebalance (which is
currently still in progress). The status shows "failed" on certain bricks
but "in progress" for others. Adding output for gluster rebalance status
below.
The glusterfs client logs are flooded with "Mismatching layouts for
<directory>,gfid = <id>"
The performance too seems to have degraded due to this, even basic commands
like `cd` and `ls` are taking more than a minute compared to sub-second
number before brick addition.
Apart from that we also experienced many binaries and files giving error
stale file handle error even though the files were present.
*gluster rebalance status :*
Node Rebalanced-files size scanned failures
skipped status run time in h:m:s
--------- ----------- ----------- ----------- -----------
----------- ------------ --------------
localhost 176 3.5GB 12790 0
8552 in progress 21:36:01
10.132.0.72 8232 394.8GB 19995 21
26 failed 14:50:30
10.132.0.44 12625 1.0TB 50023 1
10202 in progress 21:36:00
10.132.0.3 21982 956.8GB 79145 1
34571 in progress 21:36:00
10.132.0.9 7975 355.8GB 20157 6
1522 failed 14:51:45
10.132.0.73 6293 394.5GB 26414 151
8085 failed 14:51:45
10.132.0.70 6480 477.1GB 21058 27
1787 failed 14:50:32
Estimated time left for rebalance to complete : 130:56:28
*Logs from one of the clients below:*
[2020-07-02 12:30:14.971916] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-16; inode layout - 2761060380 - 3067813815 - 3995747641; disk
layout - 2761060380 - 3067813815 - 4159036738
[2020-07-02 12:30:14.971935] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
[2020-07-02 12:30:15.032013] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-19; inode layout - 3681390552 - 3988143987 - 3995747641; disk
layout - 3681390552 - 3988143987 - 4159036738
[2020-07-02 12:30:15.032059] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
[2020-07-02 12:30:15.032107] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-18; inode layout - 3374637116 - 3681390551 - 3995747641; disk
layout - 3374637116 - 3681390551 - 4159036738
[2020-07-02 12:30:15.032153] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
[2020-07-02 12:30:15.093329] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-15; inode layout - 2454306944 - 2761060379 - 3997647794; disk
layout - 2454306944 - 2761060379 - 4159036738
[2020-07-02 12:30:15.093373] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
[2020-07-02 12:30:15.093460] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-16; inode layout - 2761060380 - 3067813815 - 3997647794; disk
layout - 2761060380 - 3067813815 - 4159036738
[2020-07-02 12:30:15.093515] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
[2020-07-02 12:30:15.151063] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-19; inode layout - 3681390552 - 3988143987 - 3997647794; disk
layout - 3681390552 - 3988143987 - 4159036738
[2020-07-02 12:30:15.151108] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
[2020-07-02 12:30:15.151149] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-18; inode layout - 3374637116 - 3681390551 - 3997647794; disk
layout - 3374637116 - 3681390551 - 4159036738
[2020-07-02 12:30:15.151162] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
[2020-07-02 12:30:15.424321] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-11; inode layout - 920400036 - 1227153471 - 3997647794; disk
layout - 920400036 - 1227153471 - 4159036738
[2020-07-02 12:30:15.424380] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
[2020-07-02 12:30:15.424456] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-16; inode layout - 1840730208 - 2147483643 - 3997647794; disk
layout - 1840730208 - 2147483643 - 4159036738
[2020-07-02 12:30:15.424484] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
[2020-07-02 12:30:15.424525] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-15; inode layout - 1533976772 - 1840730207 - 3997647794; disk
layout - 1533976772 - 1840730207 - 4159036738
[2020-07-02 12:30:15.424542] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
[2020-07-02 12:30:15.424596] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-10; inode layout - 613646600 - 920400035 - 3997647794; disk
layout - 613646600 - 920400035 - 4159036738
[2020-07-02 12:30:15.424607] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
[2020-07-02 12:30:16.004482] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_DATA.dat on
data-client-7 (hashed subvol is data-client-17)
[2020-07-02 12:30:16.005523] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_CDS_1_DATA.dat
[2020-07-02 12:30:16.531047] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_METADATA.dat
on data-client-9 (hashed subvol is data-client-19)
[2020-07-02 12:30:16.532086] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_CDS_1_METADATA.dat
[2020-07-02 12:30:18.733229] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_DATA.dat on
data-client-17 (hashed subvol is data-client-9)
[2020-07-02 12:30:18.734421] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_CDS_2_DATA.dat
[2020-07-02 12:30:19.171930] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_METADATA.dat
on data-client-9 (hashed subvol is data-client-18)
[2020-07-02 12:30:19.172901] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_CDS_2_METADATA.dat
[2020-07-02 12:30:21.028495] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_DATA.dat on
data-client-6 (hashed subvol is data-client-15)
[2020-07-02 12:30:21.029836] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_EQ_2_DATA.dat
[2020-07-02 12:30:21.127648] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_METADATA.dat
on data-client-11 (hashed subvol is data-client-3)
[2020-07-02 12:30:21.128713] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_EQ_2_METADATA.dat
[2020-07-02 12:30:21.201126] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_DATA.dat on
data-client-15 (hashed subvol is data-client-7)
[2020-07-02 12:30:21.201928] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_EQ_3_DATA.dat
[2020-07-02 12:30:21.566158] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_METADATA.dat
on data-client-7 (hashed subvol is data-client-16)
[2020-07-02 12:30:21.567123] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_EQ_3_METADATA.dat
[2020-07-02 12:30:21.649357] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_DATA.dat on
data-client-2 (hashed subvol is data-client-11)
[2020-07-02 12:30:21.661381] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_EQ_4_DATA.dat
[2020-07-02 12:30:21.748937] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_METADATA.dat
on data-client-15 (hashed subvol is data-client-7)
[2020-07-02 12:30:21.749481] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_EQ_4_METADATA.dat
[2020-07-02 12:30:21.898593] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_DATA.dat on
data-client-14 (hashed subvol is data-client-7)
[2020-07-02 12:30:21.899442] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_EQ_6_DATA.dat
[2020-07-02 12:30:22.039337] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_METADATA.dat
on data-client-10 (hashed subvol is data-client-2)
[2020-07-02 12:30:22.040086] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_EQ_6_METADATA.dat
[2020-07-02 12:30:22.501877] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/MCASTNSECDS1_DATA.dat
on data-client-15 (hashed subvol is data-client-8)
[2020-07-02 12:30:22.502712] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSECDS1_DATA.dat
[2020-07-02 12:30:22.782577] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/MCASTNSECDS1_METADATA.dat on data-client-11
(hashed subvol is data-client-6)
[2020-07-02 12:30:22.783777] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSECDS1_METADATA.dat
[2020-07-02 12:30:23.146847] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/MCASTNSECM1_DATA.dat on
data-client-17 (hashed subvol is data-client-9)
[2020-07-02 12:30:23.148009] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSECM1_DATA.dat
[2020-07-02 12:30:23.229290] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/MCASTNSECM1_METADATA.dat on data-client-14 (hashed
subvol is data-client-6)
[2020-07-02 12:30:23.230151] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSECM1_METADATA.dat
[2020-07-02 12:30:23.889520] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/MCASTNSECM2_DATA.dat on
data-client-2 (hashed subvol is data-client-11)
[2020-07-02 12:30:23.896618] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSECM2_DATA.dat
[2020-07-02 12:30:24.093017] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/MCASTNSECM2_METADATA.dat on data-client-6 (hashed
subvol is data-client-15)
[2020-07-02 12:30:24.094117] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSECM2_METADATA.dat
[2020-07-02 12:30:24.345257] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/MCASTNSECM3_DATA.dat on
data-client-17 (hashed subvol is data-client-10)
[2020-07-02 12:30:24.346234] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSECM3_DATA.dat
[2020-07-02 12:30:24.425835] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/MCASTNSECM3_METADATA.dat on data-client-6 (hashed
subvol is data-client-15)
[2020-07-02 12:30:24.426880] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSECM3_METADATA.dat
[2020-07-02 12:30:25.158718] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO1_DATA.dat
on data-client-9 (hashed subvol is data-client-19)
[2020-07-02 12:30:25.159619] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO1_DATA.dat
[2020-07-02 12:30:25.531479] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/MCASTNSEFNO1_METADATA.dat on data-client-2 (hashed
subvol is data-client-10)
[2020-07-02 12:30:25.540569] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO1_METADATA.dat
[2020-07-02 12:30:25.771692] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO2_DATA.dat
on data-client-11 (hashed subvol is data-client-3)
[2020-07-02 12:30:25.772610] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO2_DATA.dat
[2020-07-02 12:30:25.866118] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/MCASTNSEFNO2_METADATA.dat on data-client-15
(hashed subvol is data-client-8)
[2020-07-02 12:30:25.866917] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO2_METADATA.dat
[2020-07-02 12:30:26.424386] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO3_DATA.dat
on data-client-9 (hashed subvol is data-client-18)
[2020-07-02 12:30:26.425309] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO3_DATA.dat
[2020-07-02 12:30:26.818852] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/MCASTNSEFNO3_METADATA.dat on data-client-10
(hashed subvol is data-client-2)
[2020-07-02 12:30:26.819890] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO3_METADATA.dat
[2020-07-02 12:30:27.352405] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO4_DATA.dat
on data-client-10 (hashed subvol is data-client-2)
[2020-07-02 12:30:27.352914] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO4_DATA.dat
[2020-07-02 12:30:27.521286] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/MCASTNSEFNO4_METADATA.dat on data-client-8 (hashed
subvol is data-client-18)
[2020-07-02 12:30:27.522325] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO4_METADATA.dat
[2020-07-02 12:30:28.566634] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO5_DATA.dat
on data-client-2 (hashed subvol is data-client-11)
[2020-07-02 12:30:28.579295] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO5_DATA.dat
[2020-07-02 12:30:28.958028] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO6_DATA.dat
on data-client-7 (hashed subvol is data-client-16)
[2020-07-02 12:30:28.959102] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO6_DATA.dat
[2020-07-02 12:30:29.012429] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/MCASTNSEFNO6_METADATA.dat on data-client-6 (hashed
subvol is data-client-15)
[2020-07-02 12:30:29.013416] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO6_METADATA.dat
[2020-07-02 12:30:29.396716] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/NSEFO_BSE_TSDATA.dat on
data-client-17 (hashed subvol is data-client-10)
[2020-07-02 12:30:29.397740] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/NSEFO_BSE_TSDATA.dat
[2020-07-02 12:30:29.556312] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/NSEFO_BSE_TSMETADATA.dat on data-client-9 (hashed
subvol is data-client-18)
[2020-07-02 12:30:29.557197] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/NSEFO_BSE_TSMETADATA.dat
[2020-07-02 12:30:30.605354] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat on data-client-9
(hashed subvol is data-client-19)
[2020-07-02 12:30:30.606117] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat
[2020-07-02 12:30:31.559206] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-18; inode layout - 613576736 - 920330171 - 1; disk layout -
613576736 - 920330171 - 4159036738
[2020-07-02 12:30:31.559255] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
[2020-07-02 12:30:31.569025] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-19; inode layout - 920330172 - 1227083607 - 1; disk layout -
920330172 - 1227083607 - 4159036738
[2020-07-02 12:30:31.569067] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
[2020-07-02 12:30:31.701849] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-18; inode layout - 3374637116 - 3681390551 - 1; disk layout -
3374637116 - 3681390551 - 4159036738
[2020-07-02 12:30:31.701895] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes/NSEINDEX, gfid =
fff324f2-f855-4881-b77c-81e856522373
[2020-07-02 12:30:31.738464] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-19; inode layout - 3681390552 - 3988143987 - 1; disk layout -
3681390552 - 3988143987 - 4159036738
[2020-07-02 12:30:31.738507] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes/NSEINDEX, gfid =
fff324f2-f855-4881-b77c-81e856522373
[2020-07-02 12:30:31.857102] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-15; inode layout - 3067883680 - 3374637115 - 3995747641; disk
layout - 3067883680 - 3374637115 - 4159036738
[2020-07-02 12:30:31.857147] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
f8447150-4801-4188-add9-ea295bb88729
[2020-07-02 12:30:31.857180] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-16; inode layout - 3374637116 - 3681390551 - 3995747641; disk
layout - 3374637116 - 3681390551 - 4159036738
[2020-07-02 12:30:31.857197] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
f8447150-4801-4188-add9-ea295bb88729
[2020-07-02 12:30:31.917705] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-19; inode layout - 0 - 306753435 - 3995747641; disk layout - 0
- 306753435 - 4159036738
[2020-07-02 12:30:31.917781] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
f8447150-4801-4188-add9-ea295bb88729
[2020-07-02 12:30:31.917855] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-18; inode layout - 3988213852 - 4294967295 - 3995747641; disk
layout - 3988213852 - 4294967295 - 4159036738
[2020-07-02 12:30:31.917874] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
f8447150-4801-4188-add9-ea295bb88729
[2020-07-02 12:30:32.390945] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-18; inode layout - 3681460416 - 3988213851 - 1; disk layout -
3681460416 - 3988213851 - 4159036738
[2020-07-02 12:30:32.390998] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes/NSEINDEX/NIFTY, gfid =
b2d4deb7-c58c-4046-b6f2-7c7f44d71311
[2020-07-02 12:30:32.391056] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-19; inode layout - 3988213852 - 4294967295 - 1; disk layout -
3988213852 - 4294967295 - 4159036738
[2020-07-02 12:30:32.391075] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes/NSEINDEX/NIFTY, gfid =
b2d4deb7-c58c-4046-b6f2-7c7f44d71311
[2020-07-02 12:33:50.915279] I [MSGID: 109066]
[dht-rename.c:1922:dht_rename] 4-data-dht: renaming
/raw_data/Brazil/20200414/.260_INCREMENTAL.dat.gz.IwE7T2
(2cb54500-814d-4e85-83e7-e33d9440b18d)
(hash=data-client-6/cache=data-client-18) =>
/raw_data/Brazil/20200414/260_INCREMENTAL.dat.gz ((null))
(hash=data-client-6/cache=<nul>)
[2020-07-02 12:34:09.799586] I [MSGID: 109066]
[dht-rename.c:1922:dht_rename] 4-data-dht: renaming
/raw_data/Brazil/20200414/.260_INSTRUMENTS.dat.gz.1jUL1k
(99938ee6-6986-4123-9d72-ec09e2310b4f)
(hash=data-client-17/cache=data-client-18) =>
/raw_data/Brazil/20200414/260_INSTRUMENTS.dat.gz ((null))
(hash=data-client-17/cache=<nul>)
....
Please look into this at top-priority if possible.
Let me know if anything else is required.
--
Regards,
Shreyansh Shah
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200702/350ba536/attachment.html>
More information about the Gluster-users
mailing list