[Gluster-users] "Mismatching layouts" in glusterfs client logs after new brick addition and rebalance

Artem Russakovskii archon810 at gmail.com
Wed Jul 8 06:02:10 UTC 2020


I think it'd be extremely helpful if gluster had a feature to grab all the
necessary logs/debug info (maybe a few variations depending on the bug) so
that all the user would have to do is enter a simple command and have
gluster generate the whole bug report, ready to be sent to to the gluster
team.

Sincerely,
Artem

--
Founder, Android Police <http://www.androidpolice.com>, APK Mirror
<http://www.apkmirror.com/>, Illogical Robot LLC
beerpla.net | @ArtemR <http://twitter.com/ArtemR>


On Tue, Jul 7, 2020 at 1:47 AM Shreyansh Shah <shreyansh.shah at alpha-grep.com>
wrote:

> Sounds good, thank you.
>
> On Tue, Jul 7, 2020 at 2:12 PM Barak Sason Rofman <bsasonro at redhat.com>
> wrote:
>
>> Thanks Shreyansh,
>>
>> I'll look into it, however I'll likely need some help from more senior
>> team members to perform RCA.
>> I'll update once I have new insights.
>>
>> My regards,
>>
>> On Tue, Jul 7, 2020 at 11:40 AM Shreyansh Shah <
>> shreyansh.shah at alpha-grep.com> wrote:
>>
>>> Hi Barak,
>>> Thanks for looking into this and helping me out,
>>> The fix-layout was successful, and I ran a rebalance after completion of
>>> fix-layout.
>>> The rebalance status though did show failure for 3 nodes.
>>>
>>> On Tue, Jul 7, 2020 at 2:07 PM Barak Sason Rofman <bsasonro at redhat.com>
>>> wrote:
>>>
>>>> Greetings again Shreyansh,
>>>>
>>>> I'm indeed seeing a lot of errors in the log file - still unsure about
>>>> the RC.
>>>> You mentioned that prior to running rebalance you ran fix-layout, was
>>>> the fix-layout successful?
>>>> Another question - did you wait until fix-layout was completed before
>>>> running rebalance?
>>>>
>>>> My thanks,
>>>>
>>>> On Mon, Jul 6, 2020 at 9:33 PM Shreyansh Shah <
>>>> shreyansh.shah at alpha-grep.com> wrote:
>>>>
>>>>> Hi,
>>>>> Attaching rebalance logs
>>>>> FYI, we ran "gluster rebalance fix-layout" followed by "gluster
>>>>> rebalance" on 20200701 and today we again ran "gluster rebalance fix-layout"
>>>>>
>>>>>
>>>>> PFA
>>>>>
>>>>> On Mon, Jul 6, 2020 at 11:08 PM Barak Sason Rofman <
>>>>> bsasonro at redhat.com> wrote:
>>>>>
>>>>>> I think it would be best.
>>>>>> As I can't say at this point where the problem is originating from,
>>>>>> brick logs might also be necessary (I assume I would have a better picture
>>>>>> once I have the rebalance logs).
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> On Mon, Jul 6, 2020 at 8:16 PM Shreyansh Shah <
>>>>>> shreyansh.shah at alpha-grep.com> wrote:
>>>>>>
>>>>>>> Hi Barak,
>>>>>>> Can provide the rebalance logs. Do you require all the brick logs
>>>>>>> (14 in total)?
>>>>>>>
>>>>>>> On Mon, Jul 6, 2020 at 10:43 PM Barak Sason Rofman <
>>>>>>> bsasonro at redhat.com> wrote:
>>>>>>>
>>>>>>>> Greetings Shreyansh,
>>>>>>>>
>>>>>>>> Off-hand I can't come up with a reason for these failures.
>>>>>>>> In order to start looking into this, access to the full rebalance
>>>>>>>> logs is required (possibly brick logs as well).
>>>>>>>> Can you provide those?
>>>>>>>>
>>>>>>>> My regards,
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Jul 6, 2020 at 11:41 AM Shreyansh Shah <
>>>>>>>> shreyansh.shah at alpha-grep.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>> Did anyone get a chance to look into this?
>>>>>>>>>
>>>>>>>>> On Thu, Jul 2, 2020 at 8:09 PM Shreyansh Shah <
>>>>>>>>> shreyansh.shah at alpha-grep.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi All,
>>>>>>>>>>
>>>>>>>>>> *We are facing "Mismatching layouts for <directory>,gfid = <id>"
>>>>>>>>>> errors.*
>>>>>>>>>>
>>>>>>>>>> We have a distributed glusterfs 5.10, no replication, 2 bricks
>>>>>>>>>> (4TB each) on each node, 7 nodes in total. We added new bricks yesterday to
>>>>>>>>>> the existing setup.
>>>>>>>>>> Post that we did a rebalance fix-layout and then a rebalance
>>>>>>>>>> (which is currently still in progress). The status shows "failed" on
>>>>>>>>>> certain bricks but "in progress" for others. Adding output for gluster
>>>>>>>>>> rebalance status below.
>>>>>>>>>>
>>>>>>>>>> The glusterfs client logs are flooded with "Mismatching layouts
>>>>>>>>>> for <directory>,gfid = <id>"
>>>>>>>>>> The performance too seems to have degraded due to this, even
>>>>>>>>>> basic commands like `cd` and `ls` are taking more than a minute compared to
>>>>>>>>>> sub-second number before brick addition.
>>>>>>>>>> Apart from that we also experienced many binaries and files
>>>>>>>>>> giving error stale file handle error even though the files were present.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *gluster rebalance status :*
>>>>>>>>>>
>>>>>>>>>> Node Rebalanced-files          size       scanned      failures
>>>>>>>>>>     skipped               status  run time in h:m:s
>>>>>>>>>> ---------      -----------   -----------   -----------
>>>>>>>>>> -----------   -----------         ------------     --------------
>>>>>>>>>> localhost              176         3.5GB         12790
>>>>>>>>>>   0          8552          in progress       21:36:01
>>>>>>>>>> 10.132.0.72             8232       394.8GB         19995
>>>>>>>>>>    21            26               failed       14:50:30
>>>>>>>>>> 10.132.0.44            12625         1.0TB         50023
>>>>>>>>>>     1         10202          in progress       21:36:00
>>>>>>>>>> 10.132.0.3            21982       956.8GB         79145
>>>>>>>>>>   1         34571          in progress       21:36:00
>>>>>>>>>> 10.132.0.9             7975       355.8GB         20157
>>>>>>>>>>   6          1522               failed       14:51:45
>>>>>>>>>> 10.132.0.73             6293       394.5GB         26414
>>>>>>>>>>   151          8085               failed       14:51:45
>>>>>>>>>> 10.132.0.70             6480       477.1GB         21058
>>>>>>>>>>    27          1787               failed       14:50:32
>>>>>>>>>> Estimated time left for rebalance to complete :      130:56:28
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *Logs from one of the clients below:*
>>>>>>>>>>
>>>>>>>>>> [2020-07-02 12:30:14.971916] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-16; inode layout - 2761060380 - 3067813815 - 3995747641; disk
>>>>>>>>>> layout - 2761060380 - 3067813815 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:14.971935] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>>>>>>>>>> [2020-07-02 12:30:15.032013] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-19; inode layout - 3681390552 - 3988143987 - 3995747641; disk
>>>>>>>>>> layout - 3681390552 - 3988143987 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:15.032059] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>>>>>>>>>> [2020-07-02 12:30:15.032107] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-18; inode layout - 3374637116 - 3681390551 - 3995747641; disk
>>>>>>>>>> layout - 3374637116 - 3681390551 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:15.032153] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>>>>>>>>>> [2020-07-02 12:30:15.093329] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-15; inode layout - 2454306944 - 2761060379 - 3997647794; disk
>>>>>>>>>> layout - 2454306944 - 2761060379 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:15.093373] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
>>>>>>>>>> [2020-07-02 12:30:15.093460] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-16; inode layout - 2761060380 - 3067813815 - 3997647794; disk
>>>>>>>>>> layout - 2761060380 - 3067813815 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:15.093515] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
>>>>>>>>>> [2020-07-02 12:30:15.151063] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-19; inode layout - 3681390552 - 3988143987 - 3997647794; disk
>>>>>>>>>> layout - 3681390552 - 3988143987 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:15.151108] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
>>>>>>>>>> [2020-07-02 12:30:15.151149] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-18; inode layout - 3374637116 - 3681390551 - 3997647794; disk
>>>>>>>>>> layout - 3374637116 - 3681390551 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:15.151162] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
>>>>>>>>>> [2020-07-02 12:30:15.424321] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-11; inode layout - 920400036 - 1227153471 - 3997647794; disk
>>>>>>>>>> layout - 920400036 - 1227153471 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:15.424380] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>>>>>>>>>> [2020-07-02 12:30:15.424456] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-16; inode layout - 1840730208 - 2147483643 - 3997647794; disk
>>>>>>>>>> layout - 1840730208 - 2147483643 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:15.424484] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>>>>>>>>>> [2020-07-02 12:30:15.424525] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-15; inode layout - 1533976772 - 1840730207 - 3997647794; disk
>>>>>>>>>> layout - 1533976772 - 1840730207 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:15.424542] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>>>>>>>>>> [2020-07-02 12:30:15.424596] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-10; inode layout - 613646600 - 920400035 - 3997647794; disk
>>>>>>>>>> layout - 613646600 - 920400035 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:15.424607] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>>>>>>>>>> [2020-07-02 12:30:16.004482] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_DATA.dat on
>>>>>>>>>> data-client-7 (hashed subvol is data-client-17)
>>>>>>>>>> [2020-07-02 12:30:16.005523] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/BSE_CDS_1_DATA.dat
>>>>>>>>>> [2020-07-02 12:30:16.531047] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_METADATA.dat
>>>>>>>>>> on data-client-9 (hashed subvol is data-client-19)
>>>>>>>>>> [2020-07-02 12:30:16.532086] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/BSE_CDS_1_METADATA.dat
>>>>>>>>>> [2020-07-02 12:30:18.733229] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_DATA.dat on
>>>>>>>>>> data-client-17 (hashed subvol is data-client-9)
>>>>>>>>>> [2020-07-02 12:30:18.734421] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/BSE_CDS_2_DATA.dat
>>>>>>>>>> [2020-07-02 12:30:19.171930] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_METADATA.dat
>>>>>>>>>> on data-client-9 (hashed subvol is data-client-18)
>>>>>>>>>> [2020-07-02 12:30:19.172901] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/BSE_CDS_2_METADATA.dat
>>>>>>>>>> [2020-07-02 12:30:21.028495] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_DATA.dat on
>>>>>>>>>> data-client-6 (hashed subvol is data-client-15)
>>>>>>>>>> [2020-07-02 12:30:21.029836] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/BSE_EQ_2_DATA.dat
>>>>>>>>>> [2020-07-02 12:30:21.127648] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_METADATA.dat
>>>>>>>>>> on data-client-11 (hashed subvol is data-client-3)
>>>>>>>>>> [2020-07-02 12:30:21.128713] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/BSE_EQ_2_METADATA.dat
>>>>>>>>>> [2020-07-02 12:30:21.201126] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_DATA.dat on
>>>>>>>>>> data-client-15 (hashed subvol is data-client-7)
>>>>>>>>>> [2020-07-02 12:30:21.201928] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/BSE_EQ_3_DATA.dat
>>>>>>>>>> [2020-07-02 12:30:21.566158] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_METADATA.dat
>>>>>>>>>> on data-client-7 (hashed subvol is data-client-16)
>>>>>>>>>> [2020-07-02 12:30:21.567123] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/BSE_EQ_3_METADATA.dat
>>>>>>>>>> [2020-07-02 12:30:21.649357] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_DATA.dat on
>>>>>>>>>> data-client-2 (hashed subvol is data-client-11)
>>>>>>>>>> [2020-07-02 12:30:21.661381] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/BSE_EQ_4_DATA.dat
>>>>>>>>>> [2020-07-02 12:30:21.748937] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_METADATA.dat
>>>>>>>>>> on data-client-15 (hashed subvol is data-client-7)
>>>>>>>>>> [2020-07-02 12:30:21.749481] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/BSE_EQ_4_METADATA.dat
>>>>>>>>>> [2020-07-02 12:30:21.898593] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_DATA.dat on
>>>>>>>>>> data-client-14 (hashed subvol is data-client-7)
>>>>>>>>>> [2020-07-02 12:30:21.899442] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/BSE_EQ_6_DATA.dat
>>>>>>>>>> [2020-07-02 12:30:22.039337] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_METADATA.dat
>>>>>>>>>> on data-client-10 (hashed subvol is data-client-2)
>>>>>>>>>> [2020-07-02 12:30:22.040086] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/BSE_EQ_6_METADATA.dat
>>>>>>>>>> [2020-07-02 12:30:22.501877] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECDS1_DATA.dat
>>>>>>>>>> on data-client-15 (hashed subvol is data-client-8)
>>>>>>>>>> [2020-07-02 12:30:22.502712] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSECDS1_DATA.dat
>>>>>>>>>> [2020-07-02 12:30:22.782577] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>> /processed_data/20200630/MCASTNSECDS1_METADATA.dat on data-client-11
>>>>>>>>>> (hashed subvol is data-client-6)
>>>>>>>>>> [2020-07-02 12:30:22.783777] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSECDS1_METADATA.dat
>>>>>>>>>> [2020-07-02 12:30:23.146847] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM1_DATA.dat on
>>>>>>>>>> data-client-17 (hashed subvol is data-client-9)
>>>>>>>>>> [2020-07-02 12:30:23.148009] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSECM1_DATA.dat
>>>>>>>>>> [2020-07-02 12:30:23.229290] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>> /processed_data/20200630/MCASTNSECM1_METADATA.dat on data-client-14 (hashed
>>>>>>>>>> subvol is data-client-6)
>>>>>>>>>> [2020-07-02 12:30:23.230151] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSECM1_METADATA.dat
>>>>>>>>>> [2020-07-02 12:30:23.889520] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM2_DATA.dat on
>>>>>>>>>> data-client-2 (hashed subvol is data-client-11)
>>>>>>>>>> [2020-07-02 12:30:23.896618] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSECM2_DATA.dat
>>>>>>>>>> [2020-07-02 12:30:24.093017] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>> /processed_data/20200630/MCASTNSECM2_METADATA.dat on data-client-6 (hashed
>>>>>>>>>> subvol is data-client-15)
>>>>>>>>>> [2020-07-02 12:30:24.094117] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSECM2_METADATA.dat
>>>>>>>>>> [2020-07-02 12:30:24.345257] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM3_DATA.dat on
>>>>>>>>>> data-client-17 (hashed subvol is data-client-10)
>>>>>>>>>> [2020-07-02 12:30:24.346234] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSECM3_DATA.dat
>>>>>>>>>> [2020-07-02 12:30:24.425835] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>> /processed_data/20200630/MCASTNSECM3_METADATA.dat on data-client-6 (hashed
>>>>>>>>>> subvol is data-client-15)
>>>>>>>>>> [2020-07-02 12:30:24.426880] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSECM3_METADATA.dat
>>>>>>>>>> [2020-07-02 12:30:25.158718] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO1_DATA.dat
>>>>>>>>>> on data-client-9 (hashed subvol is data-client-19)
>>>>>>>>>> [2020-07-02 12:30:25.159619] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO1_DATA.dat
>>>>>>>>>> [2020-07-02 12:30:25.531479] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO1_METADATA.dat on data-client-2 (hashed
>>>>>>>>>> subvol is data-client-10)
>>>>>>>>>> [2020-07-02 12:30:25.540569] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO1_METADATA.dat
>>>>>>>>>> [2020-07-02 12:30:25.771692] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO2_DATA.dat
>>>>>>>>>> on data-client-11 (hashed subvol is data-client-3)
>>>>>>>>>> [2020-07-02 12:30:25.772610] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO2_DATA.dat
>>>>>>>>>> [2020-07-02 12:30:25.866118] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO2_METADATA.dat on data-client-15
>>>>>>>>>> (hashed subvol is data-client-8)
>>>>>>>>>> [2020-07-02 12:30:25.866917] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO2_METADATA.dat
>>>>>>>>>> [2020-07-02 12:30:26.424386] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO3_DATA.dat
>>>>>>>>>> on data-client-9 (hashed subvol is data-client-18)
>>>>>>>>>> [2020-07-02 12:30:26.425309] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO3_DATA.dat
>>>>>>>>>> [2020-07-02 12:30:26.818852] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO3_METADATA.dat on data-client-10
>>>>>>>>>> (hashed subvol is data-client-2)
>>>>>>>>>> [2020-07-02 12:30:26.819890] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO3_METADATA.dat
>>>>>>>>>> [2020-07-02 12:30:27.352405] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO4_DATA.dat
>>>>>>>>>> on data-client-10 (hashed subvol is data-client-2)
>>>>>>>>>> [2020-07-02 12:30:27.352914] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO4_DATA.dat
>>>>>>>>>> [2020-07-02 12:30:27.521286] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO4_METADATA.dat on data-client-8 (hashed
>>>>>>>>>> subvol is data-client-18)
>>>>>>>>>> [2020-07-02 12:30:27.522325] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO4_METADATA.dat
>>>>>>>>>> [2020-07-02 12:30:28.566634] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO5_DATA.dat
>>>>>>>>>> on data-client-2 (hashed subvol is data-client-11)
>>>>>>>>>> [2020-07-02 12:30:28.579295] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO5_DATA.dat
>>>>>>>>>> [2020-07-02 12:30:28.958028] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO6_DATA.dat
>>>>>>>>>> on data-client-7 (hashed subvol is data-client-16)
>>>>>>>>>> [2020-07-02 12:30:28.959102] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO6_DATA.dat
>>>>>>>>>> [2020-07-02 12:30:29.012429] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO6_METADATA.dat on data-client-6 (hashed
>>>>>>>>>> subvol is data-client-15)
>>>>>>>>>> [2020-07-02 12:30:29.013416] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO6_METADATA.dat
>>>>>>>>>> [2020-07-02 12:30:29.396716] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/NSEFO_BSE_TSDATA.dat on
>>>>>>>>>> data-client-17 (hashed subvol is data-client-10)
>>>>>>>>>> [2020-07-02 12:30:29.397740] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/NSEFO_BSE_TSDATA.dat
>>>>>>>>>> [2020-07-02 12:30:29.556312] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>> /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat on data-client-9 (hashed
>>>>>>>>>> subvol is data-client-18)
>>>>>>>>>> [2020-07-02 12:30:29.557197] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat
>>>>>>>>>> [2020-07-02 12:30:30.605354] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>> /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat on data-client-9
>>>>>>>>>> (hashed subvol is data-client-19)
>>>>>>>>>> [2020-07-02 12:30:30.606117] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat
>>>>>>>>>> [2020-07-02 12:30:31.559206] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-18; inode layout - 613576736 - 920330171 - 1; disk layout -
>>>>>>>>>> 613576736 - 920330171 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:31.559255] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
>>>>>>>>>> [2020-07-02 12:30:31.569025] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-19; inode layout - 920330172 - 1227083607 - 1; disk layout -
>>>>>>>>>> 920330172 - 1227083607 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:31.569067] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
>>>>>>>>>> [2020-07-02 12:30:31.701849] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-18; inode layout - 3374637116 - 3681390551 - 1; disk layout -
>>>>>>>>>> 3374637116 - 3681390551 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:31.701895] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /processed_data/Indexes/NSEINDEX, gfid =
>>>>>>>>>> fff324f2-f855-4881-b77c-81e856522373
>>>>>>>>>> [2020-07-02 12:30:31.738464] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-19; inode layout - 3681390552 - 3988143987 - 1; disk layout -
>>>>>>>>>> 3681390552 - 3988143987 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:31.738507] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /processed_data/Indexes/NSEINDEX, gfid =
>>>>>>>>>> fff324f2-f855-4881-b77c-81e856522373
>>>>>>>>>> [2020-07-02 12:30:31.857102] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-15; inode layout - 3067883680 - 3374637115 - 3995747641; disk
>>>>>>>>>> layout - 3067883680 - 3374637115 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:31.857147] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>>>>>>>>>> f8447150-4801-4188-add9-ea295bb88729
>>>>>>>>>> [2020-07-02 12:30:31.857180] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-16; inode layout - 3374637116 - 3681390551 - 3995747641; disk
>>>>>>>>>> layout - 3374637116 - 3681390551 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:31.857197] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>>>>>>>>>> f8447150-4801-4188-add9-ea295bb88729
>>>>>>>>>> [2020-07-02 12:30:31.917705] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-19; inode layout - 0 - 306753435 - 3995747641; disk layout - 0
>>>>>>>>>> - 306753435 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:31.917781] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>>>>>>>>>> f8447150-4801-4188-add9-ea295bb88729
>>>>>>>>>> [2020-07-02 12:30:31.917855] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-18; inode layout - 3988213852 - 4294967295 - 3995747641; disk
>>>>>>>>>> layout - 3988213852 - 4294967295 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:31.917874] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>>>>>>>>>> f8447150-4801-4188-add9-ea295bb88729
>>>>>>>>>> [2020-07-02 12:30:32.390945] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-18; inode layout - 3681460416 - 3988213851 - 1; disk layout -
>>>>>>>>>> 3681460416 - 3988213851 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:32.390998] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /processed_data/Indexes/NSEINDEX/NIFTY, gfid =
>>>>>>>>>> b2d4deb7-c58c-4046-b6f2-7c7f44d71311
>>>>>>>>>> [2020-07-02 12:30:32.391056] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-19; inode layout - 3988213852 - 4294967295 - 1; disk layout -
>>>>>>>>>> 3988213852 - 4294967295 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:32.391075] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /processed_data/Indexes/NSEINDEX/NIFTY, gfid =
>>>>>>>>>> b2d4deb7-c58c-4046-b6f2-7c7f44d71311
>>>>>>>>>> [2020-07-02 12:33:50.915279] I [MSGID: 109066]
>>>>>>>>>> [dht-rename.c:1922:dht_rename] 4-data-dht: renaming
>>>>>>>>>> /raw_data/Brazil/20200414/.260_INCREMENTAL.dat.gz.IwE7T2
>>>>>>>>>> (2cb54500-814d-4e85-83e7-e33d9440b18d)
>>>>>>>>>> (hash=data-client-6/cache=data-client-18) =>
>>>>>>>>>> /raw_data/Brazil/20200414/260_INCREMENTAL.dat.gz ((null))
>>>>>>>>>> (hash=data-client-6/cache=<nul>)
>>>>>>>>>> [2020-07-02 12:34:09.799586] I [MSGID: 109066]
>>>>>>>>>> [dht-rename.c:1922:dht_rename] 4-data-dht: renaming
>>>>>>>>>> /raw_data/Brazil/20200414/.260_INSTRUMENTS.dat.gz.1jUL1k
>>>>>>>>>> (99938ee6-6986-4123-9d72-ec09e2310b4f)
>>>>>>>>>> (hash=data-client-17/cache=data-client-18) =>
>>>>>>>>>> /raw_data/Brazil/20200414/260_INSTRUMENTS.dat.gz ((null))
>>>>>>>>>> (hash=data-client-17/cache=<nul>)
>>>>>>>>>> ....
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Please look into this at top-priority if possible.
>>>>>>>>>> Let me know if anything else is required.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Regards,
>>>>>>>>>> Shreyansh Shah
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Regards,
>>>>>>>>> Shreyansh Shah
>>>>>>>>> ________
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Community Meeting Calendar:
>>>>>>>>>
>>>>>>>>> Schedule -
>>>>>>>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>>>>>>>> Bridge: https://bluejeans.com/441850968
>>>>>>>>>
>>>>>>>>> Gluster-users mailing list
>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Barak Sason Rofman*
>>>>>>>>
>>>>>>>> Gluster Storage Development
>>>>>>>>
>>>>>>>> Red Hat Israel <https://www.redhat.com/>
>>>>>>>>
>>>>>>>> 34 Jerusalem rd. Ra'anana, 43501
>>>>>>>>
>>>>>>>> bsasonro at redhat.com <adi at redhat.com>    T: *+972-9-7692304*
>>>>>>>> M: *+972-52-4326355*
>>>>>>>> @RedHat <https://twitter.com/redhat>   Red Hat
>>>>>>>> <https://www.linkedin.com/company/red-hat>  Red Hat
>>>>>>>> <https://www.facebook.com/redhat.il/>
>>>>>>>> <https://red.ht/sig>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Shreyansh Shah
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Barak Sason Rofman*
>>>>>>
>>>>>> Gluster Storage Development
>>>>>>
>>>>>> Red Hat Israel <https://www.redhat.com/>
>>>>>>
>>>>>> 34 Jerusalem rd. Ra'anana, 43501
>>>>>>
>>>>>> bsasonro at redhat.com <adi at redhat.com>    T: *+972-9-7692304*
>>>>>> M: *+972-52-4326355*
>>>>>> @RedHat <https://twitter.com/redhat>   Red Hat
>>>>>> <https://www.linkedin.com/company/red-hat>  Red Hat
>>>>>> <https://www.facebook.com/redhat.il/>
>>>>>> <https://red.ht/sig>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Shreyansh Shah
>>>>>
>>>>
>>>>
>>>> --
>>>> *Barak Sason Rofman*
>>>>
>>>> Gluster Storage Development
>>>>
>>>> Red Hat Israel <https://www.redhat.com/>
>>>>
>>>> 34 Jerusalem rd. Ra'anana, 43501
>>>>
>>>> bsasonro at redhat.com <adi at redhat.com>    T: *+972-9-7692304*
>>>> M: *+972-52-4326355*
>>>> @RedHat <https://twitter.com/redhat>   Red Hat
>>>> <https://www.linkedin.com/company/red-hat>  Red Hat
>>>> <https://www.facebook.com/redhat.il/>
>>>> <https://red.ht/sig>
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Shreyansh Shah
>>>
>>
>>
>> --
>> *Barak Sason Rofman*
>>
>> Gluster Storage Development
>>
>> Red Hat Israel <https://www.redhat.com/>
>>
>> 34 Jerusalem rd. Ra'anana, 43501
>>
>> bsasonro at redhat.com <adi at redhat.com>    T: *+972-9-7692304*
>> M: *+972-52-4326355*
>> @RedHat <https://twitter.com/redhat>   Red Hat
>> <https://www.linkedin.com/company/red-hat>  Red Hat
>> <https://www.facebook.com/redhat.il/>
>> <https://red.ht/sig>
>>
>
>
> --
> Regards,
> Shreyansh Shah
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200707/47b1369d/attachment.html>


More information about the Gluster-users mailing list