<div dir="ltr"><div><div>I think it would be best.<br></div>As I can't say at this point where the problem is originating from, brick logs might also be necessary (I assume I would have a better picture once I have the rebalance logs).<br><br></div>Cheers,<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jul 6, 2020 at 8:16 PM Shreyansh Shah <<a href="mailto:shreyansh.shah@alpha-grep.com">shreyansh.shah@alpha-grep.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Barak,<br>Can provide the rebalance logs. Do you require all the brick logs (14 in total)?</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jul 6, 2020 at 10:43 PM Barak Sason Rofman <<a href="mailto:bsasonro@redhat.com" target="_blank">bsasonro@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div>Greetings <font face="arial, helvetica, sans-serif">Shreyansh,<br><br></font></div><font face="arial, helvetica, sans-serif">Off-hand I can't come up with a reason for these failures.<br>In order to start looking into this, access to the full rebalance logs is required (possibly brick logs as well).<br></font></div><font face="arial, helvetica, sans-serif">Can you provide those?<br><br></font></div><font face="arial, helvetica, sans-serif">My regards,<br><br></font></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jul 6, 2020 at 11:41 AM Shreyansh Shah <<a href="mailto:shreyansh.shah@alpha-grep.com" target="_blank">shreyansh.shah@alpha-grep.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi, <br>Did anyone get a chance to look into this?</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jul 2, 2020 at 8:09 PM Shreyansh Shah <<a href="mailto:shreyansh.shah@alpha-grep.com" target="_blank">shreyansh.shah@alpha-grep.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi All,<div><br></div><div><div><b>We are facing "Mismatching layouts for <directory>,gfid = <id>" errors.</b></div></div><div><br></div><div>We have a distributed glusterfs 5.10, no replication, 2 bricks (4TB each) on each node, 7 nodes in total. We added new bricks yesterday to the existing setup.<br>Post that we did a rebalance fix-layout and then a rebalance (which is currently still in progress). The status shows "failed" on certain bricks but "in progress" for others. Adding output for gluster rebalance status below.<div><br></div><div><div>The glusterfs client logs are flooded with "Mismatching layouts for <directory>,gfid = <id>"</div><div>The performance too seems to have degraded due to this, even basic commands like `cd` and `ls` are taking more than a minute compared to sub-second number before brick addition.</div><div>Apart from that we also experienced many binaries and files giving error stale file handle error even though the files were present.</div><div><br></div><div><br></div><div><b>gluster rebalance status :</b></div><div><b><br></b></div><div>Node Rebalanced-files size scanned failures skipped status run time in h:m:s<br>--------- ----------- ----------- ----------- ----------- ----------- ------------ --------------<br>localhost 176 3.5GB 12790 0 8552 in progress 21:36:01<br>10.132.0.72 8232 394.8GB 19995 21 26 failed 14:50:30</div><div>10.132.0.44 12625 1.0TB 50023 1 10202 in progress 21:36:00<br>10.132.0.3 21982 956.8GB 79145 1 34571 in progress 21:36:00<br>10.132.0.9 7975 355.8GB 20157 6 1522 failed 14:51:45<br>10.132.0.73 6293 394.5GB 26414 151 8085 failed 14:51:45<br>10.132.0.70 6480 477.1GB 21058 27 1787 failed 14:50:32<br>Estimated time left for rebalance to complete : 130:56:28<br></div><div><br></div><div><br></div><div><b>Logs from one of the clients below:</b><br></div><div><b><br></b></div><div>[2020-07-02 12:30:14.971916] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-16; inode layout - 2761060380 - 3067813815 - 3995747641; disk layout - 2761060380 - 3067813815 - 4159036738<br>[2020-07-02 12:30:14.971935] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc<br>[2020-07-02 12:30:15.032013] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-19; inode layout - 3681390552 - 3988143987 - 3995747641; disk layout - 3681390552 - 3988143987 - 4159036738<br>[2020-07-02 12:30:15.032059] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc<br>[2020-07-02 12:30:15.032107] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-18; inode layout - 3374637116 - 3681390551 - 3995747641; disk layout - 3374637116 - 3681390551 - 4159036738<br>[2020-07-02 12:30:15.032153] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc<br>[2020-07-02 12:30:15.093329] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-15; inode layout - 2454306944 - 2761060379 - 3997647794; disk layout - 2454306944 - 2761060379 - 4159036738<br>[2020-07-02 12:30:15.093373] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222<br>[2020-07-02 12:30:15.093460] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-16; inode layout - 2761060380 - 3067813815 - 3997647794; disk layout - 2761060380 - 3067813815 - 4159036738<br>[2020-07-02 12:30:15.093515] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222<br>[2020-07-02 12:30:15.151063] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-19; inode layout - 3681390552 - 3988143987 - 3997647794; disk layout - 3681390552 - 3988143987 - 4159036738<br>[2020-07-02 12:30:15.151108] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222<br>[2020-07-02 12:30:15.151149] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-18; inode layout - 3374637116 - 3681390551 - 3997647794; disk layout - 3374637116 - 3681390551 - 4159036738<br>[2020-07-02 12:30:15.151162] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222<br>[2020-07-02 12:30:15.424321] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-11; inode layout - 920400036 - 1227153471 - 3997647794; disk layout - 920400036 - 1227153471 - 4159036738<br>[2020-07-02 12:30:15.424380] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1<br>[2020-07-02 12:30:15.424456] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-16; inode layout - 1840730208 - 2147483643 - 3997647794; disk layout - 1840730208 - 2147483643 - 4159036738<br>[2020-07-02 12:30:15.424484] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1<br>[2020-07-02 12:30:15.424525] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-15; inode layout - 1533976772 - 1840730207 - 3997647794; disk layout - 1533976772 - 1840730207 - 4159036738<br>[2020-07-02 12:30:15.424542] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1<br>[2020-07-02 12:30:15.424596] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-10; inode layout - 613646600 - 920400035 - 3997647794; disk layout - 613646600 - 920400035 - 4159036738<br>[2020-07-02 12:30:15.424607] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1<br>[2020-07-02 12:30:16.004482] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_DATA.dat on data-client-7 (hashed subvol is data-client-17)<br>[2020-07-02 12:30:16.005523] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_CDS_1_DATA.dat<br>[2020-07-02 12:30:16.531047] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_METADATA.dat on data-client-9 (hashed subvol is data-client-19)<br>[2020-07-02 12:30:16.532086] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_CDS_1_METADATA.dat<br>[2020-07-02 12:30:18.733229] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_DATA.dat on data-client-17 (hashed subvol is data-client-9)<br>[2020-07-02 12:30:18.734421] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_CDS_2_DATA.dat<br>[2020-07-02 12:30:19.171930] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_METADATA.dat on data-client-9 (hashed subvol is data-client-18)<br>[2020-07-02 12:30:19.172901] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_CDS_2_METADATA.dat<br>[2020-07-02 12:30:21.028495] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_DATA.dat on data-client-6 (hashed subvol is data-client-15)<br>[2020-07-02 12:30:21.029836] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_EQ_2_DATA.dat<br>[2020-07-02 12:30:21.127648] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_METADATA.dat on data-client-11 (hashed subvol is data-client-3)<br>[2020-07-02 12:30:21.128713] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_EQ_2_METADATA.dat<br>[2020-07-02 12:30:21.201126] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_DATA.dat on data-client-15 (hashed subvol is data-client-7)<br>[2020-07-02 12:30:21.201928] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_EQ_3_DATA.dat<br>[2020-07-02 12:30:21.566158] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_METADATA.dat on data-client-7 (hashed subvol is data-client-16)<br>[2020-07-02 12:30:21.567123] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_EQ_3_METADATA.dat<br>[2020-07-02 12:30:21.649357] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_DATA.dat on data-client-2 (hashed subvol is data-client-11)<br>[2020-07-02 12:30:21.661381] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_EQ_4_DATA.dat<br>[2020-07-02 12:30:21.748937] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_METADATA.dat on data-client-15 (hashed subvol is data-client-7)<br>[2020-07-02 12:30:21.749481] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_EQ_4_METADATA.dat<br>[2020-07-02 12:30:21.898593] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_DATA.dat on data-client-14 (hashed subvol is data-client-7)<br>[2020-07-02 12:30:21.899442] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_EQ_6_DATA.dat<br>[2020-07-02 12:30:22.039337] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_METADATA.dat on data-client-10 (hashed subvol is data-client-2)<br>[2020-07-02 12:30:22.040086] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_EQ_6_METADATA.dat<br>[2020-07-02 12:30:22.501877] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSECDS1_DATA.dat on data-client-15 (hashed subvol is data-client-8)<br>[2020-07-02 12:30:22.502712] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSECDS1_DATA.dat<br>[2020-07-02 12:30:22.782577] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSECDS1_METADATA.dat on data-client-11 (hashed subvol is data-client-6)<br>[2020-07-02 12:30:22.783777] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSECDS1_METADATA.dat<br>[2020-07-02 12:30:23.146847] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSECM1_DATA.dat on data-client-17 (hashed subvol is data-client-9)<br>[2020-07-02 12:30:23.148009] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSECM1_DATA.dat<br>[2020-07-02 12:30:23.229290] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSECM1_METADATA.dat on data-client-14 (hashed subvol is data-client-6)<br>[2020-07-02 12:30:23.230151] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSECM1_METADATA.dat<br>[2020-07-02 12:30:23.889520] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSECM2_DATA.dat on data-client-2 (hashed subvol is data-client-11)<br>[2020-07-02 12:30:23.896618] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSECM2_DATA.dat<br>[2020-07-02 12:30:24.093017] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSECM2_METADATA.dat on data-client-6 (hashed subvol is data-client-15)<br>[2020-07-02 12:30:24.094117] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSECM2_METADATA.dat<br>[2020-07-02 12:30:24.345257] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSECM3_DATA.dat on data-client-17 (hashed subvol is data-client-10)<br>[2020-07-02 12:30:24.346234] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSECM3_DATA.dat<br>[2020-07-02 12:30:24.425835] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSECM3_METADATA.dat on data-client-6 (hashed subvol is data-client-15)<br>[2020-07-02 12:30:24.426880] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSECM3_METADATA.dat<br>[2020-07-02 12:30:25.158718] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO1_DATA.dat on data-client-9 (hashed subvol is data-client-19)<br>[2020-07-02 12:30:25.159619] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO1_DATA.dat<br>[2020-07-02 12:30:25.531479] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO1_METADATA.dat on data-client-2 (hashed subvol is data-client-10)<br>[2020-07-02 12:30:25.540569] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO1_METADATA.dat<br>[2020-07-02 12:30:25.771692] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO2_DATA.dat on data-client-11 (hashed subvol is data-client-3)<br>[2020-07-02 12:30:25.772610] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO2_DATA.dat<br>[2020-07-02 12:30:25.866118] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO2_METADATA.dat on data-client-15 (hashed subvol is data-client-8)<br>[2020-07-02 12:30:25.866917] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO2_METADATA.dat<br>[2020-07-02 12:30:26.424386] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO3_DATA.dat on data-client-9 (hashed subvol is data-client-18)<br>[2020-07-02 12:30:26.425309] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO3_DATA.dat<br>[2020-07-02 12:30:26.818852] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO3_METADATA.dat on data-client-10 (hashed subvol is data-client-2)<br>[2020-07-02 12:30:26.819890] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO3_METADATA.dat<br>[2020-07-02 12:30:27.352405] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO4_DATA.dat on data-client-10 (hashed subvol is data-client-2)<br>[2020-07-02 12:30:27.352914] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO4_DATA.dat<br>[2020-07-02 12:30:27.521286] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO4_METADATA.dat on data-client-8 (hashed subvol is data-client-18)<br>[2020-07-02 12:30:27.522325] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO4_METADATA.dat<br>[2020-07-02 12:30:28.566634] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO5_DATA.dat on data-client-2 (hashed subvol is data-client-11)<br>[2020-07-02 12:30:28.579295] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO5_DATA.dat<br>[2020-07-02 12:30:28.958028] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO6_DATA.dat on data-client-7 (hashed subvol is data-client-16)<br>[2020-07-02 12:30:28.959102] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO6_DATA.dat<br>[2020-07-02 12:30:29.012429] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO6_METADATA.dat on data-client-6 (hashed subvol is data-client-15)<br>[2020-07-02 12:30:29.013416] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO6_METADATA.dat<br>[2020-07-02 12:30:29.396716] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/NSEFO_BSE_TSDATA.dat on data-client-17 (hashed subvol is data-client-10)<br>[2020-07-02 12:30:29.397740] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/NSEFO_BSE_TSDATA.dat<br>[2020-07-02 12:30:29.556312] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat on data-client-9 (hashed subvol is data-client-18)<br>[2020-07-02 12:30:29.557197] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat<br>[2020-07-02 12:30:30.605354] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat on data-client-9 (hashed subvol is data-client-19)<br>[2020-07-02 12:30:30.606117] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat<br>[2020-07-02 12:30:31.559206] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-18; inode layout - 613576736 - 920330171 - 1; disk layout - 613576736 - 920330171 - 4159036738<br>[2020-07-02 12:30:31.559255] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99<br>[2020-07-02 12:30:31.569025] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-19; inode layout - 920330172 - 1227083607 - 1; disk layout - 920330172 - 1227083607 - 4159036738<br>[2020-07-02 12:30:31.569067] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99<br>[2020-07-02 12:30:31.701849] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-18; inode layout - 3374637116 - 3681390551 - 1; disk layout - 3374637116 - 3681390551 - 4159036738<br>[2020-07-02 12:30:31.701895] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes/NSEINDEX, gfid = fff324f2-f855-4881-b77c-81e856522373<br>[2020-07-02 12:30:31.738464] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-19; inode layout - 3681390552 - 3988143987 - 1; disk layout - 3681390552 - 3988143987 - 4159036738<br>[2020-07-02 12:30:31.738507] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes/NSEINDEX, gfid = fff324f2-f855-4881-b77c-81e856522373<br>[2020-07-02 12:30:31.857102] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-15; inode layout - 3067883680 - 3374637115 - 3995747641; disk layout - 3067883680 - 3374637115 - 4159036738<br>[2020-07-02 12:30:31.857147] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid = f8447150-4801-4188-add9-ea295bb88729<br>[2020-07-02 12:30:31.857180] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-16; inode layout - 3374637116 - 3681390551 - 3995747641; disk layout - 3374637116 - 3681390551 - 4159036738<br>[2020-07-02 12:30:31.857197] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid = f8447150-4801-4188-add9-ea295bb88729<br>[2020-07-02 12:30:31.917705] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-19; inode layout - 0 - 306753435 - 3995747641; disk layout - 0 - 306753435 - 4159036738<br>[2020-07-02 12:30:31.917781] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid = f8447150-4801-4188-add9-ea295bb88729<br>[2020-07-02 12:30:31.917855] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-18; inode layout - 3988213852 - 4294967295 - 3995747641; disk layout - 3988213852 - 4294967295 - 4159036738<br>[2020-07-02 12:30:31.917874] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid = f8447150-4801-4188-add9-ea295bb88729<br>[2020-07-02 12:30:32.390945] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-18; inode layout - 3681460416 - 3988213851 - 1; disk layout - 3681460416 - 3988213851 - 4159036738<br>[2020-07-02 12:30:32.390998] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes/NSEINDEX/NIFTY, gfid = b2d4deb7-c58c-4046-b6f2-7c7f44d71311<br>[2020-07-02 12:30:32.391056] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-19; inode layout - 3988213852 - 4294967295 - 1; disk layout - 3988213852 - 4294967295 - 4159036738<br>[2020-07-02 12:30:32.391075] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes/NSEINDEX/NIFTY, gfid = b2d4deb7-c58c-4046-b6f2-7c7f44d71311<br>[2020-07-02 12:33:50.915279] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 4-data-dht: renaming /raw_data/Brazil/20200414/.260_INCREMENTAL.dat.gz.IwE7T2 (2cb54500-814d-4e85-83e7-e33d9440b18d) (hash=data-client-6/cache=data-client-18) => /raw_data/Brazil/20200414/260_INCREMENTAL.dat.gz ((null)) (hash=data-client-6/cache=<nul>)<br>[2020-07-02 12:34:09.799586] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 4-data-dht: renaming /raw_data/Brazil/20200414/.260_INSTRUMENTS.dat.gz.1jUL1k (99938ee6-6986-4123-9d72-ec09e2310b4f) (hash=data-client-17/cache=data-client-18) => /raw_data/Brazil/20200414/260_INSTRUMENTS.dat.gz ((null)) (hash=data-client-17/cache=<nul>)<br></div><div>....</div><div><br></div><div><br></div><div>Please look into this at top-priority if possible.</div><div>Let me know if anything else is required.</div><div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><font face="arial, helvetica, sans-serif">Regards,<br>Shreyansh Shah</font></div></div></div></div></div></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><font face="arial, helvetica, sans-serif">Regards,<br>Shreyansh Shah</font></div></div></div></div></div></div></div>
________<br>
<br>
<br>
<br>
Community Meeting Calendar:<br>
<br>
Schedule -<br>
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div style="margin-left:44px"><div style="margin-bottom:0px;margin-left:0px;padding-bottom:5px"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><font face="RedHatText, sans-serif"><span style="font-size:14px;text-transform:capitalize"><b>Barak Sason Rofman</b></span></font><br></div></div></div></div></div></div></div></div></div></div><div style="margin-left:44px"><div style="margin-bottom:0px;margin-left:0px;padding-bottom:5px"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><p style="margin:0px;font-size:14px;color:rgb(0,0,0);font-family:overpass,sans-serif;font-weight:bold;padding:0px;text-transform:uppercase"><span style="font-family:RedHatText,sans-serif;font-size:12px;text-transform:capitalize;font-weight:normal;color:rgb(34,34,34)">Gluster Storage </span><span style="font-family:RedHatText,sans-serif;font-size:12px;text-transform:capitalize;font-weight:normal;color:rgb(34,34,34)">Development</span></p></div></div></div></div></div></div></div></div></div></div><div style="margin-left:44px"><div style="margin-bottom:0px;margin-left:0px;padding-bottom:5px"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><p style="margin:0px 0px 4px;font-size:12px;font-family:RedHatText,sans-serif"><a href="https://www.redhat.com/" style="color:rgb(0,136,206);margin:0px" target="_blank">Red Hat Israel</a></p><div style="margin-bottom:4px"><p style="margin:0px;font-size:12px;font-family:RedHatText,sans-serif">34 Jerusalem rd. Ra'anana, 43501</p></div><p style="margin:0px;font-size:12px;font-family:RedHatText,sans-serif"><span style="margin:0px;padding:0px">bsasonro<a href="mailto:adi@redhat.com" style="color:rgb(0,0,0);margin:0px" target="_blank">@redhat.com</a> </span> T: <u>+972-9-7692304</u><br>M: <u>+972-52-4326355</u></p><div style="font-size:12px;font-family:RedHatText,sans-serif;margin-bottom:8px"><a href="https://twitter.com/redhat" title="twitter" style="color:rgb(0,0,0);background:transparent url("https://marketing-outfit-prod-images.s3-us-west-2.amazonaws.com/3780bd4ede961ef3cd4108b8c0e80186/web-icon-twitter.png") no-repeat scroll 0px 50%/13px;display:inline-block;line-height:20px;padding-left:13px" target="_blank"><span style="margin-left:2px">@RedHat</span></a> <a href="https://www.linkedin.com/company/red-hat" title="LinkedIn" style="color:rgb(0,0,0);background:transparent url("https://marketing-outfit-prod-images.s3-us-west-2.amazonaws.com/8d3507e3c6b6c9ad10e301accf1a4af0/web-icon-linkedin.png") no-repeat scroll 0px 50%/12px;display:inline-block;line-height:20px;padding-left:12px;margin:0px 4px 0px 3px;padding-top:1px" target="_blank"><span style="margin-left:4px">Red Hat</span></a> <a href="https://www.facebook.com/redhat.il/" title="Facebook" style="color:rgb(0,0,0);background:transparent url("https://marketing-outfit-prod-images.s3-us-west-2.amazonaws.com/220b85e2f100025e94cb1bcd993bd51d/web-icon-facebook.png") no-repeat scroll 0px 50%/11px;display:inline-block;line-height:20px;padding-left:13px" target="_blank">Red Hat</a></div><div style="margin-top:12px"><table border="0"><tbody><tr><td width="100px"><a href="https://red.ht/sig" style="color:rgb(17,85,204)" target="_blank"><img src="https://static.redhat.com/libs/redhat/brand-assets/latest/corp/logo.png" width="90" height="auto"></a></td></tr></tbody></table></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><font face="arial, helvetica, sans-serif">Regards,<br>Shreyansh Shah</font></div></div></div></div></div></div></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div style="margin-left:44px"><div style="margin-bottom:0px;margin-left:0px;padding-bottom:5px"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><font face="RedHatText, sans-serif"><span style="font-size:14px;text-transform:capitalize"><b>Barak Sason Rofman</b></span></font><br></div></div></div></div></div></div></div></div></div></div><div style="margin-left:44px"><div style="margin-bottom:0px;margin-left:0px;padding-bottom:5px"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><p style="margin:0px;font-size:14px;color:rgb(0,0,0);font-family:overpass,sans-serif;font-weight:bold;padding:0px;text-transform:uppercase"><span style="font-family:RedHatText,sans-serif;font-size:12px;text-transform:capitalize;font-weight:normal;color:rgb(34,34,34)">Gluster Storage </span><span style="font-family:RedHatText,sans-serif;font-size:12px;text-transform:capitalize;font-weight:normal;color:rgb(34,34,34)">Development</span></p></div></div></div></div></div></div></div></div></div></div><div style="margin-left:44px"><div style="margin-bottom:0px;margin-left:0px;padding-bottom:5px"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><p style="margin:0px 0px 4px;font-size:12px;font-family:RedHatText,sans-serif"><a href="https://www.redhat.com/" style="color:rgb(0,136,206);margin:0px" target="_blank">Red Hat Israel</a></p><div style="margin-bottom:4px"><p style="margin:0px;font-size:12px;font-family:RedHatText,sans-serif">34 Jerusalem rd. Ra'anana, 43501</p></div><p style="margin:0px;font-size:12px;font-family:RedHatText,sans-serif"><span style="margin:0px;padding:0px">bsasonro<a href="mailto:adi@redhat.com" style="color:rgb(0,0,0);margin:0px" target="_blank">@redhat.com</a> </span> T: <u>+972-9-7692304</u><br>M: <u>+972-52-4326355</u></p><div style="font-size:12px;font-family:RedHatText,sans-serif;margin-bottom:8px"><a href="https://twitter.com/redhat" title="twitter" style="color:rgb(0,0,0);background:transparent url("https://marketing-outfit-prod-images.s3-us-west-2.amazonaws.com/3780bd4ede961ef3cd4108b8c0e80186/web-icon-twitter.png") no-repeat scroll 0px 50%/13px;display:inline-block;line-height:20px;padding-left:13px" target="_blank"><span style="margin-left:2px">@RedHat</span></a> <a href="https://www.linkedin.com/company/red-hat" title="LinkedIn" style="color:rgb(0,0,0);background:transparent url("https://marketing-outfit-prod-images.s3-us-west-2.amazonaws.com/8d3507e3c6b6c9ad10e301accf1a4af0/web-icon-linkedin.png") no-repeat scroll 0px 50%/12px;display:inline-block;line-height:20px;padding-left:12px;margin:0px 4px 0px 3px;padding-top:1px" target="_blank"><span style="margin-left:4px">Red Hat</span></a> <a href="https://www.facebook.com/redhat.il/" title="Facebook" style="color:rgb(0,0,0);background:transparent url("https://marketing-outfit-prod-images.s3-us-west-2.amazonaws.com/220b85e2f100025e94cb1bcd993bd51d/web-icon-facebook.png") no-repeat scroll 0px 50%/11px;display:inline-block;line-height:20px;padding-left:13px" target="_blank">Red Hat</a></div><div style="margin-top:12px"><table border="0"><tbody><tr><td width="100px"><a href="https://red.ht/sig" style="color:rgb(17,85,204)" target="_blank"><img src="https://static.redhat.com/libs/redhat/brand-assets/latest/corp/logo.png" width="90" height="auto"></a></td></tr></tbody></table></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>