[Gluster-users] Tons of dht: Found anomalies in (null)

Tue May 19 06:48:05 UTC 2020

On Tue, May 19, 2020 at 12:15 PM Aravinda VK <aravinda at kadalu.io> wrote:

>
>
> On 19-May-2020, at 12:05 PM, Susant Palai <spalai at redhat.com> wrote:
>
>
>
> On Thu, Apr 30, 2020 at 6:31 AM Artem Russakovskii <archon810 at gmail.com>
> wrote:
>
>> Hi,
>>
>> Every time I ls large dirs in our 1x4 replicate gluster volume, I get a
>> ton of these in the logs.
>>
>> If I run the same ls right away again, they won't repeat, but inevitably,
>> in a couple of hours or days, they show up again.
>>
>> What is the meaning of these messages and why am I getting them so
>> frequently?
>>
>> [2020-04-30 00:57:39.020757] I [MSGID: 109063]
>> [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies
>> in (null) (gfid = 9f879856-36fc-49f9-900f-0e5d742b2e54). Holes=1 overlaps=0
>> [2020-04-30 00:57:39.021036] I [MSGID: 109063]
>> [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies
>> in (null) (gfid = d6c7f283-3607-4a4d-9099-6f76792d5bdd). Holes=1 overlaps=0
>> [2020-04-30 00:57:39.021067] I [MSGID: 109063]
>> [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies
>> in (null) (gfid = d8f88e6d-d80e-40fc-a386-9dc35212ad91). Holes=1 overlaps=0
>> [2020-04-30 00:57:39.021079] I [MSGID: 109063]
>> [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies
>> in (null) (gfid = 704b3e6e-9f55-4166-8a0f-4283fb513593). Holes=1 overlaps=0
>> [2020-04-30 00:57:39.021103] I [MSGID: 109063]
>> [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies
>> in (null) (gfid = 2de3da38-c56d-4196-b001-e4fa7d901383). Holes=1 overlaps=0
>> [2020-04-30 00:57:39.021118] I [MSGID: 109063]
>> [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies
>> in (null) (gfid = c0b2c50e-1abd-4c86-9958-2c6f87d469d1). Holes=1 overlaps=0
>> [2020-04-30 00:57:39.021400] I [MSGID: 109063]
>> [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies
>> in (null) (gfid = 5365eb39-f120-4a1c-9920-1e375c030843). Holes=1 overlaps=0
>> [2020-04-30 00:57:39.021436] I [MSGID: 109063]
>> [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies
>> in (null) (gfid = f7f46524-28d8-4c57-b244-43f953ec3656). Holes=1 overlaps=0
>> [2020-04-30 00:57:39.021448] I [MSGID: 109063]
>> [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies
>> in (null) (gfid = 6cfd94ad-3b82-4a6f-9f0d-03cc0c59e5f1). Holes=1 overlaps=0
>> [2020-04-30 00:57:39.021455] I [MSGID: 109063]
>> [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies
>> in (null) (gfid = 285061fb-f1b7-49c0-ae10-5d8d8d5cd569). Holes=1 overlaps=0
>> [2020-04-30 00:57:39.021463] I [MSGID: 109063]
>> [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies
>> in (null) (gfid = 38734b38-381f-4825-b65a-4292c5075faa). Holes=1 overlaps=0
>> [2020-04-30 00:57:39.021470] I [MSGID: 109063]
>> [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies
>> in (null) (gfid = 60c15124-18da-40e7-8776-cd78303b79ad). Holes=1 overlaps=0
>> [2020-04-30 00:57:39.021481] I [MSGID: 109063]
>> [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies
>> in (null) (gfid = e26783bf-bee7-4a59-ae3e-505ada95c5f9). Holes=1 overlaps=0
>> [2020-04-30 00:57:39.022163] I [MSGID: 109063]
>> [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies
>> in (null) (gfid = 6f7da42f-22bd-4e96-828d-02c1c1690afc). Holes=1 overlaps=0
>> [2020-04-30 00:57:39.022192] I [MSGID: 109063]
>> [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies
>> in (null) (gfid = 6d17f907-9fb3-4d13-9a2f-8266b0e3bead). Holes=1 overlaps=0
>> [2020-04-30 00:57:39.022205] I [MSGID: 109063]
>> [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies
>> in (null) (gfid = 64f40fc3-efb3-4359-98df-ba3be29b6326). Holes=1 overlaps=0
>> [2020-04-30 00:57:39.022214] I [MSGID: 109063]
>> [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies
>> in (null) (gfid = d9081c5d-40e1-4521-a4b5-78fd60d71237). Holes=1 overlaps=0
>> [2020-04-30 00:57:39.022735] I [MSGID: 109063]
>> [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies
>> in (null) (gfid = b627c175-d6eb-44de-9eed-a13efea52f88). Holes=1 overlaps=0
>> [2020-04-30 00:57:39.022771] I [MSGID: 109063]
>> [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies
>> in (null) (gfid = 9dce8380-ca5d-4947-831c-0db1aae85e09). Holes=1 overlaps=0
>> [2020-04-30 00:57:39.022788] I [MSGID: 109063]
>> [dht-layout.c:659:dht_layout_normalize] 0-SNIP_data1-dht: Found anomalies
>> in (null) (gfid = a9c24397-a521-4d1b-8543-42b331fa6794). Holes=1 overlaps=0
>>
>
> Generally, these logs indicate that there were directories missing ("hole
> = 1") on one of the bricks. These are harmless message and you can ignore
> them.
>
>
> Good opportunity to improve the log messages. Can we disable this logging
> if this can be ignored? Is it when distribute count is 1 or these messages
> are always harmless?
>

I will think about it and will update. But my current feeling is that they
are helfpful in debugging issues related to directory inconsistency. If I
see no value to these messages in INFO level, will send a patch.

>
>

>
>>
>>
>
>>
> Thanks.
>>
>> Sincerely,
>> Artem
>>
>> --
>> Founder, Android Police <http://www.androidpolice.com/>, APK Mirror
>> <http://www.apkmirror.com/>, Illogical Robot LLC
>> beerpla.net | @ArtemR <http://twitter.com/ArtemR>
>> ________
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968
>>
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>
> Aravinda Vishwanathapura
> https://kadalu.io
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200519/d555c459/attachment.html>