[Gluster-users] Gluster 3.12.14: wrong quota in Distributed Dispersed Volume

[Frank Ruehlemann]

Hi,

we're running a Distributed Dispersed volume with Gluster 3.12.14 at
Debian 9.6 (Stretch).

We migrated our data (>300TB) from a pure Distributed volume into this
Dispersed volume with cp, followed by multiple rsyncs.
After the migration was successful we enabled quotas again with "gluster
volume quota $VOLUME enable", which finished successfully.
And we set our required quotas with "gluster volume quota $VOLUME
limit-usage $PATH $QUOTA", which finished without errors too.

But our "gluster volume quota $VOLUME list" shows wrong values.
For example:
A directory with ~170TB of data shows only 40.8TB Used.
When we sum up all quoted directories we're way under the ~310TB that
"df -h /$volume" shows.
And "df -h /$volume/$directory" shows wrong values for nearly all
directories.

All 72 8TB-bricks and all quota deamons of the 6 servers are visible and
online in "gluster volume status $VOLUME".


In quotad.log I found multiple warnings like this:
> [2018-11-16 09:21:25.738901] W [dict.c:636:dict_unref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.14/xlator/features/quotad.so(+0x1d58) [0x7f6844be7d58] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.14/xlator/features/quotad.so(+0x2b92) [0x7f6844be8b92] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_unref+0xc0) [0x7f684b0db640] ) 0-dict: dict is NULL [Invalid argument]

In some brick logs I found those:
> [2018-11-19 07:23:30.932327] I [MSGID: 120020] [quota.c:2198:quota_unlink_cbk] 0-$VOLUME-quota: quota context not set inode (gfid:f100f7a9-0779-4b4c-880f-c8b3b4bdc49d) [Invalid argument]
and (replaced the volume name with "$VOLUME") those:
> The message "W [MSGID: 120003] [quota.c:821:quota_build_ancestry_cbk] 0-$VOLUME-quota: parent is NULL [Invalid argument]" repeated 13 times between [2018-11-19 15:28:54.089404] and [2018-11-19 15:30:12.792175]
> [2018-11-19 15:31:34.559348] W [MSGID: 120003] [quota.c:821:quota_build_ancestry_cbk] 0-$VOLUME-quota: parent is NULL [Invalid argument]

I already found that setting the flag "trusted.glusterfs.quota.dirty" might help, but I'm unsure about the consequences that will be triggered. And I'm unsure about the necessary version flag.

Has anyone an idea how to fix this?

Best Regards,
-- 
Frank Rühlemann
   IT-Systemtechnik

UNIVERSITÄT ZU LÜBECK
    IT-Service-Center
    
    Ratzeburger Allee 160
    23562 Lübeck
    Tel +49 451 3101 2034
    Fax +49 451 3101 2004
    ruehlemann at itsc.uni-luebeck.de
    www.itsc.uni-luebeck.de