[Gluster-users] Bug with hardlink limitation in 3.12.13 ?

[Shyam Ranganathan]

On 08/31/2018 01:06 PM, Reiner Keller wrote:
> Hello,
> 
> Am 31.08.2018 um 13:59 schrieb Shyam Ranganathan:
>> I suspect you have hit this:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1602262#c5
>>
>> I further suspect your older setup was 3.10 based and not 3.12 based.
>>
>> There is an additional feature added in 3.12 that stores GFID to path
>> conversion details using xattrs (see "GFID to path" in
>> https://docs.gluster.org/en/latest/release-notes/3.12.0/#major-changes-and-features
>> )
>>
>> Due to which xattr storage limit is reached/breached on ext4 based bricks.
>>
>> To check if you are facing similar issue to the one in the bug provided
>> above, I would check if the brick logs throw up the no space error on a
>> gfid2path set failure.
> 
> thanks for the hint.
> 
> From log output (= no gfid2path errors) it seems to be not the problem
> although the old
> gluster volume was setup with version 3.10.x (or even 3.8.x i think).
> 
> I wrote I could reproduce it on new ext4  and on old xfs gluster volumes
> with version
> 3.12.13 while it was running fine with ~ 3.12.8 (half year ago) without
> problems.
> 
> But just saw that my old main volume wasn't/isn't xfs but also ext4.
> Digging into logs I could see that I was running in January still 3.10.8
> / 3.10.9
> and initial switched in April to 3.12.9 / 3.12 version branch.
> 
> From entry sizes/differences your suggestion would fit:
> 
>     https://manpages.debian.org/testing/manpages/xattr.7.en.html or
>     http://man7.org/linux/man-pages/man5/attr.5.html
> 
>   In the current ext2, ext3, and ext4 filesystem implementations, the
>        total bytes used by the names and values of all of a file's extended
>        attributes must fit in a single filesystem block (1024, 2048 or 4096
>        bytes, depending on the block size specified when the filesystem was
>        created).
> 
> because I can see differences by volume setup type:

<huge snip>

So in short, the inode size limits in ext4 impacts the hard link counts
that can be created in Gluster, which is the limitation that you hit,
would that be a correct summary?

> 
> 
>> To check if you are facing similar issue to the one in the bug provided
>> above, I would check if the brick logs throw up the no space error on a
>> gfid2path set failure.
> 
> Is there some parameter to get more detailed error logging ? But from
> docu it looks like it has default good settings:

The error logs posted are from the client (FUSE mount) logs, the log
lines with the gfid2path that I was mentioning is on the bricks.

There is no further logging level that needs to change to see the said
errors as these are warning and above.

>>
>>> My search for documentation found only the parameter
>>> "storage.max-hardlinks" with default of 100 for version 4.0.
>>> I checked it in my gluster 3.12.13 but here the parameter is not yet
>>> implemented.
> 
> If this problem is backend filesystem related it would be good to have
> it documented also for 4.0 that the storage.max-hardlinks parameter
> would work only if the backend is e.g. xfs and has enough inode space
> for it (best with a reference/short example howto calculate it) ?

Fair point, raised a github issue around the same here [1]
(contributions welcome :) ).

Regards,
Shyam

[1] Gluster documentation github issue for hardlink and ext4
limitations: https://github.com/gluster/glusterdocs/issues/418