[Gluster-users] [EXT] link files not being created

[Xavi Hernandez]

Hi Stefan,

On Tue, Oct 18, 2022 at 10:34 AM Stefan Solbrig <stefan.solbrig at ur.de>
wrote:

> Hi Xavi,
>
> On Mon, Oct 17, 2022 at 1:03 PM Stefan Solbrig <stefan.solbrig at ur.de>
> wrote:
>
>> Dear all,
>>
>> I was doing some testing regarding to GlusterFS link files (as they are
>> created by a "move" operation). According to this document:
>> https://www.gluster.org/glusterfs-algorithms-distribution/  If a link
>> file is missing, it should be created after accessing the file.
>> However, I don't see this behaviour.  If I delete (by hand) a link file
>> on the brick, the file is still accessible, but the link file is never
>> recreated. I can do an "open" or a "stat" on the file without getting an
>> error, but the link file is not created.
>> Is this the intended behaviour? Or am I misunderstanding the above
>> mentioned document?
>>
>
> You shouldn't access or modify the backend filesystems manually, you can
> accidentally create unexpected problems if you don't fully understand what
> you are doing.
>
> That said, most probably the access to the file is still working because
> Gluster is using its cached information to locate the file. If the client
> mount is restarted, probably the file won't be accessible anymore unless
> you disable the "lookup-optimize" option (and this should recreate the link
> file).
>
> Regards,
>
> Xavi
>
>
> Thanks for the quick reply!  Maybe I should explain better my motivation
> for the above mentioned experiments. I have a large production system
> running GlusterFS with almost 5 PB of data (in approx 100G of inodes). It's
> a distributed-only system (no sharding, not dispersed).  In this system,
> the users sometimes experience the problem that they cannot delete a
> seemingly empty directory.  The cause of this problem is, that the
> directory contains leftover link files, i.e. dht link files where the
> target is gone. I haven't identified yet why this happens and I don't have
> a method to provoke this error (otherwise I would have mentioned it on this
> list already.)
>

What version of Gluster are you using ? if I remember correctly, there was
a fix in 3.10.2 (and some other following patches) to delete stale link
files when deleting empty directories to avoid precisely this problem.
Recently there have also been some patches to avoid leaving some of those
stale entries.

If you are still using 3.x I would recommend you to upgrade to a newer
version, which have many issues already fixed.


>  But my quick & dirty fix is, to delete these leftover link files by hand.
>  (These leftover link files are not being cleaned up by a "rebalance".)
>

If you only remove the file, you are leaving some data behind that should
also be removed. Each file is associated with an entry inside
.glusterfs/xx/yy in the brick, called gfid. This entry has the format of an
uuid and can be determined by reading (in hex) the "trusted.gfid" xattr of
the file you are going to delete:

# getfattr -n trusted.gfid -e hex <file>


If you manually remove files, you should also remove the gfid.


> The reason for my experiments with link files is: what happens if for some
> reason I accidentally delete a link file where the target still exists?
>
> In the experiments (not on the production system) I also tried umounting
> and remounting the system, and I already tried setting "loopup-optmize =
> off". It doesn't affect the outcome of the experiments.
>

If after remounting the volume you are still able to access the file but
the link file is not created, then it means that it's not needed. Maybe it
was one of those stale link files.

Can you give me one example of those link files (I need the name) and the
trusted.glusterfs.dht xattr of the parent directory from all bricks ?

# getfattr -n trusted.glusterfs.dht -e hex <path/to/directory>


Regards,

Xavi


> best wishes,
> Stefan
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20221018/decb4abe/attachment.html>