[Gluster-users] mkdir produces stale file handles

[Strahil]

That seems a gluster control.

Still, for me the issue is quite obvious - you are at 100% (or almost)  storage and you should rebalance your VMs.

Can you do a storage migration from the storage using gluster volumes at 100% to another storage whose gluster volumes are not so full (for example those that are 88% full)?

Best Regards,
Strahil NikolovOn Sep 19, 2019 11:43, Stefan Solbrig <stefan.solbrig at ur.de> wrote:
>
> Dear all,
>
> I have a situation where "mkdir" on a client produces stale file handles.
> This happened after upgrading from 3.12 to 6.5
>
> I believe I found the reason for it:
> 6.5 (but not 3.12) checks if there is space left on the device before doing a "mkdir", but calculates the "fullness" in percent.   In my situations I have bricks that seem 100% full although there is plenty space left on the device (several GBytes, see listing below).  In this situation, a "mkdir" is not performed on bricks that are 100% full, but the "mkdir" succeeds from a user perspective.  Then, doing a "ls" on the recently created directory leads to the message "stale file handle".
>
> I believe the call sequence is more or less this:
>
> server-rpc-fops.c:539:server_mkdir_cbk
> server-rpc-fops.c:2666:server_mkdir_resume
> server-rpc-fops.c:5242:server3_3_mkdir
> posix-entry-ops.c:625:posix_mkdir
> posix-helpers.c:2271
>
> My questions are:
> * is it meant to operate in this way?
> * is there a built-in way to fix the inconsistent directories? 
> (I tried creating the missing directories on the bricks by hand, which seemed to fix the issue, but I'm not sure if this will introduce other problems.)
>
>
> The obvious (good) fix would be to redistribute the data such that the 100% full bricks will have enough free space. However, if a user writes a really large file, the problem can re-occur any time... 
>
> best wishes,
> Stefan
>
>
> PS:
> File system listing.  Each file system is served as a brick, in a distribute-only system.
>
> Filesystem                                       Size  Used Avail Use% Mounted on
> /dev/mapper/vgosb03pool06vd03-lvosb03pool06vd03   30T   27T  3.8T  88% /gl/lvosb03pool06vd03
> /dev/mapper/vgosb03pool06vd02-lvosb03pool06vd02   30T   27T  3.8T  88% /gl/lvosb03pool06vd02
> /dev/mapper/vgosb03pool06vd01-lvosb03pool06vd01   30T   27T  3.7T  88% /gl/lvosb03pool06vd01
> /dev/mapper/vgosb03pool01vd01-lvosb03pool01vd01   30T   30T  7.8G 100% /gl/lvosb03pool01vd01
> /dev/mapper/vgosb03pool01vd02-lvosb03pool01vd02   30T   30T   41G 100% /gl/lvosb03pool01vd02
> /dev/mapper/vgosb03pool01vd03-lvosb03pool01vd03   30T   29T  1.5T  96% /gl/lvosb03pool01vd03
> /dev/mapper/vgosb03pool01vd04-lvosb03pool01vd04   30T   30T   17G 100% /gl/lvosb03pool01vd04
> /dev/mapper/vgosb03pool02vd01-lvosb03pool02vd01   30T   30T   57G 100% /gl/lvosb03pool02vd01
> /dev/mapper/vgosb03pool02vd02-lvosb03pool02vd02   30T   30T   29G 100% /gl/lvosb03pool02vd02
> /dev/mapper/vgosb03pool02vd03-lvosb03pool02vd03   30T   30T   26G 100% /gl/lvosb03pool02vd03
> /dev/mapper/vgosb03pool02vd04-lvosb03pool02vd04   31T   31T  9.7G 100% /gl/lvosb03pool02vd04
> /dev/mapper/vgosb03pool03vd01-lvosb03pool03vd01   30T   30T   93G 100% /gl/lvosb03pool03vd01
> /dev/mapper/vgosb03pool03vd02-lvosb03pool03vd02   30T   30T   23G 100% /gl/lvosb03pool03vd02
> /dev/mapper/vgosb03pool03vd03-lvosb03pool03vd03   30T   30T  163G 100% /gl/lvosb03pool03vd03
> /dev/mapper/vgosb03pool03vd04-lvosb03pool03vd04   31T   30T  384G  99% /gl/lvosb03pool03vd04
> /dev/mapper/vgosb03pool04vd01-lvosb03pool04vd01   30T   29T  1.1T  97% /gl/lvosb03pool04vd01
> /dev/mapper/vgosb03pool04vd02-lvosb03pool04vd02   30T   27T  3.9T  88% /gl/lvosb03pool04vd02
> /dev/mapper/vgosb03pool04vd03-lvosb03pool04vd03   30T   29T  1.9T  94% /gl/lvosb03pool04vd03
> /dev/mapper/vgosb03pool04vd04-lvosb03pool04vd04   31T   29T  1.9T  94% /gl/lvosb03pool04vd04
> /dev/mapper/vgosb03pool05vd01-lvosb03pool05vd01   30T   28T  2.3T  93% /gl/lvosb03pool05vd01
> /dev/mapper/vgosb03pool05vd02-lvosb03pool05vd02   30T   27T  3.9T  88% /gl/lvosb03pool05vd02
> /dev/mapper/vgosb03pool05vd03-lvosb03pool05vd03   30T   27T  3.9T  88% /gl/lvosb03pool05vd03
> /dev/mapper/vgosb03pool05vd04-lvosb03pool05vd04   31T   27T  3.9T  88% /gl/lvosb03pool05vd04
>
>
> -- 
> Dr. Stefan Solbrig
> Universität Regensburg, Fakultät für Physik,
> 93040 Regensburg, Germany
> Tel +49-941-943-2097
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190919/b02937b4/attachment.html>