[Gluster-users] mkdir produces stale file handles

[Stefan Solbrig]

>>> I think I can reduce data on the "full" bricks, solving the problem temporarily.
>>> 
>>> The thing is, that the behavior changed from 3.12 to 6.5:   3.12 didn't have problems with almost full bricks, so I thought everything was fine. Then, after the upgrade, I ran into this problem. This might be a corner case that will go away once no-one uses 3.12 any more.
>>> 
>>> But I think I can create a situation with 6.5 only that reproduces the error. Suppose I have a brick that 99% full.  So a write() will succeed. After the write, the brick can be 100% full, so a subsequent mkdir() will produce stale file handles (i.e., bricks that have different directory trees).  The funny thing is, that the mkdir() on the user side does not produce an error.   Clearly, no-one should ever let the file system get to 99%, but still, mkdir should fail then... 
>> 
>> I think there is a soft and hard limit that prevents creation of files/folders when a specific threshold is hit , but that threshold might be per brick instead of per replica set.
> 
> There is the cluster.min-free-disk, which states that the server should look for a free brick if the hash would place the file on a brick with less than "min-free-disk" bytes.   However, this seems to be a "should". If all bricks have less space than "min-free-disk", then the file is written anyway. 
> 
> Apart from that, I have some really large bricks (around 200 TB each), which means that if these are 99% full, then there are still 2 TB left (a signifikant amount).  The logic of "do not create a directory if the brick is 100% full" seems to be hard coded.  I didn't find a setting to disable this logic.

I actually found a setting that influences this behavior:
setting

gluster v set <volname> storage.reserve  0

lets me fill the bricks to 100%   So my issue is solved.
best wishes,
-Stefan