<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Thanks for the quick answer!<div class=""><br class=""></div><div class="">I think I can reduce data on the "full" bricks, solving the problem temporarily.</div><div class=""><br class=""></div><div class="">The thing is, that the behavior changed from 3.12 to 6.5: 3.12 didn't have problems with almost full bricks, so I thought everything was fine. Then, after the upgrade, I ran into this problem. This might be a corner case that will go away once no-one uses 3.12 any more.</div><div class=""><br class=""></div><div class="">But I think I can create a situation with 6.5 only that reproduces the error. Suppose I have a brick that 99% full. So a write() will succeed. After the write, the brick can be 100% full, so a subsequent mkdir() will produce stale file handles (i.e., bricks that have different directory trees). The funny thing is, that the mkdir() on the user side does not produce an error. Clearly, no-one should ever let the file system get to 99%, but still, mkdir should fail then... </div><div class=""><br class=""></div><div class="">What remains: is there a recommended way how to deal with the situation that I have some bricks that don't have all directories?</div><div class=""><br class=""></div><div class="">best wishes,</div><div class="">Stefan</div><div class=""><br class=""></div><div class=""><div><blockquote type="cite" class=""><div class=""><p dir="ltr" class="">That seems a gluster control.</p><p dir="ltr" class="">Still, for me the issue is quite obvious - you are at 100% (or almost) storage and you should rebalance your VMs.</p><p dir="ltr" class="">Can you do a storage migration from the storage using gluster volumes at 100% to another storage whose gluster volumes are not so full (for example those that are 88% full)?</p><p dir="ltr" class="">Best Regards,<br class="">
Strahil Nikolov</p>
<div class="quote">On Sep 19, 2019 11:43, Stefan Solbrig <<a href="mailto:stefan.solbrig@ur.de" class="">stefan.solbrig@ur.de</a>> wrote:<br type="attribution" class=""><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class=""><div class="">Dear all,</div><div class=""><br class=""></div><div class="">I have a situation where "mkdir" on a client produces stale file handles.</div><div class="">This happened after upgrading from 3.12 to 6.5</div><div class=""><br class=""></div><div class="">I believe I found the reason for it:</div><div class="">6.5 (but not 3.12) checks if there is space left on the device before doing a "mkdir", but calculates the "fullness" in percent. In my situations I have bricks that seem 100% full although there is plenty space left on the device (several GBytes, see listing below). In this situation, a "mkdir" is not performed on bricks that are 100% full, but the "mkdir" succeeds from a user perspective. Then, doing a "ls" on the recently created directory leads to the message "stale file handle".</div><div class=""><br class=""></div><div class="">I believe the call sequence is more or less this:</div><div class=""><br class=""></div><div class="">server-rpc-fops.c:539:server_mkdir_cbk</div><div class="">server-rpc-fops.c:2666:server_mkdir_resume</div><div class="">server-rpc-fops.c:5242:server3_3_mkdir</div><div class="">posix-entry-ops.c:625:posix_mkdir</div><div class="">posix-helpers.c:2271</div><div class=""><br class=""></div><div class="">My questions are:</div><div class="">* is it meant to operate in this way?</div><div class="">* is there a built-in way to fix the inconsistent directories? </div><div class="">(I tried creating the missing directories on the bricks by hand, which seemed to fix the issue, but I'm not sure if this will introduce other problems.)</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">The obvious (good) fix would be to redistribute the data such that the 100% full bricks will have enough free space. However, if a user writes a really large file, the problem can re-occur any time... </div><div class=""><br class=""></div><div class="">best wishes,</div><div class="">Stefan</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">PS:</div><div class="">File system listing. Each file system is served as a brick, in a distribute-only system.</div><div class=""><br class=""></div><div class=""><div style="margin:0px;font-size:11px;line-height:normal;font-family:'menlo';background-color:rgb( 255 , 248 , 232 )" class="">Filesystem Size Used Avail Use% Mounted on</div><div style="margin:0px;font-size:11px;line-height:normal;font-family:'menlo';background-color:rgb( 255 , 248 , 232 )" class="">/dev/mapper/vgosb03pool06vd03-lvosb03pool06vd03 30T 27T 3.8T 88% /gl/lvosb03pool06vd03</div><div style="margin:0px;font-size:11px;line-height:normal;font-family:'menlo';background-color:rgb( 255 , 248 , 232 )" class="">/dev/mapper/vgosb03pool06vd02-lvosb03pool06vd02 30T 27T 3.8T 88% /gl/lvosb03pool06vd02</div><div style="margin:0px;font-size:11px;line-height:normal;font-family:'menlo';background-color:rgb( 255 , 248 , 232 )" class="">/dev/mapper/vgosb03pool06vd01-lvosb03pool06vd01 30T 27T 3.7T 88% /gl/lvosb03pool06vd01</div><div style="margin:0px;font-size:11px;line-height:normal;font-family:'menlo';background-color:rgb( 255 , 248 , 232 )" class="">/dev/mapper/vgosb03pool01vd01-lvosb03pool01vd01 30T 30T 7.8G 100% /gl/lvosb03pool01vd01</div><div style="margin:0px;font-size:11px;line-height:normal;font-family:'menlo';background-color:rgb( 255 , 248 , 232 )" class="">/dev/mapper/vgosb03pool01vd02-lvosb03pool01vd02 30T 30T 41G 100% /gl/lvosb03pool01vd02</div><div style="margin:0px;font-size:11px;line-height:normal;font-family:'menlo';background-color:rgb( 255 , 248 , 232 )" class="">/dev/mapper/vgosb03pool01vd03-lvosb03pool01vd03 30T 29T 1.5T 96% /gl/lvosb03pool01vd03</div><div style="margin:0px;font-size:11px;line-height:normal;font-family:'menlo';background-color:rgb( 255 , 248 , 232 )" class="">/dev/mapper/vgosb03pool01vd04-lvosb03pool01vd04 30T 30T 17G 100% /gl/lvosb03pool01vd04</div><div style="margin:0px;font-size:11px;line-height:normal;font-family:'menlo';background-color:rgb( 255 , 248 , 232 )" class="">/dev/mapper/vgosb03pool02vd01-lvosb03pool02vd01 30T 30T 57G 100% /gl/lvosb03pool02vd01</div><div style="margin:0px;font-size:11px;line-height:normal;font-family:'menlo';background-color:rgb( 255 , 248 , 232 )" class="">/dev/mapper/vgosb03pool02vd02-lvosb03pool02vd02 30T 30T 29G 100% /gl/lvosb03pool02vd02</div><div style="margin:0px;font-size:11px;line-height:normal;font-family:'menlo';background-color:rgb( 255 , 248 , 232 )" class="">/dev/mapper/vgosb03pool02vd03-lvosb03pool02vd03 30T 30T 26G 100% /gl/lvosb03pool02vd03</div><div style="margin:0px;font-size:11px;line-height:normal;font-family:'menlo';background-color:rgb( 255 , 248 , 232 )" class="">/dev/mapper/vgosb03pool02vd04-lvosb03pool02vd04 31T 31T 9.7G 100% /gl/lvosb03pool02vd04</div><div style="margin:0px;font-size:11px;line-height:normal;font-family:'menlo';background-color:rgb( 255 , 248 , 232 )" class="">/dev/mapper/vgosb03pool03vd01-lvosb03pool03vd01 30T 30T 93G 100% /gl/lvosb03pool03vd01</div><div style="margin:0px;font-size:11px;line-height:normal;font-family:'menlo';background-color:rgb( 255 , 248 , 232 )" class="">/dev/mapper/vgosb03pool03vd02-lvosb03pool03vd02 30T 30T 23G 100% /gl/lvosb03pool03vd02</div><div style="margin:0px;font-size:11px;line-height:normal;font-family:'menlo';background-color:rgb( 255 , 248 , 232 )" class="">/dev/mapper/vgosb03pool03vd03-lvosb03pool03vd03 30T 30T 163G 100% /gl/lvosb03pool03vd03</div><div style="margin:0px;font-size:11px;line-height:normal;font-family:'menlo';background-color:rgb( 255 , 248 , 232 )" class="">/dev/mapper/vgosb03pool03vd04-lvosb03pool03vd04 31T 30T 384G 99% /gl/lvosb03pool03vd04</div><div style="margin:0px;font-size:11px;line-height:normal;font-family:'menlo';background-color:rgb( 255 , 248 , 232 )" class="">/dev/mapper/vgosb03pool04vd01-lvosb03pool04vd01 30T 29T 1.1T 97% /gl/lvosb03pool04vd01</div><div style="margin:0px;font-size:11px;line-height:normal;font-family:'menlo';background-color:rgb( 255 , 248 , 232 )" class="">/dev/mapper/vgosb03pool04vd02-lvosb03pool04vd02 30T 27T 3.9T 88% /gl/lvosb03pool04vd02</div><div style="margin:0px;font-size:11px;line-height:normal;font-family:'menlo';background-color:rgb( 255 , 248 , 232 )" class="">/dev/mapper/vgosb03pool04vd03-lvosb03pool04vd03 30T 29T 1.9T 94% /gl/lvosb03pool04vd03</div><div style="margin:0px;font-size:11px;line-height:normal;font-family:'menlo';background-color:rgb( 255 , 248 , 232 )" class="">/dev/mapper/vgosb03pool04vd04-lvosb03pool04vd04 31T 29T 1.9T 94% /gl/lvosb03pool04vd04</div><div style="margin:0px;font-size:11px;line-height:normal;font-family:'menlo';background-color:rgb( 255 , 248 , 232 )" class="">/dev/mapper/vgosb03pool05vd01-lvosb03pool05vd01 30T 28T 2.3T 93% /gl/lvosb03pool05vd01</div><div style="margin:0px;font-size:11px;line-height:normal;font-family:'menlo';background-color:rgb( 255 , 248 , 232 )" class="">/dev/mapper/vgosb03pool05vd02-lvosb03pool05vd02 30T 27T 3.9T 88% /gl/lvosb03pool05vd02</div><div style="margin:0px;font-size:11px;line-height:normal;font-family:'menlo';background-color:rgb( 255 , 248 , 232 )" class="">/dev/mapper/vgosb03pool05vd03-lvosb03pool05vd03 30T 27T 3.9T 88% /gl/lvosb03pool05vd03</div><div style="margin:0px;font-size:11px;line-height:normal;font-family:'menlo';background-color:rgb( 255 , 248 , 232 )" class="">/dev/mapper/vgosb03pool05vd04-lvosb03pool05vd04 31T 27T 3.9T 88% /gl/lvosb03pool05vd04</div></div><div class=""><br class=""></div><br class=""><div class="">-- <br class="">Dr. Stefan Solbrig<br class="">Universität Regensburg, Fakultät für Physik,<br class="">93040 Regensburg, Germany<br class="">Tel +49-941-943-2097<br class=""><br class=""></div><br class=""></div></blockquote></div></div></blockquote></div><br class=""></div></body></html>