<div dir="ltr">Hi,<div><div class="gmail_extra"><br><div class="gmail_quote">On 23 April 2018 at 18:52, Frank Ruehlemann <span dir="ltr"><<a href="mailto:ruehlemann@itsc.uni-luebeck.de" target="_blank">ruehlemann@itsc.uni-luebeck.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
after 2 years running GlusterFS without bigger problems we're facing<br>
some strange errors lately.<br>
<br>
After updating to 3.12.7 some user reported at least 4 broken<br>
directories with some invisible files. The files are at the bricks and<br>
don't start with a dot, but aren't visible in "ls". Clients still can<br>
interact with them by using the explicit path.<br>
More information: <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1564071" rel="noreferrer" target="_blank">https://bugzilla.redhat.com/<wbr>show_bug.cgi?id=1564071</a></blockquote><div><br></div><div>I will continue the analysis for this issue in the bug. </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
<br>
And since this update gluster reported for the rebalance of >16900 PB<br>
(Petabyte!) of data for one of our 2 server, when using „gluster volume<br>
rebalance $myvolume status“. The time looks right, but the size of<br>
transfered files is absurd. The rebalance was with 3.12.6 in March 2018.<br>
The last rebalance log file listed no errors and a realistic size at the<br>
end.<br></blockquote><div><br></div><div>This has been seen a few times and is because an incorrect value is stored in the <a href="http://node_state.info">node_state.info</a> file . However, I don't know what causes this incorrect value to be stored. It is harmless and can be ignored.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
We started a new rebalance today during a downtime of our corresponding<br>
compute cluster, since these errors started to spread and this might<br>
help. The output of „gluster volume rebalance $myvolume status“ doesn't<br>
list any errors so far and the numbers look like realistic values.<br>
But we're seeing some strange errors (every few minutes) reports in the<br>
journald:<br>
„[2018-04-23 12:31:24.942377] E [MSGID: 113001]<br>
[posix.c:5983:_posix_handle_<wbr>xattr_keyvalue_pair] 0-$myvolume-posix:<br>
setxattr failed<br>
on /srv/glusterfs/bricks/DATA112/<wbr>data/.glusterfs/e6/a8/<wbr>e6a8ce50-fda5-4bad-8d4d-<wbr>acd25dafcaa2 while doing xattrop: key=trusted.glusterfs.quota.<wbr>1ce02d3b-b7ae-4485-903c-<wbr>2991de5350b6.contri.1 [No such file or directory]“<br>
The rebalance log file lists no errors.<br>
<br>
Has anybody seen similar error messages during a rebalance?<br></blockquote><div><br></div><div>Are any directories being deleted/renamed during the rebalance? If yes, this could be a valid message. </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
And we see some files dublicated. There are two copies on different<br>
bricks (we're running a distributed volume). <br>
One copy looks like this: <br>
$ ls -lah<br>
-rwxr--r-- 2 $user $group 293 May 11 2017 config<br>
<br>
The other one looks rather strange:<br>
$ ls -lah<br>
---------T 2 root $group 0 May 11 2017 config<br>
<br>
Has anybody seen similar broken files?<br></blockquote><div><br></div><div>This is fine as long as you only see a single file from the mount point. The 'T' files are internal gluster files (called linkto files) and should be invisible from the mount point. </div><div><br></div><div><br></div><div>Regards,</div><div>Nithya</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
We're using gluster 3.12 from the gluster.org-repositories on a standard<br>
Debian 9 with XFS formatted bricks.<br>
<br>
Hopefully somebody might have an answer how to fix this.<br>
<br>
At least somebody in the future might find this, since we didn't found<br>
anything while searching after these errors. If you're from the future:<br>
Good luck! (^_^)<br>
<br>
So far,<br>
<br>
-- <br>
Frank Rühlemann<br>
IT-Systemtechnik<br>
<br>
UNIVERSITÄT ZU LÜBECK<br>
IT-Service-Center<br>
<br>
Ratzeburger Allee 160<br>
23562 Lübeck<br>
Tel +49 451 3101 2034<br>
Fax +49 451 3101 2004<br>
<a href="mailto:ruehlemann@itsc.uni-luebeck.de">ruehlemann@itsc.uni-luebeck.de</a><br>
<a href="http://www.itsc.uni-luebeck.de" rel="noreferrer" target="_blank">www.itsc.uni-luebeck.de</a><br>
<br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a></blockquote></div><br></div></div></div>