[Gluster-devel] Bad file access in bit-rot-detection

[Raghavendra Bhat]

Hi,

As part of Bit-rot detection feature a file that has its data changed 
due to some backend errors is marked as a bad file by the scrubber (sets 
an extended attribute indicating its a bad file). Now, the access to the 
bad file has to be denied (to prevent wrong data being served).

In bit-rot-stub xlator (the xlator which does object versioning and 
sends notifications to BitD upon object modification) the check for 
whether the file is bad or not can be done in lookup where if the xattr 
is set, then the object can be marked as bad within its inode context as 
well. But the problem is what if the object was not marked as bad at the 
time of lookup and later it was marked bad. Now when a fop such as open, 
readv or writev comes, the fops should not be allowed. If its fuse 
client from which the file is being accessed, then probably its ok to 
rely only on lookups (to check if its bad or not), as fuse sends lookups 
before sending fops. But for NFS, once the lookup is done and filehandle 
is available further lookups are not sent. In that case relying only on 
lookup to check if its a bad file or not is suffecient.

Below 3 solutions in bit-rot-stub xlator seem to address the above issue.

1) Whenever a fop such as open, readv or writev comes, check in the 
inode context if its a bad file or not. If not, then send a getxattr of 
bad file xattr on that file. If its present, then set the bad file 
attribute in the inode context and fail the fop.

But for above operation, a getxattr call has to be sent downwards for 
almost each open or readv or writev. If the file is identified as bad, 
then getxattr might not be necessary. But for good files extra getxattr 
might affect the performance.

OR

2) Set a key in xdata whenever open, readv, or writev comes (in 
bit-rot-stub xlator) and send it downwards. The posix xlator can look 
into the xdata and if the key for bad file identification is present, 
then it can do getxattr as part of open or readv or writev itself and 
send the response back in xdata itself.

Not sure whether the above method is ok or not as it overloads open, 
readv and writev. Apart from that, the getxattr disk operation is still 
done.

OR

3) Once the file is identified as bad, the scrubber marks it as bad (via 
setxattr) by sending a call to to bit-rot-stub xlator. Bit-rot-stub 
xlator marks the file as bad in the inode context once it receives the 
notification from scrubber that a file is bad. This saves those getxattr 
calls being made from other fops (either in bit-rot-stub xlator or posix 
xlator).

But the trick is what if the inode gets forgotten or the brick restarts. 
But I think in that case, checking in lookup call is suffecient (as in 
both inode forgets and brick restarts, a lookup will definitely come if 
there is an accss to that file).

Please provide feedback on above 3 methods. If there are any other 
solutions which might solve this issue, they are welcome.

Regards,
Raghavendra Bhat