[Gluster-devel] RFC - structuring indices/xattrop directory for faster self heal

[raghav]

Hi,

This is basically a RFC for bringing more structure into the 
indices/xattrop directory thus improving heal performance.

Problem statement - Right now the indices/xattrop directory is a flat 
directory with a bunch of files with their names as the gfids of the 
files/directories that need to be healed. These files are handlinks to 
each other. The way self heal daemon works is that it does a directory 
read of xattrop directory and does a heal entry on per entry basis. 
Since the dependencies are not built across the entries, the heal 
ordering is dependent on how the entries (on reading the directory) are 
returned to the self heal daemon. And this causes self heal to execute 
among multiple crawls because the parents MAY not get healed before the 
children. The number of crawls becomes large when there is a big 
hierarchy in the directory in the brick; that would degrade the 
performance of the self heal.

Solution-
To maintain a hierarchy among dependent entries - This can be done in 
memory and if ncecessary be logged into a file in the same 
directory(indices/xattrop). There is a patch sent

http://review.gluster.org/#/c/5140/

that allows for a parent gfid to be stored per file. So when a brick 
daemon needs to create a new entry in xattrop directory, it checks if 
its parent gfid is also required to be healed. If yes, it inserts this 
entry against the parent.  This was we can build a hierarchy of linked 
lists which is two dimensional; each member of list points to its peer 
because they have the same parent (in one dimension) and they point to 
their children (in second dimension). This normally would happen when 
say in a N way replicate scheme, one brick was down and many 
creates/writes are hapenning on other bricks. AT time of healing we heal 
all the parents followed by its children and so on. This way we will be 
sure that in one crawl, all heals will pass thru. All independent 
(unrelated) files that require self heal will have their gfids stored 
seperately; these could be healed at a higher priority since these are 
low hanging fruits as far as heal is concerned since there are no 
dependencies.
Having such a mechanism could  thus create a more deterministic 
mechanism to heal. Also the errors that we see during heal (basically 
because of the file non existing because parent directory entry is still 
unhealed) would go away making the self heal process cleaner. This is 
observed in the following bug:

https://bugzilla.redhat.com/show_bug.cgi?id=978335

Like I mentioned above this can be logged into a file. Flushing into a 
file could make this information persistent across reboots; though its 
use cases needs to be better evaluated. Doing this on every writable fop 
that could cause an entry creation into index/xattrop could make the 
performance slower. Hence this can be done at some periodic intervals. 
Since this is an additional log file that is used as an input to heal, 
it should not cause any problems on upgrade. Worst case is that self 
heal does not look into this file and does the old style healing. 
Implementation wise this can be implemented using a new translator (in 
front of index translator) Again doing this would help in upgrade scenarios.

Please revert back with ur suggestions.

Thanks & Regards
Raghav