[Gluster-users] glusterfs performance issues

[Mario Kadastik]

> You have a replicated filesystem, brick1 and brick2.
> Brick 2 goes down and you edit a 4k file, appending data to it.
> That change, and the fact that there is a pending change, is stored on brick1.
> Brick2 returns to service.
> Your app wants to append to the file again. It calls stat on the file. Brick2 answers first stating that the file is 4k long. Your app seeks to 4k and writes. Now the data you wrote before is gone.
> 
> This is one of the processes by which stale stat data can cause data loss. That's why each lookup() (which precedes the stat) causes a self-heal check and why it's a problem that hasn't been resolved in the last two years.
> 
> I don't know the answer. I know that they want this problem to be solved, but right now the best solution is hardware. The lower the latency, the less of a problem you'll have.

Well I'd assume that the brick that comes online has to check everything from the other online bricks before it's authoritative in answering any client calls. This way if a brick comes up and sees another brick(s) in its replica blocks online the assumption should be that the data can be bad on this brick. Therefore until a complete self heal is performed the brick should be considered bad for this information. The next step from this is how to guarantee that a brick actually returns to healthy state in a busy filesystem. The basic way would be that any new writes of files are written to all bricks (including the bad) and declared good on the healing brick and as a background process all files on the brick are hashed and checked against the bricks that were live before. In a reasonable environment this should complete in a reasonable amount of time and at worst means that you'll be running at a reduced performance while this sync is happening, but it would guarantee that you don't have data loss unless you lose all your previously online bricks in which case you're anyway in disaster recovery where this semi-live brick can help in recovering files from the time it last went down or better. 

Mario Kadastik, PhD
Researcher

---
  "Physics is like sex, sure it may have practical reasons, but that's not why we do it" 
     -- Richard P. Feynman