[Gluster-devel] Regression test failures - Call for Action

[Jeff Darcy]

> Also, one of us should
> go through the last however-many failures and determine the relative
> frequency of failures caused by each test, so we can prioritize.

I started doing this, and very quickly found a runaway "winner" -
data-self-heal.t, which also happens to be the very first test we
run.  Hmmm.  The failures seem to have a common signature:

  Running all the regression test cases (new way)
  mkdir: cannot create directory `/mnt/glusterfs/2': File exists
  rm: cannot remove `/mnt/glusterfs/2': Is a directory
  mkdir: cannot create directory `/mnt/glusterfs/2': File exists
  rm: cannot remove `/mnt/glusterfs/2': Is a directory
  mkdir: cannot create directory `/mnt/glusterfs/2': File exists
  [18:38:06] ./tests/basic/afr/data-self-heal.t .. 
  Dubious, test returned 1 (wstat 256, 0x100)

That mkdir is the last thing in cleanup().  Because that's the
last thing each test script calls, that failure turns into a bad
exit code for the entire test.  The problem is that cleanup()
never unmounts that directory, like it does for the others we
use.  There are only two tests that use it, but if either of
them should ever fail to unmount the directory themselves then
their failure will become rather persistent - often across the
next several runs.  I'll be looking into why this condition
isn't *completely* permanent, as well as why those tests aren't
doing the unmount.  Meanwhile, I've implemented a general
workaround.

  http://review.gluster.org/#/c/10536/

With that, I think we'll see enough of a reduction in spurious
failures that further drastic action might be unnecessary.