[Omake] Clutter in the tests

[David Kågedal]

Jason Hickey <jyh at cs.caltech.edu> writes:

> I don't have a problem with tests for feature requests.  But they
> "pollute" the test output, making it more likely that regressions are
> missed.  We need a way, perhaps "omake regression" that would tell us
> only about errors.

At work, we have a file listing all the "expected failures", which
means that those test failures won't be highlighted by test runs
unless the unexpectedly succeed.  Any such expected failure must be
entered as a bug in the bug database so it's still on the radar.

The problem with RFEs is similar, but not identical. So you have three
kinds of test statuses:

 1) Tests that are expected to succeed
 2) Tests that are expected to fail because of known bugs
 3) Tests that are expected to fail because it's an RFE

and, of course

  4) Test that fail because they are broken, or not updated

A "regression" test run would highlight the tests that fail any of the
expectations above, but not say anything about failures in categories
2 and 3.  A "bugs" test should note failures in category 2, and a
"full" test should note all failures.  And in all cases, an unexpected
success should be highlighted.

If you have a working bug tracking system, I think that the
"regression" runs work very well if you mark all known bugs as
expected failures.  This will make sure that a failed test run is
really something to take note of.  Otherwise you will run the test
suite and always expect failures and not notice if there are 15 or 16
of them.

So a simple solution is to have a file that lists tests like this:

    parse/C/Test        bug 123
    object/Test14       rfe 650
    simple/Test8        rfe 651

And as soon as you discover a test failure in the regression tests,
you either fix it at once, or you bug report it and add it to the
exceptions file.

-- 
David Kågedal