[m-rev.] for review: improve test framework

[Zoltan Somogyi]

On 15-Aug-2002, Mark Brown <dougl at cs.mu.OZ.AU> wrote:
> As I implied earlier, I don't agree.  It is all too likely that
> fixing the bug will introduce another bug, possibly in one of the
> sub-directories.  The second bug will be caught by the full bootcheck,
> of course, but not until much later.  Needless to say, a wasted full
> bootcheck will take much more time than merely running the sub-directory
> tests, even if you have to do this several times.

You are overlooking my point.

I am trying to shorten the time between "have candidate fix" and "confirm fix
works on its intended test cases". It is worth shortening this time, because
if the fix doesn't work, which is reasonably likely, I want to go back to
working on the bug involved. The shorter the cycle time, the more times
I can go around the cycle in a session, and the less information I have swapped
out of my head by reading mail or working in some other workspace by the time
during the wait for the testing to end. This is why I like Simon's suggestion,
which is even better than mine, of capturing the set of failed test cases
and rerunning only those.

If the cycle ends because I have a fix that does handle the problems it was
meant to handle, then the probability of the full bootcheck picking up problems
is much lower. The test cases that originally failed were typically the
toughest tests in the given area, so if you pass them you are very likely
to pass the easier tests as well. Typically I set off such final bootchecks
just before the end of a session. I don't really care long that bootcheck
takes, as long as it is done by the start of my next work session.

Zoltan.
--------------------------------------------------------------------------
mercury-reviews mailing list
post:  mercury-reviews at cs.mu.oz.au
administrative address: owner-mercury-reviews at cs.mu.oz.au
unsubscribe: Address: mercury-reviews-request at cs.mu.oz.au Message: unsubscribe
subscribe:   Address: mercury-reviews-request at cs.mu.oz.au Message: subscribe
--------------------------------------------------------------------------