[m-rev.] for review: improve test framework

[Zoltan Somogyi]

On 15-Aug-2002, Mark Brown <dougl at cs.mu.OZ.AU> wrote:
> In other words, can we draw useful conclusions about a
> given change if only some particular class of tests is run?

Yes, although in general it is a conclusion about probabilities, not
certainties. If you pass all the tough tests in an area, by definition
(of what it means for a test to be tough) you are likely (not guaranteed,
of course) to pass the easy ones as well.

And of course, if I change an aspect of implementation (e.g. formatting of
debugger output) that is tested in only a fixed set of directories, then
you can get more solid conclusions as well.

> Can we omit
> test cases outside of this class without risking the overlooking of
> failures?

No, but that doesn't matter. Nobody (except TY Chen :-) was proposing omitting
the execution of any test cases from the final bootcheck.

> However, if the debugger tests are run, the debugger/declarative tests
> should also be run.  The declarative debugger depends on the internal
> debugger to work correctly, and it (necessarily) makes a lot of
> assumptions about how the internal debugger works.  Therefore, any
> change to the internal debugger could easily break the declarative
> debugger, even if the internal debugger works fine.  Omitting the
> debugger/declarative tests would in general bear a significant risk of
> breaking the declarative debugger.

I agree, but some other test directories don't have this relationship.

If I fix a bug in how we handle code with commits, and the change only affects
code with commits, I don't want to reexecute the tests in the subdirectories
of hard_coded until the final bootcheck, because they don't have any more
commits than the tests in any other directory, on average. (Hard_coded itself
has several torture tests specifically designed to test behaviour on commits.)

For that matter, the split between general and hard_coded has been obsolete
for a while now, ever since we stopped using NU-Prolog.

> To conclude, I think that this _is_ a meaningful way to classify these
> tests, so my argument is that we should organise and use our testing
> framework, as above, on the principle of what dependencies exist between
> implementations of the different sub-systems.

So you agree that where dependencies don't exist in either direction, as e.g.
between hard_coded and hard_coded/typeclasses, there should be no parent/child
relationship between those directories of tests, and you should be able to run
them independently?

Zoltan.
--------------------------------------------------------------------------
mercury-reviews mailing list
post:  mercury-reviews at cs.mu.oz.au
administrative address: owner-mercury-reviews at cs.mu.oz.au
unsubscribe: Address: mercury-reviews-request at cs.mu.oz.au Message: unsubscribe
subscribe:   Address: mercury-reviews-request at cs.mu.oz.au Message: subscribe
--------------------------------------------------------------------------