[m-rev.] for review: improve test framework

[Mark Brown]

On 15-Aug-2002, Zoltan Somogyi <zs at cs.mu.OZ.AU> wrote:
> On 15-Aug-2002, Simon Taylor <stayl at cs.mu.OZ.AU> wrote:
> > +# SUBDIRS - the names of any subdirectories containing tests.
> 
> In think one of the problems with the current system is that you can't run the
> tests in e.g. the hard_coded directory without also running the tests in the
> hard_coded/exceptions, hard_coded/purity, hard_coded/sub-modules and
> hard_coded/typeclasses directories.

You also can't easily run a subset of the test cases in the hard_coded
directory without running all of them.  But the question is: why would
you want to?  Is it just a case of sacrificing stability in order to
speed development, or are there _meaningful_ ways that the tests can be
classified?  In other words, can we draw useful conclusions about a
given change if only some particular class of tests is run?  Can we omit
test cases outside of this class without risking the overlooking of
failures?

If a change is made that only affects the internal debugger, I think it
is reasonable to only run the tests in the debugger directory during
development (of course, all tests should be run before posting for
review).  My reason for saying this is that any bug that triggers a test
failure outside of the debugger directory is fairly likely to trigger one
inside the debugger directory, so there is not much increased risk of
failures going unnoticed by omitting the non-debugger tests.

However, if the debugger tests are run, the debugger/declarative tests
should also be run.  The declarative debugger depends on the internal
debugger to work correctly, and it (necessarily) makes a lot of
assumptions about how the internal debugger works.  Therefore, any
change to the internal debugger could easily break the declarative
debugger, even if the internal debugger works fine.  Omitting the
debugger/declarative tests would in general bear a significant risk of
breaking the declarative debugger.

On the other hand, for a change which only affects the declarative debugger
I think it is reasonable to only run the tests in the debugger/declarative
directory during development, without also running the tests in the
parent directory.  As before, any bugs that cause the failure of
debugger tests would almost certainly cause the failure of
debugger/declarative tests, so there is not much increased risk of
failures going unnoticed if the former are omitted.

My rationale for putting the declarative debugger tests into a subdirectory
of the debugger tests was mainly to support the view stated above.
There is a dependency between the implementation of the declarative
debugger and the implementation of the internal debugger, and the
hierarchical structure of the tests was intended to reflect this.
To conclude, I think that this _is_ a meaningful way to classify these
tests, so my argument is that we should organise and use our testing
framework, as above, on the principle of what dependencies exist between
implementations of the different sub-systems.

Cheers,
Mark.

--------------------------------------------------------------------------
mercury-reviews mailing list
post:  mercury-reviews at cs.mu.oz.au
administrative address: owner-mercury-reviews at cs.mu.oz.au
unsubscribe: Address: mercury-reviews-request at cs.mu.oz.au Message: unsubscribe
subscribe:   Address: mercury-reviews-request at cs.mu.oz.au Message: subscribe
--------------------------------------------------------------------------