[m-rev.] for review: improve test framework

[Mark Brown]

On 15-Aug-2002, Zoltan Somogyi <zs at cs.mu.OZ.AU> wrote:
> On 15-Aug-2002, Mark Brown <dougl at cs.mu.OZ.AU> wrote:
> > However, if the debugger tests are run, the debugger/declarative tests
> > should also be run.  The declarative debugger depends on the internal
> > debugger to work correctly, and it (necessarily) makes a lot of
> > assumptions about how the internal debugger works.  Therefore, any
> > change to the internal debugger could easily break the declarative
> > debugger, even if the internal debugger works fine.  Omitting the
> > debugger/declarative tests would in general bear a significant risk of
> > breaking the declarative debugger.
>
> I agree, but some other test directories don't have this relationship.

I'm suggesting that maybe they should (but see below).

> If I fix a bug in how we handle code with commits, and the change only affects
> code with commits, I don't want to reexecute the tests in the subdirectories
> of hard_coded until the final bootcheck, because they don't have any more
> commits than the tests in any other directory, on average. (Hard_coded itself
> has several torture tests specifically designed to test behaviour on commits.)

It would be logical and useful to put those in a sub-directory.  (And, yes,
if this were done you would want people who run tests in hard_coded to
also run tests in hard_coded/commits.)

> For that matter, the split between general and hard_coded has been obsolete
> for a while now, ever since we stopped using NU-Prolog.

Yep, I know about that; I imagine this is because of CVS's inflexibility
when it comes to moving files around.  So this will probably throw a
spanner in the works as far as a sub-directory based test case hierarchy
is concerned.

Using the directory structure of test cases, however, isn't the only
way to achieve a hierarchy of test cases.  The hierachy itself could be
programmed in to the relevant Mmakefiles, and I'd still argue that such a
hierarchy would be useful.  And if Mmakefile programming is used, the
"meaningful subsets" of test cases don't even have to form a hierachy;
my only requirement would be that they have some valid rationale to
back them up.

> > To conclude, I think that this _is_ a meaningful way to classify these
> > tests, so my argument is that we should organise and use our testing
> > framework, as above, on the principle of what dependencies exist between
> > implementations of the different sub-systems.
>
> So you agree that where dependencies don't exist in either direction, as e.g.
> between hard_coded and hard_coded/typeclasses, there should be no parent/child
> relationship between those directories of tests, and you should be able to run
> them independently?

I'd be inclined to argue that there is a dependency there.  Insofar as
the hard_coded test cases test the operation of complete Mercury
programs, then the typeclass tests, which are also complete Mercury
programs, do depend on the hard_coded tests.  For example, if you made
a change to the unused argument optimization, this change could just as
easily affect arguments that are typeclass_infos as it could affect any
other type of argument.  The typeclass implementation depends on unused
argument optimization being performed correctly just as much as many other
aspects of running Mercury programs do.

Even if there was no dependency, I still don't see the rationale for
testing all of the hard_coded tests, but nothing else.

Cheers,
Mark.

--------------------------------------------------------------------------
mercury-reviews mailing list
post:  mercury-reviews at cs.mu.oz.au
administrative address: owner-mercury-reviews at cs.mu.oz.au
unsubscribe: Address: mercury-reviews-request at cs.mu.oz.au Message: unsubscribe
subscribe:   Address: mercury-reviews-request at cs.mu.oz.au Message: subscribe
--------------------------------------------------------------------------