[m-rev.] for review: improve test framework

[Fergus Henderson]

On 16-Aug-2002, Mark Brown <dougl at cs.mu.OZ.AU> wrote:
> On 15-Aug-2002, Fergus Henderson <fjh at cs.mu.OZ.AU> wrote:
> > On 15-Aug-2002, Mark Brown <dougl at cs.mu.OZ.AU> wrote:
> > > You also can't easily run a subset of the test cases in the hard_coded
> > > directory without running all of them.
> > 
> > Actually, you can.
> > 
> > 	mmake foo.res bar.res
> > 
> > > But the question is: why would you want to?
> > 
> > I do this all the time, for exactly the reason that Zoltan described.
> 
> Allow me to rephrase my point.  You can't easily run an _arbitrary_
> subset of test cases in a directory, but why would you want to?
> Only certain subsets are meaningful to run in a given circumstance.

Right.  But in general what you want to run is a subset of the
tests that you think are most likely to fail, but which will run in
a short amount of time.

> So far, I've seen a few suggested subsets, each with an explanation of how
> it is a meaningful subset to run:
> 
> 	- Simon's suggestion that the tests which just failed should be
> 	  run.

This is a good heuristic for tests which are likely to fail.

Another good heuristic is "any tests which have just been added".

> 	- My suggestion that all tests pertaining to the sub-system
> 	  being modified, and any sub-system that depends on it, should
> 	  be run.

This is often a good heuristic.

> What I haven't seen is a rationale for running all the tests pertaining
> to a particular sub-system, without also running the tests for any
> sub-system that depends on it.

I agree, generally that's not very useful IMHO.

-- 
Fergus Henderson <fjh at cs.mu.oz.au>  |  "I have always known that the pursuit
The University of Melbourne         |  of excellence is a lethal habit"
WWW: <http://www.cs.mu.oz.au/~fjh>  |     -- the last words of T. S. Garp.
--------------------------------------------------------------------------
mercury-reviews mailing list
post:  mercury-reviews at cs.mu.oz.au
administrative address: owner-mercury-reviews at cs.mu.oz.au
unsubscribe: Address: mercury-reviews-request at cs.mu.oz.au Message: unsubscribe
subscribe:   Address: mercury-reviews-request at cs.mu.oz.au Message: subscribe
--------------------------------------------------------------------------