[m-rev.] for review: improve test framework

[Mark Brown]

On 15-Aug-2002, Fergus Henderson <fjh at cs.mu.OZ.AU> wrote:
> On 15-Aug-2002, Mark Brown <dougl at cs.mu.OZ.AU> wrote:
> > You also can't easily run a subset of the test cases in the hard_coded
> > directory without running all of them.
> 
> Actually, you can.
> 
> 	mmake foo.res bar.res
> 
> > But the question is: why would you want to?
> 
> I do this all the time, for exactly the reason that Zoltan described.

Allow me to rephrase my point.  You can't easily run an _arbitrary_
subset of test cases in a directory, but why would you want to?  Only
certain subsets are meaningful to run in a given circumstance.

So far, I've seen a few suggested subsets, each with an explanation of how
it is a meaningful subset to run:

	- Simon's suggestion that the tests which just failed should be
	  run.  This is useful when you are part of the way through a
	  change, and you have had some tests that have already failed
	  on an earlier version of the change.  I agree that this is
	  the most useful suggestion so far.

	- My suggestion that all tests pertaining to the sub-system
	  being modified, and any sub-system that depends on it, should
	  be run.  This is useful when you have just begun to add a new
	  feature, and you haven't yet run any tests.  The first point
	  can't apply here, because there won't have been any failures.

	- The obvious suggestion that all tests (including bootstrap
	  test) should be run before posting for review.

What I haven't seen is a rationale for running all the tests pertaining
to a particular sub-system, without also running the tests for any
sub-system that depends on it.  (Actually, there is a reason for having
the *_local targets, but that is an implementation detail and wasn't
intended for use.)

> 
> > Is it just a case of sacrificing stability in order to speed development,
> 
> No.  It is a case of running the tests in the right order in order
> to speed development by getting quick notification of the most likely
> failures.

In this case, the "right order" is the one suggested by Simon.

> No stability is sacrificed so long as you run a full
> bootcheck before committing.
> 

Ok, you're right.  The risk we are dealing with here is the risk of slowing
down the edit-test cycle unnecessarily, not the risk of introducing bugs.

Cheers,
Mark.

--------------------------------------------------------------------------
mercury-reviews mailing list
post:  mercury-reviews at cs.mu.oz.au
administrative address: owner-mercury-reviews at cs.mu.oz.au
unsubscribe: Address: mercury-reviews-request at cs.mu.oz.au Message: unsubscribe
subscribe:   Address: mercury-reviews-request at cs.mu.oz.au Message: subscribe
--------------------------------------------------------------------------