[m-dev.] Test cases (was: for review: big ints)

[Tyson Dowd]

On 07-Apr-1998, Andrew Bromage <bromage at cs.mu.OZ.AU> wrote:
> G'day.
> 
> Tyson Dowd wrote:
> 
> > The tests directory is getting pretty full.  It's sometimes difficult
> > to tell feature tests from regression tests, and lots of the feature
> > tests are together. 
> 
> As a matter of interest, what does this buy you apart from general
> cleanliness and good karma?

Mainly the problems with the current setup are:
	- many features don't have test cases (or have regression tests
	  masquerading as test cases) and it's almost impossible to see
	  that.
	- test cases are difficult to find if you don't know what
	  you are looking for ("is there a test case for...?")
	- the directory design is not transparent for developers
	  ("so where do I put my tests?").  
	- no place for programs (which means about half the bug reports
	  we are sent don't produce regression tests).

> 
> In one sense, I agree with you.  Knowing what the test was testing
> for can help you diagnose the cause.  The problem is that last
> year's regression tests for the code generator often double as
> this year's regression tests for some semantic pass (a pathological
> case is a pathological case), so such a neat filing system seems to
> me unhelpful in the general case.
> 

Well, the purpose of a test will not always be its only use.
I'm not concerned if the tests also show a problem elsewhere, that's
fine, but I am concerned if there are large gaps in what we are
testing.

> I think what would make a bigger difference is to discipline developers
> to attach to test cases a summary of the log message when a bug is
> fixed.  This would be more helpful information, because if a test fails,
> what you have is some indication of where to look.
> 
> For example, while this might be useful information:
> 
> 	tests/language/regression/lambda_mode_check_bug.m
> 
> I'd much rather see:
> 
> 	% Non-local variables of lambdas were not mode checked
> 	% correctly by the 8 May 1997 compiler.  Problem was a
> 	% bug in modecheck_unification where the non-locals were
> 	% not computed correctly.
> 	%			- ahacker (9 May 1997)
> 	%
> 	% This test failed 2 Feb 1998 with a run-time abort
> 	% "variable P (8) not found" at opt level -O3.  The lambda
> 	% goal was not correctly quantified after eliminating the
> 	% excess assignment during simplify2.
> 	%			- anotherhacker (3 Feb 1998)
> 
> [Disclaimer: The above bugs are made up.  Any resemblance to real
> bugs living or dead is purely coincidental.]
> 

Well, this is ok, but the CVS logs actually give much the same
information (if it is a good log message, that is).

My main point is that although it would have been nice to have the
neat filing scheme we have above, it is still possible to have
something close from now on if subdirectories are created for new
features.

-- 
       Tyson Dowd           # So I asked Sarah: what's the serial number on
                            # your computer? She replied:
     trd at cs.mu.oz.au        #          A-C-2-4-0-V-/-5-0-H-Z
http://www.cs.mu.oz.au/~trd #