[m-dev.] Test cases (was: for review: big ints)

[Fergus Henderson]

On 06-Apr-1998, Tyson Dowd <trd at stimpy.cs.mu.oz.au> wrote:
> On adding new test cases:
> 
> The tests directory is getting pretty full.

Well, --use-subdirs helps a lot.

> It's sometimes difficult
> to tell feature tests from regression tests, and lots of the feature
> tests are together. 

What's the difference between a feature test and a regression test?
The boundary is not entirely clear.  Is the distinguishing factor
related to history rather than the nature of the test case itself?
It would be strange if the "correct" location for a test case depended
on historical factors rather than just being determined by the test
case itself.

[quoted out of order]
> And then further subdivisions, including regression tests: e.g.
> 	language/errors
> 	language/warnings
> 	language/general
> 	language/regression

If a bug causes an incorrect warning message,
where does the regression test go: in `warnings' or in `regression'?

I think the distinction between feature tests and regression tests is a
relatively incidental one that we should not use in deciding the
location of test cases.

> I don't really see the point of dividing tests on how they are tested,
> so eventually I'd prefer not to have `hard_coded' (you can do the same
> thing with some make rules). 

I think dividing tests on how they are tested is useful.  For example,
you may want to run tests that are tested using some particular tool
only on machines for which that tool is available.  If you're testing
a cross-compiler, you may want to run only those tests which don't
require running the generated programs.  Etc.

However, I could be persuaded into making this categorization
be a secondary one, with the primary classification being
based on what it is that each test tests.

> I think the best scheme would be to divide
> tests into
> 	language features (types, modes, plain old code) 
> 	implementation features (pragmas, non-language compile options)
> 	library tests (unit tests, sanity tests, library regression)
> 	build environment (mmake, scripts, dependencies, mdemangle)
> 	programs (full programs, e.g. samples and extras and others)

I think this is a fairly good categorization scheme.

In particular, distinguishing between tests of Mercury language
features and features specific to the current implementation would
be very useful to anyone developing a new Mercury implementation.

Currently we divide the tests of language features into those that test
features which should also work in Prolog, and those that don't.
This is a potentially useful distinction and so I think we should keep it.
But is this a distinction on *what* to test, or on *how*? 

I guess in truth it is both.  Perhaps considering it as the latter is
most useful.

In addition to the categories you mentioned above, we need a place
to put tests of the debugger and profiler. These could be considered
part of the build environment, but probably it makes more sense to
make them new top-level categories.

This leads to the following set of subdirectories:

	Proposed name		Purpose
	-------------		-------
	language		Standard Mercury language features
				(types, modes, plain old code) 
	extensions		Implementation-specific pragmas, etc.
	library			Tests of library modules
				(unit tests, sanity tests, library regression)
	implementation		Tests of the build environment
				(mmake, scripts, dependencies, mdemangle)
	debugger		Tests of the debugger(s)
 	programs		Full programs, e.g. samples and extras
				and others

The existing tests would be moved and renamed as follows:

	Current name		Proposed location
	------------		------------------
	general			language/general		
	hard_coded		language/hard_coded	
	valid			language/valid
	invalid			language/invalid
	warnings		language/warnings
	term			extensions/term
	misc_tests		implementation
	benchmarks		programs/benchmarks

I'm open to suggestions about the names of the subdirectories of `language'.
Hmm, now that I think about it, `warnings' should not be there, instead
it should be a subdirectory of `implementation' -- at least unless
we change the language reference manual to require warnings for certain
programs.

In your original scheme, you had a distinction between
"implementation features" and "build environment".
Above, I changed this to a distinction between
`extensions' (implementation-specific things that involve
changes to the source code) and `implementation'
(implementation-specific things that do not involve changes
to the source code).  I'm not sure which is better.

-- 
Fergus Henderson <fjh at cs.mu.oz.au>  |  "I have always known that the pursuit
WWW: <http://www.cs.mu.oz.au/~fjh>  |  of excellence is a lethal habit"
PGP: finger fjh at 128.250.37.3        |     -- the last words of T. S. Garp.