[m-dev.] Test cases (was: for review: big ints)
fjh at cs.mu.OZ.AU
Wed Apr 8 03:20:12 AEST 1998
On 06-Apr-1998, Tyson Dowd <trd at stimpy.cs.mu.oz.au> wrote:
> On adding new test cases:
> The tests directory is getting pretty full.
Well, --use-subdirs helps a lot.
> It's sometimes difficult
> to tell feature tests from regression tests, and lots of the feature
> tests are together.
What's the difference between a feature test and a regression test?
The boundary is not entirely clear. Is the distinguishing factor
related to history rather than the nature of the test case itself?
It would be strange if the "correct" location for a test case depended
on historical factors rather than just being determined by the test
[quoted out of order]
> And then further subdivisions, including regression tests: e.g.
If a bug causes an incorrect warning message,
where does the regression test go: in `warnings' or in `regression'?
I think the distinction between feature tests and regression tests is a
relatively incidental one that we should not use in deciding the
location of test cases.
> I don't really see the point of dividing tests on how they are tested,
> so eventually I'd prefer not to have `hard_coded' (you can do the same
> thing with some make rules).
I think dividing tests on how they are tested is useful. For example,
you may want to run tests that are tested using some particular tool
only on machines for which that tool is available. If you're testing
a cross-compiler, you may want to run only those tests which don't
require running the generated programs. Etc.
However, I could be persuaded into making this categorization
be a secondary one, with the primary classification being
based on what it is that each test tests.
> I think the best scheme would be to divide
> tests into
> language features (types, modes, plain old code)
> implementation features (pragmas, non-language compile options)
> library tests (unit tests, sanity tests, library regression)
> build environment (mmake, scripts, dependencies, mdemangle)
> programs (full programs, e.g. samples and extras and others)
I think this is a fairly good categorization scheme.
In particular, distinguishing between tests of Mercury language
features and features specific to the current implementation would
be very useful to anyone developing a new Mercury implementation.
Currently we divide the tests of language features into those that test
features which should also work in Prolog, and those that don't.
This is a potentially useful distinction and so I think we should keep it.
But is this a distinction on *what* to test, or on *how*?
I guess in truth it is both. Perhaps considering it as the latter is
In addition to the categories you mentioned above, we need a place
to put tests of the debugger and profiler. These could be considered
part of the build environment, but probably it makes more sense to
make them new top-level categories.
This leads to the following set of subdirectories:
Proposed name Purpose
language Standard Mercury language features
(types, modes, plain old code)
extensions Implementation-specific pragmas, etc.
library Tests of library modules
(unit tests, sanity tests, library regression)
implementation Tests of the build environment
(mmake, scripts, dependencies, mdemangle)
debugger Tests of the debugger(s)
programs Full programs, e.g. samples and extras
The existing tests would be moved and renamed as follows:
Current name Proposed location
I'm open to suggestions about the names of the subdirectories of `language'.
Hmm, now that I think about it, `warnings' should not be there, instead
it should be a subdirectory of `implementation' -- at least unless
we change the language reference manual to require warnings for certain
In your original scheme, you had a distinction between
"implementation features" and "build environment".
Above, I changed this to a distinction between
`extensions' (implementation-specific things that involve
changes to the source code) and `implementation'
(implementation-specific things that do not involve changes
to the source code). I'm not sure which is better.
Fergus Henderson <fjh at cs.mu.oz.au> | "I have always known that the pursuit
WWW: <http://www.cs.mu.oz.au/~fjh> | of excellence is a lethal habit"
PGP: finger fjh at 220.127.116.11 | -- the last words of T. S. Garp.
More information about the developers