[m-rev.] for review: Disable tabling progs in parallel grades in tests/debugger.

[Peter Wang]

On Fri, 21 Jan 2022 18:09:48 +1100 "Zoltan Somogyi" <zoltan.somogyi at runbox.com> wrote:
> 
> 2022-01-21 17:55 GMT+11:00 "Peter Wang" <novalazy at gmail.com>:
> > The whole test infrastructure has always been horrible to work with.
> 
> I wouldn't call it horrible, but I agree, it is "highly suboptimal".
> 
> > If we were to start over,
> 
> I think we should, after the release, though we should keep
> the pieces that do work.
> 
> > I think my suggestions would be:
> > 
> >  1. To create a test case, just add the .m file to the directory.
> >     Don't require it to be listed in a Mmakefile or whatever.
> >     We can establish a convention for .m files that aren't
> >     the top-level module of a test case.
> 
> What convention would you propose? Simply a filename
> containing the string "submodule"?

I suggest "_helper", which should be pretty obvious, or perhaps "_aux"
for something shorter. Not all additional modules are submodules.
I think we want to leave "submodule" for tests about submodules.

> >  2. Each test case FOO can be controlled by creating other files named
> >     FOO.something. To disable 'loopcheck' in .par grades, there could be
> >     a file 'loopcheck.disable' that contains something like:
> > 
> > 	.par
> > 
> >     which tells the test infrastructure to skip it in .par grades.
> 
> Agreed, though in the interest of keeping ls output easier to read,
> I would use DISABLE/loopcheck instead.
> 

I'd prefer to see everything related to foo with "ls foo*".
If I want to see the list of test cases I'd run "ls *.m".

> >     Other things you might want to control are: how to invoke the test
> >     case, how to normalise the output, how to determine a passing test,
> 
> Agreed.
> 
> >   3. Make it easier to run individual test cases.
> 
> For single-module test cases, and if you don't need to use mmc --make,
> lmc works fine for me in most cases. I would love to lift both of those restrictions.
> I have an idea for lifting the former, but don't know enough mmc --make
> to lift the latter.

To run the test case exactly as the test system would, you currently
have to run something like:

    WORKSPACE=/path mmake MC=lmc GRADE=grade TESTS=foo runtests_local

(I suppose I should have written a shell script to do that long ago.)

Compare that with the Prince test system, which I'm also familiar with.
To run all tests in a directory, you run:

    ./runtests.sh

To run a subset of the tests, you'd run:

    ./runtests.sh foo*.html

While we're on the subject, I'd also appreciate if the test system would
be less noisy by default. Again, for comparison, the Prince test system
produces no output for a passing test case, and a failing test produces
only a single line of output, something like:

    foo.html:   Fails pdf.  Missing log.

The output file (foo.out) and contents of stderr (foo.err) will be there
if I need to investigate the failure.

Peter