[m-rev.] for review: Disable tabling progs in parallel grades in tests/debugger.
Peter Wang
novalazy at gmail.com
Sat Jan 22 11:59:24 AEDT 2022
On Fri, 21 Jan 2022 18:09:48 +1100 "Zoltan Somogyi" <zoltan.somogyi at runbox.com> wrote:
>
> 2022-01-21 17:55 GMT+11:00 "Peter Wang" <novalazy at gmail.com>:
> > The whole test infrastructure has always been horrible to work with.
>
> I wouldn't call it horrible, but I agree, it is "highly suboptimal".
>
> > If we were to start over,
>
> I think we should, after the release, though we should keep
> the pieces that do work.
>
> > I think my suggestions would be:
> >
> > 1. To create a test case, just add the .m file to the directory.
> > Don't require it to be listed in a Mmakefile or whatever.
> > We can establish a convention for .m files that aren't
> > the top-level module of a test case.
>
> What convention would you propose? Simply a filename
> containing the string "submodule"?
I suggest "_helper", which should be pretty obvious, or perhaps "_aux"
for something shorter. Not all additional modules are submodules.
I think we want to leave "submodule" for tests about submodules.
> > 2. Each test case FOO can be controlled by creating other files named
> > FOO.something. To disable 'loopcheck' in .par grades, there could be
> > a file 'loopcheck.disable' that contains something like:
> >
> > .par
> >
> > which tells the test infrastructure to skip it in .par grades.
>
> Agreed, though in the interest of keeping ls output easier to read,
> I would use DISABLE/loopcheck instead.
>
I'd prefer to see everything related to foo with "ls foo*".
If I want to see the list of test cases I'd run "ls *.m".
> > Other things you might want to control are: how to invoke the test
> > case, how to normalise the output, how to determine a passing test,
>
> Agreed.
>
> > 3. Make it easier to run individual test cases.
>
> For single-module test cases, and if you don't need to use mmc --make,
> lmc works fine for me in most cases. I would love to lift both of those restrictions.
> I have an idea for lifting the former, but don't know enough mmc --make
> to lift the latter.
To run the test case exactly as the test system would, you currently
have to run something like:
WORKSPACE=/path mmake MC=lmc GRADE=grade TESTS=foo runtests_local
(I suppose I should have written a shell script to do that long ago.)
Compare that with the Prince test system, which I'm also familiar with.
To run all tests in a directory, you run:
./runtests.sh
To run a subset of the tests, you'd run:
./runtests.sh foo*.html
While we're on the subject, I'd also appreciate if the test system would
be less noisy by default. Again, for comparison, the Prince test system
produces no output for a passing test case, and a failing test produces
only a single line of output, something like:
foo.html: Fails pdf. Missing log.
The output file (foo.out) and contents of stderr (foo.err) will be there
if I need to investigate the failure.
Peter
More information about the reviews
mailing list