[m-rev.] for review: improve test framework

[Simon Taylor]

On 15-Aug-2002, Fergus Henderson <fjh at cs.mu.OZ.AU> wrote:
> On 15-Aug-2002, Simon Taylor <stayl at cs.mu.OZ.AU> wrote:
> > 
> > Improve the test framework to make it easier to find out which tests
> > failed and to reduce disk usage (important in debug grades). The
> > disadvantage is that the tests can no longer be run in parallel.

> > Allow the tests to be run with `mmc --make' (still some failures).
> > 
> > tests/Mmake.common:
> > tests/*/Mmakefile:
> > 	Move common code (such as the code to deal with subdirectories)
> > 	to Mmake.common.
> > 
> > 	Run the tests using `mmake runtests' rather than using slightly
> > 	different runtests scripts in each directory.
> 
> The standard name for this would be "mmake check".
> (This is specified by the GNU coding guidelines.)
> 
> It's OK to have "runtests" as an alternative name if you like,
> but we should also support "mmake check".

I used "runtests" rather than "check" because `mmake X.check' usually
means build all the `.err' files (but I have added the check target).
 
> > Index: tests/Mmake.common
> ...
> > +ifeq ($(RUNTESTS_IN_SUBDIR),true)
> > +    STARTUP_LOG=:
> > +    REDIRECT_TO_LOG=
> > +    PROCESS_LOG=:
> > +    CLEANUP_LOG=:
> > +    REPORT_SUCCESS=:
> > +    REPORT_FAILURE=:
> > +else
> > +    STARTUP_LOG=rm -f $(LOG_FILE); touch $(LOG_FILE)
> > +    REDIRECT_TO_LOG= | tee -a $(LOG_FILE)
> > +    PROCESS_LOG=echo ERRORS; \
> > +        echo END ERRORS
> > +    CLEANUP_LOG=rm -f $(LOG_FILE)
> > +    REPORT_SUCCESS=echo ALL TESTS SUCCEEDED; rm -f $(ERROR_FILE)
> > +    REPORT_FAILURE= \
> > +        awk -f $(TESTS_DIR)/process_log.awk < $(LOG_FILE) > $(ERROR_FILE); \
> > +        echo SOME TESTS FAILED: see $(ERROR_FILE)
> > +endif
> 
> A comment or two there might be nice.
> Why are these set to null if RUNTESTS_IN_SUBDIR is true?
> What's the PROCESS_LOG command for?

This has all been removed. runtests_local now generates the
runtests.errs file, and the runtests.errs files from each
subdirectory are concatenated onto the end of the local
runtests.errs file.
 
> > +# If the tests in the subdirectories, still run the local tests,
> > +# but return a status of 1.
> > +# 
> > +runtests:
> > +	@$(STARTUP_LOG)
> > +	@rm -f subdirs_failed local_failed
> > +	+@{ mmake runtests_subdirs || touch subdirs_failed; } $(REDIRECT_TO_LOG)
> > +	+@{ mmake runtests_local || touch local_failed; } $(REDIRECT_TO_LOG)
> > +	@if [ -f subdirs_failed -o -f local_failed ] ; then \
> > +		$(REPORT_FAILURE); \
> > +		$(CLEANUP_LOG); \
> > +		rm -f subdirs_failed local_failed; \
> > +		exit 1; \
> > +	else \
> > +		$(REPORT_SUCCESS); \
> > +		$(CLEANUP_LOG); \
> > +		rm -f subdirs_failed local_failed; \
> > +	fi
> 
> Why is this code serialized by writing it as the body of a single
> target, rather than using multiple targets (with appropriate
> dependencies between then, of course) so that the different
> parts which don't depend on each other can run in parallel?
> E.g.
>
> ...

A simpler solution is to run `mmake -k runtests_subdirs runtests_local'.

> > +runtests_subdirs:
> > +	succeeded=true; \
> > +	for dir in $(SUBDIRS); do \
> > +		(cd $$dir && RUNTESTS_IN_SUBDIR=true mmake runtests) || \
> > +			succeeded=false; \
> > +	done; \
> > +	case $$succeeded in false) exit 1 ;; esac
> > +
> 
> Likewise here.
> 
> Why not
> 
> 	RUNTESTS_IN_SUBDIRS=$(SUBDIRS:%=runtests_in_%)
> 
> 	runtests_subdirs: $(RUNTESTS_IN_SUBDIRS)
> 
> 	.PHONY: $(RUNTESTS_IN_SUBDIRS)
> 	$(RUNTESTS_IN_SUBDIRS):
> 		cd $* && RUNTESTS_IN_SUBDIRS=true mmake runtests
> 
> ?

Done.
 
> > +#
> > +# Run all tests in the directory one at a time, running
> > +# `mmake realclean' for the tests which succeed and
> > +# gzipping the executables for those which don't.
> > +#
> > +runtests_local:
 
> Likewise here.  If the issue is log file intermingling, then the results
> of each test `foo' could be written out to `foo.log' and then these log
> files can all be concatenated at the end.

Done.

> > +realclean_subdirs:
> > +	+succeeded=true; \
> > +	for dir in $(SUBDIRS); do \
> > +	    (cd $$dir && mmake realclean) || succeeded=false; \
> > +	done
> > +	case $$succeeded in false) exit 1 ;; esac
> > +
> > +clean_subdirs:
> > +	+succeeded=true; \
> > +	for dir in $(SUBDIRS); do \
> > +		(cd $$dir && mmake clean) || succeeded=false; \
> > +	done; \
> > +	case $$succeeded in false) exit 1 ;; esac
> 
> Likewise here.

I haven't changed this. The rule you gave above for runtests_subdirs
only works with `mmake -k'. That's fine because runtests_subdirs is always
invoked from `mmake -k' in runtests. It would be possible to do this for
`realclean_subdirs', but there should be more than enough parallelism
within each directory, and the extra `mmake' invocation would possibly
slow things down more than any potential speedup.
 
> > Index: tests/recompilation/test_functions
> 
> That file wasn't mentioned in the log message.
> 
> > Index: tests/warnings/Mmakefile
> > -%.res_compile:	%.exp $(cs_subdir)%.c
> > +%.res_compile:	%.exp %.c
> 
> That change isn't mentioned in the log message.
> It also looks a bit suspicious to me -- the $(cs_subdir)
> reference is needed for the --use-subdirs case, isn't it?

The $(cs_subdir) part doesn't work with `mmc --make',
and isn't necessary any more (modules.m generates a rule
`module.c: $(cs_subdir)module.c').

Below are full diffs for the modified files, the interdiffs
weren't very useful.

Simon.


Estimated hours taken: 30
Branches: main

Improve the test framework to make it easier to find out which tests
failed and to reduce disk usage (important in debug grades).

Allow the tests to be run with `mmc --make' (still some failures).

Allow the user to run only the failing tests from a previous
run by using `mmake ERROR_FILE=runtests.errs', where runtests.errs
is the log file from the previous run.

tests/Mmake.common:
tests/*/Mmakefile:
	Move common code (such as the code to deal with subdirectories)
	to Mmake.common.

	Run the tests using `mmake runtests' rather than using slightly
	different runtests scripts in each directory.

	Add to the output from `mmake runtests' to make it easier to
	identify which tests failed in which grades.

	Move per-module options into Mercury.options files so they
	can be read by `mmc --make'.

	Remove the last of the NU-Prolog support.

	Consistently use the main module name when listing tests.
	Some directories (e.g. invalid) were using the source file
	name.

tests/process_log.awk:
	Collect the parts of the output relating to failing tests.

tests/generate_exp:
tests/handle_options:
tests/subdir_runtests:
tests/startup:
tests/shutdown:
tests/*/runtests:
tests/recompilation/TESTS:
	Removed.

tests/recompilation/test_functions:
	Make sure the old result file is removed before starting
	each test.

	Put the mmake output for tests which are supposed to fail
	into a different file for each test.

tests/warnings/Mmakefile:
	Use %.c rather than $(cs_subdir)%.c in a rule.
	The $(cs_subdir) part doesn't work with `mmc --make',
	and isn't necessary any more (modules.m generates a rule
	`module.c: $(cs_subdir)module.c').

tests/README:
	Updated.

tools/bootcheck:
tools/test_mercury:
	Use `mmake runtests' instead of the `runtests' script.

	Add a `-f' (`--failing-tests') option to bootcheck which
	runs only the failing tests from the last run.

tools/test_mercury:
tools/run_all_tests_from_cron:
	Use the new framework to summarize test failures.


Index: tests/README
===================================================================
RCS file: /home/mercury1/repository/tests/README,v
retrieving revision 1.6
diff -u -u -r1.6 README
--- tests/README	27 Jun 2001 05:04:42 -0000	1.6
+++ tests/README	15 Aug 2002 13:38:44 -0000
@@ -1,17 +1,23 @@
 The Mercury test suite is (dis)organized into a directory hierarchy.
-Each subdirectory has a script named "runtests" that runs the tests
-in that subdirectory; these indicate the presence of any errors by
-returning a nonzero exit status. The runtests script in each directory
-invokes the runtests scripts in all subdirectories, and propagates
-any nonzero exit status upwards.
 
-Both the bootcheck script and the nightly script use the main runtests
-script to run all the tests.
+`mmake' will run the tests in the directory it is
+invoked in and all subdirectories of that subdirectory.
+Use `mmake runtests_local' to just run the local tests without
+the subdirectories.
+
+To run just the tests that failed in a previous test run,
+use `mmake ERROR_FILE=FILE', where FILE is a copy of the
+runtests.errs file from the previous run.
+
+Both the bootcheck script and the nightly script use `mmake'
+to run all the tests.
+
+Unless otherwise stated, the tests in each directory compile
+the test programs and compare their output against hand-coded
+`.exp' (or `.exp2', `.exp3', etc) files.
 
 benchmarks
 	This directory contains Mercury versions of the benchmarks.
-	These tests work by comparing the output of the Mercury and
-	the NU-Prolog versions.
 
 debugger
 	This directory is for testing mdb, the Mercury debugger.
@@ -24,22 +30,27 @@
 	features of mdb.
 
 general
-	This directory is for general test cases.
+hard_coded
+	These directories are for general test cases.
+
 	(It might be a good idea to split this into tests
 	of particular features and regression tests that check
 	for old bugs.  But for the moment, just about everything
 	goes in here.)
-	These tests work by comparing the output of the Mercury and
-	the NU-Prolog versions.
 
-hard_coded
-	This directory is for tests of features that don't work in
-	NU-Prolog.  The expected output of the program has to be
-	hard-coded in a hand-written `.exp' file, rather having the
-	`.exp' file be generated automatically using NU-Prolog.
+	The historical reason for the separate `general' and `hard_coded'
+	directories was that the tests in `general' worked with NU-Prolog
+	and compared the Mercury output with the NU-Prolog output,
+	but the tests in `hard_coded' didn't work with NU-Prolog, so
+	their expected output needed to be hard-coded. We no longer
+	support compilation with NU-Prolog, so everything goes
+	in hard_coded now.
 
 recompilation
 	This directory contains tests of the smart recompilation system.
+	As well as checking for the correct output from the test
+	programs these tests also examine the `.err' files to
+	make sure that all necessary recompilations are performed.
 
 valid
 	This directory is for test cases that are not complete
@@ -47,12 +58,12 @@
 
 invalid
 	This directory is for test cases that are invalid
-	programs. We check that the files do *not* compile.
+	programs. We check that the files do *not* compile,
+	and check that the errors match those in the hand-written
+	`.err_exp' file.
 
 warnings
 	This directory is for tests of compiler warnings. These work by
 	comparing the warnings emitted by the compiler with those given
 	in the hand-written `.exp' file. 
 
-To regenerate the expected output files for the benchmark and general
-directories, execute the "generate_exp" script in this directory.
Index: tests/Mmake.common
===================================================================
RCS file: /home/mercury1/repository/tests/Mmake.common,v
retrieving revision 1.24
diff -u -u -r1.24 Mmake.common
--- tests/Mmake.common	29 Apr 2002 08:22:05 -0000	1.24
+++ tests/Mmake.common	16 Aug 2002 04:22:15 -0000
@@ -1,4 +1,28 @@
 #-----------------------------------------------------------------------------#
+# Mmake.common - shared Mmake variables and rules for the test directories.
+#
+# The including Mmakefile must set the following variables:
+# THIS_DIR - the name of the directory containing tests
+#		(e.g. benchmarks, hard_coded/typeclasses)
+# TESTS_DIR - the directory containing this file.
+# SUBDIRS - the names of any subdirectories containing tests.
+# PROGS - the names of the main modules of the test programs.
+# TESTS- the names of the tests. For each test there must be targets
+#	test.depend, test.runtest and test.realclean, unless the test ends
+#	in `-nodepend', in which case the `-nodepend' suffix will be stripped
+#	off and the test.depend target is not required.
+#
+# If the variable ERROR_FILE is set, only the tests which failed in
+# the test run which produced the specified error log file will be run.
+#
+#-----------------------------------------------------------------------------#
+
+MAIN_TARGET = runtests
+
+-include $(TESTS_DIR)/Mmake.params
+
+# Avoid trying to make this file with `mmc --make' if it doesn't exist.
+$(TESTS_DIR)/Mmake.params: ;
 
 # Set up to test a particular workspace.
 ifdef WORKSPACE
@@ -15,10 +39,6 @@
 # file by setting both GRADEFLAGS-foo.
 #
 
-# Override this with `mmake HAVE_NUPROLOG=yes'
-# if you want to rebuild the `.exp' files.
-HAVE_NUPROLOG=no
-
 DIFF_OPTS=-c
 
 # Override this with `RUN_RECOMPILATION_TESTS=no'
@@ -31,7 +51,7 @@
 
 #-----------------------------------------------------------------------------#
 
-# .PRECIOUS: %.mod %.c %.o %_init.c %.no %.nu %_init.nl %_init.no
+# .PRECIOUS: %.c %.o %_init.c
 
 %_init.c: Entry
 
@@ -82,44 +102,214 @@
 		echo "** $*.out did not match the expected output"; \
 		echo "** (closest match was $$shortest)"; \
 		cp $$shortest $@; \
+		cat $@; \
 		exit 1; }
 
+# Some tests have more than one possible valid output, so
+# we allow the test to pass if it matches any of the .err_exp* files.
+%.err_res: %.err %.err_exp
+	@-rm -f $@
+	@diff $(DIFF_OPTS) $*.err_exp $*.err > $@ || \
+		{ [ -f $*.err_exp2 ] && \
+		  diff $(DIFF_OPTS) $*.err_exp2 $*.err > $@; } || \
+		{ [ -f $*.err_exp3 ] && \
+		  diff $(DIFF_OPTS) $*.err_exp3 $*.err > $@; } || \
+		{ [ -f $*.err_exp4 ] && \
+		  diff $(DIFF_OPTS) $*.err_exp4 $*.err > $@; } || \
+		{ cat $*.err && exit 1; }
+
 #-----------------------------------------------------------------------------#
 
-clean_local: clean_out clean_res
+MERCURY_MAIN_MODULES = $(PROGS)
+
+ERROR_OUTPUT_FILE = runtests.errs
+
+realclean_local: clean_errors
+clean_local: clean_out clean_res clean_zip
 
 clean_mc: clean_c clean_o clean_out clean_res
 
 clean_out:
 	rm -f *.out
 
-clean_exp:
-	rm -f *.exp
-
 clean_res:
 	rm -f *.res*
 
+clean_errors:
+	rm -f $(ERROR_OUTPUT_FILE)
+
+# Remove gzipped executables for tests which have failed.
+clean_zip:
+	for test in $(TESTS); do \
+		rm -f `basename $$test -nodepend`.gz; \
+	done
+
 #
-# The `foo' targets make `foo_local' in the current directory before
-# recursively making `foo' in all subdirectories.  The recursive part
-# is handled in individual Mmakefiles.
+# The `foo' targets make recursively make `foo' in all subdirectories.
+# before making `foo_local' in the current directory 
 # 
 
-.PHONY: check_local dep_local depend_local all_local
+.PHONY: runtests runtests_local runtests_subdirs
+.PHONY: check check_local check_subdirs
+.PHONY: realclean_subdirs clean_subdirs
 
-.PHONY: check_subdirs dep_subdirs depend_subdirs realclean_subdirs \
-        clean_subdirs all_subdirs
-
-check:          check_local check_subdirs
-dep:            dep_local dep_subdirs
-depend:         depend_local depend_subdirs
 realclean:      realclean_subdirs
 clean:          clean_subdirs
-all:            all_local all_subdirs
+all:		runtests
+
+check:		runtests
+check_local:	runtests_local
+check_subdirs:	runtests_subdirs
+
+#
+# We want to run all the tests, even if some fail, so
+# run the tests using `mmake -k'.
+# 
+runtests:
+	+ at if mmake -k runtests_local runtests_subdirs; then \
+		echo ALL TESTS SUCCEEDED; \
+		rm -f $(ERROR_OUTPUT_FILE); \
+	else \
+		for subdir in $(SUBDIRS) x; do \
+		    if [ -f $$subdir/$(ERROR_OUTPUT_FILE) ]; then \
+			cat $$subdir/$(ERROR_OUTPUT_FILE) \
+				>> $(ERROR_OUTPUT_FILE); \
+			rm -f $$subdir/$(ERROR_OUTPUT_FILE); \
+		    fi; \
+		done; \
+		echo SOME TESTS FAILED: see $(ERROR_OUTPUT_FILE); \
+		exit 1; \
+	fi
+
+#
+# If the variable ERROR_FILE is set, only the tests which failed in
+# the test run which produced the specified error log file will be run.
+#
+ifdef ERROR_FILE
+ifndef TESTS_TO_RUN
+TESTS_TO_RUN := $(shell awk '/^FAILED TEST/ { print $$3 }' $(ERROR_FILE))
+export TESTS_TO_RUN
+endif
+endif
+
+ifdef TESTS_TO_RUN
+THIS_DIR_TESTS_TO_RUN := $(patsubst $(THIS_DIR)/%,%,\
+		$(filter $(THIS_DIR)/%,$(TESTS_TO_RUN)))
+else
+THIS_DIR_TESTS_TO_RUN := $(TESTS)
+endif
+
+ifeq ($(THIS_DIR_TESTS_TO_RUN),)
+
+runtests_local: ;
+
+else
+
+# Run multiple rm commands at once to speed up
+# `mmake realclean' on slow NFS filesystems.
+RM_JFACTOR=-j10
+
+#
+# Clean up after any previous test run.
+#
+.PHONY: start_runtests_local
+start_runtests_local:
+	@echo STARTING tests in $(THIS_DIR) in grade $(GRADE)
+	@echo cleaning up the directory before the tests
+	+ at if ls -lt | head -2 | egrep CLEAN > /dev/null 2>&1; then \
+		rm -f CLEAN > /dev/null 2>&1; \
+	else \
+		rm -f CLEAN > /dev/null 2>&1; \
+		mmake $(RM_JFACTOR) realclean_local > /dev/null 2>&1; \
+		rm -f *.d *.dep *.int *.int2 *.int3 > /dev/null 2>&1; \
+		rm -f *.date *.date3 *.opt *.optdate > /dev/null 2>&1; \
+		rm -f *.trans_opt *.trans_opt_date > /dev/null 2>&1; \
+	fi
+
+#
+# Run a single test, cleaning up if it succeeds, or producing
+# a log file and gzipping the executable the test fails.
+# Note that the %.log target does not fail if the test fails --
+# we really want `--keep-going' to be the default for this Mmakefile.
+#
+.PHONY: %.log
+%.log: start_runtests_local
+	+ at test=$(*:%-nodepend=%); \
+	echo RUNNING TEST $(THIS_DIR)/$* in grade $(GRADE) | tee $@; \
+	case $* in \
+		*-nodepend) \
+			rm -f $*.failed;; \
+		*) \
+			rm -f $*.failed; \
+			{ mmake $$test.depend 2>&1 || touch $*.failed; } \
+				| tee -a $@ ;; \
+	esac; \
+	if [ -f $*.failed ]; then \
+		echo FAILED TEST $(THIS_DIR)/$* in grade $(GRADE) \
+			| tee -a $@; \
+	else \
+		{ mmake $$test.runtest 2>&1 || touch $*.failed; } \
+			| tee -a $@; \
+		if [ -f $*.failed ]; then \
+			rm -f $*.failed; \
+		    	if [ -f $$test ]; then \
+				rm -f $$test.gz; gzip $$test; \
+			fi; \
+			echo FAILED TEST $(THIS_DIR)/$* in grade $(GRADE) \
+				| tee -a $@; \
+		else \
+			rm -f $@; \
+			rm -f $$test.out* $$test.*res*; \
+			mmake $$test.realclean > /dev/null 2>&1; \
+			echo FINISHED TEST $(THIS_DIR)/$*; \
+		fi; \
+	fi
+
+TEST_LOGS = $(THIS_DIR_TESTS_TO_RUN:%=%.log)
+runtests_local: $(TEST_LOGS)
+	@touch DUMMY.log
+	@cat *.log > $(ERROR_OUTPUT_FILE)
+	@rm -f *.log
+	@if [ -s $(ERROR_OUTPUT_FILE) ]; then \
+		echo "FAILED tests in $(THIS_DIR) in grade $(GRADE)"; \
+		exit 1; \
+	else \
+		echo "FINISHED tests in $(THIS_DIR) in grade $(GRADE)"; \
+		echo cleaning up the directory after the tests; \
+		mmake $(RM_JFACTOR) realclean_local > /dev/null 2>&1; \
+		rm core > /dev/null 2>&1; \
+		touch CLEAN; \
+	fi
+
+endif	# THIS_DIR_TESTS_TO_RUN != ""
+
+ifeq ($(SUBDIRS),)
+
+runtests_subdirs realclean_subdirs clean_subdirs : ;
+
+else
+
+RUNTESTS_IN_SUBDIRS=$(SUBDIRS:%=runtests_in_%)
+runtests_subdirs: $(RUNTESTS_IN_SUBDIRS)
+
+.PHONY: $(RUNTESTS_IN_SUBDIRS)
+$(RUNTESTS_IN_SUBDIRS): runtests_in_%:
+	+cd $* && mmake runtests
+
+realclean_subdirs:
+	+succeeded=true; \
+	for dir in $(SUBDIRS); do \
+	    (cd $$dir && mmake realclean) || succeeded=false; \
+	done
+	case $$succeeded in false) exit 1 ;; esac
+
+clean_subdirs:
+	+succeeded=true; \
+	for dir in $(SUBDIRS); do \
+		(cd $$dir && mmake clean) || succeeded=false; \
+	done; \
+	case $$succeeded in false) exit 1 ;; esac
 
-SUBDIR_MMAKE = mmake \
-                GRADE='$(GRADE)' \
-                EXTRA_CFLAGS='$(EXTRA_CFLAGS)' \
-                EXTRA_MCFLAGS='$(EXTRA_MCFLAGS)'
+endif	# SUBDIRS != ""
 
 #-----------------------------------------------------------------------------#
Index: tools/bootcheck
===================================================================
RCS file: /home/mercury1/repository/mercury/tools/bootcheck,v
retrieving revision 1.133
diff -u -u -r1.133 bootcheck
--- tools/bootcheck	9 Aug 2002 07:00:47 -0000	1.133
+++ tools/bootcheck	16 Aug 2002 07:48:53 -0000
@@ -16,6 +16,9 @@
 		as a whole is disabled).
 	-e, --extras
 		Test the programs in the extras directory.
+	-f, --failing-tests
+		Run only the tests which failed on the last run. This
+		requires the tests/runtests.errs file from the previous run.
 	-h, --help
 		Display this usage message.
 	-j <num-jobs>, --jobs <num-jobs>
@@ -115,6 +118,7 @@
 
 unset WORKSPACE
 testdirs=""
+failing_tests_only=false
 extras=false
 jfactor=""
 keep_objs=false
@@ -185,6 +189,9 @@
 	-e|--extras)
 		extras=true ;;
 
+	-f|--failing-tests)
+		failing_tests_only=true ;;
+
 	-h|--help)
 		echo "$usage"
 		exit 0 ;;
@@ -967,7 +974,7 @@
 
 	if test "$test_grade" != ""
 	then
-		test_grade_opt="-g $test_grade"
+		test_grade_opt="GRADE=$test_grade"
 	else
 		test_grade_opt=""
 	fi
@@ -995,18 +1002,46 @@
 		fi
 
 		cp $root/doc/mdb_command_test.inp ${tests_prefix}tests/debugger
+
+		case $failing_tests_only in
+			true)
+				if [ ! -f ${tests_prefix}tests/runtests.errs ]
+				then
+					echo \
+		"bootcheck: \`--failing-tests' specified but" 1>&2
+					echo \
+		"\`${tests_prefix}tests/runtests.errs' does not exist." 1>&2
+					exit 1
+				fi
+
+				mv ${tests_prefix}tests/runtests.errs \
+					${tests_prefix}tests/runtests.$$
+				test_log_opt=\
+			"ERROR_FILE=${tests_prefix}tests/runtests.$$"
+				;;
+			false)
+				test_log_opt=""
+				;;
+		esac
+			
 		if test "$testdirs" = ""
 		then
 			cd ${tests_prefix}tests
-			./runtests $target_opt $jfactor $test_grade_opt
+			mmake $target_opt $jfactor $test_grade_opt \
+				$test_log_opt runtests
 		else
 			for testdir in $testdirs
 			do
 				cd $root/${tests_prefix}tests/$testdir
-				./runtests $target_opt $jfactor $test_grade_opt
+				mmake $target_opt $jfactor $test_grade_opt \
+					$test_log_opt runtests
 			done
 		fi
 		test_status=$?
+
+		case $failing_tests_only in
+			true)	rm -f ${tests_prefix}tests/runtests.$$ ;;
+		esac
 
 		if test "$type_stats" != ""
 		then
Index: tests/recompilation//Mmakefile
===================================================================
RCS file: /home/mercury1/repository/tests/recompilation/Mmakefile,v
retrieving revision 1.8
diff -u -u -r1.8 Mmakefile
--- tests/recompilation//Mmakefile	23 Feb 2002 07:30:58 -0000	1.8
+++ tests/recompilation//Mmakefile	16 Aug 2002 03:36:37 -0000
@@ -1,37 +1,98 @@
 #-----------------------------------------------------------------------------#
 
-main_target: check
+THIS_DIR = recompilation
 
-include ../Mmake.common
--include ../Mmake.params
+TESTS_SHOULD_SUCCEED = \
+	add_constructor_nr \
+	add_constructor_r \
+	add_instance_r \
+	add_instance_2_r \
+	add_type_nr \
+	change_class_r \
+	change_func_r \
+	change_instance_r \
+	change_mode_r \
+	change_type_nr \
+	combined_type_mode_nr \
+	field_r \
+	func_overloading_nr \
+	func_overloading_r \
+	lambda_mode_r \
+	no_version_numbers_r \
+	pragma_type_spec_r \
+	pred_ctor_ambiguity_r \
+	pred_overloading_r \
+	typeclass_method_pragma_r \
+	type_spec_rename_var_r \
+	type_spec_unname_var_r \
+	unchanged_pred_nr \
+	unchanged_with_type_nr
+
+# Parallel mmake with nested sub-modules is broken.
+# The commands to create `.c' files from the `.m' file containing the
+# nested sub-module are run twice, resulting in incorrect output in
+# the `.err' file.
+NO_PARALLEL_MAKE_TESTS = \
+	nested_module_r \
+	nested_module_2_r
+
+TESTS_SHOULD_FAIL = \
+	add_type_re \
+	remove_type_re \
+	type_qual_re \
+	with_type_re
+
+PROGS= $(TESTS_SHOULD_SUCCEED) $(NO_PARALLEL_MAKE_TESTS) $(TESTS_SHOULD_FAIL)
+TESTS= $(PROGS:%=%-nodepend)
+SUBDIRS=
+TESTS_DIR=..
+include $(TESTS_DIR)/Mmake.common
+
+# Module-specific options should go in Mercury.options so they
+# can be found by `mmc --make'.
+include Mercury.options
+
+$(TESTS_SHOULD_SUCCEED:%=%.runtest): %.runtest:
+	+ at if ./two_module_test false $* $*_2; then \
+		:; \
+	else \
+		cat $*.res; \
+		exit 1; \
+	fi
+
+$(NO_PARALLEL_MAKE_TESTS:%=%.runtest): %.runtest:
+	+ at if mmakeopts=-j1 ./two_module_test false $* $*_2; then \
+		:; \
+	else \
+		cat $*.res; \
+		exit 1; \
+	fi
+
+$(TESTS_SHOULD_FAIL:%=%.runtest): %.runtest:
+	+ at if ./two_module_test true $* $*_2; then \
+		rm $*.failing_make_output; \
+	else \
+		cat $*.res; \
+		exit 1; \
+	fi
 
 #-----------------------------------------------------------------------------#
-#-----------------------------------------------------------------------------#
 
 clean_local:
 
 realclean_local:
-	. ./TESTS; \
-	for module in $$ALL_TESTS; do \
+	for module in $$PROGS; do \
 		rm -rf $$module.m $${module}_2.m ;\
 	done
 
 should_run_recompilation_tests:
 	@echo $(RUN_RECOMPILATION_TESTS)
 
-	# typeclass_method_pragma_r calls a predicate with a
-	# `:- pragma obsolete' declaration.
-MCFLAGS-typeclass_method_pragma_r = --no-halt-at-warn
-
 # Smart recompilation doesn't yet work with --intermodule-optimization.
 # The `override' is needed because otherwise make ignores the assignment if
 # EXTRA_MCFLAGS is set on the command line, as it is for the nightly tests.
 override EXTRA_MCFLAGS += --no-intermodule-optimization --smart-recompilation \
 			--find-all-recompilation-reasons
-
-# This module tests recompilation of a module which depends on a module
-# for which no version numbers have been computed.
-MCFLAGS-no_version_numbers_r_2 = --no-smart-recompilation
 
 $(dates_subdir)nested_module_2_r_2.date: \
 				$(int0s_subdir)nested_module_2_r_2.int0
Index: tests/recompilation//test_functions
===================================================================
RCS file: /home/mercury1/repository/tests/recompilation/test_functions,v
retrieving revision 1.5
diff -u -u -r1.5 test_functions
--- tests/recompilation//test_functions	31 Jul 2001 16:59:48 -0000	1.5
+++ tests/recompilation//test_functions	16 Aug 2002 06:58:11 -0000
@@ -27,6 +27,9 @@
 		chmod -w $module.m
 	done
 
+	rm -f $main_module.res
+	touch $main_module.res
+
 	# XXX Avoid producing output files with the same timestamp
 	# as the input source file. The up-to-date check for the output
 	# file in recompilation_check.m checks that the output file's
@@ -134,7 +137,8 @@
 		# The `-k' option to mmake avoids differences in the output
 		# when using parallel mmakes.
 		#
-		eval mmake $mmakeopts -k $main_module > $failing_make_output 2>&1
+		eval mmake $mmakeopts -k \
+			$main_module > $main_module.failing_make_output 2>&1
 		;;
 	    false)
 		eval mmake $mmakeopts $main_module
--------------------------------------------------------------------------
mercury-reviews mailing list
post:  mercury-reviews at cs.mu.oz.au
administrative address: owner-mercury-reviews at cs.mu.oz.au
unsubscribe: Address: mercury-reviews-request at cs.mu.oz.au Message: unsubscribe
subscribe:   Address: mercury-reviews-request at cs.mu.oz.au Message: subscribe
--------------------------------------------------------------------------