[m-rev.] for review: improve test framework

[Fergus Henderson]

About a dozen tests failed on murlibobo, due to problems with parallel
make.  The code causing the problem is this:

On 16-Aug-2002, Simon Taylor <stayl at cs.mu.OZ.AU> wrote:
> +#
> +# Run a single test, cleaning up if it succeeds, or producing
> +# a log file and gzipping the executable the test fails.
> +# Note that the %.log target does not fail if the test fails --
> +# we really want `--keep-going' to be the default for this Mmakefile.
> +#
> +.PHONY: %.log
> +%.log: start_runtests_local
> +	+ at test=$(*:%-nodepend=%); \
> +	echo RUNNING TEST $(THIS_DIR)/$* in grade $(GRADE) | tee $@; \
> +	case $* in \
> +		*-nodepend) \
> +			rm -f $*.failed;; \
> +		*) \
> +			rm -f $*.failed; \
> +			{ mmake $$test.depend 2>&1 || touch $*.failed; } \
> +				| tee -a $@ ;; \
> +	esac; \
> +	if [ -f $*.failed ]; then \
> +		echo FAILED TEST $(THIS_DIR)/$* in grade $(GRADE) \
> +			| tee -a $@; \
> +	else \
> +		{ mmake $$test.runtest 2>&1 || touch $*.failed; } \
> +			| tee -a $@; \
> +		if [ -f $*.failed ]; then \
> +			rm -f $*.failed; \
> +		    	if [ -f $$test ]; then \
> +				rm -f $$test.gz; gzip $$test; \
> +			fi; \
> +			echo FAILED TEST $(THIS_DIR)/$* in grade $(GRADE) \
> +				| tee -a $@; \
> +		else \
> +			rm -f $@; \
> +			rm -f $$test.out* $$test.*res*; \
> +			mmake $$test.realclean > /dev/null 2>&1; \
> +			echo FINISHED TEST $(THIS_DIR)/$*; \
> +		fi; \
> +	fi

Using recursive mmake invocations here breaks parallel make,
because in cases where several different test cases depend on the
the same target, the different recursive mmake invocations may
both try to make that target at the same time.

For example, you can have a situation where there are two tests,
e.g. `aditi_test_1' and `aditi_test_2', which both depend on
the same target `aditi.int'.  Because you are invoking mmake
recursively, the different mmake invocations won't communicate
with each other about which targets have been made, and instead
both recursive invocations may try to make `aditi.int' at the same
time.

Also, with parallel makes, it's not OK to do `mmake foo.realclean'
until all of the tests are complete, because different tests may share
files, and so doing `mmake foo.realclean' might remove files needed
by another test `bar' while the test of `bar' is currently in progress.

-- 
Fergus Henderson <fjh at cs.mu.oz.au>  |  "I have always known that the pursuit
The University of Melbourne         |  of excellence is a lethal habit"
WWW: <http://www.cs.mu.oz.au/~fjh>  |     -- the last words of T. S. Garp.
--------------------------------------------------------------------------
mercury-reviews mailing list
post:  mercury-reviews at cs.mu.oz.au
administrative address: owner-mercury-reviews at cs.mu.oz.au
unsubscribe: Address: mercury-reviews-request at cs.mu.oz.au Message: unsubscribe
subscribe:   Address: mercury-reviews-request at cs.mu.oz.au Message: subscribe
--------------------------------------------------------------------------