[m-dev.] for review: uniform interface to tests (take 2)

[Mark Anthony BROWN]

Estimated hours taken: 9

Provide a more uniform interface to the test directories.  Before this
change, the tests directory as well as direct descendents had `runtests'
scripts to execute the tests.  These would run `mmake', which in turn
would recursively run `mmake' in any subdirectories.  The subdirectories
did not have their own `runtests' scripts.

This change adds a `runtests' script to those subdirectories that did
not have them before.  The scripts have the same meaning as they did
previously---run all tests in the current directory and below, and
return a status of 0 only if all tests pass.

The rationale for this change is so that each directory in the tests
is treated the same, regardless of whether it is a top-level
directory or not.  This means, for example, that any test directory can
be used as an argument to the `--test-dir' option to tools/bootcheck.

This change also links tests/general/accumulator into the automated
testing suite.

tests/README:
	Update comments.

tests/Mmake.common:
	Introduce new target dependencies.  Targets of the form
	`foo' depend on the targets `foo_local' and `foo_subdirs'.

tests/subdir_runtests:
	New script to recursively call runtests scripts in subdirectories.
	Most of the code for this script originally came from
	tests/runtests.

tests/shutdown:
tests/startup:
	Use `realclean_local' as a target instead of `realclean', since
	the subdirectories will be handled by other runtests scripts.

tests/runtests:
tests/debugger/runtests:
tests/general/runtests:
tests/hard_coded/runtests:
	Use the new script to run tests in the subdirectories, rather
	than doing a recursive mmake.  Make sure that non-zero return
	values are propagated upwards where appropriate.

tests/debugger/declarative/runtests:
tests/hard_coded/typeclasses/runtests:
tests/hard_coded/sub-modules/runtests:
	New scripts to handle running tests in the subdirectories.

tests/debugger/Mmakefile:
tests/general/Mmakefile:
tests/hard_coded/Mmakefile:
	Add `*_subdirs' targets that do a recursive Mmake.  Change the
	existing targets to `*_local' ones.

tests/warnings/Mmakefile:
	Manually add the extra dependencies, since this Mmakefile does
	not include tests/Mmake.common.

Index: tests/Mmake.common
===================================================================
RCS file: /home/mercury1/repository/tests/Mmake.common,v
retrieving revision 1.13
diff -u -r1.13 Mmake.common
--- Mmake.common	1998/11/04 07:53:21	1.13
+++ Mmake.common	1999/07/09 05:20:48
@@ -48,4 +48,27 @@
 clean_res:
 	rm -f *.res
 
+#
+# The `foo' targets make `foo_local' in the current directory before
+# recursively making `foo' in all subdirectories.  The recursive part
+# is handled in individual Mmakefiles.
+# 
+
+.PHONY: check_local dep_local depend_local all_local
+
+.PHONY: check_subdirs dep_subdirs depend_subdirs realclean_subdirs \
+        clean_subdirs all_subdirs
+
+check:          check_local check_subdirs
+dep:            dep_local dep_subdirs
+depend:         depend_local depend_subdirs
+realclean:      realclean_subdirs
+clean:          clean_subdirs
+all:            all_local all_subdirs
+
+SUBDIR_MMAKE = mmake \
+                GRADE='$(GRADE)' \
+                EXTRA_CFLAGS='$(EXTRA_CFLAGS)' \
+                EXTRA_MCFLAGS='$(EXTRA_MCFLAGS)'
+
 #-----------------------------------------------------------------------------#
Index: tests/README
===================================================================
RCS file: /home/mercury1/repository/tests/README,v
retrieving revision 1.4
diff -u -r1.4 README
--- README	1997/07/19 14:41:26	1.4
+++ README	1999/07/09 06:42:09
@@ -1,8 +1,9 @@
-The Mercury test suite is (dis)organized into several subdirectories.
+The Mercury test suite is (dis)organized into a directory hierarchy.
 Each subdirectory has a script named "runtests" that runs the tests
 in that subdirectory; these indicate the presence of any errors by
-returning a nonzero exit status. The runtests script in this directory
-invokes the runtests scripts in all the subdirectories.
+returning a nonzero exit status. The runtests script in each directory
+invokes the runtests scripts in all subdirectories, and propagates
+any nonzero exit status upwards.
 
 Both the bootcheck script and the nightly script use the main runtests
 script to run all the tests.
@@ -11,6 +12,16 @@
 	This directory contains Mercury versions of the benchmarks.
 	These tests work by comparing the output of the Mercury and
 	the NU-Prolog versions.
+
+debugger
+	This directory is for testing mdb, the Mercury debugger.
+	Programs are compiled with deep tracing turned on.  The
+	tests in this directory are not performed if the base grade
+	is `jump' or `fast'.
+
+debugger/declarative
+	This directory is for testing the declarative debugging
+	features of mdb.
 
 general
 	This directory is for general test cases.
Index: tests/runtests
===================================================================
RCS file: /home/mercury1/repository/tests/runtests,v
retrieving revision 1.4
diff -u -r1.4 runtests
--- runtests	1998/04/23 09:27:02	1.4
+++ runtests	1999/07/09 05:22:47
@@ -1,33 +1,18 @@
 #!/bin/sh
-# Run the tests in each of the subdirectories.
 
+# Process command line options.
 . ./handle_options
 
-failures=""
-for dir in *
-do
-	if test -d $dir -a -x $dir/runtests
-	then
-		cd $dir
-		# we need to use `eval' here to get the quoting right in
-		# the case when $fflag or $cflag contains embedded spaces
-		if eval ./runtests $jfactor $gflag $fflag $cflag
-		then
-			true
-		else
-			failures="$failures $dir"
-		fi
-		cd ..
-	fi
-done
+# Run the tests in each of the subdirectories.
+. ./subdir_runtests
 
-if test "$failures" = ""
+if test "$subdir_failures" = ""
 then
 	echo "all tests have succeeded"
 	echo "gradeopt=$gflag, flagsopt=$fflag, cflagsopt=$cflag"
 	exit 0
 else
-	echo "some tests have failed in: $failures"
+	echo "some tests have failed in: $subdir_failures"
 	echo "gradeopt=$gflag, flagsopt=$fflag, cflagsopt=$cflag"
 	exit 1
 fi
Index: tests/shutdown
===================================================================
RCS file: /home/mercury1/repository/tests/shutdown,v
retrieving revision 1.1
diff -u -r1.1 shutdown
--- shutdown	1998/05/13 04:05:14	1.1
+++ shutdown	1999/07/09 06:16:33
@@ -1,4 +1,4 @@
 echo cleaning up the directory after the tests
-mmake $gradeopt $jfactor realclean > /dev/null 2>&1
+mmake $gradeopt $jfactor realclean_local > /dev/null 2>&1
 rm core > /dev/null 2>&1
 touch CLEAN
Index: tests/startup
===================================================================
RCS file: /home/mercury1/repository/tests/startup,v
retrieving revision 1.1
diff -u -r1.1 startup
--- startup	1998/05/13 04:05:15	1.1
+++ startup	1999/07/09 06:16:17
@@ -4,5 +4,5 @@
 	rm -f CLEAN > /dev/null 2>&1
 else
 	rm -f CLEAN > /dev/null 2>&1
-	mmake $gradeopt $jfactor realclean > /dev/null 2>&1
+	mmake $gradeopt $jfactor realclean_local > /dev/null 2>&1
 fi
Index: tests/debugger/Mmakefile
===================================================================
RCS file: /home/mercury1/repository/tests/debugger/Mmakefile,v
retrieving revision 1.25
diff -u -r1.25 Mmakefile
--- Mmakefile	1999/06/02 07:28:30	1.25
+++ Mmakefile	1999/07/09 05:28:25
@@ -134,29 +134,44 @@
 OUTS=		$(PROGS:%=%.out)
 RESS=		$(PROGS:%=%.res)
 
-#-----------------------------------------------------------------------------#
-
-SUBDIR_MMAKE = mmake $(MMAKEFLAGS) \
-			GRADE='$(GRADE)' \
-			EXTRA_CFLAGS='$(EXTRA_CFLAGS)' \
-			EXTRA_MCFLAGS='$(EXTRA_MCFLAGS)'
-
-dep:		$(DEPS)
-		cd declarative && $(SUBDIR_MMAKE) dep || exit 1;
-
-depend:		$(DEPENDS)
-		cd declarative && $(SUBDIR_MMAKE) depend || exit 1;
+dep_local:	$(DEPS)
+depend_local:	$(DEPENDS)
+check_local:	$(OUTS) $(RESS)
+all_local:	$(PROGS)
 
-check:		$(OUTS) $(RESS)
-		cd declarative && $(SUBDIR_MMAKE) check || exit 1;
-
-all:		$(PROGS)
-		cd declarative && $(SUBDIR_MMAKE) all || exit 1;
+#-----------------------------------------------------------------------------#
 
-clean:
-		cd declarative && $(SUBDIR_MMAKE) clean || exit 1;
+SUBDIRS = declarative
 
-realclean:
-		cd declarative && $(SUBDIR_MMAKE) realclean || exit 1;
+check_subdirs:
+	for dir in $(SUBDIRS); do \
+		(cd $$dir && $(SUBDIR_MMAKE) check) || exit 1; \
+	done
+
+dep_subdirs:
+	for dir in $(SUBDIRS); do \
+		(cd $$dir && $(SUBDIR_MMAKE) dep) || exit 1; \
+	done
+
+depend_subdirs:
+	for dir in $(SUBDIRS); do \
+		(cd $$dir && $(SUBDIR_MMAKE) depend) || exit 1; \
+	done
+
+realclean_subdirs:
+	for dir in $(SUBDIRS); do \
+		(cd $$dir && $(SUBDIR_MMAKE) realclean) || exit 1; \
+	done
+
+clean_subdirs:
+	for dir in $(SUBDIRS); do \
+		(cd $$dir && $(SUBDIR_MMAKE) clean) || exit 1; \
+	done
+
+all_subdirs:
+	for dir in $(SUBDIRS); do \
+		(cd $$dir && $(SUBDIR_MMAKE) all) || exit 1; \
+	done
 
 #-----------------------------------------------------------------------------#
+
Index: tests/debugger/runtests
===================================================================
RCS file: /home/mercury1/repository/tests/debugger/runtests,v
retrieving revision 1.6
diff -u -r1.6 runtests
--- runtests	1998/12/16 17:28:24	1.6
+++ runtests	1999/07/09 06:14:19
@@ -4,10 +4,25 @@
 # Return a status of 0 (true) if everything is all right, and 1 otherwise.
 
 . ../handle_options
+
+# Run the tests in any subdirectories of this one.  If any of these
+# fail, we still perform the tests in this directory but return a
+# status of 1 regardless of the outcome.
+
+. ../subdir_runtests
+
+if test "$subdir_failures" = ""
+then
+	subdir_status=0
+else
+	subdir_status=1
+fi
+
+# Now run all the tests in this directory.
 . ../startup
 
-mmake $jfactor depend || exit 1
-eval mmake -k $jfactor $gradeopt $flagsopt $cflagsopt check
+mmake $jfactor depend_local || exit 1
+eval mmake -k $jfactor $gradeopt $flagsopt $cflagsopt check_local
 checkstatus=$?
 
 cat *.res > .allres
@@ -17,7 +32,7 @@
 	echo "gradeopt=$gradeopt, flagsopt=$flagsopt, cflagsopt=$cflagsopt"
 	rm -f .allres
 	. ../shutdown
-	exit 0
+	exit $subdir_status
 else
 	echo "the tests in the debugger directory failed"
 	echo "gradeopt=$gradeopt, flagsopt=$flagsopt, cflagsopt=$cflagsopt"
Index: tests/general/Mmakefile
===================================================================
RCS file: /home/mercury1/repository/tests/general/Mmakefile,v
retrieving revision 1.24
diff -u -r1.24 Mmakefile
--- Mmakefile	1998/10/27 15:36:40	1.24
+++ Mmakefile	1999/07/09 05:39:35
@@ -105,12 +105,44 @@
 EXPS=	$(PROGS:%=%.exp)
 RESS=	$(PROGS:%=%.res)
 
-depend: $(DEPENDS)
-
 exp:	$(EXPS)
+
+depend_local:	$(DEPENDS)
+check_local:	$(OUTS) $(RESS)
+all_local:	$(PROGS)
+
+#-----------------------------------------------------------------------------#
+
+SUBDIRS = accumulator
+
+check_subdirs:
+	for dir in $(SUBDIRS); do \
+		(cd $$dir && $(SUBDIR_MMAKE) check_subdirs) || exit 1; \
+	done
+
+dep_subdirs:
+	for dir in $(SUBDIRS); do \
+		(cd $$dir && $(SUBDIR_MMAKE) dep_subdirs) || exit 1; \
+	done
+
+depend_subdirs:
+	for dir in $(SUBDIRS); do \
+		(cd $$dir && $(SUBDIR_MMAKE) depend_subdirs) || exit 1; \
+	done
+
+realclean_subdirs:
+	for dir in $(SUBDIRS); do \
+		(cd $$dir && $(SUBDIR_MMAKE) realclean_subdirs) || exit 1; \
+	done
 
-check:	$(OUTS) $(RESS)
+clean_subdirs:
+	for dir in $(SUBDIRS); do \
+		(cd $$dir && $(SUBDIR_MMAKE) clean_subdirs) || exit 1; \
+	done
 
-all:	$(PROGS)
+all_subdirs:
+	for dir in $(SUBDIRS); do \
+		(cd $$dir && $(SUBDIR_MMAKE) all_subdirs) || exit 1; \
+	done
 
 #-----------------------------------------------------------------------------#
Index: tests/general/runtests
===================================================================
RCS file: /home/mercury1/repository/tests/general/runtests,v
retrieving revision 1.6
diff -u -r1.6 runtests
--- runtests	1998/05/13 04:05:54	1.6
+++ runtests	1999/07/09 06:15:39
@@ -13,10 +13,25 @@
 # on a machine that has NU-Prolog installed.
 
 . ../handle_options
+
+# Run the tests in any subdirectories of this one.  If any of these
+# fail, we still perform the tests in this directory but return a
+# status of 1 regardless of the outcome.
+
+. ../subdir_runtests
+
+if test "$subdir_failures" = ""
+then
+        subdir_status=0
+else
+        subdir_status=1
+fi
+
+# Now run all the tests in this directory.
 . ../startup
 
-mmake $jfactor depend || exit 1
-eval mmake -k $jfactor $gradeopt $flagsopt $cflagsopt check
+mmake $jfactor depend_local || exit 1
+eval mmake -k $jfactor $gradeopt $flagsopt $cflagsopt check_local
 checkstatus=$?
 
 cat *.res > .allres
@@ -26,7 +41,7 @@
 	echo "gradeopt=$gradeopt, flagsopt=$flagsopt, cflagsopt=$cflagsopt"
 	rm -f .allres
 	. ../shutdown
-	exit 0
+	exit $subdir_status
 else
 	echo "the tests in the general directory failed"
 	echo "gradeopt=$gradeopt, flagsopt=$flagsopt, cflagsopt=$cflagsopt"
Index: tests/hard_coded/Mmakefile
===================================================================
RCS file: /home/mercury1/repository/tests/hard_coded/Mmakefile,v
retrieving revision 1.61
diff -u -r1.61 Mmakefile
--- Mmakefile	1999/07/06 06:32:31	1.61
+++ Mmakefile	1999/07/09 06:58:23
@@ -143,45 +143,41 @@
 OUTS=	$(PROGS:%=%.out)
 RESS=	$(PROGS:%=%.res)
 
-#-----------------------------------------------------------------------------#
-
-SUBDIRS = typeclasses sub-modules
+dep_local:	$(DEPS)
+depend_local:	$(DEPENDS)
+check_local:	$(OUTS) $(RESS)
+all_local:	$(PROGS)
 
 #-----------------------------------------------------------------------------#
 
-MMAKEFLAGS =
-
-SUBDIR_MMAKE = mmake $(MMAKEFLAGS) \
-			GRADE='$(GRADE)' \
-			EXTRA_CFLAGS='$(EXTRA_CFLAGS)' \
-			EXTRA_MCFLAGS='$(EXTRA_MCFLAGS)'
+SUBDIRS = typeclasses sub-modules
 
-dep:	$(DEPS)
+dep_subdirs:
 	for dir in $(SUBDIRS); do \
 		(cd $$dir && $(SUBDIR_MMAKE) dep) || exit 1; \
 	done
 
-depend:	$(DEPENDS)
+depend_subdirs:
 	for dir in $(SUBDIRS); do \
 		(cd $$dir && $(SUBDIR_MMAKE) depend) || exit 1; \
 	done
 
-check:	$(OUTS) $(RESS)
+check_subdirs:
 	for dir in $(SUBDIRS); do \
 		(cd $$dir && $(SUBDIR_MMAKE) check) || exit 1; \
 	done
 
-all:	$(PROGS)
+all_subdirs:
 	for dir in $(SUBDIRS); do \
 		(cd $$dir && $(SUBDIR_MMAKE) all) || exit 1; \
 	done
 
-clean:
+clean_subdirs:
 	for dir in $(SUBDIRS); do \
 		(cd $$dir && $(SUBDIR_MMAKE) clean) || exit 1; \
 	done
 
-realclean:
+realclean_subdirs:
 	for dir in $(SUBDIRS); do \
 		(cd $$dir && $(SUBDIR_MMAKE) realclean) || exit 1; \
 	done
Index: tests/hard_coded/runtests
===================================================================
RCS file: /home/mercury1/repository/tests/hard_coded/runtests,v
retrieving revision 1.6
diff -u -r1.6 runtests
--- runtests	1998/05/13 04:06:05	1.6
+++ runtests	1999/07/09 06:42:41
@@ -4,10 +4,25 @@
 # Return a status of 0 (true) if everything is all right, and 1 otherwise.
 
 . ../handle_options
+
+# Run the tests in any subdirectories of this one.  If any of these
+# fail, we still perform the tests in this directory but return a
+# status of 1 regardless of the outcome.
+
+. ../subdir_runtests
+
+if test "$subdir_failures" = ""
+then
+        subdir_status=0
+else
+        subdir_status=1
+fi
+
+# Now run all the tests in this directory.
 . ../startup
 
-mmake $jfactor depend || exit 1
-eval mmake -k $jfactor $gradeopt $flagsopt $cflagsopt check
+mmake $jfactor depend_local || exit 1
+eval mmake -k $jfactor $gradeopt $flagsopt $cflagsopt check_local
 checkstatus=$?
 
 cat *.res > .allres
@@ -17,7 +32,7 @@
 	echo "gradeopt=$gradeopt, flagsopt=$flagsopt, cflagsopt=$cflagsopt"
 	rm -f .allres
 	. ../shutdown
-	exit 0
+	exit $subdir_status
 else
 	echo "the tests in the hard_coded directory failed"
 	echo "gradeopt=$gradeopt, flagsopt=$flagsopt, cflagsopt=$cflagsopt"


New file: tests/subdir_runtests

# Run the tests in each of the subdirectories.

subdir_failures=""
for dir in *
do
        if test -d $dir -a -x $dir/runtests
        then
                cd $dir
                # we need to use `eval' here to get the quoting right in
                # the case when $fflag or $cflag contains embedded spaces
                if eval ./runtests $jfactor $gflag $fflag $cflag
                then
                        true
                else
                        subdir_failures="$subdir_failures $dir"
                fi
                cd ..
        fi
done


New file: tests/debugger/declarative/runtests

#!/bin/sh
# Test whether the code generated by the Mercury compiler
# is producing the expected output.
# Return a status of 0 (true) if everything is all right, and 1 otherwise.

. ../../handle_options
. ../../startup

mmake $jfactor depend || exit 1
eval mmake -k $jfactor $gradeopt $flagsopt $cflagsopt check
checkstatus=$?

cat *.res > .allres
if test ! -s .allres -a "$checkstatus" = 0
then
	echo "the tests in the debugger/declarative directory succeeded"
	echo "gradeopt=$gradeopt, flagsopt=$flagsopt, cflagsopt=$cflagsopt"
	rm -f .allres
	. ../../shutdown
	exit 0
else
	echo "the tests in the debugger/declarative directory failed"
	echo "gradeopt=$gradeopt, flagsopt=$flagsopt, cflagsopt=$cflagsopt"
	echo "the differences are:"
	cat .allres
	exit 1
fi


The other two new files (tests/hard_coded/*/runtests) are exactly the
same as the above file, except that the two references to the
directory name have been fixed.

-- 
Mark Brown, PhD student            )O+  |  "Another of Fortran's breakthroughs
(m.brown at cs.mu.oz.au)                   |  was the GOTO statement, which was...
Dept. of Computer Science and Software  |  uniquely simple and understandable"
Engineering, University of Melbourne    |              -- IEEE, 1994
--------------------------------------------------------------------------
mercury-developers mailing list
Post messages to:       mercury-developers at cs.mu.oz.au
Administrative Queries: owner-mercury-developers at cs.mu.oz.au
Subscriptions:          mercury-developers-request at cs.mu.oz.au
--------------------------------------------------------------------------