[m-dev.] for review: uniform interface to tests

[Mark Anthony BROWN]

Fergus, this is the change we discussed a few days ago.  Could you
review it, please?

Cheers,
Mark.

Estimated hours taken: 8

Provide a more uniform interface to the test directories.  Before this
change, the tests directory as well as direct descendents had `runtests'
scripts to execute the tests.  These would run `mmake', which in turn
would recursively run `mmake' in any subdirectories.  The subdirectories
did not have their own `runtests' scripts.

This change adds a `runtests' script to those subdirectories that did
not have them before.  The scripts have the same meaning as they did
previously---run all tests in the current directory and below, and
return a status of 0 only if all tests pass.  However, the Mmake
targets 'check', 'dep', 'depend', 'realclean', 'clean' and 'all'  now
work in the current directory only.  The targets that also invoke Mmake
recursively in subdirectories are now called `check_subdirs',
`dep_subdirs', etc.

The rationale for this change is so that each directory in the tests
is treated the same, regardless of whether it is a top-level
directory or not.  This means, for example, that any test directory can
be used as an argument to the `--test-dir' option to tools/bootcheck.

This change also links tests/general/accumulator into the automated
testing suite.

tests/Mmake.common:
	Introduce new target dependencies.  Targets of the form
	`foo_subdirs' depend on the target `foo'.

tests/Mmakefile:
	Use the new *_subdirs targets in recursive mmakes.  Also, add
	`debugger' to the list of subdirectories.

tests/subdir_runtests:
	New script to recursively call runtests scripts in subdirectories.

tests/runtests:
tests/debugger/runtests:
tests/general/runtests:
tests/hard_coded/runtests:
	Use the new script to do the recursive calls.  Make sure that
	non-zero return values are propagated upwards where appropriate.

tests/debugger/declarative/runtests:
tests/hard_coded/typeclasses/runtests:
tests/hard_coded/sub-modules/runtests:
	New scripts to handle running tests in the subdirectories.

tests/debugger/Mmakefile:
tests/general/Mmakefile:
tests/hard_coded/Mmakefile:
	Add `*_subdirs' targets that do a recursive Mmake.

tests/warnings/Mmakefile:
	Manually add the extra dependencies, since this Mmakefile does
	not include tests/Mmake.common.

Index: tests/Mmake.common
===================================================================
RCS file: /home/mercury1/repository/tests/Mmake.common,v
retrieving revision 1.13
diff -u -r1.13 Mmake.common
--- Mmake.common	1998/11/04 07:53:21	1.13
+++ Mmake.common	1999/07/07 08:53:58
@@ -48,4 +48,17 @@
 clean_res:
 	rm -f *.res
 
+#
+# The `foo_subdirs' targets make `foo' in the current directory before
+# recursively making `foo_subdirs' in all subdirectories.  The recursive
+# part is handled in individual Mmakefiles.
+#
+
+check_subdirs:		check
+dep_subdirs:		dep
+depend_subdirs:		depend
+realclean_subdirs:	realclean
+clean_subdirs:		clean
+all_subdirs:		all
+
 #-----------------------------------------------------------------------------#
Index: tests/Mmakefile
===================================================================
RCS file: /home/mercury1/repository/tests/Mmakefile,v
retrieving revision 1.3
diff -u -r1.3 Mmakefile
--- Mmakefile	1998/08/14 06:26:07	1.3
+++ Mmakefile	1999/07/07 07:03:03
@@ -1,4 +1,4 @@
-SUBDIRS = benchmarks general hard_coded invalid misc_tests tabling term valid warnings
+SUBDIRS = benchmarks debugger general hard_coded invalid misc_tests tabling term valid warnings
 NUPROLOG_SUBDIRS = benchmarks general
 
 main_target: check
@@ -13,25 +13,26 @@
 # run the tests
 check:
 	for dir in $(SUBDIRS); do \
-		(cd $$dir && mmake $(MMAKEFLAGS) check) || exit 1; \
+		(cd $$dir && mmake $(MMAKEFLAGS) check_subdirs) || exit 1; \
 	done
 
 dep:
 	for dir in $(SUBDIRS); do \
-		(cd $$dir && mmake $(MMAKEFLAGS) dep) || exit 1; \
+		(cd $$dir && mmake $(MMAKEFLAGS) dep_subdirs) || exit 1; \
 	done
 
 depend:
 	for dir in $(SUBDIRS); do \
-		(cd $$dir && mmake $(MMAKEFLAGS) depend) || exit 1; \
+		(cd $$dir && mmake $(MMAKEFLAGS) depend_subdirs) || exit 1; \
 	done
 
 realclean:
 	for dir in $(SUBDIRS); do \
-		(cd $$dir && mmake $(MMAKEFLAGS) realclean) || exit 1; \
+		(cd $$dir && mmake $(MMAKEFLAGS) realclean_subdirs) \
+			|| exit 1; \
 	done
 
 clean:
 	for dir in $(SUBDIRS); do \
-		(cd $$dir && mmake $(MMAKEFLAGS) clean) || exit 1; \
+		(cd $$dir && mmake $(MMAKEFLAGS) clean_subdirs) || exit 1; \
 	done
Index: tests/runtests
===================================================================
RCS file: /home/mercury1/repository/tests/runtests,v
retrieving revision 1.4
diff -u -r1.4 runtests
--- runtests	1998/04/23 09:27:02	1.4
+++ runtests	1999/07/05 06:24:45
@@ -1,33 +1,18 @@
 #!/bin/sh
-# Run the tests in each of the subdirectories.
 
+# Process command line options.
 . ./handle_options
 
-failures=""
-for dir in *
-do
-	if test -d $dir -a -x $dir/runtests
-	then
-		cd $dir
-		# we need to use `eval' here to get the quoting right in
-		# the case when $fflag or $cflag contains embedded spaces
-		if eval ./runtests $jfactor $gflag $fflag $cflag
-		then
-			true
-		else
-			failures="$failures $dir"
-		fi
-		cd ..
-	fi
-done
+# Run the tests in each of the subdirectories.
+. ./subdir_runtests
 
-if test "$failures" = ""
+if test "$subdir_failures" = ""
 then
 	echo "all tests have succeeded"
 	echo "gradeopt=$gflag, flagsopt=$fflag, cflagsopt=$cflag"
 	exit 0
 else
-	echo "some tests have failed in: $failures"
+	echo "some tests have failed in: $subdir_failures"
 	echo "gradeopt=$gflag, flagsopt=$fflag, cflagsopt=$cflag"
 	exit 1
 fi
Index: tests/debugger/Mmakefile
===================================================================
RCS file: /home/mercury1/repository/tests/debugger/Mmakefile,v
retrieving revision 1.25
diff -u -r1.25 Mmakefile
--- Mmakefile	1999/06/02 07:28:30	1.25
+++ Mmakefile	1999/07/07 07:21:03
@@ -136,27 +136,45 @@
 
 #-----------------------------------------------------------------------------#
 
-SUBDIR_MMAKE = mmake $(MMAKEFLAGS) \
-			GRADE='$(GRADE)' \
-			EXTRA_CFLAGS='$(EXTRA_CFLAGS)' \
-			EXTRA_MCFLAGS='$(EXTRA_MCFLAGS)'
-
 dep:		$(DEPS)
-		cd declarative && $(SUBDIR_MMAKE) dep || exit 1;
 
 depend:		$(DEPENDS)
-		cd declarative && $(SUBDIR_MMAKE) depend || exit 1;
 
 check:		$(OUTS) $(RESS)
-		cd declarative && $(SUBDIR_MMAKE) check || exit 1;
 
 all:		$(PROGS)
-		cd declarative && $(SUBDIR_MMAKE) all || exit 1;
 
-clean:
-		cd declarative && $(SUBDIR_MMAKE) clean || exit 1;
+SUBDIRS = declarative
 
-realclean:
-		cd declarative && $(SUBDIR_MMAKE) realclean || exit 1;
+check_subdirs:
+	for dir in $(SUBDIRS); do \
+		(cd $$dir && mmake $(MMAKEFLAGS) check_subdirs) || exit 1; \
+	done
+
+dep_subdirs:
+	for dir in $(SUBDIRS); do \
+		(cd $$dir && mmake $(MMAKEFLAGS) dep_subdirs) || exit 1; \
+	done
+
+depend_subdirs:
+	for dir in $(SUBDIRS); do \
+		(cd $$dir && mmake $(MMAKEFLAGS) depend_subdirs) || exit 1; \
+	done
+
+realclean_subdirs:
+	for dir in $(SUBDIRS); do \
+		(cd $$dir && mmake $(MMAKEFLAGS) realclean_subdirs) \
+			|| exit 1; \
+	done
+
+clean_subdirs:
+	for dir in $(SUBDIRS); do \
+		(cd $$dir && mmake $(MMAKEFLAGS) clean_subdirs) || exit 1; \
+	done
+
+all_subdirs:
+	for dir in $(SUBDIRS); do \
+		(cd $$dir && mmake $(MMAKEFLAGS) all_subdirs) || exit 1; \
+	done
 
 #-----------------------------------------------------------------------------#
Index: tests/debugger/runtests
===================================================================
RCS file: /home/mercury1/repository/tests/debugger/runtests,v
retrieving revision 1.6
diff -u -r1.6 runtests
--- runtests	1998/12/16 17:28:24	1.6
+++ runtests	1999/07/07 05:28:00
@@ -4,6 +4,21 @@
 # Return a status of 0 (true) if everything is all right, and 1 otherwise.
 
 . ../handle_options
+
+# Run the tests in any subdirectories of this one.  If any of these
+# fail, we still perform the tests in this directory but return a
+# status of 1 regardless of the outcome.
+
+. ../subdir_runtests
+
+if test "$subdir_failures" = ""
+then
+	subdir_status=0
+else
+	subdir_status=1
+fi
+
+# Now run all the tests in this directory.
 . ../startup
 
 mmake $jfactor depend || exit 1
@@ -17,7 +32,7 @@
 	echo "gradeopt=$gradeopt, flagsopt=$flagsopt, cflagsopt=$cflagsopt"
 	rm -f .allres
 	. ../shutdown
-	exit 0
+	exit $subdir_status
 else
 	echo "the tests in the debugger directory failed"
 	echo "gradeopt=$gradeopt, flagsopt=$flagsopt, cflagsopt=$cflagsopt"
Index: tests/general/Mmakefile
===================================================================
RCS file: /home/mercury1/repository/tests/general/Mmakefile,v
retrieving revision 1.24
diff -u -r1.24 Mmakefile
--- Mmakefile	1998/10/27 15:36:40	1.24
+++ Mmakefile	1999/07/07 07:26:05
@@ -113,4 +113,37 @@
 
 all:	$(PROGS)
 
+SUBDIRS = accumulator
+
+check_subdirs:
+	for dir in $(SUBDIRS); do \
+		(cd $$dir && mmake $(MMAKEFLAGS) check_subdirs) || exit 1; \
+	done
+
+dep_subdirs:
+	for dir in $(SUBDIRS); do \
+		(cd $$dir && mmake $(MMAKEFLAGS) dep_subdirs) || exit 1; \
+	done
+
+depend_subdirs:
+	for dir in $(SUBDIRS); do \
+		(cd $$dir && mmake $(MMAKEFLAGS) depend_subdirs) || exit 1; \
+	done
+
+realclean_subdirs:
+	for dir in $(SUBDIRS); do \
+		(cd $$dir && mmake $(MMAKEFLAGS) realclean_subdirs) \
+			|| exit 1; \
+	done
+
+clean_subdirs:
+	for dir in $(SUBDIRS); do \
+		(cd $$dir && mmake $(MMAKEFLAGS) clean_subdirs) || exit 1; \
+	done
+
+all_subdirs:
+	for dir in $(SUBDIRS); do \
+		(cd $$dir && mmake $(MMAKEFLAGS) all_subdirs) || exit 1; \
+	done
+
 #-----------------------------------------------------------------------------#
Index: tests/general/runtests
===================================================================
RCS file: /home/mercury1/repository/tests/general/runtests,v
retrieving revision 1.6
diff -u -r1.6 runtests
--- runtests	1998/05/13 04:05:54	1.6
+++ runtests	1999/07/07 08:17:10
@@ -13,6 +13,21 @@
 # on a machine that has NU-Prolog installed.
 
 . ../handle_options
+
+# Run the tests in any subdirectories of this one.  If any of these
+# fail, we still perform the tests in this directory but return a
+# status of 1 regardless of the outcome.
+
+. ../subdir_runtests
+
+if test "$subdir_failures" = ""
+then
+        subdir_status=0
+else
+        subdir_status=1
+fi
+
+# Now run all the tests in this directory.
 . ../startup
 
 mmake $jfactor depend || exit 1
@@ -26,7 +41,7 @@
 	echo "gradeopt=$gradeopt, flagsopt=$flagsopt, cflagsopt=$cflagsopt"
 	rm -f .allres
 	. ../shutdown
-	exit 0
+	exit $subdir_status
 else
 	echo "the tests in the general directory failed"
 	echo "gradeopt=$gradeopt, flagsopt=$flagsopt, cflagsopt=$cflagsopt"
Index: tests/hard_coded/Mmakefile
===================================================================
RCS file: /home/mercury1/repository/tests/hard_coded/Mmakefile,v
retrieving revision 1.61
diff -u -r1.61 Mmakefile
--- Mmakefile	1999/07/06 06:32:31	1.61
+++ Mmakefile	1999/07/07 08:35:16
@@ -143,12 +143,15 @@
 OUTS=	$(PROGS:%=%.out)
 RESS=	$(PROGS:%=%.res)
 
+dep:	$(DEPS)
+depend:	$(DEPENDS)
+check:	$(OUTS) $(RESS)
+all:	$(PROGS)
+
 #-----------------------------------------------------------------------------#
 
 SUBDIRS = typeclasses sub-modules
 
-#-----------------------------------------------------------------------------#
-
 MMAKEFLAGS =
 
 SUBDIR_MMAKE = mmake $(MMAKEFLAGS) \
@@ -156,34 +159,34 @@
 			EXTRA_CFLAGS='$(EXTRA_CFLAGS)' \
 			EXTRA_MCFLAGS='$(EXTRA_MCFLAGS)'
 
-dep:	$(DEPS)
+check_subdirs:
 	for dir in $(SUBDIRS); do \
-		(cd $$dir && $(SUBDIR_MMAKE) dep) || exit 1; \
+		(cd $$dir && $(SUBDIR_MMAKE) check_subdirs) || exit 1; \
 	done
 
-depend:	$(DEPENDS)
+dep_subdirs:
 	for dir in $(SUBDIRS); do \
-		(cd $$dir && $(SUBDIR_MMAKE) depend) || exit 1; \
+		(cd $$dir && $(SUBDIR_MMAKE) dep_subdirs) || exit 1; \
 	done
 
-check:	$(OUTS) $(RESS)
+depend_subdirs:
 	for dir in $(SUBDIRS); do \
-		(cd $$dir && $(SUBDIR_MMAKE) check) || exit 1; \
+		(cd $$dir && $(SUBDIR_MMAKE) depend_subdirs) || exit 1; \
 	done
 
-all:	$(PROGS)
+realclean_subdirs:
 	for dir in $(SUBDIRS); do \
-		(cd $$dir && $(SUBDIR_MMAKE) all) || exit 1; \
+		(cd $$dir && $(SUBDIR_MMAKE) realclean_subdirs) || exit 1; \
 	done
 
-clean:
+clean_subdirs:
 	for dir in $(SUBDIRS); do \
-		(cd $$dir && $(SUBDIR_MMAKE) clean) || exit 1; \
+		(cd $$dir && $(SUBDIR_MMAKE) clean_subdirs) || exit 1; \
 	done
 
-realclean:
+all_subdirs:
 	for dir in $(SUBDIRS); do \
-		(cd $$dir && $(SUBDIR_MMAKE) realclean) || exit 1; \
+		(cd $$dir && $(SUBDIR_MMAKE) all_subdirs) || exit 1; \
 	done
 
 #-----------------------------------------------------------------------------#
Index: tests/hard_coded/runtests
===================================================================
RCS file: /home/mercury1/repository/tests/hard_coded/runtests,v
retrieving revision 1.6
diff -u -r1.6 runtests
--- runtests	1998/05/13 04:06:05	1.6
+++ runtests	1999/07/07 09:50:47
@@ -4,6 +4,21 @@
 # Return a status of 0 (true) if everything is all right, and 1 otherwise.
 
 . ../handle_options
+
+# Run the tests in any subdirectories of this one.  If any of these
+# fail, we still perform the tests in this directory but return a
+# status of 1 regardless of the outcome.
+
+. ../subdir_runtests
+
+if test "$subdir_failures" = ""
+then
+        subdir_status=0
+else
+        subdir_status=1
+fi
+
+# Now run all the tests in this directory.
 . ../startup
 
 mmake $jfactor depend || exit 1
@@ -17,7 +32,7 @@
 	echo "gradeopt=$gradeopt, flagsopt=$flagsopt, cflagsopt=$cflagsopt"
 	rm -f .allres
 	. ../shutdown
-	exit 0
+	exit $subdir_status
 else
 	echo "the tests in the hard_coded directory failed"
 	echo "gradeopt=$gradeopt, flagsopt=$flagsopt, cflagsopt=$cflagsopt"
Index: tests/warnings/Mmakefile
===================================================================
RCS file: /home/mercury1/repository/tests/warnings/Mmakefile,v
retrieving revision 1.5
diff -u -r1.5 Mmakefile
--- Mmakefile	1999/06/23 04:17:56	1.5
+++ Mmakefile	1999/07/07 06:59:10
@@ -55,4 +55,10 @@
 clean_res:
 	rm -f *.res
 
+check_subdirs:		check
+dep_subdirs:		dep
+depend_subdirs:		depend
+realclean_subdirs:	realclean
+clean_subdirs:		clean
+
 #-----------------------------------------------------------------------------#

New file: tests/subdir_runtests

# Run the tests in each of the subdirectories.

subdir_failures=""
for dir in *
do
        if test -d $dir -a -x $dir/runtests
        then
                cd $dir
                # we need to use `eval' here to get the quoting right in
                # the case when $fflag or $cflag contains embedded spaces
                if eval ./runtests $jfactor $gflag $fflag $cflag
                then
                        true
                else
                        subdir_failures="$subdir_failures $dir"
                fi
                cd ..
        fi
done

New file: tests/debugger/declarative/runtests

#!/bin/sh
# Test whether the code generated by the Mercury compiler
# is producing the expected output.
# Return a status of 0 (true) if everything is all right, and 1 otherwise.

. ../../handle_options
. ../../startup

mmake $jfactor depend || exit 1
eval mmake -k $jfactor $gradeopt $flagsopt $cflagsopt check
checkstatus=$?

cat *.res > .allres
if test ! -s .allres -a "$checkstatus" = 0
then
	echo "the tests in the debugger/declarative directory succeeded"
	echo "gradeopt=$gradeopt, flagsopt=$flagsopt, cflagsopt=$cflagsopt"
	rm -f .allres
	. ../../shutdown
	exit 0
else
	echo "the tests in the debugger/declarative directory failed"
	echo "gradeopt=$gradeopt, flagsopt=$flagsopt, cflagsopt=$cflagsopt"
	echo "the differences are:"
	cat .allres
	exit 1
fi


The other two new files (tests/hard_coded/*/runtests) are exactly the
same as the above file, except that the two references to the
directory name have been fixed.

-- 
Mark Brown, PhD student            )O+  |  "Another of Fortran's breakthroughs
(m.brown at cs.mu.oz.au)                   |  was the GOTO statement, which was...
Dept. of Computer Science and Software  |  uniquely simple and understandable"
Engineering, University of Melbourne    |              -- IEEE, 1994
--------------------------------------------------------------------------
mercury-developers mailing list
Post messages to:       mercury-developers at cs.mu.oz.au
Administrative Queries: owner-mercury-developers at cs.mu.oz.au
Subscriptions:          mercury-developers-request at cs.mu.oz.au
--------------------------------------------------------------------------