[m-dev.] diff: tools improvements

[Zoltan Somogyi]

Improvements to the infrastructure for debugging code generator changes.

tools/binary:
	If either stage2.ok or stage2.bad is missing object files, then do not
	complain: instead, recreate the missing object files.

	Fix a bug: copy the library's .pic_o files together with its .o files.

	Fix a bug: make sure that we link *all* the possible relevant .init
	files from the runtime to the stage2 directory.

	If the search narrows down to one file, and the script is trying to
	find out which part of the file is in error, then consider all the
	parts that are identical between the stage2.ok and stage2.bad to be
	known good from the start. This can reduce the number of bootchecks
	needed by one or two.

tools/binary:
tools/binary_step:
	Allow the test to be the successful compilation of the stage 3 library
	directory. (In almost all cases, bad stage 2 compilers fail while
	compiling the stage 3 library. By not compiling the stage 3 compiler 
	if the compilation of the stage 3 library succeeds, we can save a lot
	of time.)

	If the search narrows down to one file, and the script is trying to
	find out which part of the file is in error, watch out for errors that
	prevent the stage 2 executable from being built. If such an error
	occurs, then stop right then and there. In such cases, there is no
	point in further binary search, since each further invocation of
	binary_step will just indicate that the source files in stage2.bad and
	stage2.ok are not compatible (e.g. because they do not use the same
	mapping from static term numbers to static term contents.)

tools/binary_step:
	Reduce the time for declaring a program to be in an infinite loop,
	since the slowest machine we may want to use is faster now.

tools/makebatch:
	Fix some glaring bugs: e.g. test uses -lt, not < for comparisons.

	Add an option, --stop-at-failure, that stops makebatch if a bootcheck
	fails and thus preserves the stage[23] directories involved in the
	failure. This allows one to verify that a change works in many grades
	without sacrificing the ability to debug any problems.

	Add another option --c-files, that gathers the C files created
	in each bootcheck. This allows the C files to be compared, e.g.
	for efficiency problems.

Zoltan.

cvs diff: Diffing .
Index: binary
===================================================================
RCS file: /home/mercury1/repository/mercury/tools/binary,v
retrieving revision 1.19
diff -u -b -r1.19 binary
--- binary	2000/03/28 10:43:20	1.19
+++ binary	2000/07/17 06:07:59
@@ -12,8 +12,9 @@
 # versions, then the script stops after identifying only the file.
 #
 # The test for the composite stage2 compiler is either bootstrap checking
-# (the default), the successful execution of all the test cases in one or
-# more subdirectories of the tests directory, or the successful execution
+# (the default), some initial part of bootstrap checking (making dependencies,
+# compiling the library), the successful execution of all the test cases in
+# one or more subdirectories of the tests directory, or the successful execution
 # of a single command.
 
 usage="\
@@ -40,6 +41,9 @@
 		Display this usage message.
 	-j <num-jobs>, --jobs <num-jobs>
 		Run using <num-jobs> different parallel processes.
+	-l, --library-only
+		Check the successful creation of the stage3 .c files in the
+		library, but do not compile the compiler directory.
 	-m <mmake-args>, --mmake-args <mmake-args>
 		Pass <mmake-args> as options to \`mmake'.
 	-n, --negative-search
@@ -66,10 +70,12 @@
 bootcheck=""
 compile_only=""
 dependency_only=""
+library_only=""
 jfactor=
 mmake_opts=""
 outfile=""
 copy_runtime=false
+single_command=""
 testdirs=""
 negative=false
 alldirs=""
@@ -114,6 +120,9 @@
 	--jobs*)
 		jfactor="--jobs` expr $1 : '--jobs\(.*\)' `" ;;
 
+	-l|--library-only)
+		library_only="-l" ;;
+
 	-m|--mmake)
 		mmake_opts="$mmake_opts $2"; shift ;;
 
@@ -171,16 +180,6 @@
 	exit 1
 fi
 
-if test -f stage2.ok/library/builtin.o -a -f stage2.bad/library/builtin.o
-then
-	echo "stage2.ok and stage2.bad both seem to have object files"
-else
-	echo "at least one of stage2.ok and stage2.bad is missing object files"
-	exit 1
-fi
-
-echo "starting at `date`"
-
 root=`/bin/pwd`
 trueroot=`echo $root | sed '
 s:/mount/munkora/mercury:/home/mercury:
@@ -194,6 +193,50 @@
 	RMSTAGECMD="/bin/rm -fr"
 fi
 
+echo "starting at `date`"
+
+if test -f stage2.ok/library/builtin.o
+then
+	echo "stage2.ok/library seems to have its object files"
+else
+	echo "reconstructing object files in stage2.ok/library"
+	cd stage2.ok/library
+	mmake
+	cd $root
+fi
+
+if test -f stage2.bad/library/builtin.o
+then
+	echo "stage2.bad/library seems to have its object files"
+else
+	echo "reconstructing object files in stage2.bad/library"
+	cd stage2.bad/library
+	mmake
+	cd $root
+fi
+
+if test -f stage2.ok/compiler/arg_info.o
+then
+	echo "stage2.ok/compiler seems to have its object files"
+else
+	echo "reconstructing object files in stage2.ok/compiler"
+	cd stage2.ok/compiler
+	mmake
+	cd $root
+fi
+
+if test -f stage2.bad/compiler/arg_info.o
+then
+	echo "stage2.bad/compiler seems to have its object files"
+else
+	echo "reconstructing object files in stage2.bad/compiler"
+	cd stage2.bad/compiler
+	mmake
+	cd $root
+fi
+
+echo "starting binary at `date`"
+
 set +x
 [ -d stage2 ] || mkdir stage2
 $RMSTAGECMD $trueroot/stage2/compiler < /dev/null &
@@ -215,7 +258,7 @@
 cd library
 ln -s $root/library/[a-l]*.m .
 ln -s $root/library/[m-z]*.m .
-ln -s $root/library/lib$STD_LIB_NAME.init .
+ln -s $root/library/*.init .
 cp $root/library/Mmake* .
 cd $root/stage2
 if "$copy_runtime"
@@ -297,7 +340,7 @@
 	cd library
 	ln -s $root/library/[a-l]*.m .
 	ln -s $root/library/[m-z]*.m .
-	ln -s $root/library/lib$STD_LIB_NAME.init .
+	ln -s $root/library/*.init .
 	cp $root/library/Mmake* .
 	cd $root/stage3
 	ln -s $root/boehm_gc .
@@ -344,7 +387,7 @@
 
 	if "$negative"
 	then
-		if binary_step $bootcheck $compile_only $compare_to_bad $dependency_only $jfactor -m "$mmake_opts" $outfile $testdirs -s "$single_command"
+		if binary_step $bootcheck $compile_only $compare_to_bad $dependency_only $library_only $jfactor -m "$mmake_opts" $outfile $testdirs -s "$single_command"
 		then
 			testeddir=library
 		else
@@ -352,7 +395,7 @@
 		fi
 		echo "solution seems to be in the $testeddir directory"
 	else
-		if binary_step $bootcheck $compile_only $compare_to_bad $dependency_only $jfactor -m "$mmake_opts" $outfile $testdirs -s "$single_command"
+		if binary_step $bootcheck $compile_only $compare_to_bad $dependency_only $library_only $jfactor -m "$mmake_opts" $outfile $testdirs -s "$single_command"
 		then
 			testeddir=compiler
 		else
@@ -410,12 +453,15 @@
 	for module in $tested
 	do
 		cp stage2.$trial/$testeddir/$module.[co] stage2/$testeddir
-		cp stage2.$trial/$testeddir/$module.pic_o stage2/$testeddir
+		if test $testeddir = library
+		then
+			cp stage2.$trial/library/$module.pic_o stage2/library
+		fi
 	done
 
 	if "$negative"
 	then
-		if binary_step $bootcheck $compile_only $compare_to_bad $dependency_only $jfactor -m "$mmake_opts" $outfile $testdirs -s "$single_command"
+		if binary_step $bootcheck $compile_only $compare_to_bad $dependency_only $library_only $jfactor -m "$mmake_opts" $outfile $testdirs -s "$single_command"
 		then
 			echo "test succeeded"
 			lasttest=success
@@ -436,7 +482,7 @@
 			doubtful="$newdoubtful"
 		fi
 	else
-		if binary_step $bootcheck $compile_only $compare_to_bad $dependency_only $jfactor -m "$mmake_opts" $outfile $testdirs -s "$single_command"
+		if binary_step $bootcheck $compile_only $compare_to_bad $dependency_only $library_only $jfactor -m "$mmake_opts" $outfile $testdirs -s "$single_command"
 		then
 			echo "test succeeded"
 			lasttest=success
@@ -462,6 +508,10 @@
 	for module in $tested
 	do
 		cp stage2.$base/$testeddir/$module.[co] stage2/$testeddir
+		if test $testeddir = library
+		then
+			cp stage2.$base/library/$module.pic_o stage2/library
+		fi
 	done
 
 	tested=`half $doubtful`
@@ -511,19 +561,29 @@
 
 set +x
 doubtful=
+knowngood=
 i=0
 while test $i -le $basecnt
 do
+	# If two corresponding parts from stage2.ok and stage.bad are the same,
+	# then the bug cannot be in that part.
+	if cmp -s stage2.$base/$testeddir/$module.c.part.$i stage2.$trial/$testeddir/$module.c.part.$i
+	then
+		knowngood="$knowngood $i"
+	else
 	doubtful="$doubtful $i"
+	fi
 	i=`expr $i + 1`
 done
 set -x
 
 tested=`half $doubtful`
-knowngood=
+
+/bin/rm -fr .stage2_problem
 
 while test "$tested" != ""
 do
+	echo "knowngood: $knowngood"
 	echo "doubtful: $doubtful"
 	echo "testing:  $tested"
 
@@ -535,13 +595,22 @@
 
 	if "$negative"
 	then
-		if binary_step $bootcheck $compile_only $compare_to_bad $dependency_only $jfactor -m "$mmake_opts" $outfile $testdirs -s "$single_command"
+		if binary_step $bootcheck $compile_only $compare_to_bad $dependency_only $library_only $jfactor -m "$mmake_opts" $outfile $testdirs -s "$single_command"
 		then
 			echo "test succeeded"
 			lasttest=success
 			doubtful="$tested"
 		else
 			echo "test failed"
+
+			if test -f .stage2_problem
+			then
+				/bin/rm .stage2_problem
+				echo "problem with the creation of stage 2:"
+				echo "search of $testeddir/$module abandoned"
+				exit 1
+			fi
+
 			lasttest=failure
 			set +x
 			newdoubtful=""
@@ -556,7 +625,7 @@
 			doubtful="$newdoubtful"
 		fi
 	else
-		if binary_step $bootcheck $compile_only $compare_to_bad $dependency_only $jfactor -m "$mmake_opts" $outfile $testdirs -s "$single_command"
+		if binary_step $bootcheck $compile_only $compare_to_bad $dependency_only $library_only $jfactor -m "$mmake_opts" $outfile $testdirs -s "$single_command"
 		then
 			echo "test succeeded"
 			lasttest=success
@@ -574,6 +643,15 @@
 			doubtful="$newdoubtful"
 		else
 			echo "test failed"
+
+			if test -f .stage2_problem
+			then
+				/bin/rm .stage2_problem
+				echo "problem with the creation of stage 2:"
+				echo "search of $testeddir/$module abandoned"
+				exit 1
+			fi
+
 			lasttest=failure
 			doubtful="$tested"
 		fi
Index: binary_step
===================================================================
RCS file: /home/mercury1/repository/mercury/tools/binary_step,v
retrieving revision 1.13
diff -u -b -r1.13 binary_step
--- binary_step	2000/03/28 10:43:20	1.13
+++ binary_step	2000/07/17 06:07:59
@@ -27,6 +27,9 @@
 		Display this usage message.
 	-j <num-jobs>, --jobs <num-jobs>
 		Run using <num-jobs> different parallel processes.
+	-l, --library-only
+		Check the successful creation of the stage3 .c files in the
+		library, but do not compile the compiler directory.
 	-m <mmake-args>, --mmake-args <mmake-args>
 		Pass <mmake-args> as options to \`mmake'.
 	-o <filename>, --output-file <filename>
@@ -49,6 +52,7 @@
 bootcheck="true"
 compile_only="false"
 dependency_only="false"
+library_only="false"
 jfactor=""
 mmake_opts=""
 outfile="DIFF.BINARY"
@@ -82,6 +86,9 @@
 	--jobs*)
 		jfactor="--jobs` expr $1 : '--jobs\(.*\)' `" ;;
 
+	-l|--library-only)
+		library_only="true" ;;
+
 	-m|--mmake)
 		mmake_opts="$mmake_opts $2"; shift ;;
 
@@ -203,6 +210,7 @@
 	echo "building of stage 2 library successful"
 else
 	echo "building of stage 2 library not successful"
+	touch .stage2_problem
 	exit 1
 fi
 
@@ -211,6 +219,7 @@
 	echo "building of stage 2 compiler successful"
 else
 	echo "building of stage 2 compiler not successful"
+	touch .stage2_problem
 	exit 1
 fi
 
@@ -222,7 +231,7 @@
 MERCURY_INT_DIR=$root/stage3/library
 export MERCURY_INT_DIR
 
-ulimit -t 600
+ulimit -t 200
 
 if test "$single_command" != ""
 then
@@ -268,7 +277,7 @@
 	if (cd stage3 ; mmake $mmake_opts depend_library depend_compiler)
 	then
 		echo "building of stage 3 dependencies successful"
-		if test "$dependency_only" = "true"
+		if $dependency_only
 		then
 			exit 0
 		fi
@@ -285,6 +294,10 @@
 	if (cd stage3/library ; mmake -S $mmake_opts $jfactor ints ; mmake -S $mmake_opts $jfactor cs)
 	then
 		echo "building of stage 3 library successful"
+		if $library_only
+		then
+			exit 0
+		fi
 	else
 		echo "building of stage 3 library not successful"
 		exit 1
Index: makebatch
===================================================================
RCS file: /home/mercury1/repository/mercury/tools/makebatch,v
retrieving revision 1.12
diff -u -b -r1.12 makebatch
--- makebatch	2000/01/06 01:28:36	1.12
+++ makebatch	2000/07/17 06:08:00
@@ -26,16 +26,20 @@
 # that version. When makebatch exits normally, it removes the file to indicate
 # completion.
 
-usage="Usage: makebatch [-jN] [-fot] batchname"
+usage="Usage: makebatch [-jN] [-fost] batchname"
 jfactor=-j1
 runtests=""
 objects=""
+cfiles="false"
 failed="notwanted"
 
 while test $# -gt 0
 do
 	case $1 in
 
+	-c|--c-files)
+		cfiles="true" ;;
+
 	-f|--failed-compilers)
 		failed="wanted" ;;
 
@@ -49,6 +53,9 @@
 	-o|--object-files)
 		objects="-k" ;;
 
+	-s|--stop-at-failure)
+		failed="stop" ;;
+
 	-t|--no-test-suite)
 		runtests="-t" ;;
 
@@ -139,12 +146,12 @@
 		gradeopt=""
 	fi
 
-	if test $n < 10
+	if test $n -lt 10
 	then
 		visn="0$n"
 	else
 		visn="$n"
-	else
+	fi
 
 	cp Mmake.stage.params batch/$batch.params.$visn
 	echo starting bootcheck of version $visn
@@ -153,28 +160,28 @@
 		echo bootcheck of version $visn succeeded
 		echo bootcheck succeeded > batch/$batch.out.$visn
 		mv stage2/compiler/mercury_compile batch/$batch.mercury_compile.$visn
-		echo -n "batch/$batch.mercury_compile.$visn	" >> batch/$batch.sizes
-		size batch/$batch.mercury_compile.$visn | tail -1 >> batch/$batch.sizes
+		size batch/$batch.mercury_compile.$visn | tail -1 | sed -e 's/mercury_compile.//' | sed -e 's/batch\///' >> batch/$batch.sizes
 		/bin/rm -f batch/$batch.mercury_compile.$visn.gz > /dev/null 2>&1
 		gzip batch/$batch.mercury_compile.$visn
-	else
-		if test "$failed" = "wanted"
+	elif test "$failed" = "stop"
 		then
+		echo bootcheck of version $visn failed
+		exit 1
+	elif test "$failed" = "wanted"
+	then
 			if test -x stage2/compiler/mercury_compile
 			then
 				echo bootcheck of version $visn failed but produced compiler
 				mv stage2/compiler/mercury_compile batch/$batch.mercury_compile.$visn
-				size batch/$batch.mercury_compile.$visn | tail -1 >> batch/$batch.sizes
+			size batch/$batch.mercury_compile.$visn | tail -1 | sed -e 's/mercury_compile.//' | sed -e 's/batch\///' >> batch/$batch.sizes
 				/bin/rm -f batch/$batch.mercury_compile.$visn.gz > /dev/null 2>&1
 				gzip batch/$batch.mercury_compile.$visn
-
 			else
 				echo bootcheck of version $visn failed and did not produce compiler
 			fi
 		else
 			echo bootcheck of version $visn failed
 		fi
-	fi
 
 	if test "$objects" = "-k"
 	then
@@ -187,6 +194,19 @@
 		/bin/rm -fr batch/objs/$batch.compiler.$visn/*
 		cp stage2/compiler/*.o batch/objs/$batch.compiler.$visn
 		gzip batch/objs/$batch.compiler.$visn/*
+	fi
+
+	if "$cfiles"
+	then
+		echo saving c files
+		mkdir -p batch/cfiles/$batch.library.$visn
+		/bin/rm -fr batch/cfiles/$batch.library.$visn/*
+		cp stage2/library/*.c batch/cfiles/$batch.library.$visn
+		gzip batch/cfiles/$batch.library.$visn/*
+		mkdir -p batch/cfiles/$batch.compiler.$visn
+		/bin/rm -fr batch/cfiles/$batch.compiler.$visn/*
+		cp stage2/compiler/*.c batch/cfiles/$batch.compiler.$visn
+		gzip batch/cfiles/$batch.compiler.$visn/*
 	fi
 
 	n=`expr $n + 1`
Index: speedtest
===================================================================
RCS file: /home/mercury1/repository/mercury/tools/speedtest,v
retrieving revision 1.6
diff -u -b -r1.6 speedtest
--- speedtest	1999/09/21 04:33:27	1.6
+++ speedtest	2000/07/17 06:08:00
@@ -4,7 +4,7 @@
 
 usage="Usage: speedtest [-c cmd] [-dns] batchname"
 limit=4
-cmd="mmc -C -O2 -I../compiler make_hlds.m"
+cmd="mmc -C -O2 -I../compiler --no-lazy-code make_hlds.m"
 debug=false
 size=false
 
--------------------------------------------------------------------------
mercury-developers mailing list
Post messages to:       mercury-developers at cs.mu.oz.au
Administrative Queries: owner-mercury-developers at cs.mu.oz.au
Subscriptions:          mercury-developers-request at cs.mu.oz.au
--------------------------------------------------------------------------