[m-dev.] for review: rename the lexer and parser modules

[Zoltan Somogyi]

Ralph Becket's bug report is about the fourth problem I can remember being
caused by the fact that we effectively don't allow people to call their modules
"lexer" or "parser", both of which are natural names. I think it better to fix
this problem now, without waiting for submodules to be used in the library.

The diff is entirely boring, having been produced mostly by global replace
commands, and does not need review; what may need review is the idea
of the rename. The only downside I can see is that looking up the CVS histories
of the two affected modules will be somewhat harder. Any other opinions?

Zoltan.

----------------------

Rename the lexer and parser modules of the standard library as mer_lexer and
mer_parser. This should allow people to write their own lexer and parser
modules without getting linker errors.

Once all the modules in the library are submodules of module std, we can
revisit this decision.

library/lexer.m:
library/parser.m:
	Deleted these files under their old names.

library/mer_lexer.m:
library/mer_parser.m:
	And created them under their new names.

library/io.m:
library/library.m:
library/term_io.m:
browser/browse.m:
browser/interactive_query.m:
compiler/fact_table.m:
compiler/mercury_to_mercury.m:
compiler/prog_io.m:
	Refer to lexer and parser by their new names. In one case, delete a
	unused import.

cvs diff: Diffing .
cvs diff: Diffing bindist
cvs diff: Diffing boehm_gc
cvs diff: Diffing boehm_gc/Mac_files
cvs diff: Diffing boehm_gc/cord
cvs diff: Diffing boehm_gc/cord/private
cvs diff: Diffing boehm_gc/include
cvs diff: Diffing boehm_gc/include/private
cvs diff: Diffing browser
Index: browser/browse.m
===================================================================
RCS file: /home/mercury1/repository/mercury/browser/browse.m,v
retrieving revision 1.15
diff -u -r1.15 browse.m
--- browser/browse.m	2000/08/17 08:06:55	1.15
+++ browser/browse.m	2000/08/22 04:10:13
@@ -66,7 +66,7 @@
 :- implementation.
 
 :- import_module mdb__parse, mdb__util, mdb__frame.
-:- import_module string, list, parser, require, std_util, int, char, pprint.
+:- import_module string, list, int, char, require, std_util, pprint.
 
 %---------------------------------------------------------------------------%
 %
Index: browser/interactive_query.m
===================================================================
RCS file: /home/mercury1/repository/mercury/browser/interactive_query.m,v
retrieving revision 1.7
diff -u -r1.7 interactive_query.m
--- browser/interactive_query.m	2000/02/04 03:45:29	1.7
+++ browser/interactive_query.m	2000/08/22 04:09:29
@@ -33,7 +33,7 @@
 :- type options == string.
 
 :- implementation.
-:- import_module std_util, bool, string, term, varset, term_io, parser.
+:- import_module std_util, bool, string, term, varset, term_io, mer_parser.
 :- import_module mdb__dl, mdb__name_mangle.
 :- import_module mdb__util.
 
@@ -52,7 +52,7 @@
 		io__write_string(MDB_Stdout, Msg), io__nl(MDB_Stdout),
 		query(QueryType, Imports, Options, MDB_Stdin, MDB_Stdout)
 	; { Result = ok(Line) },
-		{ parser__read_term_from_string("", Line, _, ReadTerm) },
+		{ mer_parser__read_term_from_string("", Line, _, ReadTerm) },
 		query_2(QueryType, Imports, Options, MDB_Stdin, MDB_Stdout,
 				ReadTerm)
 	).
cvs diff: Diffing bytecode
cvs diff: Diffing compiler
Index: compiler/fact_table.m
===================================================================
RCS file: /home/mercury1/repository/mercury/compiler/fact_table.m,v
retrieving revision 1.32
diff -u -r1.32 fact_table.m
--- compiler/fact_table.m	2000/05/08 13:48:05	1.32
+++ compiler/fact_table.m	2000/08/22 02:19:11
@@ -86,13 +86,13 @@
 
 :- implementation.
 
-:- import_module int, map, std_util, assoc_list, char, require, library, bool.
-:- import_module float, math, getopt, string.
-:- import_module parser, term_io.
-
 :- import_module prog_util, prog_out, llds_out, modules, hlds_out, hlds_data.
 :- import_module globals, options, passes_aux, arg_info, llds, mode_util.
-:- import_module prog_io, code_util, export, inst_match, term.
+:- import_module prog_io, code_util, export, inst_match.
+
+:- import_module int, map, std_util, assoc_list, char, require, library, bool.
+:- import_module float, math, getopt, string.
+:- import_module mer_parser, term, term_io.
 
 :- type fact_result
 	--->	ok ; error.
@@ -290,7 +290,7 @@
 
 compile_facts(PredName, Arity, PredInfo, ModuleInfo, FactArgInfos, ProcStreams,
 		MaybeOutput, NumFacts0, NumFacts) -->
-	parser__read_term(Result0),
+	mer_parser__read_term(Result0),
 	(
 		{ Result0 = eof},
 		{ NumFacts = NumFacts0 }
Index: compiler/mercury_to_mercury.m
===================================================================
RCS file: /home/mercury1/repository/mercury/compiler/mercury_to_mercury.m,v
retrieving revision 1.172
diff -u -r1.172 mercury_to_mercury.m
--- compiler/mercury_to_mercury.m	2000/08/13 13:18:44	1.172
+++ compiler/mercury_to_mercury.m	2000/08/22 02:34:08
@@ -210,7 +210,7 @@
 :- import_module purity, term_util.
 :- import_module globals, options, termination.
 
-:- import_module assoc_list, char, int, string, set, lexer, require.
+:- import_module assoc_list, char, int, string, set, mer_lexer, require.
 :- import_module term, term_io, varset.
 
 %-----------------------------------------------------------------------------%
@@ -2740,7 +2740,7 @@
 		{ NextToGraphicToken = next_to_graphic_token },
 		{ string__to_char_list(Name, Chars) },
 		{ \+ (  list__member(Char, Chars),
-			\+ lexer__graphic_token_char(Char)) }
+			\+ mer_lexer__graphic_token_char(Char)) }
 	->
 		io__write_string("'"),
 		term_io__write_escaped_string(Name),
Index: compiler/prog_io.m
===================================================================
RCS file: /home/mercury1/repository/mercury/compiler/prog_io.m,v
retrieving revision 1.192
diff -u -r1.192 prog_io.m
--- compiler/prog_io.m	2000/08/16 08:54:04	1.192
+++ compiler/prog_io.m	2000/08/22 02:18:36
@@ -187,7 +187,7 @@
 :- import_module hlds_data, hlds_pred, prog_util, prog_out.
 :- import_module globals, options, (inst).
 
-:- import_module int, string, std_util, parser, term_io, dir, require.
+:- import_module int, string, std_util, mer_parser, term_io, dir, require.
 :- import_module assoc_list.
 
 %-----------------------------------------------------------------------------%
@@ -445,7 +445,7 @@
 	% (so that any `:- module' declaration is taken to
 	% be a non-nested module unless explicitly qualified).
 	%
-	parser__read_term(SourceFileName, MaybeFirstTerm),
+	mer_parser__read_term(SourceFileName, MaybeFirstTerm),
 	{ root_module_name(RootModuleName) },
 	{ process_read_term(RootModuleName, MaybeFirstTerm, MaybeFirstItem) },
 	(
@@ -673,7 +673,7 @@
 :- mode read_item(in, in, out, di, uo) is det.
 
 read_item(ModuleName, SourceFileName, MaybeItem) -->
-	parser__read_term(SourceFileName, MaybeTerm),
+	mer_parser__read_term(SourceFileName, MaybeTerm),
 	{ process_read_term(ModuleName, MaybeTerm, MaybeItem) }.
 
 :- pred process_read_term(module_name, read_term, maybe_item_or_eof).
cvs diff: Diffing compiler/notes
cvs diff: Diffing debian
cvs diff: Diffing doc
cvs diff: Diffing extras
cvs diff: Diffing extras/aditi
cvs diff: Diffing extras/cgi
cvs diff: Diffing extras/complex_numbers
cvs diff: Diffing extras/complex_numbers/samples
cvs diff: Diffing extras/complex_numbers/tests
cvs diff: Diffing extras/concurrency
cvs diff: Diffing extras/curses
cvs diff: Diffing extras/curses/sample
cvs diff: Diffing extras/dynamic_linking
cvs diff: Diffing extras/graphics
cvs diff: Diffing extras/graphics/mercury_opengl
cvs diff: Diffing extras/graphics/mercury_tcltk
cvs diff: Diffing extras/graphics/samples
cvs diff: Diffing extras/graphics/samples/calc
cvs diff: Diffing extras/graphics/samples/maze
cvs diff: Diffing extras/graphics/samples/pent
cvs diff: Diffing extras/lazy_evaluation
cvs diff: Diffing extras/logged_output
cvs diff: Diffing extras/moose
cvs diff: Diffing extras/moose/samples
cvs diff: Diffing extras/morphine
cvs diff: Diffing extras/morphine/non-regression-tests
cvs diff: Diffing extras/morphine/scripts
cvs diff: Diffing extras/morphine/source
cvs diff: Diffing extras/odbc
cvs diff: Diffing extras/posix
cvs diff: Diffing extras/references
cvs diff: Diffing extras/references/samples
cvs diff: Diffing extras/references/tests
cvs diff: Diffing extras/trailed_update
cvs diff: Diffing extras/trailed_update/samples
cvs diff: Diffing extras/trailed_update/tests
cvs diff: Diffing library
Index: library/io.m
===================================================================
RCS file: /home/mercury1/repository/mercury/library/io.m,v
retrieving revision 1.201
diff -u -r1.201 io.m
--- library/io.m	2000/08/11 16:50:15	1.201
+++ library/io.m	2000/08/22 02:08:23
@@ -1089,7 +1089,7 @@
 
 :- implementation.
 :- import_module map, dir, term, term_io, varset, require, benchmarking, array.
-:- import_module int, parser, exception.
+:- import_module int, mer_parser, exception.
 
 :- type io__state ---> io__state(c_pointer).
 	% Values of type `io__state' are never really used:
@@ -1717,7 +1717,8 @@
 	{ io__process_read_term(ReadResult, LineNumber, Result) }.
 
 io__read_from_string(FileName, String, Len, Result, Posn0, Posn) :-
-	parser__read_term_from_string(FileName, String, Len, Posn0, Posn, ReadResult),
+	mer_parser__read_term_from_string(FileName, String, Len,
+		Posn0, Posn, ReadResult),
 	Posn = posn(LineNumber, _, _),
 	io__process_read_term(ReadResult, LineNumber, Result).
 
Index: library/lexer.m
===================================================================
RCS file: lexer.m
diff -N lexer.m
--- /tmp/cvs2Yu1XU	Tue Aug 22 14:19:31 2000
+++ /dev/null	Thu Sep  2 15:00:04 1999
@@ -1,2000 +0,0 @@
-%---------------------------------------------------------------------------%
-% Copyright (C) 1993-2000 The University of Melbourne.
-% This file may only be copied under the terms of the GNU Library General
-% Public License - see the file COPYING.LIB in the Mercury distribution.
-%---------------------------------------------------------------------------%
-%
-% file: lexer.m.
-% main author: fjh.
-% stability: high.
-%
-% Lexical analysis.  This module defines the representation of tokens
-% and exports predicates for reading in tokens from an input stream.
-%
-% See ISO Prolog 6.4.  Also see the comments at the top of parser.m.
-%
-%-----------------------------------------------------------------------------%
-
-:- module lexer.
-:- interface.
-:- import_module io, char.
-
-:- type	token
-	--->	name(string)
-	;	variable(string)
-	;	integer(int)
-	;	float(float)
-	;	string(string)		% "...."
-	;	open			% '('
-	;	open_ct			% '(' without any preceding whitespace
-	;	close			% ')'
-	;	open_list		% '['
-	;	close_list		% ']'
-	;	open_curly		% '}'
-	;	close_curly		% '{'
-	;	ht_sep			% '|'
-	;	comma			% ','
-	;	end			% '.'
-	;	junk(char)		% junk character in the input stream
-	;	error(string)		% some other invalid token
-	;	io_error(io__error)	% error reading from the input stream
-	;	eof.			% end-of-file
-
-% For every token, we record the line number of the line on
-% which the token occurred.
-
-:- type token_context == int.	% line number
-
-% This "fat list" representation is more efficient than a list of pairs.
-:- type token_list	--->	token_cons(token, token_context, token_list)
-			;	token_nil.
-
-:- pred lexer__get_token_list(token_list, io__state, io__state).
-:- mode lexer__get_token_list(out, di, uo) is det.
-%	Read a list of tokens from the current input stream.
-%	Keep reading until we encounter either an `end' token
-%	(i.e. a full stop followed by whitespace) or the end-of-file.
-
-% The type `offset' represents a (zero-based) offset into a string.
-:- type offset == int.
-
-:- pred lexer__string_get_token_list(string, offset, token_list, posn, posn).
-:- mode lexer__string_get_token_list(in, in, out, in, out) is det.
-% lexer__string_get_token_list(String, MaxOffset, Tokens,
-%		InitialPos, FinalPos):
-%	Scan a list of tokens from a string,
-%	starting at the current offset specified by InitialPos.
-%	Keep scanning until either we encounter either an `end' token
-%	(i.e. a full stop followed by whitespace) or until
-%	we reach MaxOffset.  (MaxOffset must be =< the length of the string.)
-%	Return the tokens scanned in Tokens, and return the position
-%	one character past the end of the last token in FinalPos.
-
-:- pred lexer__string_get_token_list(string, token_list, posn, posn).
-:- mode lexer__string_get_token_list(in, out, in, out) is det.
-% lexer__string_get_token_list(String, Tokens, InitialPos, FinalPos):
-%	calls string_get_token_list/5 above with MaxPos = length of String.
-
-:- pred lexer__token_to_string(token, string).
-:- mode lexer__token_to_string(in, out) is det.
-%	Convert a token to a human-readable string describing the token.
-
-%-----------------------------------------------------------------------------%
-:- implementation.
-%-----------------------------------------------------------------------------%
-:- interface.
-
-	% lexer__graphic_token_char(Char): true iff `Char'
-	% is "graphic token char" (ISO Prolog 6.4.2).
-	% This is exported for use by term_io__quote_atom.
-:- pred lexer__graphic_token_char(char).
-:- mode lexer__graphic_token_char(in) is semidet.
-%-----------------------------------------------------------------------------%
-
-:- implementation.
-:- import_module list, term, require, string, int.
-
-%
-% Note that there are two implementations of most predicates here:
-% one which deals with strings, the other that deals with io__states.
-% We can't write the io__state version in terms of the string
-% version because we don't know how much string to slurp up
-% until after we've lexically analysed it.  Some interactive
-% applications require the old Prolog behaviour of stopping
-% after an end token (i.e. `.' plus whitespace) rather than
-% reading in whole lines.
-% Conversely, we can't write the string version using the io__state
-% version, since that would require either cheating with the io__state
-% or ruining the string interface.
-% 
-% An alternative would be to write both versions in terms
-% of a generic "char_stream" typeclass, with instances
-% for io__states and for strings.
-% However, for this to be acceptably efficient it would
-% require the compiler to specialize the code, which
-% currently (13 May 98) it is not capable of doing.
-%
-% In fact, the string version is still not as efficient as I would
-% like.  The compiler ought to (but currently doesn't) unfold all
-% the instances of the `posn' type.  We could do this type unfolding
-% by hand, but it would be very tedious and it would make the code
-% less readable.  If/when there is compiler support for this, we
-% should also think about moving the `String' and `Len' arguments
-% into the posn (or making a new `lexer_state' struct which contains
-% both the posn and the String and Len arguments).
-%
-
-%-----------------------------------------------------------------------------%
-
-lexer__token_to_string(name(Name), String) :-
-	string__append_list(["token '", Name, "'"], String).
-lexer__token_to_string(variable(Var), String) :-
-	string__append_list(["variable `", Var, "'"], String).
-lexer__token_to_string(integer(Int), String) :-
-	string__int_to_string(Int, IntString),
-	string__append_list(["integer `", IntString, "'"], String).
-lexer__token_to_string(float(Float), String) :-
-	string__float_to_string(Float, FloatString),
-	string__append_list(["float `", FloatString, "'"], String).
-lexer__token_to_string(string(Token), String) :-
-	string__append_list(["string """, Token, """"], String).
-lexer__token_to_string(open, "token ` ('").
-lexer__token_to_string(open_ct, "token `('").
-lexer__token_to_string(close, "token `)'").
-lexer__token_to_string(open_list, "token `['").
-lexer__token_to_string(close_list, "token `]'").
-lexer__token_to_string(open_curly, "token `{'").
-lexer__token_to_string(close_curly, "token `}'").
-lexer__token_to_string(ht_sep, "token `|'").
-lexer__token_to_string(comma, "token `,'").
-lexer__token_to_string(end, "token `. '").
-lexer__token_to_string(eof, "end-of-file").
-lexer__token_to_string(junk(JunkChar), String) :-
-	char__to_int(JunkChar, Code),
-	string__int_to_base_string(Code, 16, Hex),
-	string__append_list(["illegal character <<0x", Hex, ">>"], String).
-lexer__token_to_string(io_error(IO_Error), String) :-
-	io__error_message(IO_Error, IO_ErrorMessage),
-	string__append("I/O error: ", IO_ErrorMessage, String).
-lexer__token_to_string(error(Message), String) :-
-	string__append_list(["illegal token (", Message, ")"], String).
-
-	% We build the tokens up as lists of characters in reverse order.
-	% When we get to the end of each token, we call
-	% `lexer__rev_char_list_to_string/2' to convert that representation
-	% into a string.
-
-	% Comments of the form
-	%	foo --> bar . baz
-	% mean that we are parsing a `foo', and we've already scanned
-	% past the `bar', so now we need to match with a `baz'.
-
-lexer__get_token_list(Tokens) -->
-	lexer__get_token(Token, Context),
-	( { Token = eof } ->
-		{ Tokens = token_nil }
-	; { Token = end ; Token = error(_) ; Token = io_error(_) } ->
-		{ Tokens = token_cons(Token, Context, token_nil) }
-	;
-		{ Tokens = token_cons(Token, Context, Tokens1) },
-		lexer__get_token_list(Tokens1)
-	).
-
-lexer__string_get_token_list(String, Tokens) -->
-	{ string__length(String, Len) },
-	lexer__string_get_token_list(String, Len, Tokens).
-
-lexer__string_get_token_list(String, Len, Tokens) -->
-	lexer__string_get_token(String, Len, Token, Context),
-	( { Token = eof } ->
-		{ Tokens = token_nil }
-	; { Token = end ; Token = error(_) ; Token = io_error(_) } ->
-		{ Tokens = token_cons(Token, Context, token_nil) }
-	;
-		{ Tokens = token_cons(Token, Context, Tokens1) },
-		lexer__string_get_token_list(String, Len, Tokens1)
-	).
-
-%-----------------------------------------------------------------------------%
-
-% some low-level routines
-
-:- pred lexer__get_context(token_context, io__state, io__state).
-:- mode lexer__get_context(out, di, uo) is det.
-
-lexer__get_context(Context) -->
-	io__get_line_number(Context).
-
-:- type string_token_context == token_context.
-
-:- pred lexer__string_get_context(posn, string_token_context, posn, posn).
-:- mode lexer__string_get_context(in, out, in, out) is det.
-
-lexer__string_get_context(StartPosn, Context, EndPosn, EndPosn) :-
-	StartPosn = posn(StartLineNum, _, _),
-	Context = StartLineNum.
-	% In future, we might want to modify this code to read something
-	% like this:
-	%	posn_to_line_and_column(StartPosn, StartLineNum, StartColumn),
-	%	posn_to_line_and_column(EndPosn, EndLineNum, EndColumn),
-	%	Context = detailed(StartLine, StartColumn, EndLine, EndColumn).
-
-:- pred lexer__string_read_char(string, int, char, posn, posn).
-:- mode lexer__string_read_char(in, in, out, in, out) is semidet.
-
-:- pragma inline(lexer__string_read_char/5).
-
-lexer__string_read_char(String, Len, Char, Posn0, Posn) :-
-	Posn0 = posn(LineNum0, LineOffset0, Offset0),
-	Offset0 < Len,
-	string__unsafe_index(String, Offset0, Char),
-	Offset is Offset0 + 1,
-	( Char = '\n' ->
-		LineNum is LineNum0 + 1,
-		Posn = posn(LineNum, Offset, Offset)
-	;
-		Posn = posn(LineNum0, LineOffset0, Offset)
-	).
-
-:- pred lexer__string_ungetchar(string, posn, posn).
-:- mode lexer__string_ungetchar(in, in, out) is det.
-
-lexer__string_ungetchar(String, Posn0, Posn) :-
-	Posn0 = posn(LineNum0, LineOffset0, Offset0),
-	Offset is Offset0 - 1,
-	string__unsafe_index(String, Offset, Char),
-	( Char = '\n' ->
-		LineNum is LineNum0 - 1,
-		Posn = posn(LineNum, Offset, Offset)
-	;
-		Posn = posn(LineNum0, LineOffset0, Offset)
-	).
-
-:- pred lexer__grab_string(string, posn, string, posn, posn).
-:- mode lexer__grab_string(in, in, out, in, out) is det.
-
-lexer__grab_string(String, Posn0, SubString, Posn, Posn) :-
-	Posn0 = posn(_, _, Offset0),
-	Posn = posn(_, _, Offset),
-	Count is Offset - Offset0,
-	string__unsafe_substring(String, Offset0, Count, SubString).
-
-:- pred lexer__string_set_line_number(int, posn, posn).
-:- mode lexer__string_set_line_number(in, in, out) is det.
-
-lexer__string_set_line_number(LineNumber, Posn0, Posn) :-
-	Posn0 = posn(_, _, Offset),
-	Posn = posn(LineNumber, Offset, Offset).
-
-%-----------------------------------------------------------------------------%
-
-:- pred lexer__get_token(token, token_context, io__state, io__state).
-:- mode lexer__get_token(out, out, di, uo) is det.
-
-lexer__get_token(Token, Context) -->
-	io__read_char(Result),
-	( { Result = error(Error) },
-		lexer__get_context(Context),
-		{ Token = io_error(Error) }
-	; { Result = eof },
-		lexer__get_context(Context),
-		{ Token = eof }
-	; { Result = ok(Char) },
-		( { Char = ' ' ; Char = '\t' ; Char = '\n' } ->
-			lexer__get_token_2(Token, Context)
-		; { char__is_upper(Char) ; Char = '_' } ->
-			lexer__get_context(Context),
-			lexer__get_variable([Char], Token)
-		; { char__is_lower(Char) } ->
-			lexer__get_context(Context),
-			lexer__get_name([Char], Token)
-		; { Char = '0' } ->
-			lexer__get_context(Context),
-			lexer__get_zero(Token)
-		; { char__is_digit(Char) } ->
-			lexer__get_context(Context),
-			lexer__get_number([Char], Token)
-		; { lexer__special_token(Char, SpecialToken) } ->
-			lexer__get_context(Context),
-			{ SpecialToken = open ->
-				Token = open_ct
-			;
-				Token = SpecialToken
-			}
-		; { Char = ('.') } ->
-			lexer__get_context(Context),
-			lexer__get_dot(Token)
-		; { Char = ('%') } ->
-			lexer__skip_to_eol(Token, Context)
-		; { Char = '"' ; Char = '''' } ->
-			lexer__get_context(Context),
-			lexer__get_quoted_name(Char, [], Token)
-		; { Char = ('/') } ->
-			lexer__get_slash(Token, Context)
-		; { Char = ('#') } ->
-			lexer__get_source_line_number([], Token, Context)
-		; { Char = ('`') } ->
-			lexer__get_context(Context),
-			{ Token = name("`") }
-		; { lexer__graphic_token_char(Char) } ->
-			lexer__get_context(Context),
-			lexer__get_graphic([Char], Token)
-		;
-			lexer__get_context(Context),
-			{ Token = junk(Char) }
-		)
-	).
-
-:- pred lexer__string_get_token(string, int, token, token_context, posn, posn).
-:- mode lexer__string_get_token(in, in, out, out, in, out) is det.
-
-lexer__string_get_token(String, Len, Token, Context) -->
-	=(Posn0),
-	( lexer__string_read_char(String, Len, Char) ->
-		( { Char = ' ' ; Char = '\t' ; Char = '\n' } ->
-			lexer__string_get_token_2(String, Len, Token, Context)
-		; { char__is_upper(Char) ; Char = '_' } ->
-			lexer__string_get_variable(String, Len, Posn0,
-				Token, Context)
-		; { char__is_lower(Char) } ->
-			lexer__string_get_name(String, Len, Posn0,
-				Token, Context)
-		; { Char = '0' } ->
-			lexer__string_get_zero(String, Len, Posn0,
-				Token, Context)
-		; { char__is_digit(Char) } ->
-			lexer__string_get_number(String, Len, Posn0,
-				Token, Context)
-		; { lexer__special_token(Char, SpecialToken) } ->
-			lexer__string_get_context(Posn0, Context),
-			{ SpecialToken = open ->
-				Token = open_ct
-			;
-				Token = SpecialToken
-			}
-		; { Char = ('.') } ->
-			lexer__string_get_dot(String, Len, Posn0,
-				Token, Context)
-		; { Char = ('%') } ->
-			lexer__string_skip_to_eol(String, Len, Token, Context)
-		; { Char = '"' ; Char = '''' } ->
-			lexer__string_get_quoted_name(String, Len, Char, [], 
-				Posn0, Token, Context)
-		; { Char = ('/') } ->
-			lexer__string_get_slash(String, Len, Posn0,
-				Token, Context)
-		; { Char = ('#') } ->
-			=(Posn1),
-			lexer__string_get_source_line_number(String, Len,
-				Posn1, Token, Context)
-		; { Char = ('`') } ->
-			lexer__string_get_context(Posn0, Context),
-			{ Token = name("`") }
-		; { lexer__graphic_token_char(Char) } ->
-			lexer__string_get_graphic(String, Len, Posn0,
-				Token, Context)
-		;
-			lexer__string_get_context(Posn0, Context),
-			{ Token = junk(Char) }
-		)
-	;
-		lexer__string_get_context(Posn0, Context),
-		{ Token = eof }
-	).
-
-%-----------------------------------------------------------------------------%
-
-:- pred lexer__get_token_2(token, token_context, io__state, io__state).
-:- mode lexer__get_token_2(out, out, di, uo) is det.
-
-	% This is just like get_token, except that we have already
-	% scanned past some whitespace, so '(' gets scanned as `open'
-	% rather than `open_ct'.
-
-lexer__get_token_2(Token, Context) -->
-	io__read_char(Result),
-	( { Result = error(Error) },
-		lexer__get_context(Context),
-		{ Token = io_error(Error) }
-	; { Result = eof },
-		lexer__get_context(Context),
-		{ Token = eof }
-	; { Result = ok(Char) },
-		( { Char = ' ' ; Char = '\t' ; Char = '\n' } ->
-			lexer__get_token_2(Token, Context)
-		; { char__is_upper(Char) ; Char = '_' } ->
-			lexer__get_context(Context),
-			lexer__get_variable([Char], Token)
-		; { char__is_lower(Char) } ->
-			lexer__get_context(Context),
-			lexer__get_name([Char], Token)
-		; { Char = '0' } ->
-			lexer__get_context(Context),
-			lexer__get_zero(Token)
-		; { char__is_digit(Char) } ->
-			lexer__get_context(Context),
-			lexer__get_number([Char], Token)
-		; { lexer__special_token(Char, SpecialToken) } ->
-			lexer__get_context(Context),
-			{ Token = SpecialToken }
-		; { Char = ('.') } ->
-			lexer__get_context(Context),
-			lexer__get_dot(Token)
-		; { Char = ('%') } ->
-			lexer__skip_to_eol(Token, Context)
-		; { Char = '"' ; Char = '''' } ->
-			lexer__get_context(Context),
-			lexer__get_quoted_name(Char, [], Token)
-		; { Char = ('/') } ->
-			lexer__get_slash(Token, Context)
-		; { Char = ('#') } ->
-			lexer__get_source_line_number([], Token, Context)
-		; { Char = ('`') } ->
-			lexer__get_context(Context),
-			{ Token = name("`") }
-		; { lexer__graphic_token_char(Char) } ->
-			lexer__get_context(Context),
-			lexer__get_graphic([Char], Token)
-		;
-			lexer__get_context(Context),
-			{ Token = junk(Char) }
-		)
-	).
-
-:- pred lexer__string_get_token_2(string, int, token, token_context,
-					posn, posn).
-:- mode lexer__string_get_token_2(in, in, out, out, in, out) is det.
-
-lexer__string_get_token_2(String, Len, Token, Context) -->
-	=(Posn0),
-	( lexer__string_read_char(String, Len, Char) ->
-		( { Char = ' ' ; Char = '\t' ; Char = '\n' } ->
-			lexer__string_get_token_2(String, Len, Token, Context)
-		; { char__is_upper(Char) ; Char = '_' } ->
-			lexer__string_get_variable(String, Len, Posn0,
-				Token, Context)
-		; { char__is_lower(Char) } ->
-			lexer__string_get_name(String, Len, Posn0,
-				Token, Context)
-		; { Char = '0' } ->
-			lexer__string_get_zero(String, Len, Posn0,
-				Token, Context)
-		; { char__is_digit(Char) } ->
-			lexer__string_get_number(String, Len, Posn0,
-				Token, Context)
-		; { lexer__special_token(Char, SpecialToken) } ->
-			lexer__string_get_context(Posn0, Context),
-			{ Token = SpecialToken }
-		; { Char = ('.') } ->
-			lexer__string_get_dot(String, Len, Posn0,
-				Token, Context)
-		; { Char = ('%') } ->
-			lexer__string_skip_to_eol(String, Len, Token, Context)
-		; { Char = '"' ; Char = '''' } ->
-			lexer__string_get_quoted_name(String, Len, Char, [],
-				Posn0, Token, Context)
-		; { Char = ('/') } ->
-			lexer__string_get_slash(String, Len, Posn0,
-				Token, Context)
-		; { Char = ('#') } ->
-			=(Posn1),
-			lexer__string_get_source_line_number(String, Len,
-				Posn1, Token, Context)
-		; { Char = ('`') } ->
-			lexer__string_get_context(Posn0, Context),
-			{ Token = name("`") }
-		; { lexer__graphic_token_char(Char) } ->
-			lexer__string_get_graphic(String, Len, Posn0,
-				Token, Context)
-		;
-			lexer__string_get_context(Posn0, Context),
-			{ Token = junk(Char) }
-		)
-	;
-		lexer__string_get_context(Posn0, Context),
-		{ Token = eof }
-	).
-
-%-----------------------------------------------------------------------------%
-
-%-----------------------------------------------------------------------------%
-
-:- pred lexer__special_token(char, token).
-:- mode lexer__special_token(in, out) is semidet.
-
-lexer__special_token('(', open).	% May get converted to open_ct
-lexer__special_token(')', close).
-lexer__special_token('[', open_list).
-lexer__special_token(']', close_list).
-lexer__special_token('{', open_curly).
-lexer__special_token('}', close_curly).
-lexer__special_token('|', ht_sep).
-lexer__special_token(',', comma).
-lexer__special_token(';', name(";")).
-lexer__special_token('!', name("!")).
-
-lexer__graphic_token_char('#').
-lexer__graphic_token_char('$').
-lexer__graphic_token_char('&').
-lexer__graphic_token_char('*').
-lexer__graphic_token_char('+').
-lexer__graphic_token_char('-').
-lexer__graphic_token_char('.').
-lexer__graphic_token_char('/').
-lexer__graphic_token_char(':').
-lexer__graphic_token_char('<').
-lexer__graphic_token_char('=').
-lexer__graphic_token_char('>').
-lexer__graphic_token_char('?').
-lexer__graphic_token_char('@').
-lexer__graphic_token_char('^').
-lexer__graphic_token_char('~').
-lexer__graphic_token_char('\\').
-
-%-----------------------------------------------------------------------------%
-
-:- pred lexer__get_dot(token, io__state, io__state).
-:- mode lexer__get_dot(out, di, uo) is det.
-
-lexer__get_dot(Token) -->
-	io__read_char(Result),
-	( { Result = error(Error) },
-		{ Token = io_error(Error) }
-	; { Result = eof },
-		{ Token = end }
-	; { Result = ok(Char) },
-		( { lexer__whitespace_after_dot(Char) } ->
-			io__putback_char(Char),
-			{ Token = end }
-		; { lexer__graphic_token_char(Char) } ->
-			lexer__get_graphic([Char, '.'], Token)
-		;
-			io__putback_char(Char),
-			{ Token = name(".") }
-		)
-	).
-
-:- pred lexer__string_get_dot(string, int, posn, token, string_token_context,
-				posn, posn).
-:- mode lexer__string_get_dot(in, in, in, out, out, in, out) is det.
-
-lexer__string_get_dot(String, Len, Posn0, Token, Context) -->
-	( lexer__string_read_char(String, Len, Char) ->
-		( { lexer__whitespace_after_dot(Char) } ->
-			lexer__string_ungetchar(String),
-			lexer__string_get_context(Posn0, Context),
-			{ Token = end }
-		; { lexer__graphic_token_char(Char) } ->
-			lexer__string_get_graphic(String, Len, Posn0,
-				Token, Context)
-		;
-			lexer__string_ungetchar(String),
-			lexer__string_get_context(Posn0, Context),
-			{ Token = name(".") }
-		)
-	;
-		lexer__string_get_context(Posn0, Context),
-		{ Token = end }
-	).
-
-:- pred lexer__whitespace_after_dot(char).
-:- mode lexer__whitespace_after_dot(in) is semidet.
-
-lexer__whitespace_after_dot(' ').
-lexer__whitespace_after_dot('\t').
-lexer__whitespace_after_dot('\n').
-lexer__whitespace_after_dot('%').
-
-%-----------------------------------------------------------------------------%
-
-	% comments
-
-:- pred lexer__skip_to_eol(token, token_context, io__state, io__state).
-:- mode lexer__skip_to_eol(out, out, di, uo) is det.
-
-lexer__skip_to_eol(Token, Context) -->
-	io__read_char(Result),
-	( { Result = error(Error) },
-		lexer__get_context(Context),
-		{ Token = io_error(Error) }
-	; { Result = eof },
-		lexer__get_context(Context),
-		{ Token = eof }
-	; { Result = ok(Char) },
-		( { Char = '\n' } ->
-			lexer__get_token_2(Token, Context)
-		;
-			lexer__skip_to_eol(Token, Context)
-		)
-	).
-
-:- pred lexer__string_skip_to_eol(string, int, token, token_context,
-					posn, posn).
-:- mode lexer__string_skip_to_eol(in, in, out, out, in, out) is det.
-
-lexer__string_skip_to_eol(String, Len, Token, Context) -->
-	( lexer__string_read_char(String, Len, Char) ->
-		( { Char = '\n' } ->
-			lexer__string_get_token_2(String, Len, Token, Context)
-		;
-			lexer__string_skip_to_eol(String, Len, Token, Context)
-		)
-	;
-		=(Posn),
-		lexer__string_get_context(Posn, Context),
-		{ Token = eof }
-	).
-
-:- pred lexer__get_slash(token, token_context, io__state, io__state).
-:- mode lexer__get_slash(out, out, di, uo) is det.
-
-lexer__get_slash(Token, Context) -->
-	io__read_char(Result),
-	( { Result = error(Error) },
-		lexer__get_context(Context),
-		{ Token = io_error(Error) }
-	; { Result = eof },
-		lexer__get_context(Context),
-		{ Token = name("/") }
-	; { Result = ok(Char) },
-		( { Char = ('*') } ->
-			lexer__get_comment(Token, Context)
-		; { lexer__graphic_token_char(Char) } ->
-			lexer__get_context(Context),
-			lexer__get_graphic([Char, '/'], Token)
-		;
-			io__putback_char(Char),
-			lexer__get_context(Context),
-			{ Token = name("/") }
-		)
-	).
-
-:- pred lexer__string_get_slash(string, int, posn, token, string_token_context,
-				posn, posn).
-:- mode lexer__string_get_slash(in, in, in, out, out, in, out) is det.
-
-lexer__string_get_slash(String, Len, Posn0, Token, Context) -->
-	( lexer__string_read_char(String, Len, Char) ->
-		( { Char = ('*') } ->
-			lexer__string_get_comment(String, Len, Posn0,
-				Token, Context)
-		; { lexer__graphic_token_char(Char) } ->
-			lexer__string_get_graphic(String, Len, Posn0,
-				Token, Context)
-		;
-			lexer__string_ungetchar(String),
-			lexer__string_get_context(Posn0, Context),
-			{ Token = name("/") }
-		)
-	;
-		lexer__string_get_context(Posn0, Context),
-		{ Token = name("/") }
-	).
-
-:- pred lexer__get_comment(token, token_context, io__state, io__state).
-:- mode lexer__get_comment(out, out, di, uo) is det.
-
-lexer__get_comment(Token, Context) -->
-	io__read_char(Result),
-	( { Result = error(Error) },
-		lexer__get_context(Context),
-		{ Token = io_error(Error) }
-	; { Result = eof },
-		lexer__get_context(Context),
-		{ Token = error("unterminated '/*' comment") }
-	; { Result = ok(Char) },
-		( { Char = ('*') } ->
-			lexer__get_comment_2(Token, Context)
-		;
-			lexer__get_comment(Token, Context)
-		)
-	).
-
-:- pred lexer__string_get_comment(string, int, posn, token,
-				string_token_context, posn, posn).
-:- mode lexer__string_get_comment(in, in, in, out, out, in, out) is det.
-
-lexer__string_get_comment(String, Len, Posn0, Token, Context) -->
-	( lexer__string_read_char(String, Len, Char) ->
-		( { Char = ('*') } ->
-			lexer__string_get_comment_2(String, Len, Posn0,
-					Token, Context)
-		;
-			lexer__string_get_comment(String, Len, Posn0,
-					Token, Context)
-		)
-	;
-		lexer__string_get_context(Posn0, Context),
-		{ Token = error("unterminated '/*' comment") }
-	).
-
-:- pred lexer__get_comment_2(token, token_context, io__state, io__state).
-:- mode lexer__get_comment_2(out, out, di, uo) is det.
-
-lexer__get_comment_2(Token, Context) -->
-	io__read_char(Result),
-	( { Result = error(Error) },
-		lexer__get_context(Context),
-		{ Token = io_error(Error) }
-	; { Result = eof },
-		lexer__get_context(Context),
-		{ Token = error("unterminated '/*' comment") }
-	; { Result = ok(Char) },
-		( { Char = ('/') } ->
-			% end of /* ... */ comment, so get next token
-			lexer__get_token_2(Token, Context)
-		; { Char = ('*') } ->
-			lexer__get_comment_2(Token, Context)
-		;
-			lexer__get_comment(Token, Context)
-		)
-	).
-
-:- pred lexer__string_get_comment_2(string, int, posn, token,
-				string_token_context, posn, posn).
-:- mode lexer__string_get_comment_2(in, in, in, out, out, in, out) is det.
-
-lexer__string_get_comment_2(String, Len, Posn0, Token, Context) -->
-	( lexer__string_read_char(String, Len, Char) ->
-		( { Char = ('/') } ->
-			% end of /* ... */ comment, so get next token
-			lexer__string_get_token_2(String, Len, Token, Context)
-		; { Char = ('*') } ->
-			lexer__string_get_comment_2(String, Len, Posn0,
-				Token, Context)
-		;
-			lexer__string_get_comment(String, Len, Posn0,
-				Token, Context)
-		)
-	;
-		lexer__string_get_context(Posn0, Context),
-		{ Token = error("unterminated '/*' comment") }
-	).
-
-%-----------------------------------------------------------------------------%
-
-	% quoted names and quoted strings
-
-:- pred lexer__get_quoted_name(char, list(char), token,
-				io__state, io__state).
-:- mode lexer__get_quoted_name(in, in, out, di, uo) is det.
-
-lexer__get_quoted_name(QuoteChar, Chars, Token) -->
-	io__read_char(Result),
-	( { Result = error(Error) },
-		{ Token = io_error(Error) }
-	; { Result = eof },
-		{ Token = error("unterminated quote") }
-	; { Result = ok(Char) },
-		( { Char = QuoteChar } ->
-			lexer__get_quoted_name_quote(QuoteChar, Chars, Token)
-		; { Char = ('\\') } ->
-			lexer__get_quoted_name_escape(QuoteChar, Chars, Token)
-		;
-			lexer__get_quoted_name(QuoteChar, [Char | Chars], Token)
-		)
-	).
-
-:- pred lexer__string_get_quoted_name(string, int, char, list(char), posn,
-				token, string_token_context, posn, posn).
-:- mode lexer__string_get_quoted_name(in, in, in, in, in, out, out, in, out)
-	is det.
-
-lexer__string_get_quoted_name(String, Len, QuoteChar, Chars, Posn0,
-		Token, Context) -->
-	( lexer__string_read_char(String, Len, Char) ->
-		( { Char = QuoteChar } ->
-			lexer__string_get_quoted_name_quote(String, Len,
-				QuoteChar, Chars, Posn0, Token, Context)
-		; { Char = ('\\') } ->
-			lexer__string_get_quoted_name_escape(String, Len,
-				QuoteChar, Chars, Posn0, Token, Context)
-		;
-			lexer__string_get_quoted_name(String, Len, QuoteChar,
-				[Char | Chars], Posn0, Token, Context)
-		)
-	;
-		lexer__string_get_context(Posn0, Context),
-		{ Token = error("unterminated quote") }
-	).
-
-:- pred lexer__get_quoted_name_quote(char, list(char), token,
-				io__state, io__state).
-:- mode lexer__get_quoted_name_quote(in, in, out, di, uo) is det.
-
-lexer__get_quoted_name_quote(QuoteChar, Chars, Token) -->
-	io__read_char(Result),
-	( { Result = error(Error) },
-		{ Token = io_error(Error) }
-	; { Result = eof },
-		{ lexer__finish_quoted_name(QuoteChar, Chars, Token) }
-	; { Result = ok(Char) },
-		( { Char = QuoteChar } ->
-			lexer__get_quoted_name(QuoteChar, [Char | Chars], Token)
-		;
-			io__putback_char(Char),
-			{ lexer__finish_quoted_name(QuoteChar, Chars, Token) }
-		)
-	).
-
-:- pred lexer__string_get_quoted_name_quote(string, int, char, list(char),
-				posn, token, string_token_context, posn, posn).
-:- mode lexer__string_get_quoted_name_quote(in, in, in, in, in, out, out,
-				in, out) is det.
-
-lexer__string_get_quoted_name_quote(String, Len, QuoteChar, Chars, Posn0,
-		Token, Context) -->
-	( lexer__string_read_char(String, Len, Char) ->
-		( { Char = QuoteChar } ->
-			lexer__string_get_quoted_name(String, Len, QuoteChar,
-				[Char | Chars], Posn0, Token, Context)
-		;
-			lexer__string_ungetchar(String),
-			lexer__string_get_context(Posn0, Context),
-			{ lexer__finish_quoted_name(QuoteChar, Chars, Token) }
-		)
-	;
-		lexer__string_get_context(Posn0, Context),
-		{ lexer__finish_quoted_name(QuoteChar, Chars, Token) }
-	).
-
-:- pred lexer__finish_quoted_name(char, list(char), token).
-:- mode lexer__finish_quoted_name(in, in, out) is det.
-
-lexer__finish_quoted_name(QuoteChar, Chars, Token) :-
-	lexer__rev_char_list_to_string(Chars, String),
-	( QuoteChar = '''' ->
-		Token = name(String)
-	; QuoteChar = '"' ->
-		Token = string(String)
-	;
-		error("lexer.m: unknown quote character")
-	).
-
-:- pred lexer__get_quoted_name_escape(char, list(char), token,
-					io__state, io__state).
-:- mode lexer__get_quoted_name_escape(in, in, out, di, uo) is det.
-
-lexer__get_quoted_name_escape(QuoteChar, Chars, Token) -->
-	io__read_char(Result),
-	( { Result = error(Error) },
-		{ Token = io_error(Error) }
-	; { Result = eof },
-		{ Token = error("unterminated quoted name") }
-	; { Result = ok(Char) },
-		( { Char = '\n' } ->
-			lexer__get_quoted_name(QuoteChar, Chars, Token)
-		; { lexer__escape_char(Char, EscapedChar) } ->
-			{ Chars1 = [EscapedChar | Chars] },
-			lexer__get_quoted_name(QuoteChar, Chars1, Token)
-		; { Char = 'x' } ->
-			lexer__get_hex_escape(QuoteChar, Chars, [], Token)
-		; { char__is_octal_digit(Char) } ->
-			lexer__get_octal_escape(QuoteChar, Chars, [Char],
-				Token)
-		;
-			{ Token = error("invalid escape character") }
-		)
-	).
-
-:- pred lexer__string_get_quoted_name_escape(string, int, char, list(char),
-					posn, token, string_token_context,
-					posn, posn).
-:- mode lexer__string_get_quoted_name_escape(in, in, in, in, in, out, out,
-					in, out) is det.
-
-lexer__string_get_quoted_name_escape(String, Len, QuoteChar, Chars, Posn0,
-		Token, Context) -->
-	=(Posn1),
-	( lexer__string_read_char(String, Len, Char) ->
-		( { Char = '\n' } ->
-			lexer__string_get_quoted_name(String, Len, QuoteChar,
-				Chars, Posn0, Token, Context)
-		; { lexer__escape_char(Char, EscapedChar) } ->
-			{ Chars1 = [EscapedChar | Chars] },
-			lexer__string_get_quoted_name(String, Len, QuoteChar,
-				Chars1, Posn0, Token, Context)
-		; { Char = 'x' } ->
-			lexer__string_get_hex_escape(String, Len, QuoteChar,
-				Chars, [], Posn0, Token, Context)
-		; { char__is_octal_digit(Char) } ->
-			lexer__string_get_octal_escape(String, Len, QuoteChar,
-				Chars, [Char], Posn0, Token, Context)
-		;
-			lexer__string_get_context(Posn1, Context),
-			{ Token = error("invalid escape character") }
-		)
-	;
-		lexer__string_get_context(Posn0, Context),
-		{ Token = error("unterminated quoted name") }
-	).
-
-:- pred lexer__escape_char(char, char).
-:- mode lexer__escape_char(in, out) is semidet.
-
-lexer__escape_char('a', '\a').
-lexer__escape_char('b', '\b').
-lexer__escape_char('r', '\r').
-lexer__escape_char('f', '\f').
-lexer__escape_char('t', '\t').
-lexer__escape_char('n', '\n').
-lexer__escape_char('v', '\v').
-lexer__escape_char('\\', '\\').
-lexer__escape_char('''', '''').
-lexer__escape_char('"', '"').
-lexer__escape_char('`', '`').
-
-:- pred lexer__get_hex_escape(char, list(char), list(char),
-				token, io__state, io__state).
-:- mode lexer__get_hex_escape(in, in, in, out, di, uo) is det.
-
-lexer__get_hex_escape(QuoteChar, Chars, HexChars, Token) -->
-	io__read_char(Result),
-	( { Result = error(Error) },
-		{ Token = io_error(Error) }
-	; { Result = eof },
-		{ Token = error("unterminated quote") }
-	; { Result = ok(Char) },
-		( { char__is_hex_digit(Char) } ->
-			lexer__get_hex_escape(QuoteChar, Chars,
-						[Char | HexChars], Token)
-		; { Char = ('\\') } ->
-			lexer__finish_hex_escape(QuoteChar, Chars, HexChars,
-				Token)
-		;
-			{ Token = error("unterminated hex escape") }
-		)
-	).
-
-:- pred lexer__string_get_hex_escape(string, int, char, list(char), list(char),
-				posn, token, string_token_context, posn, posn).
-:- mode lexer__string_get_hex_escape(in, in, in, in, in, in, out, out, in, out)
-	is det.
-
-lexer__string_get_hex_escape(String, Len, QuoteChar, Chars, HexChars, Posn0,
-		Token, Context) -->
-	( lexer__string_read_char(String, Len, Char) ->
-		( { char__is_hex_digit(Char) } ->
-			lexer__string_get_hex_escape(String, Len, QuoteChar,
-				    Chars, [Char | HexChars], Posn0,
-				    Token, Context)
-		; { Char = ('\\') } ->
-			lexer__string_finish_hex_escape(String, Len, QuoteChar,
-				    Chars, HexChars, Posn0, Token, Context)
-		;
-			lexer__string_get_context(Posn0, Context),
-			{ Token = error("unterminated hex escape") }
-		)
-	;
-		lexer__string_get_context(Posn0, Context),
-		{ Token = error("unterminated quote") }
-	).
-
-:- pred lexer__finish_hex_escape(char, list(char), list(char),
-				token, io__state, io__state).
-:- mode lexer__finish_hex_escape(in, in, in, out, di, uo) is det.
-
-lexer__finish_hex_escape(QuoteChar, Chars, HexChars, Token) -->
-	( { HexChars = [] } ->
-		{ Token = error("empty hex escape") }
-	;
-		{ lexer__rev_char_list_to_string(HexChars, HexString) },
-		(
-			{ string__base_string_to_int(16, HexString, Int) },
-			{ char__to_int(Char, Int) }
-		->
-			lexer__get_quoted_name(QuoteChar, [Char|Chars], Token) 
-		;
-			{ Token = error("invalid hex escape") }
-		)
-	).
-
-:- pred lexer__string_finish_hex_escape(string, int, char, list(char),
-				list(char), posn, token, string_token_context,
-				posn, posn).
-:- mode lexer__string_finish_hex_escape(in, in, in, in, in, in, out, out, in, out)
-				is det.
-
-lexer__string_finish_hex_escape(String, Len, QuoteChar, Chars, HexChars, Posn0,
-		Token, Context) -->
-	( { HexChars = [] } ->
-		lexer__string_get_context(Posn0, Context),
-		{ Token = error("empty hex escape") }
-	;
-		{ lexer__rev_char_list_to_string(HexChars, HexString) },
-		(
-			{ string__base_string_to_int(16, HexString, Int) },
-			{ char__to_int(Char, Int) }
-		->
-			lexer__string_get_quoted_name(String, Len, QuoteChar,
-				[Char|Chars], Posn0, Token, Context)
-		;
-			lexer__string_get_context(Posn0, Context),
-			{ Token = error("invalid hex escape") }
-		)
-	).
-
-:- pred lexer__get_octal_escape(char, list(char), list(char),
-				token, io__state, io__state).
-:- mode lexer__get_octal_escape(in, in, in, out, di, uo) is det.
-
-lexer__get_octal_escape(QuoteChar, Chars, OctalChars, Token) -->
-	io__read_char(Result),
-	( { Result = error(Error) },
-		{ Token = io_error(Error) }
-	; { Result = eof },
-		{ Token = error("unterminated quote") }
-	; { Result = ok(Char) },
-		( { char__is_octal_digit(Char) } ->
-			lexer__get_octal_escape(QuoteChar, Chars,
-						[Char | OctalChars], Token)
-		; { Char = ('\\') } ->
-			lexer__finish_octal_escape(QuoteChar, Chars, OctalChars,
-				Token)
-		;
-			/****** 
-				% We don't report this as an error since
-				% we need bug-for-bug compatibility with
-				% NU-Prolog
-			{ Token = error("unterminated octal escape") }
-			******/
-			io__putback_char(Char),
-			lexer__finish_octal_escape(QuoteChar, Chars, OctalChars,
-				Token)
-		)
-	).
-
-:- pred lexer__string_get_octal_escape(string, int, char, list(char),
-				list(char), posn, token, string_token_context,
-				posn, posn).
-:- mode lexer__string_get_octal_escape(in, in, in, in, in, in, out, out,
-				in, out) is det.
-
-lexer__string_get_octal_escape(String, Len, QuoteChar, Chars, OctalChars,
-		Posn0, Token, Context) -->
-	( lexer__string_read_char(String, Len, Char) ->
-		( { char__is_octal_digit(Char) } ->
-			lexer__string_get_octal_escape(String, Len,
-				QuoteChar, Chars, [Char | OctalChars], Posn0,
-				Token, Context)
-		; { Char = ('\\') } ->
-			lexer__string_finish_octal_escape(String, Len,
-				QuoteChar, Chars, OctalChars, Posn0,
-				Token, Context)
-		;
-			/****** 
-				% We don't report this as an error since
-				% we need bug-for-bug compatibility with
-				% NU-Prolog
-			{ Token = error("unterminated octal escape") }
-			******/
-			lexer__string_ungetchar(String),
-			lexer__string_finish_octal_escape(String, Len,
-				QuoteChar, Chars, OctalChars, Posn0,
-				Token, Context)
-		)
-	;
-		{ Token = error("unterminated quote") },
-		lexer__string_get_context(Posn0, Context)
-	).
-
-:- pred lexer__finish_octal_escape(char, list(char), list(char),
-				token, io__state, io__state).
-:- mode lexer__finish_octal_escape(in, in, in, out, di, uo) is det.
-
-lexer__finish_octal_escape(QuoteChar, Chars, OctalChars, Token) -->
-	( { OctalChars = [] } ->
-		{ Token = error("empty octal escape") }
-	;
-		{ lexer__rev_char_list_to_string(OctalChars, OctalString) },
-		(
-			{ string__base_string_to_int(8, OctalString, Int) },
-			{ char__to_int(Char, Int) }
-		->
-			lexer__get_quoted_name(QuoteChar, [Char|Chars], Token) 
-		;
-			{ Token = error("invalid octal escape") }
-		)
-	).
-
-:- pred lexer__string_finish_octal_escape(string, int, char, list(char),
-				list(char), posn, token,
-				string_token_context, posn, posn).
-:- mode lexer__string_finish_octal_escape(in, in, in, in, in, in, out, out,
-				in, out) is det.
-
-lexer__string_finish_octal_escape(String, Len, QuoteChar, Chars, OctalChars,
-		Posn0, Token, Context) -->
-	( { OctalChars = [] } ->
-		{ Token = error("empty octal escape") },
-		lexer__string_get_context(Posn0, Context)
-	;
-		{ lexer__rev_char_list_to_string(OctalChars, OctalString) },
-		(
-			{ string__base_string_to_int(8, OctalString, Int) },
-			{ char__to_int(Char, Int) }
-		->
-			lexer__string_get_quoted_name(String, Len, QuoteChar,
-				[Char|Chars], Posn0, Token, Context) 
-		;
-			{ Token = error("invalid octal escape") },
-			lexer__string_get_context(Posn0, Context)
-		)
-	).
-
-%-----------------------------------------------------------------------------%
-
-	% names and variables
-
-:- pred lexer__get_name(list(char), token, io__state, io__state).
-:- mode lexer__get_name(in, out, di, uo) is det.
-
-lexer__get_name(Chars, Token) -->
-	io__read_char(Result),
-	( { Result = error(Error) },
-		{ Token = io_error(Error) }
-	; { Result = eof },
-		{ lexer__rev_char_list_to_string(Chars, Name) },
-		{ Token = name(Name) }
-	; { Result = ok(Char) },
-		( { char__is_alnum_or_underscore(Char) } ->
-			lexer__get_name([Char | Chars], Token)
-		;
-			io__putback_char(Char),
-			{ lexer__rev_char_list_to_string(Chars, Name) },
-			{ Token = name(Name) }
-		)
-	).
-
-:- pred lexer__string_get_name(string, int, posn, token, string_token_context,
-				posn, posn).
-:- mode lexer__string_get_name(in, in, in, out, out, in, out) is det.
-
-lexer__string_get_name(String, Len, Posn0, Token, Context) -->
-	( lexer__string_read_char(String, Len, Char) ->
-		( { char__is_alnum_or_underscore(Char) } ->
-			lexer__string_get_name(String, Len, Posn0,
-				Token, Context)
-		;
-			lexer__string_ungetchar(String),
-			lexer__grab_string(String, Posn0, Name),
-			{ Token = name(Name) },
-			lexer__string_get_context(Posn0, Context)
-		)
-	;
-		lexer__grab_string(String, Posn0, Name),
-		{ Token = name(Name) },
-		lexer__string_get_context(Posn0, Context)
-	).
-
-	%
-	% A line number directive token is `#' followed by an integer
-	% (specifying the line number) followed by a newline.
-	% Such a token sets the source line number for the next
-	% line, but it is otherwise ignored.  This means that line number
-	% directives may appear anywhere that a token may appear, including
-	% in the middle of terms.
-	% (The source file name can be set with a `:- pragma source_file' 
-	% declaration.)
-	%
-
-:- pred lexer__get_source_line_number(list(char), token, token_context,
-	io__state, io__state).
-:- mode lexer__get_source_line_number(in, out, out, di, uo) is det.
-
-lexer__get_source_line_number(Chars, Token, Context) -->
-	io__read_char(Result),
-	( { Result = error(Error) },
-		lexer__get_context(Context),
-		{ Token = io_error(Error) }
-	; { Result = eof },
-		lexer__get_context(Context),
-		{ Token = error(
-			"unexpected end-of-file in `#' line number directive") }
-	; { Result = ok(Char) },
-		( { char__is_digit(Char) } ->
-			lexer__get_source_line_number([Char | Chars],
-				Token, Context)
-		; { Char = '\n' } ->
-			{ lexer__rev_char_list_to_string(Chars, String) },
-			(
-				{ string__base_string_to_int(10, String, Int) },
-				{ Int > 0 }
-			->
-				io__set_line_number(Int),
-				lexer__get_token(Token, Context)
-			;
-				lexer__get_context(Context),
-				{ string__append_list([
-					"invalid line number `", String,
-					"' in `#' line number directive"],
-					Message) },
-				{ Token = error(Message) }
-			)
-		;
-			lexer__get_context(Context),
-			{ string__from_char_list([Char], String) },
-			{ string__append_list([
-				"invalid character `", String,
-				"' in `#' line number directive"],
-				Message) },
-			{ Token = error(Message) }
-		)
-	).
-
-:- pred lexer__string_get_source_line_number(string, int, posn,
-		token, token_context, posn, posn).
-:- mode lexer__string_get_source_line_number(in, in, in, out, out, in, out)
-		is det.
-
-lexer__string_get_source_line_number(String, Len, Posn1, Token, Context) -->
-	( lexer__string_read_char(String, Len, Char) ->
-		( { char__is_digit(Char) } ->
-			lexer__string_get_source_line_number(String, Len,
-				Posn1, Token, Context)
-		; { Char = '\n' } ->
-			lexer__grab_string(String, Posn1, LineNumString),
-			(
-				{ string__base_string_to_int(10, LineNumString,
-					LineNum) },
-				{ LineNum > 0 }
-			->
-				lexer__string_set_line_number(LineNum),
-				lexer__string_get_token(String, Len, Token, Context)
-			;
-				lexer__string_get_context(Posn1, Context),
-				{ string__append_list([
-					"invalid line number `", LineNumString,
-					"' in `#' line number directive"],
-					Message) },
-				{ Token = error(Message) }
-			)
-		;
-			lexer__string_get_context(Posn1, Context),
-			{ string__from_char_list([Char], DirectiveString) },
-			{ string__append_list([
-				"invalid character `", DirectiveString,
-				"' in `#' line number directive"],
-				Message) },
-			{ Token = error(Message) }
-		)
-	;
-		lexer__string_get_context(Posn1, Context),
-		{ Token = error(
-			"unexpected end-of-file in `#' line number directive") }
-	).
-
-:- pred lexer__get_graphic(list(char), token, io__state, io__state).
-:- mode lexer__get_graphic(in, out, di, uo) is det.
-
-lexer__get_graphic(Chars, Token) -->
-	io__read_char(Result),
-	( { Result = error(Error) },
-		{ Token = io_error(Error) }
-	; { Result = eof },
-		{ lexer__rev_char_list_to_string(Chars, Name) },
-		{ Token = name(Name) }
-	; { Result = ok(Char) },
-		( { lexer__graphic_token_char(Char) } ->
-			lexer__get_graphic([Char | Chars], Token)
-		;
-			io__putback_char(Char),
-			{ lexer__rev_char_list_to_string(Chars, Name) },
-			{ Token = name(Name) }
-		)
-	).
-
-:- pred lexer__string_get_graphic(string, int, posn, token,
-			string_token_context, posn, posn).
-:- mode lexer__string_get_graphic(in, in, in, out, out, in, out) is det.
-
-lexer__string_get_graphic(String, Len, Posn0, Token, Context) -->
-	( lexer__string_read_char(String, Len, Char) ->
-		( { lexer__graphic_token_char(Char) } ->
-			lexer__string_get_graphic(String, Len, Posn0,
-				Token, Context)
-		;
-			lexer__string_ungetchar(String),
-			lexer__grab_string(String, Posn0, Name),
-			{ Token = name(Name) },
-			lexer__string_get_context(Posn0, Context)
-		)
-	;
-		lexer__grab_string(String, Posn0, Name),
-		lexer__string_get_context(Posn0, Context),
-		{ Token = name(Name) }
-	).
-
-:- pred lexer__get_variable(list(char), token, io__state, io__state).
-:- mode lexer__get_variable(in, out, di, uo) is det.
-
-lexer__get_variable(Chars, Token) -->
-	io__read_char(Result),
-	( { Result = error(Error) },
-		{ Token = io_error(Error) }
-	; { Result = eof },
-		{ lexer__rev_char_list_to_string(Chars, VariableName) },
-		{ Token = variable(VariableName) }
-	; { Result = ok(Char) },
-		( { char__is_alnum_or_underscore(Char) } ->
-			lexer__get_variable([Char | Chars], Token)
-		;
-			io__putback_char(Char),
-			{ lexer__rev_char_list_to_string(Chars, VariableName) },
-			{ Token = variable(VariableName) }
-		)
-	).
-
-:- pred lexer__string_get_variable(string, int, posn, token,
-					string_token_context, posn, posn).
-:- mode lexer__string_get_variable(in, in, in, out, out, in, out) is det.
-
-lexer__string_get_variable(String, Len, Posn0, Token, Context) -->
-	( lexer__string_read_char(String, Len, Char) ->
-		( { char__is_alnum_or_underscore(Char) } ->
-			lexer__string_get_variable(String, Len, Posn0,
-				Token, Context)
-		;
-			lexer__string_ungetchar(String),
-			lexer__grab_string(String, Posn0, VariableName),
-			{ Token = variable(VariableName) },
-			lexer__string_get_context(Posn0, Context)
-		)
-	;
-		lexer__grab_string(String, Posn0, VariableName),
-		{ Token = variable(VariableName) },
-		lexer__string_get_context(Posn0, Context)
-	).
-
-%-----------------------------------------------------------------------------%
-
-	% integer and float literals
-
-:- pred lexer__get_zero(token, io__state, io__state).
-:- mode lexer__get_zero(out, di, uo) is det.
-
-lexer__get_zero(Token) -->
-	io__read_char(Result),
-	( { Result = error(Error) },
-		{ Token = io_error(Error) }
-	; { Result = eof },
-		{ Token = integer(0) }
-	; { Result = ok(Char) },
-		( { char__is_digit(Char) } ->
-			lexer__get_number([Char], Token)
-		; { Char = '''' } ->
-			lexer__get_char_code(Token)
-		; { Char = 'b' } ->
-			lexer__get_binary(Token)
-		; { Char = 'o' } ->
-			lexer__get_octal(Token)
-		; { Char = 'x' } ->
-			lexer__get_hex(Token)
-		; { Char = ('.') } ->
-			lexer__get_int_dot(['0'], Token)
-		; { Char = 'e' ; Char = 'E' } ->
-			lexer__get_float_exponent([Char, '0'], Token)
-		;
-			io__putback_char(Char),
-			{ Token = integer(0) }
-		)
-	).
-
-:- pred lexer__string_get_zero(string, int, posn, token, string_token_context,
-				posn, posn).
-:- mode lexer__string_get_zero(in, in, in, out, out, in, out) is det.
-
-lexer__string_get_zero(String, Len, Posn0, Token, Context) -->
-	( lexer__string_read_char(String, Len, Char) ->
-		( { char__is_digit(Char) } ->
-			lexer__string_get_number(String, Len, Posn0,
-				Token, Context)
-		; { Char = '''' } ->
-			lexer__string_get_char_code(String, Len, Posn0,
-				Token, Context)
-		; { Char = 'b' } ->
-			lexer__string_get_binary(String, Len, Posn0,
-				Token, Context)
-		; { Char = 'o' } ->
-			lexer__string_get_octal(String, Len, Posn0,
-				Token, Context)
-		; { Char = 'x' } ->
-			lexer__string_get_hex(String, Len, Posn0,
-				Token, Context)
-		; { Char = ('.') } ->
-			lexer__string_get_int_dot(String, Len, Posn0,
-				Token, Context)
-		; { Char = 'e' ; Char = 'E' } ->
-			lexer__string_get_float_exponent(String, Len, Posn0,
-				Token, Context)
-		;
-			lexer__string_ungetchar(String),
-			lexer__string_get_context(Posn0, Context),
-			{ Token = integer(0) }
-		)
-	;
-		lexer__string_get_context(Posn0, Context),
-		{ Token = integer(0) }
-	).
-
-:- pred lexer__get_char_code(token, io__state, io__state).
-:- mode lexer__get_char_code(out, di, uo) is det.
-
-lexer__get_char_code(Token) -->
-	io__read_char(Result),
-	( { Result = error(Error) },
-		{ Token = io_error(Error) }
-	; { Result = eof },
-		{ Token = error("unterminated char code constant") }
-	; { Result = ok(Char) },
-		{ char__to_int(Char, CharCode) },
-		{ Token = integer(CharCode) }
-	).
-
-:- pred lexer__string_get_char_code(string, int, posn, token,
-				string_token_context, posn, posn).
-:- mode lexer__string_get_char_code(in, in, in, out, out, in, out) is det.
-
-lexer__string_get_char_code(String, Len, Posn0, Token, Context) -->
-	( lexer__string_read_char(String, Len, Char) ->
-		{ char__to_int(Char, CharCode) },
-		{ Token = integer(CharCode) },
-		lexer__string_get_context(Posn0, Context)
-	;
-		{ Token = error("unterminated char code constant") },
-		lexer__string_get_context(Posn0, Context)
-	).
-
-:- pred lexer__get_binary(token, io__state, io__state).
-:- mode lexer__get_binary(out, di, uo) is det.
-
-lexer__get_binary(Token) -->
-	io__read_char(Result),
-	( { Result = error(Error) },
-		{ Token = io_error(Error) }
-	; { Result = eof },
-		{ Token = error("unterminated binary constant") }
-	; { Result = ok(Char) },
-		( { char__is_binary_digit(Char) } ->
-			lexer__get_binary_2([Char], Token)
-		;
-			io__putback_char(Char),
-			{ Token = error("unterminated binary constant") }
-		)
-	).
-
-:- pred lexer__string_get_binary(string, int, posn, token, string_token_context,
-		posn, posn).
-:- mode lexer__string_get_binary(in, in, in, out, out, in, out) is det.
-
-lexer__string_get_binary(String, Len, Posn0, Token, Context) -->
-	( lexer__string_read_char(String, Len, Char) ->
-		( { char__is_binary_digit(Char) } ->
-			lexer__string_get_binary_2(String, Len, Posn0,
-				Token, Context)
-		;
-			lexer__string_ungetchar(String),
-			{ Token = error("unterminated binary constant") },
-			lexer__string_get_context(Posn0, Context)
-		)
-	;
-		{ Token = error("unterminated binary constant") },
-		lexer__string_get_context(Posn0, Context)
-	).
-
-:- pred lexer__get_binary_2(list(char), token, io__state, io__state).
-:- mode lexer__get_binary_2(in, out, di, uo) is det.
-
-lexer__get_binary_2(Chars, Token) -->
-	io__read_char(Result),
-	( { Result = error(Error) },
-		{ Token = io_error(Error) }
-	; { Result = eof },
-		{ lexer__rev_char_list_to_int(Chars, 2, Token) }
-	; { Result = ok(Char) },
-		( { char__is_binary_digit(Char) } ->
-			lexer__get_binary_2([Char | Chars], Token)
-		;
-			io__putback_char(Char),
-			{ lexer__rev_char_list_to_int(Chars, 2, Token) }
-		)
-	).
-
-:- pred lexer__string_get_binary_2(string, int, posn, token,
-			string_token_context, posn, posn).
-:- mode lexer__string_get_binary_2(in, in, in, out, out, in, out) is det.
-
-lexer__string_get_binary_2(String, Len, Posn0, Token, Context) -->
-	( lexer__string_read_char(String, Len, Char) ->
-		( { char__is_binary_digit(Char) } ->
-			lexer__string_get_binary_2(String, Len, Posn0,
-				Token, Context)
-		;
-			lexer__string_ungetchar(String),
-			lexer__grab_string(String, Posn0, BinaryString),
-			{ lexer__conv_string_to_int(BinaryString, 2, Token) },
-			lexer__string_get_context(Posn0, Context)
-		)
-	;
-		lexer__grab_string(String, Posn0, BinaryString),
-		{ lexer__conv_string_to_int(BinaryString, 2, Token) },
-		lexer__string_get_context(Posn0, Context)
-	).
-
-:- pred lexer__get_octal(token, io__state, io__state).
-:- mode lexer__get_octal(out, di, uo) is det.
-
-lexer__get_octal(Token) -->
-	io__read_char(Result),
-	( { Result = error(Error) },
-		{ Token = io_error(Error) }
-	; { Result = eof },
-		{ Token = error("unterminated octal constant") }
-	; { Result = ok(Char) },
-		( { char__is_octal_digit(Char) } ->
-			lexer__get_octal_2([Char], Token)
-		;
-			io__putback_char(Char),
-			{ Token = error("unterminated octal constant") }
-		)
-	).
-
-:- pred lexer__string_get_octal(string, int, posn, token, string_token_context,
-				posn, posn).
-:- mode lexer__string_get_octal(in, in, in, out, out, in, out) is det.
-
-lexer__string_get_octal(String, Len, Posn0, Token, Context) -->
-	( lexer__string_read_char(String, Len, Char) ->
-		( { char__is_octal_digit(Char) } ->
-			lexer__string_get_octal_2(String, Len, Posn0,
-				Token, Context)
-		;
-			lexer__string_ungetchar(String),
-			{ Token = error("unterminated octal constant") },
-			lexer__string_get_context(Posn0, Context)
-		)
-	;
-		{ Token = error("unterminated octal constant") },
-		lexer__string_get_context(Posn0, Context)
-	).
-
-:- pred lexer__get_octal_2(list(char), token, io__state, io__state).
-:- mode lexer__get_octal_2(in, out, di, uo) is det.
-
-lexer__get_octal_2(Chars, Token) -->
-	io__read_char(Result),
-	( { Result = error(Error) },
-		{ Token = io_error(Error) }
-	; { Result = eof },
-		{ lexer__rev_char_list_to_int(Chars, 8, Token) }
-	; { Result = ok(Char) },
-		( { char__is_octal_digit(Char) } ->
-			lexer__get_octal_2([Char | Chars], Token)
-		;
-			io__putback_char(Char),
-			{ lexer__rev_char_list_to_int(Chars, 8, Token) }
-		)
-	).
-
-:- pred lexer__string_get_octal_2(string, int, posn, token,
-				string_token_context, posn, posn).
-:- mode lexer__string_get_octal_2(in, in, in, out, out, in, out) is det.
-
-lexer__string_get_octal_2(String, Len, Posn0, Token, Context) -->
-	( lexer__string_read_char(String, Len, Char) ->
-		( { char__is_octal_digit(Char) } ->
-			lexer__string_get_octal_2(String, Len, Posn0,
-				Token, Context)
-		;
-			lexer__string_ungetchar(String),
-			lexer__grab_string(String, Posn0, BinaryString),
-			{ lexer__conv_string_to_int(BinaryString, 8, Token) },
-			lexer__string_get_context(Posn0, Context)
-		)
-	;
-		lexer__grab_string(String, Posn0, BinaryString),
-		{ lexer__conv_string_to_int(BinaryString, 8, Token) },
-		lexer__string_get_context(Posn0, Context)
-	).
-
-:- pred lexer__get_hex(token, io__state, io__state).
-:- mode lexer__get_hex(out, di, uo) is det.
-
-lexer__get_hex(Token) -->
-	io__read_char(Result),
-	( { Result = error(Error) },
-		{ Token = io_error(Error) }
-	; { Result = eof },
-		{ Token = error("unterminated hex constant") }
-	; { Result = ok(Char) },
-		( { char__is_hex_digit(Char) } ->
-			lexer__get_hex_2([Char], Token)
-		;
-			io__putback_char(Char),
-			{ Token = error("unterminated hex constant") }
-		)
-	).
-
-:- pred lexer__string_get_hex(string, int, posn, token, string_token_context,
-				posn, posn).
-:- mode lexer__string_get_hex(in, in, in, out, out, in, out) is det.
-
-lexer__string_get_hex(String, Len, Posn0, Token, Context) -->
-	( lexer__string_read_char(String, Len, Char) ->
-		( { char__is_hex_digit(Char) } ->
-			lexer__string_get_hex_2(String, Len, Posn0,
-				Token, Context)
-		;
-			lexer__string_ungetchar(String),
-			{ Token = error("unterminated hex constant") },
-			lexer__string_get_context(Posn0, Context)
-		)
-	;
-		{ Token = error("unterminated hex constant") },
-		lexer__string_get_context(Posn0, Context)
-	).
-
-
-:- pred lexer__get_hex_2(list(char), token, io__state, io__state).
-:- mode lexer__get_hex_2(in, out, di, uo) is det.
-
-lexer__get_hex_2(Chars, Token) -->
-	io__read_char(Result),
-	( { Result = error(Error) },
-		{ Token = io_error(Error) }
-	; { Result = eof },
-		{ lexer__rev_char_list_to_int(Chars, 16, Token) }
-	; { Result = ok(Char) },
-		( { char__is_hex_digit(Char) } ->
-			lexer__get_hex_2([Char | Chars], Token)
-		;
-			io__putback_char(Char),
-			{ lexer__rev_char_list_to_int(Chars, 16, Token) }
-		)
-	).
-
-:- pred lexer__string_get_hex_2(string, int, posn, token,
-				string_token_context, posn, posn).
-:- mode lexer__string_get_hex_2(in, in, in, out, out, in, out) is det.
-
-lexer__string_get_hex_2(String, Len, Posn0, Token, Context) -->
-	( lexer__string_read_char(String, Len, Char) ->
-		( { char__is_hex_digit(Char) } ->
-			lexer__string_get_hex_2(String, Len, Posn0,
-				Token, Context)
-		;
-			lexer__string_ungetchar(String),
-			lexer__grab_string(String, Posn0, BinaryString),
-			{ lexer__conv_string_to_int(BinaryString, 16, Token) },
-			lexer__string_get_context(Posn0, Context)
-		)
-	;
-		lexer__grab_string(String, Posn0, BinaryString),
-		{ lexer__conv_string_to_int(BinaryString, 16, Token) },
-		lexer__string_get_context(Posn0, Context)
-	).
-
-
-:- pred lexer__get_number(list(char), token, io__state, io__state).
-:- mode lexer__get_number(in, out, di, uo) is det.
-
-lexer__get_number(Chars, Token) -->
-	io__read_char(Result),
-	( { Result = error(Error) },
-		{ Token = io_error(Error) }
-	; { Result = eof },
-		{ lexer__rev_char_list_to_int(Chars, 10, Token) }
-	; { Result = ok(Char) },
-		( { char__is_digit(Char) } ->
-			lexer__get_number([Char | Chars], Token)
-		; { Char = ('.') } ->
-			lexer__get_int_dot(Chars, Token)
-		; { Char = 'e' ; Char = 'E' } ->
-			lexer__get_float_exponent([Char | Chars], Token)
-		;
-			io__putback_char(Char),
-			{ lexer__rev_char_list_to_int(Chars, 10, Token) }
-		)
-	).
-
-:- pred lexer__string_get_number(string, int, posn, token,
-					string_token_context, posn, posn).
-:- mode lexer__string_get_number(in, in, in, out, out, in, out) is det.
-
-lexer__string_get_number(String, Len, Posn0, Token, Context) -->
-	( lexer__string_read_char(String, Len, Char) ->
-		( { char__is_digit(Char) } ->
-			lexer__string_get_number(String, Len, Posn0,
-				Token, Context)
-		; { Char = ('.') } ->
-			lexer__string_get_int_dot(String, Len, Posn0,
-				Token, Context)
-		; { Char = 'e' ; Char = 'E' } ->
-			lexer__string_get_float_exponent(String, Len, Posn0,
-				Token, Context)
-		;
-			lexer__string_ungetchar(String),
-			lexer__grab_string(String, Posn0, NumberString),
-			{ lexer__conv_string_to_int(NumberString, 10, Token) },
-			lexer__string_get_context(Posn0, Context)
-		)
-	;
-		lexer__grab_string(String, Posn0, NumberString),
-		{ lexer__conv_string_to_int(NumberString, 10, Token) },
-		lexer__string_get_context(Posn0, Context)
-	).
-
-	% XXX the float literal syntax doesn't match ISO Prolog
-
-:- pred lexer__get_int_dot(list(char), token, io__state, io__state).
-:- mode lexer__get_int_dot(in, out, di, uo) is det.
-
-lexer__get_int_dot(Chars, Token) -->
-	io__read_char(Result),
-	( { Result = error(Error) },
-		{ Token = io_error(Error) }
-	; { Result = eof },
-		io__putback_char('.'),
-		{ lexer__rev_char_list_to_int(Chars, 10, Token) }
-	; { Result = ok(Char) },
-		( { char__is_digit(Char) } ->
-			lexer__get_float_decimals([Char, '.' | Chars], Token)
-		;
-			io__putback_char(Char),
-			io__putback_char('.'),
-			{ lexer__rev_char_list_to_int(Chars, 10, Token) }
-		)
-	).
-
-:- pred lexer__string_get_int_dot(string, int, posn, token,
-					string_token_context, posn, posn).
-:- mode lexer__string_get_int_dot(in, in, in, out, out, in, out) is det.
-
-lexer__string_get_int_dot(String, Len, Posn0, Token, Context) -->
-	( lexer__string_read_char(String, Len, Char) ->
-		( { char__is_digit(Char) } ->
-			lexer__string_get_float_decimals(String, Len, Posn0,
-				Token, Context)
-		;
-			lexer__string_ungetchar(String),
-			lexer__string_ungetchar(String),
-			lexer__grab_string(String, Posn0, NumberString),
-			{ lexer__conv_string_to_int(NumberString, 10, Token) },
-			lexer__string_get_context(Posn0, Context)
-		)
-	;
-		lexer__string_ungetchar(String),
-		lexer__grab_string(String, Posn0, NumberString),
-		{ lexer__conv_string_to_int(NumberString, 10, Token) },
-		lexer__string_get_context(Posn0, Context)
-	).
-
-:- pred lexer__get_float_decimals(list(char), token, io__state, io__state).
-:- mode lexer__get_float_decimals(in, out, di, uo) is det.
-
-	% we've read past the decimal point, so now get the decimals
-
-lexer__get_float_decimals(Chars, Token) -->
-	io__read_char(Result),
-	( { Result = error(Error) },
-		{ Token = io_error(Error) }
-	; { Result = eof },
-		{ lexer__rev_char_list_to_float(Chars, Token) }
-	; { Result = ok(Char) },
-		( { char__is_digit(Char) } ->
-			lexer__get_float_decimals([Char | Chars], Token)
-		; { Char = 'e' ; Char = 'E' } ->
-			lexer__get_float_exponent([Char | Chars], Token)
-		;
-			io__putback_char(Char),
-			{ lexer__rev_char_list_to_float(Chars, Token) }
-		)
-	).
-
-:- pred lexer__string_get_float_decimals(string, int, posn, token,
-					string_token_context, posn, posn).
-:- mode lexer__string_get_float_decimals(in, in, in, out, out, in, out) is det.
-
-lexer__string_get_float_decimals(String, Len, Posn0, Token, Context) -->
-	( lexer__string_read_char(String, Len, Char) ->
-		( { char__is_digit(Char) } ->
-			lexer__string_get_float_decimals(String, Len, Posn0,
-				Token, Context)
-		; { Char = 'e' ; Char = 'E' } ->
-			lexer__string_get_float_exponent(String, Len, Posn0,
-				Token, Context)
-		;
-			lexer__string_ungetchar(String),
-			lexer__grab_string(String, Posn0, FloatString),
-			{ lexer__conv_to_float(FloatString, Token) },
-			lexer__string_get_context(Posn0, Context)
-		)
-	;
-		lexer__grab_string(String, Posn0, FloatString),
-		{ lexer__conv_to_float(FloatString, Token) },
-		lexer__string_get_context(Posn0, Context)
-	).
-
-:- pred lexer__get_float_exponent(list(char), token, io__state, io__state).
-:- mode lexer__get_float_exponent(in, out, di, uo) is det.
-
-lexer__get_float_exponent(Chars, Token) -->
-	io__read_char(Result),
-	( { Result = error(Error) },
-		{ Token = io_error(Error) }
-	; { Result = eof },
-		{ lexer__rev_char_list_to_float(Chars, Token) }
-	; { Result = ok(Char) },
-		( { Char = ('+') ; Char = ('-') } ->
-			lexer__get_float_exponent_2([Char | Chars], Token)
-		; { char__is_digit(Char) } ->
-			lexer__get_float_exponent_3([Char | Chars], Token)
-		;
-			io__putback_char(Char),
-			{ Token =
-			  error("unterminated exponent in float token") }
-		)
-	).
-
-:- pred lexer__string_get_float_exponent(string, int, posn, token, 	
-				string_token_context, posn, posn).
-:- mode lexer__string_get_float_exponent(in, in, in, out, out, in, out) is det.
-
-lexer__string_get_float_exponent(String, Len, Posn0, Token, Context) -->
-	( lexer__string_read_char(String, Len, Char) ->
-		( { Char = ('+') ; Char = ('-') } ->
-			lexer__string_get_float_exponent_2(String, Len, Posn0,
-				Token, Context)
-		; { char__is_digit(Char) } ->
-			lexer__string_get_float_exponent_3(String, Len, Posn0,
-				Token, Context)
-		;
-			lexer__string_ungetchar(String),
-			{ Token =
-			  error("unterminated exponent in float token") },
-			lexer__string_get_context(Posn0, Context)
-		)
-	;
-		lexer__grab_string(String, Posn0, FloatString),
-		{ lexer__conv_to_float(FloatString, Token) },
-		lexer__string_get_context(Posn0, Context)
-	).
-
-:- pred lexer__get_float_exponent_2(list(char), token,
-				io__state, io__state).
-:- mode lexer__get_float_exponent_2(in, out, di, uo) is det.
-
-	% we've read past the E signalling the start of the exponent -
-	% make sure that there's at least one digit following,
-	% and then get the remaining digits
-
-lexer__get_float_exponent_2(Chars, Token) -->
-	io__read_char(Result),
-	( { Result = error(Error) },
-		{ Token = io_error(Error) }
-	; { Result = eof },
-		{ Token = error("unterminated exponent in float token") }
-	; { Result = ok(Char) },
-		( { char__is_digit(Char) } ->
-			lexer__get_float_exponent_3([Char | Chars], Token)
-		;
-			io__putback_char(Char),
-			{ Token =
-			  error("unterminated exponent in float token") }
-		)
-	).
-
-:- pred lexer__string_get_float_exponent_2(string, int, posn, token,
-				string_token_context, posn, posn).
-:- mode lexer__string_get_float_exponent_2(in, in, in, out, out, in, out)
-				is det.
-
-	% we've read past the E signalling the start of the exponent -
-	% make sure that there's at least one digit following,
-	% and then get the remaining digits
-
-lexer__string_get_float_exponent_2(String, Len, Posn0, Token, Context) -->
-	( lexer__string_read_char(String, Len, Char) ->
-		( { char__is_digit(Char) } ->
-			lexer__string_get_float_exponent_3(String, Len, Posn0,
-				Token, Context)
-		;
-			lexer__string_ungetchar(String),
-			{ Token =
-			  error("unterminated exponent in float token") },
-			lexer__string_get_context(Posn0, Context)
-		)
-	;
-		{ Token = error("unterminated exponent in float token") },
-		lexer__string_get_context(Posn0, Context)
-	).
-
-:- pred lexer__get_float_exponent_3(list(char), token,
-					io__state, io__state).
-:- mode lexer__get_float_exponent_3(in, out, di, uo) is det.
-
-	% we've read past the first digit of the exponent -
-	% now get the remaining digits
-
-lexer__get_float_exponent_3(Chars, Token) -->
-	io__read_char(Result),
-	( { Result = error(Error) },
-		{ Token = io_error(Error) }
-	; { Result = eof },
-		{ lexer__rev_char_list_to_float(Chars, Token) }
-	; { Result = ok(Char) },
-		( { char__is_digit(Char) } ->
-			lexer__get_float_exponent_3([Char | Chars], Token)
-		;
-			io__putback_char(Char),
-			{ lexer__rev_char_list_to_float(Chars, Token) }
-		)
-	).
-
-:- pred lexer__string_get_float_exponent_3(string, int, posn, token,
-				string_token_context, posn, posn).
-:- mode lexer__string_get_float_exponent_3(in, in, in, out, out, in, out)
-				is det.
-
-lexer__string_get_float_exponent_3(String, Len, Posn0, Token, Context) -->
-	( lexer__string_read_char(String, Len, Char) ->
-		( { char__is_digit(Char) } ->
-			lexer__string_get_float_exponent_3(String, Len, Posn0,
-				Token, Context)
-		;
-			lexer__string_ungetchar(String),
-			lexer__grab_string(String, Posn0, FloatString),
-			{ lexer__conv_to_float(FloatString, Token) },
-			lexer__string_get_context(Posn0, Context)
-		)
-	;
-		lexer__grab_string(String, Posn0, FloatString),
-		{ lexer__conv_to_float(FloatString, Token) },
-		lexer__string_get_context(Posn0, Context)
-	).
-
-%-----------------------------------------------------------------------------%
-
-	% Utility routines
-
-:- pred lexer__rev_char_list_to_int(list(char), int, token).
-:- mode lexer__rev_char_list_to_int(in, in, out) is det.
-
-lexer__rev_char_list_to_int(RevChars, Base, Token) :-
-	lexer__rev_char_list_to_string(RevChars, String),
-	lexer__conv_string_to_int(String, Base, Token).
-
-:- pred lexer__conv_string_to_int(string, int, token).
-:- mode lexer__conv_string_to_int(in, in, out) is det.
-
-lexer__conv_string_to_int(String, Base, Token) :-
-	( string__base_string_to_int(Base, String, Int) ->
-		Token = integer(Int)
-	;
-		Token = error("invalid integer token")
-	).
-
-:- pred lexer__rev_char_list_to_float(list(char), token).
-:- mode lexer__rev_char_list_to_float(in, out) is det.
-
-lexer__rev_char_list_to_float(RevChars, Token) :-
-	lexer__rev_char_list_to_string(RevChars, String),
-	lexer__conv_to_float(String, Token).
-
-:- pred lexer__conv_to_float(string, token).
-:- mode lexer__conv_to_float(in, out) is det.
-
-lexer__conv_to_float(String, Token) :-
-	( string__to_float(String, Float) ->
-		Token = float(Float)
-	;
-		Token = error("invalid float token")
-	).
-
-:- pred lexer__rev_char_list_to_string(list(char), string).
-:- mode lexer__rev_char_list_to_string(in, out) is det.
-
-lexer__rev_char_list_to_string(RevChars, String) :-
-       string__from_rev_char_list(RevChars, String).
-
-%-----------------------------------------------------------------------------%
Index: library/library.m
===================================================================
RCS file: /home/mercury1/repository/mercury/library/library.m,v
retrieving revision 1.49
diff -u -r1.49 library.m
--- library/library.m	2000/05/15 03:33:06	1.49
+++ library/library.m	2000/08/22 02:08:38
@@ -32,7 +32,7 @@
 :- import_module io, list, map, multi_map, pqueue, queue, random, relation.
 :- import_module require, set, set_bbbtree, set_ordlist, set_unordlist, stack.
 :- import_module std_util, string, term, term_io, tree234, varset.
-:- import_module store, rbtree, parser, lexer, ops.
+:- import_module store, rbtree, mer_parser, mer_lexer, ops.
 :- import_module prolog.
 :- import_module integer, rational.
 :- import_module exception, gc.
Index: library/mer_lexer.m
===================================================================
RCS file: mer_lexer.m
diff -N mer_lexer.m
--- /dev/null	Thu Sep  2 15:00:04 1999
+++ mer_lexer.m	Tue Aug 22 12:09:00 2000
@@ -0,0 +1,2000 @@
+%---------------------------------------------------------------------------%
+% Copyright (C) 1993-2000 The University of Melbourne.
+% This file may only be copied under the terms of the GNU Library General
+% Public License - see the file COPYING.LIB in the Mercury distribution.
+%---------------------------------------------------------------------------%
+%
+% file: mer_lexer.m.
+% main author: fjh.
+% stability: high.
+%
+% Lexical analysis.  This module defines the representation of tokens
+% and exports predicates for reading in tokens from an input stream.
+%
+% See ISO Prolog 6.4.  Also see the comments at the top of mer_parser.m.
+%
+%-----------------------------------------------------------------------------%
+
+:- module mer_lexer.
+:- interface.
+:- import_module io, char.
+
+:- type	token
+	--->	name(string)
+	;	variable(string)
+	;	integer(int)
+	;	float(float)
+	;	string(string)		% "...."
+	;	open			% '('
+	;	open_ct			% '(' without any preceding whitespace
+	;	close			% ')'
+	;	open_list		% '['
+	;	close_list		% ']'
+	;	open_curly		% '}'
+	;	close_curly		% '{'
+	;	ht_sep			% '|'
+	;	comma			% ','
+	;	end			% '.'
+	;	junk(char)		% junk character in the input stream
+	;	error(string)		% some other invalid token
+	;	io_error(io__error)	% error reading from the input stream
+	;	eof.			% end-of-file
+
+% For every token, we record the line number of the line on
+% which the token occurred.
+
+:- type token_context == int.	% line number
+
+% This "fat list" representation is more efficient than a list of pairs.
+:- type token_list	--->	token_cons(token, token_context, token_list)
+			;	token_nil.
+
+:- pred mer_lexer__get_token_list(token_list, io__state, io__state).
+:- mode mer_lexer__get_token_list(out, di, uo) is det.
+%	Read a list of tokens from the current input stream.
+%	Keep reading until we encounter either an `end' token
+%	(i.e. a full stop followed by whitespace) or the end-of-file.
+
+% The type `offset' represents a (zero-based) offset into a string.
+:- type offset == int.
+
+:- pred mer_lexer__string_get_token_list(string, offset, token_list, posn, posn).
+:- mode mer_lexer__string_get_token_list(in, in, out, in, out) is det.
+% mer_lexer__string_get_token_list(String, MaxOffset, Tokens,
+%		InitialPos, FinalPos):
+%	Scan a list of tokens from a string,
+%	starting at the current offset specified by InitialPos.
+%	Keep scanning until either we encounter either an `end' token
+%	(i.e. a full stop followed by whitespace) or until
+%	we reach MaxOffset.  (MaxOffset must be =< the length of the string.)
+%	Return the tokens scanned in Tokens, and return the position
+%	one character past the end of the last token in FinalPos.
+
+:- pred mer_lexer__string_get_token_list(string, token_list, posn, posn).
+:- mode mer_lexer__string_get_token_list(in, out, in, out) is det.
+% mer_lexer__string_get_token_list(String, Tokens, InitialPos, FinalPos):
+%	calls string_get_token_list/5 above with MaxPos = length of String.
+
+:- pred mer_lexer__token_to_string(token, string).
+:- mode mer_lexer__token_to_string(in, out) is det.
+%	Convert a token to a human-readable string describing the token.
+
+%-----------------------------------------------------------------------------%
+:- implementation.
+%-----------------------------------------------------------------------------%
+:- interface.
+
+	% mer_lexer__graphic_token_char(Char): true iff `Char'
+	% is "graphic token char" (ISO Prolog 6.4.2).
+	% This is exported for use by term_io__quote_atom.
+:- pred mer_lexer__graphic_token_char(char).
+:- mode mer_lexer__graphic_token_char(in) is semidet.
+%-----------------------------------------------------------------------------%
+
+:- implementation.
+:- import_module list, term, require, string, int.
+
+%
+% Note that there are two implementations of most predicates here:
+% one which deals with strings, the other that deals with io__states.
+% We can't write the io__state version in terms of the string
+% version because we don't know how much string to slurp up
+% until after we've lexically analysed it.  Some interactive
+% applications require the old Prolog behaviour of stopping
+% after an end token (i.e. `.' plus whitespace) rather than
+% reading in whole lines.
+% Conversely, we can't write the string version using the io__state
+% version, since that would require either cheating with the io__state
+% or ruining the string interface.
+% 
+% An alternative would be to write both versions in terms
+% of a generic "char_stream" typeclass, with instances
+% for io__states and for strings.
+% However, for this to be acceptably efficient it would
+% require the compiler to specialize the code, which
+% currently (13 May 98) it is not capable of doing.
+%
+% In fact, the string version is still not as efficient as I would
+% like.  The compiler ought to (but currently doesn't) unfold all
+% the instances of the `posn' type.  We could do this type unfolding
+% by hand, but it would be very tedious and it would make the code
+% less readable.  If/when there is compiler support for this, we
+% should also think about moving the `String' and `Len' arguments
+% into the posn (or making a new `lexer_state' struct which contains
+% both the posn and the String and Len arguments).
+%
+
+%-----------------------------------------------------------------------------%
+
+mer_lexer__token_to_string(name(Name), String) :-
+	string__append_list(["token '", Name, "'"], String).
+mer_lexer__token_to_string(variable(Var), String) :-
+	string__append_list(["variable `", Var, "'"], String).
+mer_lexer__token_to_string(integer(Int), String) :-
+	string__int_to_string(Int, IntString),
+	string__append_list(["integer `", IntString, "'"], String).
+mer_lexer__token_to_string(float(Float), String) :-
+	string__float_to_string(Float, FloatString),
+	string__append_list(["float `", FloatString, "'"], String).
+mer_lexer__token_to_string(string(Token), String) :-
+	string__append_list(["string """, Token, """"], String).
+mer_lexer__token_to_string(open, "token ` ('").
+mer_lexer__token_to_string(open_ct, "token `('").
+mer_lexer__token_to_string(close, "token `)'").
+mer_lexer__token_to_string(open_list, "token `['").
+mer_lexer__token_to_string(close_list, "token `]'").
+mer_lexer__token_to_string(open_curly, "token `{'").
+mer_lexer__token_to_string(close_curly, "token `}'").
+mer_lexer__token_to_string(ht_sep, "token `|'").
+mer_lexer__token_to_string(comma, "token `,'").
+mer_lexer__token_to_string(end, "token `. '").
+mer_lexer__token_to_string(eof, "end-of-file").
+mer_lexer__token_to_string(junk(JunkChar), String) :-
+	char__to_int(JunkChar, Code),
+	string__int_to_base_string(Code, 16, Hex),
+	string__append_list(["illegal character <<0x", Hex, ">>"], String).
+mer_lexer__token_to_string(io_error(IO_Error), String) :-
+	io__error_message(IO_Error, IO_ErrorMessage),
+	string__append("I/O error: ", IO_ErrorMessage, String).
+mer_lexer__token_to_string(error(Message), String) :-
+	string__append_list(["illegal token (", Message, ")"], String).
+
+	% We build the tokens up as lists of characters in reverse order.
+	% When we get to the end of each token, we call
+	% `mer_lexer__rev_char_list_to_string/2' to convert that representation
+	% into a string.
+
+	% Comments of the form
+	%	foo --> bar . baz
+	% mean that we are parsing a `foo', and we've already scanned
+	% past the `bar', so now we need to match with a `baz'.
+
+mer_lexer__get_token_list(Tokens) -->
+	mer_lexer__get_token(Token, Context),
+	( { Token = eof } ->
+		{ Tokens = token_nil }
+	; { Token = end ; Token = error(_) ; Token = io_error(_) } ->
+		{ Tokens = token_cons(Token, Context, token_nil) }
+	;
+		{ Tokens = token_cons(Token, Context, Tokens1) },
+		mer_lexer__get_token_list(Tokens1)
+	).
+
+mer_lexer__string_get_token_list(String, Tokens) -->
+	{ string__length(String, Len) },
+	mer_lexer__string_get_token_list(String, Len, Tokens).
+
+mer_lexer__string_get_token_list(String, Len, Tokens) -->
+	mer_lexer__string_get_token(String, Len, Token, Context),
+	( { Token = eof } ->
+		{ Tokens = token_nil }
+	; { Token = end ; Token = error(_) ; Token = io_error(_) } ->
+		{ Tokens = token_cons(Token, Context, token_nil) }
+	;
+		{ Tokens = token_cons(Token, Context, Tokens1) },
+		mer_lexer__string_get_token_list(String, Len, Tokens1)
+	).
+
+%-----------------------------------------------------------------------------%
+
+% some low-level routines
+
+:- pred mer_lexer__get_context(token_context, io__state, io__state).
+:- mode mer_lexer__get_context(out, di, uo) is det.
+
+mer_lexer__get_context(Context) -->
+	io__get_line_number(Context).
+
+:- type string_token_context == token_context.
+
+:- pred mer_lexer__string_get_context(posn, string_token_context, posn, posn).
+:- mode mer_lexer__string_get_context(in, out, in, out) is det.
+
+mer_lexer__string_get_context(StartPosn, Context, EndPosn, EndPosn) :-
+	StartPosn = posn(StartLineNum, _, _),
+	Context = StartLineNum.
+	% In future, we might want to modify this code to read something
+	% like this:
+	%	posn_to_line_and_column(StartPosn, StartLineNum, StartColumn),
+	%	posn_to_line_and_column(EndPosn, EndLineNum, EndColumn),
+	%	Context = detailed(StartLine, StartColumn, EndLine, EndColumn).
+
+:- pred mer_lexer__string_read_char(string, int, char, posn, posn).
+:- mode mer_lexer__string_read_char(in, in, out, in, out) is semidet.
+
+:- pragma inline(mer_lexer__string_read_char/5).
+
+mer_lexer__string_read_char(String, Len, Char, Posn0, Posn) :-
+	Posn0 = posn(LineNum0, LineOffset0, Offset0),
+	Offset0 < Len,
+	string__unsafe_index(String, Offset0, Char),
+	Offset is Offset0 + 1,
+	( Char = '\n' ->
+		LineNum is LineNum0 + 1,
+		Posn = posn(LineNum, Offset, Offset)
+	;
+		Posn = posn(LineNum0, LineOffset0, Offset)
+	).
+
+:- pred mer_lexer__string_ungetchar(string, posn, posn).
+:- mode mer_lexer__string_ungetchar(in, in, out) is det.
+
+mer_lexer__string_ungetchar(String, Posn0, Posn) :-
+	Posn0 = posn(LineNum0, LineOffset0, Offset0),
+	Offset is Offset0 - 1,
+	string__unsafe_index(String, Offset, Char),
+	( Char = '\n' ->
+		LineNum is LineNum0 - 1,
+		Posn = posn(LineNum, Offset, Offset)
+	;
+		Posn = posn(LineNum0, LineOffset0, Offset)
+	).
+
+:- pred mer_lexer__grab_string(string, posn, string, posn, posn).
+:- mode mer_lexer__grab_string(in, in, out, in, out) is det.
+
+mer_lexer__grab_string(String, Posn0, SubString, Posn, Posn) :-
+	Posn0 = posn(_, _, Offset0),
+	Posn = posn(_, _, Offset),
+	Count is Offset - Offset0,
+	string__unsafe_substring(String, Offset0, Count, SubString).
+
+:- pred mer_lexer__string_set_line_number(int, posn, posn).
+:- mode mer_lexer__string_set_line_number(in, in, out) is det.
+
+mer_lexer__string_set_line_number(LineNumber, Posn0, Posn) :-
+	Posn0 = posn(_, _, Offset),
+	Posn = posn(LineNumber, Offset, Offset).
+
+%-----------------------------------------------------------------------------%
+
+:- pred mer_lexer__get_token(token, token_context, io__state, io__state).
+:- mode mer_lexer__get_token(out, out, di, uo) is det.
+
+mer_lexer__get_token(Token, Context) -->
+	io__read_char(Result),
+	( { Result = error(Error) },
+		mer_lexer__get_context(Context),
+		{ Token = io_error(Error) }
+	; { Result = eof },
+		mer_lexer__get_context(Context),
+		{ Token = eof }
+	; { Result = ok(Char) },
+		( { Char = ' ' ; Char = '\t' ; Char = '\n' } ->
+			mer_lexer__get_token_2(Token, Context)
+		; { char__is_upper(Char) ; Char = '_' } ->
+			mer_lexer__get_context(Context),
+			mer_lexer__get_variable([Char], Token)
+		; { char__is_lower(Char) } ->
+			mer_lexer__get_context(Context),
+			mer_lexer__get_name([Char], Token)
+		; { Char = '0' } ->
+			mer_lexer__get_context(Context),
+			mer_lexer__get_zero(Token)
+		; { char__is_digit(Char) } ->
+			mer_lexer__get_context(Context),
+			mer_lexer__get_number([Char], Token)
+		; { mer_lexer__special_token(Char, SpecialToken) } ->
+			mer_lexer__get_context(Context),
+			{ SpecialToken = open ->
+				Token = open_ct
+			;
+				Token = SpecialToken
+			}
+		; { Char = ('.') } ->
+			mer_lexer__get_context(Context),
+			mer_lexer__get_dot(Token)
+		; { Char = ('%') } ->
+			mer_lexer__skip_to_eol(Token, Context)
+		; { Char = '"' ; Char = '''' } ->
+			mer_lexer__get_context(Context),
+			mer_lexer__get_quoted_name(Char, [], Token)
+		; { Char = ('/') } ->
+			mer_lexer__get_slash(Token, Context)
+		; { Char = ('#') } ->
+			mer_lexer__get_source_line_number([], Token, Context)
+		; { Char = ('`') } ->
+			mer_lexer__get_context(Context),
+			{ Token = name("`") }
+		; { mer_lexer__graphic_token_char(Char) } ->
+			mer_lexer__get_context(Context),
+			mer_lexer__get_graphic([Char], Token)
+		;
+			mer_lexer__get_context(Context),
+			{ Token = junk(Char) }
+		)
+	).
+
+:- pred mer_lexer__string_get_token(string, int, token, token_context, posn, posn).
+:- mode mer_lexer__string_get_token(in, in, out, out, in, out) is det.
+
+mer_lexer__string_get_token(String, Len, Token, Context) -->
+	=(Posn0),
+	( mer_lexer__string_read_char(String, Len, Char) ->
+		( { Char = ' ' ; Char = '\t' ; Char = '\n' } ->
+			mer_lexer__string_get_token_2(String, Len, Token, Context)
+		; { char__is_upper(Char) ; Char = '_' } ->
+			mer_lexer__string_get_variable(String, Len, Posn0,
+				Token, Context)
+		; { char__is_lower(Char) } ->
+			mer_lexer__string_get_name(String, Len, Posn0,
+				Token, Context)
+		; { Char = '0' } ->
+			mer_lexer__string_get_zero(String, Len, Posn0,
+				Token, Context)
+		; { char__is_digit(Char) } ->
+			mer_lexer__string_get_number(String, Len, Posn0,
+				Token, Context)
+		; { mer_lexer__special_token(Char, SpecialToken) } ->
+			mer_lexer__string_get_context(Posn0, Context),
+			{ SpecialToken = open ->
+				Token = open_ct
+			;
+				Token = SpecialToken
+			}
+		; { Char = ('.') } ->
+			mer_lexer__string_get_dot(String, Len, Posn0,
+				Token, Context)
+		; { Char = ('%') } ->
+			mer_lexer__string_skip_to_eol(String, Len, Token, Context)
+		; { Char = '"' ; Char = '''' } ->
+			mer_lexer__string_get_quoted_name(String, Len, Char, [], 
+				Posn0, Token, Context)
+		; { Char = ('/') } ->
+			mer_lexer__string_get_slash(String, Len, Posn0,
+				Token, Context)
+		; { Char = ('#') } ->
+			=(Posn1),
+			mer_lexer__string_get_source_line_number(String, Len,
+				Posn1, Token, Context)
+		; { Char = ('`') } ->
+			mer_lexer__string_get_context(Posn0, Context),
+			{ Token = name("`") }
+		; { mer_lexer__graphic_token_char(Char) } ->
+			mer_lexer__string_get_graphic(String, Len, Posn0,
+				Token, Context)
+		;
+			mer_lexer__string_get_context(Posn0, Context),
+			{ Token = junk(Char) }
+		)
+	;
+		mer_lexer__string_get_context(Posn0, Context),
+		{ Token = eof }
+	).
+
+%-----------------------------------------------------------------------------%
+
+:- pred mer_lexer__get_token_2(token, token_context, io__state, io__state).
+:- mode mer_lexer__get_token_2(out, out, di, uo) is det.
+
+	% This is just like get_token, except that we have already
+	% scanned past some whitespace, so '(' gets scanned as `open'
+	% rather than `open_ct'.
+
+mer_lexer__get_token_2(Token, Context) -->
+	io__read_char(Result),
+	( { Result = error(Error) },
+		mer_lexer__get_context(Context),
+		{ Token = io_error(Error) }
+	; { Result = eof },
+		mer_lexer__get_context(Context),
+		{ Token = eof }
+	; { Result = ok(Char) },
+		( { Char = ' ' ; Char = '\t' ; Char = '\n' } ->
+			mer_lexer__get_token_2(Token, Context)
+		; { char__is_upper(Char) ; Char = '_' } ->
+			mer_lexer__get_context(Context),
+			mer_lexer__get_variable([Char], Token)
+		; { char__is_lower(Char) } ->
+			mer_lexer__get_context(Context),
+			mer_lexer__get_name([Char], Token)
+		; { Char = '0' } ->
+			mer_lexer__get_context(Context),
+			mer_lexer__get_zero(Token)
+		; { char__is_digit(Char) } ->
+			mer_lexer__get_context(Context),
+			mer_lexer__get_number([Char], Token)
+		; { mer_lexer__special_token(Char, SpecialToken) } ->
+			mer_lexer__get_context(Context),
+			{ Token = SpecialToken }
+		; { Char = ('.') } ->
+			mer_lexer__get_context(Context),
+			mer_lexer__get_dot(Token)
+		; { Char = ('%') } ->
+			mer_lexer__skip_to_eol(Token, Context)
+		; { Char = '"' ; Char = '''' } ->
+			mer_lexer__get_context(Context),
+			mer_lexer__get_quoted_name(Char, [], Token)
+		; { Char = ('/') } ->
+			mer_lexer__get_slash(Token, Context)
+		; { Char = ('#') } ->
+			mer_lexer__get_source_line_number([], Token, Context)
+		; { Char = ('`') } ->
+			mer_lexer__get_context(Context),
+			{ Token = name("`") }
+		; { mer_lexer__graphic_token_char(Char) } ->
+			mer_lexer__get_context(Context),
+			mer_lexer__get_graphic([Char], Token)
+		;
+			mer_lexer__get_context(Context),
+			{ Token = junk(Char) }
+		)
+	).
+
+:- pred mer_lexer__string_get_token_2(string, int, token, token_context,
+					posn, posn).
+:- mode mer_lexer__string_get_token_2(in, in, out, out, in, out) is det.
+
+mer_lexer__string_get_token_2(String, Len, Token, Context) -->
+	=(Posn0),
+	( mer_lexer__string_read_char(String, Len, Char) ->
+		( { Char = ' ' ; Char = '\t' ; Char = '\n' } ->
+			mer_lexer__string_get_token_2(String, Len, Token, Context)
+		; { char__is_upper(Char) ; Char = '_' } ->
+			mer_lexer__string_get_variable(String, Len, Posn0,
+				Token, Context)
+		; { char__is_lower(Char) } ->
+			mer_lexer__string_get_name(String, Len, Posn0,
+				Token, Context)
+		; { Char = '0' } ->
+			mer_lexer__string_get_zero(String, Len, Posn0,
+				Token, Context)
+		; { char__is_digit(Char) } ->
+			mer_lexer__string_get_number(String, Len, Posn0,
+				Token, Context)
+		; { mer_lexer__special_token(Char, SpecialToken) } ->
+			mer_lexer__string_get_context(Posn0, Context),
+			{ Token = SpecialToken }
+		; { Char = ('.') } ->
+			mer_lexer__string_get_dot(String, Len, Posn0,
+				Token, Context)
+		; { Char = ('%') } ->
+			mer_lexer__string_skip_to_eol(String, Len, Token, Context)
+		; { Char = '"' ; Char = '''' } ->
+			mer_lexer__string_get_quoted_name(String, Len, Char, [],
+				Posn0, Token, Context)
+		; { Char = ('/') } ->
+			mer_lexer__string_get_slash(String, Len, Posn0,
+				Token, Context)
+		; { Char = ('#') } ->
+			=(Posn1),
+			mer_lexer__string_get_source_line_number(String, Len,
+				Posn1, Token, Context)
+		; { Char = ('`') } ->
+			mer_lexer__string_get_context(Posn0, Context),
+			{ Token = name("`") }
+		; { mer_lexer__graphic_token_char(Char) } ->
+			mer_lexer__string_get_graphic(String, Len, Posn0,
+				Token, Context)
+		;
+			mer_lexer__string_get_context(Posn0, Context),
+			{ Token = junk(Char) }
+		)
+	;
+		mer_lexer__string_get_context(Posn0, Context),
+		{ Token = eof }
+	).
+
+%-----------------------------------------------------------------------------%
+
+%-----------------------------------------------------------------------------%
+
+:- pred mer_lexer__special_token(char, token).
+:- mode mer_lexer__special_token(in, out) is semidet.
+
+mer_lexer__special_token('(', open).	% May get converted to open_ct
+mer_lexer__special_token(')', close).
+mer_lexer__special_token('[', open_list).
+mer_lexer__special_token(']', close_list).
+mer_lexer__special_token('{', open_curly).
+mer_lexer__special_token('}', close_curly).
+mer_lexer__special_token('|', ht_sep).
+mer_lexer__special_token(',', comma).
+mer_lexer__special_token(';', name(";")).
+mer_lexer__special_token('!', name("!")).
+
+mer_lexer__graphic_token_char('#').
+mer_lexer__graphic_token_char('$').
+mer_lexer__graphic_token_char('&').
+mer_lexer__graphic_token_char('*').
+mer_lexer__graphic_token_char('+').
+mer_lexer__graphic_token_char('-').
+mer_lexer__graphic_token_char('.').
+mer_lexer__graphic_token_char('/').
+mer_lexer__graphic_token_char(':').
+mer_lexer__graphic_token_char('<').
+mer_lexer__graphic_token_char('=').
+mer_lexer__graphic_token_char('>').
+mer_lexer__graphic_token_char('?').
+mer_lexer__graphic_token_char('@').
+mer_lexer__graphic_token_char('^').
+mer_lexer__graphic_token_char('~').
+mer_lexer__graphic_token_char('\\').
+
+%-----------------------------------------------------------------------------%
+
+:- pred mer_lexer__get_dot(token, io__state, io__state).
+:- mode mer_lexer__get_dot(out, di, uo) is det.
+
+mer_lexer__get_dot(Token) -->
+	io__read_char(Result),
+	( { Result = error(Error) },
+		{ Token = io_error(Error) }
+	; { Result = eof },
+		{ Token = end }
+	; { Result = ok(Char) },
+		( { mer_lexer__whitespace_after_dot(Char) } ->
+			io__putback_char(Char),
+			{ Token = end }
+		; { mer_lexer__graphic_token_char(Char) } ->
+			mer_lexer__get_graphic([Char, '.'], Token)
+		;
+			io__putback_char(Char),
+			{ Token = name(".") }
+		)
+	).
+
+:- pred mer_lexer__string_get_dot(string, int, posn, token, string_token_context,
+				posn, posn).
+:- mode mer_lexer__string_get_dot(in, in, in, out, out, in, out) is det.
+
+mer_lexer__string_get_dot(String, Len, Posn0, Token, Context) -->
+	( mer_lexer__string_read_char(String, Len, Char) ->
+		( { mer_lexer__whitespace_after_dot(Char) } ->
+			mer_lexer__string_ungetchar(String),
+			mer_lexer__string_get_context(Posn0, Context),
+			{ Token = end }
+		; { mer_lexer__graphic_token_char(Char) } ->
+			mer_lexer__string_get_graphic(String, Len, Posn0,
+				Token, Context)
+		;
+			mer_lexer__string_ungetchar(String),
+			mer_lexer__string_get_context(Posn0, Context),
+			{ Token = name(".") }
+		)
+	;
+		mer_lexer__string_get_context(Posn0, Context),
+		{ Token = end }
+	).
+
+:- pred mer_lexer__whitespace_after_dot(char).
+:- mode mer_lexer__whitespace_after_dot(in) is semidet.
+
+mer_lexer__whitespace_after_dot(' ').
+mer_lexer__whitespace_after_dot('\t').
+mer_lexer__whitespace_after_dot('\n').
+mer_lexer__whitespace_after_dot('%').
+
+%-----------------------------------------------------------------------------%
+
+	% comments
+
+:- pred mer_lexer__skip_to_eol(token, token_context, io__state, io__state).
+:- mode mer_lexer__skip_to_eol(out, out, di, uo) is det.
+
+mer_lexer__skip_to_eol(Token, Context) -->
+	io__read_char(Result),
+	( { Result = error(Error) },
+		mer_lexer__get_context(Context),
+		{ Token = io_error(Error) }
+	; { Result = eof },
+		mer_lexer__get_context(Context),
+		{ Token = eof }
+	; { Result = ok(Char) },
+		( { Char = '\n' } ->
+			mer_lexer__get_token_2(Token, Context)
+		;
+			mer_lexer__skip_to_eol(Token, Context)
+		)
+	).
+
+:- pred mer_lexer__string_skip_to_eol(string, int, token, token_context,
+					posn, posn).
+:- mode mer_lexer__string_skip_to_eol(in, in, out, out, in, out) is det.
+
+mer_lexer__string_skip_to_eol(String, Len, Token, Context) -->
+	( mer_lexer__string_read_char(String, Len, Char) ->
+		( { Char = '\n' } ->
+			mer_lexer__string_get_token_2(String, Len, Token, Context)
+		;
+			mer_lexer__string_skip_to_eol(String, Len, Token, Context)
+		)
+	;
+		=(Posn),
+		mer_lexer__string_get_context(Posn, Context),
+		{ Token = eof }
+	).
+
+:- pred mer_lexer__get_slash(token, token_context, io__state, io__state).
+:- mode mer_lexer__get_slash(out, out, di, uo) is det.
+
+mer_lexer__get_slash(Token, Context) -->
+	io__read_char(Result),
+	( { Result = error(Error) },
+		mer_lexer__get_context(Context),
+		{ Token = io_error(Error) }
+	; { Result = eof },
+		mer_lexer__get_context(Context),
+		{ Token = name("/") }
+	; { Result = ok(Char) },
+		( { Char = ('*') } ->
+			mer_lexer__get_comment(Token, Context)
+		; { mer_lexer__graphic_token_char(Char) } ->
+			mer_lexer__get_context(Context),
+			mer_lexer__get_graphic([Char, '/'], Token)
+		;
+			io__putback_char(Char),
+			mer_lexer__get_context(Context),
+			{ Token = name("/") }
+		)
+	).
+
+:- pred mer_lexer__string_get_slash(string, int, posn, token, string_token_context,
+				posn, posn).
+:- mode mer_lexer__string_get_slash(in, in, in, out, out, in, out) is det.
+
+mer_lexer__string_get_slash(String, Len, Posn0, Token, Context) -->
+	( mer_lexer__string_read_char(String, Len, Char) ->
+		( { Char = ('*') } ->
+			mer_lexer__string_get_comment(String, Len, Posn0,
+				Token, Context)
+		; { mer_lexer__graphic_token_char(Char) } ->
+			mer_lexer__string_get_graphic(String, Len, Posn0,
+				Token, Context)
+		;
+			mer_lexer__string_ungetchar(String),
+			mer_lexer__string_get_context(Posn0, Context),
+			{ Token = name("/") }
+		)
+	;
+		mer_lexer__string_get_context(Posn0, Context),
+		{ Token = name("/") }
+	).
+
+:- pred mer_lexer__get_comment(token, token_context, io__state, io__state).
+:- mode mer_lexer__get_comment(out, out, di, uo) is det.
+
+mer_lexer__get_comment(Token, Context) -->
+	io__read_char(Result),
+	( { Result = error(Error) },
+		mer_lexer__get_context(Context),
+		{ Token = io_error(Error) }
+	; { Result = eof },
+		mer_lexer__get_context(Context),
+		{ Token = error("unterminated '/*' comment") }
+	; { Result = ok(Char) },
+		( { Char = ('*') } ->
+			mer_lexer__get_comment_2(Token, Context)
+		;
+			mer_lexer__get_comment(Token, Context)
+		)
+	).
+
+:- pred mer_lexer__string_get_comment(string, int, posn, token,
+				string_token_context, posn, posn).
+:- mode mer_lexer__string_get_comment(in, in, in, out, out, in, out) is det.
+
+mer_lexer__string_get_comment(String, Len, Posn0, Token, Context) -->
+	( mer_lexer__string_read_char(String, Len, Char) ->
+		( { Char = ('*') } ->
+			mer_lexer__string_get_comment_2(String, Len, Posn0,
+					Token, Context)
+		;
+			mer_lexer__string_get_comment(String, Len, Posn0,
+					Token, Context)
+		)
+	;
+		mer_lexer__string_get_context(Posn0, Context),
+		{ Token = error("unterminated '/*' comment") }
+	).
+
+:- pred mer_lexer__get_comment_2(token, token_context, io__state, io__state).
+:- mode mer_lexer__get_comment_2(out, out, di, uo) is det.
+
+mer_lexer__get_comment_2(Token, Context) -->
+	io__read_char(Result),
+	( { Result = error(Error) },
+		mer_lexer__get_context(Context),
+		{ Token = io_error(Error) }
+	; { Result = eof },
+		mer_lexer__get_context(Context),
+		{ Token = error("unterminated '/*' comment") }
+	; { Result = ok(Char) },
+		( { Char = ('/') } ->
+			% end of /* ... */ comment, so get next token
+			mer_lexer__get_token_2(Token, Context)
+		; { Char = ('*') } ->
+			mer_lexer__get_comment_2(Token, Context)
+		;
+			mer_lexer__get_comment(Token, Context)
+		)
+	).
+
+:- pred mer_lexer__string_get_comment_2(string, int, posn, token,
+				string_token_context, posn, posn).
+:- mode mer_lexer__string_get_comment_2(in, in, in, out, out, in, out) is det.
+
+mer_lexer__string_get_comment_2(String, Len, Posn0, Token, Context) -->
+	( mer_lexer__string_read_char(String, Len, Char) ->
+		( { Char = ('/') } ->
+			% end of /* ... */ comment, so get next token
+			mer_lexer__string_get_token_2(String, Len, Token, Context)
+		; { Char = ('*') } ->
+			mer_lexer__string_get_comment_2(String, Len, Posn0,
+				Token, Context)
+		;
+			mer_lexer__string_get_comment(String, Len, Posn0,
+				Token, Context)
+		)
+	;
+		mer_lexer__string_get_context(Posn0, Context),
+		{ Token = error("unterminated '/*' comment") }
+	).
+
+%-----------------------------------------------------------------------------%
+
+	% quoted names and quoted strings
+
+:- pred mer_lexer__get_quoted_name(char, list(char), token,
+				io__state, io__state).
+:- mode mer_lexer__get_quoted_name(in, in, out, di, uo) is det.
+
+mer_lexer__get_quoted_name(QuoteChar, Chars, Token) -->
+	io__read_char(Result),
+	( { Result = error(Error) },
+		{ Token = io_error(Error) }
+	; { Result = eof },
+		{ Token = error("unterminated quote") }
+	; { Result = ok(Char) },
+		( { Char = QuoteChar } ->
+			mer_lexer__get_quoted_name_quote(QuoteChar, Chars, Token)
+		; { Char = ('\\') } ->
+			mer_lexer__get_quoted_name_escape(QuoteChar, Chars, Token)
+		;
+			mer_lexer__get_quoted_name(QuoteChar, [Char | Chars], Token)
+		)
+	).
+
+:- pred mer_lexer__string_get_quoted_name(string, int, char, list(char), posn,
+				token, string_token_context, posn, posn).
+:- mode mer_lexer__string_get_quoted_name(in, in, in, in, in, out, out, in, out)
+	is det.
+
+mer_lexer__string_get_quoted_name(String, Len, QuoteChar, Chars, Posn0,
+		Token, Context) -->
+	( mer_lexer__string_read_char(String, Len, Char) ->
+		( { Char = QuoteChar } ->
+			mer_lexer__string_get_quoted_name_quote(String, Len,
+				QuoteChar, Chars, Posn0, Token, Context)
+		; { Char = ('\\') } ->
+			mer_lexer__string_get_quoted_name_escape(String, Len,
+				QuoteChar, Chars, Posn0, Token, Context)
+		;
+			mer_lexer__string_get_quoted_name(String, Len, QuoteChar,
+				[Char | Chars], Posn0, Token, Context)
+		)
+	;
+		mer_lexer__string_get_context(Posn0, Context),
+		{ Token = error("unterminated quote") }
+	).
+
+:- pred mer_lexer__get_quoted_name_quote(char, list(char), token,
+				io__state, io__state).
+:- mode mer_lexer__get_quoted_name_quote(in, in, out, di, uo) is det.
+
+mer_lexer__get_quoted_name_quote(QuoteChar, Chars, Token) -->
+	io__read_char(Result),
+	( { Result = error(Error) },
+		{ Token = io_error(Error) }
+	; { Result = eof },
+		{ mer_lexer__finish_quoted_name(QuoteChar, Chars, Token) }
+	; { Result = ok(Char) },
+		( { Char = QuoteChar } ->
+			mer_lexer__get_quoted_name(QuoteChar, [Char | Chars], Token)
+		;
+			io__putback_char(Char),
+			{ mer_lexer__finish_quoted_name(QuoteChar, Chars, Token) }
+		)
+	).
+
+:- pred mer_lexer__string_get_quoted_name_quote(string, int, char, list(char),
+				posn, token, string_token_context, posn, posn).
+:- mode mer_lexer__string_get_quoted_name_quote(in, in, in, in, in, out, out,
+				in, out) is det.
+
+mer_lexer__string_get_quoted_name_quote(String, Len, QuoteChar, Chars, Posn0,
+		Token, Context) -->
+	( mer_lexer__string_read_char(String, Len, Char) ->
+		( { Char = QuoteChar } ->
+			mer_lexer__string_get_quoted_name(String, Len, QuoteChar,
+				[Char | Chars], Posn0, Token, Context)
+		;
+			mer_lexer__string_ungetchar(String),
+			mer_lexer__string_get_context(Posn0, Context),
+			{ mer_lexer__finish_quoted_name(QuoteChar, Chars, Token) }
+		)
+	;
+		mer_lexer__string_get_context(Posn0, Context),
+		{ mer_lexer__finish_quoted_name(QuoteChar, Chars, Token) }
+	).
+
+:- pred mer_lexer__finish_quoted_name(char, list(char), token).
+:- mode mer_lexer__finish_quoted_name(in, in, out) is det.
+
+mer_lexer__finish_quoted_name(QuoteChar, Chars, Token) :-
+	mer_lexer__rev_char_list_to_string(Chars, String),
+	( QuoteChar = '''' ->
+		Token = name(String)
+	; QuoteChar = '"' ->
+		Token = string(String)
+	;
+		error("lexer.m: unknown quote character")
+	).
+
+:- pred mer_lexer__get_quoted_name_escape(char, list(char), token,
+					io__state, io__state).
+:- mode mer_lexer__get_quoted_name_escape(in, in, out, di, uo) is det.
+
+mer_lexer__get_quoted_name_escape(QuoteChar, Chars, Token) -->
+	io__read_char(Result),
+	( { Result = error(Error) },
+		{ Token = io_error(Error) }
+	; { Result = eof },
+		{ Token = error("unterminated quoted name") }
+	; { Result = ok(Char) },
+		( { Char = '\n' } ->
+			mer_lexer__get_quoted_name(QuoteChar, Chars, Token)
+		; { mer_lexer__escape_char(Char, EscapedChar) } ->
+			{ Chars1 = [EscapedChar | Chars] },
+			mer_lexer__get_quoted_name(QuoteChar, Chars1, Token)
+		; { Char = 'x' } ->
+			mer_lexer__get_hex_escape(QuoteChar, Chars, [], Token)
+		; { char__is_octal_digit(Char) } ->
+			mer_lexer__get_octal_escape(QuoteChar, Chars, [Char],
+				Token)
+		;
+			{ Token = error("invalid escape character") }
+		)
+	).
+
+:- pred mer_lexer__string_get_quoted_name_escape(string, int, char, list(char),
+					posn, token, string_token_context,
+					posn, posn).
+:- mode mer_lexer__string_get_quoted_name_escape(in, in, in, in, in, out, out,
+					in, out) is det.
+
+mer_lexer__string_get_quoted_name_escape(String, Len, QuoteChar, Chars, Posn0,
+		Token, Context) -->
+	=(Posn1),
+	( mer_lexer__string_read_char(String, Len, Char) ->
+		( { Char = '\n' } ->
+			mer_lexer__string_get_quoted_name(String, Len, QuoteChar,
+				Chars, Posn0, Token, Context)
+		; { mer_lexer__escape_char(Char, EscapedChar) } ->
+			{ Chars1 = [EscapedChar | Chars] },
+			mer_lexer__string_get_quoted_name(String, Len, QuoteChar,
+				Chars1, Posn0, Token, Context)
+		; { Char = 'x' } ->
+			mer_lexer__string_get_hex_escape(String, Len, QuoteChar,
+				Chars, [], Posn0, Token, Context)
+		; { char__is_octal_digit(Char) } ->
+			mer_lexer__string_get_octal_escape(String, Len, QuoteChar,
+				Chars, [Char], Posn0, Token, Context)
+		;
+			mer_lexer__string_get_context(Posn1, Context),
+			{ Token = error("invalid escape character") }
+		)
+	;
+		mer_lexer__string_get_context(Posn0, Context),
+		{ Token = error("unterminated quoted name") }
+	).
+
+:- pred mer_lexer__escape_char(char, char).
+:- mode mer_lexer__escape_char(in, out) is semidet.
+
+mer_lexer__escape_char('a', '\a').
+mer_lexer__escape_char('b', '\b').
+mer_lexer__escape_char('r', '\r').
+mer_lexer__escape_char('f', '\f').
+mer_lexer__escape_char('t', '\t').
+mer_lexer__escape_char('n', '\n').
+mer_lexer__escape_char('v', '\v').
+mer_lexer__escape_char('\\', '\\').
+mer_lexer__escape_char('''', '''').
+mer_lexer__escape_char('"', '"').
+mer_lexer__escape_char('`', '`').
+
+:- pred mer_lexer__get_hex_escape(char, list(char), list(char),
+				token, io__state, io__state).
+:- mode mer_lexer__get_hex_escape(in, in, in, out, di, uo) is det.
+
+mer_lexer__get_hex_escape(QuoteChar, Chars, HexChars, Token) -->
+	io__read_char(Result),
+	( { Result = error(Error) },
+		{ Token = io_error(Error) }
+	; { Result = eof },
+		{ Token = error("unterminated quote") }
+	; { Result = ok(Char) },
+		( { char__is_hex_digit(Char) } ->
+			mer_lexer__get_hex_escape(QuoteChar, Chars,
+						[Char | HexChars], Token)
+		; { Char = ('\\') } ->
+			mer_lexer__finish_hex_escape(QuoteChar, Chars, HexChars,
+				Token)
+		;
+			{ Token = error("unterminated hex escape") }
+		)
+	).
+
+:- pred mer_lexer__string_get_hex_escape(string, int, char, list(char), list(char),
+				posn, token, string_token_context, posn, posn).
+:- mode mer_lexer__string_get_hex_escape(in, in, in, in, in, in, out, out, in, out)
+	is det.
+
+mer_lexer__string_get_hex_escape(String, Len, QuoteChar, Chars, HexChars, Posn0,
+		Token, Context) -->
+	( mer_lexer__string_read_char(String, Len, Char) ->
+		( { char__is_hex_digit(Char) } ->
+			mer_lexer__string_get_hex_escape(String, Len, QuoteChar,
+				    Chars, [Char | HexChars], Posn0,
+				    Token, Context)
+		; { Char = ('\\') } ->
+			mer_lexer__string_finish_hex_escape(String, Len, QuoteChar,
+				    Chars, HexChars, Posn0, Token, Context)
+		;
+			mer_lexer__string_get_context(Posn0, Context),
+			{ Token = error("unterminated hex escape") }
+		)
+	;
+		mer_lexer__string_get_context(Posn0, Context),
+		{ Token = error("unterminated quote") }
+	).
+
+:- pred mer_lexer__finish_hex_escape(char, list(char), list(char),
+				token, io__state, io__state).
+:- mode mer_lexer__finish_hex_escape(in, in, in, out, di, uo) is det.
+
+mer_lexer__finish_hex_escape(QuoteChar, Chars, HexChars, Token) -->
+	( { HexChars = [] } ->
+		{ Token = error("empty hex escape") }
+	;
+		{ mer_lexer__rev_char_list_to_string(HexChars, HexString) },
+		(
+			{ string__base_string_to_int(16, HexString, Int) },
+			{ char__to_int(Char, Int) }
+		->
+			mer_lexer__get_quoted_name(QuoteChar, [Char|Chars], Token) 
+		;
+			{ Token = error("invalid hex escape") }
+		)
+	).
+
+:- pred mer_lexer__string_finish_hex_escape(string, int, char, list(char),
+				list(char), posn, token, string_token_context,
+				posn, posn).
+:- mode mer_lexer__string_finish_hex_escape(in, in, in, in, in, in, out, out, in, out)
+				is det.
+
+mer_lexer__string_finish_hex_escape(String, Len, QuoteChar, Chars, HexChars, Posn0,
+		Token, Context) -->
+	( { HexChars = [] } ->
+		mer_lexer__string_get_context(Posn0, Context),
+		{ Token = error("empty hex escape") }
+	;
+		{ mer_lexer__rev_char_list_to_string(HexChars, HexString) },
+		(
+			{ string__base_string_to_int(16, HexString, Int) },
+			{ char__to_int(Char, Int) }
+		->
+			mer_lexer__string_get_quoted_name(String, Len, QuoteChar,
+				[Char|Chars], Posn0, Token, Context)
+		;
+			mer_lexer__string_get_context(Posn0, Context),
+			{ Token = error("invalid hex escape") }
+		)
+	).
+
+:- pred mer_lexer__get_octal_escape(char, list(char), list(char),
+				token, io__state, io__state).
+:- mode mer_lexer__get_octal_escape(in, in, in, out, di, uo) is det.
+
+mer_lexer__get_octal_escape(QuoteChar, Chars, OctalChars, Token) -->
+	io__read_char(Result),
+	( { Result = error(Error) },
+		{ Token = io_error(Error) }
+	; { Result = eof },
+		{ Token = error("unterminated quote") }
+	; { Result = ok(Char) },
+		( { char__is_octal_digit(Char) } ->
+			mer_lexer__get_octal_escape(QuoteChar, Chars,
+						[Char | OctalChars], Token)
+		; { Char = ('\\') } ->
+			mer_lexer__finish_octal_escape(QuoteChar, Chars, OctalChars,
+				Token)
+		;
+			/****** 
+				% We don't report this as an error since
+				% we need bug-for-bug compatibility with
+				% NU-Prolog
+			{ Token = error("unterminated octal escape") }
+			******/
+			io__putback_char(Char),
+			mer_lexer__finish_octal_escape(QuoteChar, Chars, OctalChars,
+				Token)
+		)
+	).
+
+:- pred mer_lexer__string_get_octal_escape(string, int, char, list(char),
+				list(char), posn, token, string_token_context,
+				posn, posn).
+:- mode mer_lexer__string_get_octal_escape(in, in, in, in, in, in, out, out,
+				in, out) is det.
+
+mer_lexer__string_get_octal_escape(String, Len, QuoteChar, Chars, OctalChars,
+		Posn0, Token, Context) -->
+	( mer_lexer__string_read_char(String, Len, Char) ->
+		( { char__is_octal_digit(Char) } ->
+			mer_lexer__string_get_octal_escape(String, Len,
+				QuoteChar, Chars, [Char | OctalChars], Posn0,
+				Token, Context)
+		; { Char = ('\\') } ->
+			mer_lexer__string_finish_octal_escape(String, Len,
+				QuoteChar, Chars, OctalChars, Posn0,
+				Token, Context)
+		;
+			/****** 
+				% We don't report this as an error since
+				% we need bug-for-bug compatibility with
+				% NU-Prolog
+			{ Token = error("unterminated octal escape") }
+			******/
+			mer_lexer__string_ungetchar(String),
+			mer_lexer__string_finish_octal_escape(String, Len,
+				QuoteChar, Chars, OctalChars, Posn0,
+				Token, Context)
+		)
+	;
+		{ Token = error("unterminated quote") },
+		mer_lexer__string_get_context(Posn0, Context)
+	).
+
+:- pred mer_lexer__finish_octal_escape(char, list(char), list(char),
+				token, io__state, io__state).
+:- mode mer_lexer__finish_octal_escape(in, in, in, out, di, uo) is det.
+
+mer_lexer__finish_octal_escape(QuoteChar, Chars, OctalChars, Token) -->
+	( { OctalChars = [] } ->
+		{ Token = error("empty octal escape") }
+	;
+		{ mer_lexer__rev_char_list_to_string(OctalChars, OctalString) },
+		(
+			{ string__base_string_to_int(8, OctalString, Int) },
+			{ char__to_int(Char, Int) }
+		->
+			mer_lexer__get_quoted_name(QuoteChar, [Char|Chars], Token) 
+		;
+			{ Token = error("invalid octal escape") }
+		)
+	).
+
+:- pred mer_lexer__string_finish_octal_escape(string, int, char, list(char),
+				list(char), posn, token,
+				string_token_context, posn, posn).
+:- mode mer_lexer__string_finish_octal_escape(in, in, in, in, in, in, out, out,
+				in, out) is det.
+
+mer_lexer__string_finish_octal_escape(String, Len, QuoteChar, Chars, OctalChars,
+		Posn0, Token, Context) -->
+	( { OctalChars = [] } ->
+		{ Token = error("empty octal escape") },
+		mer_lexer__string_get_context(Posn0, Context)
+	;
+		{ mer_lexer__rev_char_list_to_string(OctalChars, OctalString) },
+		(
+			{ string__base_string_to_int(8, OctalString, Int) },
+			{ char__to_int(Char, Int) }
+		->
+			mer_lexer__string_get_quoted_name(String, Len, QuoteChar,
+				[Char|Chars], Posn0, Token, Context) 
+		;
+			{ Token = error("invalid octal escape") },
+			mer_lexer__string_get_context(Posn0, Context)
+		)
+	).
+
+%-----------------------------------------------------------------------------%
+
+	% names and variables
+
+:- pred mer_lexer__get_name(list(char), token, io__state, io__state).
+:- mode mer_lexer__get_name(in, out, di, uo) is det.
+
+mer_lexer__get_name(Chars, Token) -->
+	io__read_char(Result),
+	( { Result = error(Error) },
+		{ Token = io_error(Error) }
+	; { Result = eof },
+		{ mer_lexer__rev_char_list_to_string(Chars, Name) },
+		{ Token = name(Name) }
+	; { Result = ok(Char) },
+		( { char__is_alnum_or_underscore(Char) } ->
+			mer_lexer__get_name([Char | Chars], Token)
+		;
+			io__putback_char(Char),
+			{ mer_lexer__rev_char_list_to_string(Chars, Name) },
+			{ Token = name(Name) }
+		)
+	).
+
+:- pred mer_lexer__string_get_name(string, int, posn, token, string_token_context,
+				posn, posn).
+:- mode mer_lexer__string_get_name(in, in, in, out, out, in, out) is det.
+
+mer_lexer__string_get_name(String, Len, Posn0, Token, Context) -->
+	( mer_lexer__string_read_char(String, Len, Char) ->
+		( { char__is_alnum_or_underscore(Char) } ->
+			mer_lexer__string_get_name(String, Len, Posn0,
+				Token, Context)
+		;
+			mer_lexer__string_ungetchar(String),
+			mer_lexer__grab_string(String, Posn0, Name),
+			{ Token = name(Name) },
+			mer_lexer__string_get_context(Posn0, Context)
+		)
+	;
+		mer_lexer__grab_string(String, Posn0, Name),
+		{ Token = name(Name) },
+		mer_lexer__string_get_context(Posn0, Context)
+	).
+
+	%
+	% A line number directive token is `#' followed by an integer
+	% (specifying the line number) followed by a newline.
+	% Such a token sets the source line number for the next
+	% line, but it is otherwise ignored.  This means that line number
+	% directives may appear anywhere that a token may appear, including
+	% in the middle of terms.
+	% (The source file name can be set with a `:- pragma source_file' 
+	% declaration.)
+	%
+
+:- pred mer_lexer__get_source_line_number(list(char), token, token_context,
+	io__state, io__state).
+:- mode mer_lexer__get_source_line_number(in, out, out, di, uo) is det.
+
+mer_lexer__get_source_line_number(Chars, Token, Context) -->
+	io__read_char(Result),
+	( { Result = error(Error) },
+		mer_lexer__get_context(Context),
+		{ Token = io_error(Error) }
+	; { Result = eof },
+		mer_lexer__get_context(Context),
+		{ Token = error(
+			"unexpected end-of-file in `#' line number directive") }
+	; { Result = ok(Char) },
+		( { char__is_digit(Char) } ->
+			mer_lexer__get_source_line_number([Char | Chars],
+				Token, Context)
+		; { Char = '\n' } ->
+			{ mer_lexer__rev_char_list_to_string(Chars, String) },
+			(
+				{ string__base_string_to_int(10, String, Int) },
+				{ Int > 0 }
+			->
+				io__set_line_number(Int),
+				mer_lexer__get_token(Token, Context)
+			;
+				mer_lexer__get_context(Context),
+				{ string__append_list([
+					"invalid line number `", String,
+					"' in `#' line number directive"],
+					Message) },
+				{ Token = error(Message) }
+			)
+		;
+			mer_lexer__get_context(Context),
+			{ string__from_char_list([Char], String) },
+			{ string__append_list([
+				"invalid character `", String,
+				"' in `#' line number directive"],
+				Message) },
+			{ Token = error(Message) }
+		)
+	).
+
+:- pred mer_lexer__string_get_source_line_number(string, int, posn,
+		token, token_context, posn, posn).
+:- mode mer_lexer__string_get_source_line_number(in, in, in, out, out, in, out)
+		is det.
+
+mer_lexer__string_get_source_line_number(String, Len, Posn1, Token, Context) -->
+	( mer_lexer__string_read_char(String, Len, Char) ->
+		( { char__is_digit(Char) } ->
+			mer_lexer__string_get_source_line_number(String, Len,
+				Posn1, Token, Context)
+		; { Char = '\n' } ->
+			mer_lexer__grab_string(String, Posn1, LineNumString),
+			(
+				{ string__base_string_to_int(10, LineNumString,
+					LineNum) },
+				{ LineNum > 0 }
+			->
+				mer_lexer__string_set_line_number(LineNum),
+				mer_lexer__string_get_token(String, Len, Token, Context)
+			;
+				mer_lexer__string_get_context(Posn1, Context),
+				{ string__append_list([
+					"invalid line number `", LineNumString,
+					"' in `#' line number directive"],
+					Message) },
+				{ Token = error(Message) }
+			)
+		;
+			mer_lexer__string_get_context(Posn1, Context),
+			{ string__from_char_list([Char], DirectiveString) },
+			{ string__append_list([
+				"invalid character `", DirectiveString,
+				"' in `#' line number directive"],
+				Message) },
+			{ Token = error(Message) }
+		)
+	;
+		mer_lexer__string_get_context(Posn1, Context),
+		{ Token = error(
+			"unexpected end-of-file in `#' line number directive") }
+	).
+
+:- pred mer_lexer__get_graphic(list(char), token, io__state, io__state).
+:- mode mer_lexer__get_graphic(in, out, di, uo) is det.
+
+mer_lexer__get_graphic(Chars, Token) -->
+	io__read_char(Result),
+	( { Result = error(Error) },
+		{ Token = io_error(Error) }
+	; { Result = eof },
+		{ mer_lexer__rev_char_list_to_string(Chars, Name) },
+		{ Token = name(Name) }
+	; { Result = ok(Char) },
+		( { mer_lexer__graphic_token_char(Char) } ->
+			mer_lexer__get_graphic([Char | Chars], Token)
+		;
+			io__putback_char(Char),
+			{ mer_lexer__rev_char_list_to_string(Chars, Name) },
+			{ Token = name(Name) }
+		)
+	).
+
+:- pred mer_lexer__string_get_graphic(string, int, posn, token,
+			string_token_context, posn, posn).
+:- mode mer_lexer__string_get_graphic(in, in, in, out, out, in, out) is det.
+
+mer_lexer__string_get_graphic(String, Len, Posn0, Token, Context) -->
+	( mer_lexer__string_read_char(String, Len, Char) ->
+		( { mer_lexer__graphic_token_char(Char) } ->
+			mer_lexer__string_get_graphic(String, Len, Posn0,
+				Token, Context)
+		;
+			mer_lexer__string_ungetchar(String),
+			mer_lexer__grab_string(String, Posn0, Name),
+			{ Token = name(Name) },
+			mer_lexer__string_get_context(Posn0, Context)
+		)
+	;
+		mer_lexer__grab_string(String, Posn0, Name),
+		mer_lexer__string_get_context(Posn0, Context),
+		{ Token = name(Name) }
+	).
+
+:- pred mer_lexer__get_variable(list(char), token, io__state, io__state).
+:- mode mer_lexer__get_variable(in, out, di, uo) is det.
+
+mer_lexer__get_variable(Chars, Token) -->
+	io__read_char(Result),
+	( { Result = error(Error) },
+		{ Token = io_error(Error) }
+	; { Result = eof },
+		{ mer_lexer__rev_char_list_to_string(Chars, VariableName) },
+		{ Token = variable(VariableName) }
+	; { Result = ok(Char) },
+		( { char__is_alnum_or_underscore(Char) } ->
+			mer_lexer__get_variable([Char | Chars], Token)
+		;
+			io__putback_char(Char),
+			{ mer_lexer__rev_char_list_to_string(Chars, VariableName) },
+			{ Token = variable(VariableName) }
+		)
+	).
+
+:- pred mer_lexer__string_get_variable(string, int, posn, token,
+					string_token_context, posn, posn).
+:- mode mer_lexer__string_get_variable(in, in, in, out, out, in, out) is det.
+
+mer_lexer__string_get_variable(String, Len, Posn0, Token, Context) -->
+	( mer_lexer__string_read_char(String, Len, Char) ->
+		( { char__is_alnum_or_underscore(Char) } ->
+			mer_lexer__string_get_variable(String, Len, Posn0,
+				Token, Context)
+		;
+			mer_lexer__string_ungetchar(String),
+			mer_lexer__grab_string(String, Posn0, VariableName),
+			{ Token = variable(VariableName) },
+			mer_lexer__string_get_context(Posn0, Context)
+		)
+	;
+		mer_lexer__grab_string(String, Posn0, VariableName),
+		{ Token = variable(VariableName) },
+		mer_lexer__string_get_context(Posn0, Context)
+	).
+
+%-----------------------------------------------------------------------------%
+
+	% integer and float literals
+
+:- pred mer_lexer__get_zero(token, io__state, io__state).
+:- mode mer_lexer__get_zero(out, di, uo) is det.
+
+mer_lexer__get_zero(Token) -->
+	io__read_char(Result),
+	( { Result = error(Error) },
+		{ Token = io_error(Error) }
+	; { Result = eof },
+		{ Token = integer(0) }
+	; { Result = ok(Char) },
+		( { char__is_digit(Char) } ->
+			mer_lexer__get_number([Char], Token)
+		; { Char = '''' } ->
+			mer_lexer__get_char_code(Token)
+		; { Char = 'b' } ->
+			mer_lexer__get_binary(Token)
+		; { Char = 'o' } ->
+			mer_lexer__get_octal(Token)
+		; { Char = 'x' } ->
+			mer_lexer__get_hex(Token)
+		; { Char = ('.') } ->
+			mer_lexer__get_int_dot(['0'], Token)
+		; { Char = 'e' ; Char = 'E' } ->
+			mer_lexer__get_float_exponent([Char, '0'], Token)
+		;
+			io__putback_char(Char),
+			{ Token = integer(0) }
+		)
+	).
+
+:- pred mer_lexer__string_get_zero(string, int, posn, token, string_token_context,
+				posn, posn).
+:- mode mer_lexer__string_get_zero(in, in, in, out, out, in, out) is det.
+
+mer_lexer__string_get_zero(String, Len, Posn0, Token, Context) -->
+	( mer_lexer__string_read_char(String, Len, Char) ->
+		( { char__is_digit(Char) } ->
+			mer_lexer__string_get_number(String, Len, Posn0,
+				Token, Context)
+		; { Char = '''' } ->
+			mer_lexer__string_get_char_code(String, Len, Posn0,
+				Token, Context)
+		; { Char = 'b' } ->
+			mer_lexer__string_get_binary(String, Len, Posn0,
+				Token, Context)
+		; { Char = 'o' } ->
+			mer_lexer__string_get_octal(String, Len, Posn0,
+				Token, Context)
+		; { Char = 'x' } ->
+			mer_lexer__string_get_hex(String, Len, Posn0,
+				Token, Context)
+		; { Char = ('.') } ->
+			mer_lexer__string_get_int_dot(String, Len, Posn0,
+				Token, Context)
+		; { Char = 'e' ; Char = 'E' } ->
+			mer_lexer__string_get_float_exponent(String, Len, Posn0,
+				Token, Context)
+		;
+			mer_lexer__string_ungetchar(String),
+			mer_lexer__string_get_context(Posn0, Context),
+			{ Token = integer(0) }
+		)
+	;
+		mer_lexer__string_get_context(Posn0, Context),
+		{ Token = integer(0) }
+	).
+
+:- pred mer_lexer__get_char_code(token, io__state, io__state).
+:- mode mer_lexer__get_char_code(out, di, uo) is det.
+
+mer_lexer__get_char_code(Token) -->
+	io__read_char(Result),
+	( { Result = error(Error) },
+		{ Token = io_error(Error) }
+	; { Result = eof },
+		{ Token = error("unterminated char code constant") }
+	; { Result = ok(Char) },
+		{ char__to_int(Char, CharCode) },
+		{ Token = integer(CharCode) }
+	).
+
+:- pred mer_lexer__string_get_char_code(string, int, posn, token,
+				string_token_context, posn, posn).
+:- mode mer_lexer__string_get_char_code(in, in, in, out, out, in, out) is det.
+
+mer_lexer__string_get_char_code(String, Len, Posn0, Token, Context) -->
+	( mer_lexer__string_read_char(String, Len, Char) ->
+		{ char__to_int(Char, CharCode) },
+		{ Token = integer(CharCode) },
+		mer_lexer__string_get_context(Posn0, Context)
+	;
+		{ Token = error("unterminated char code constant") },
+		mer_lexer__string_get_context(Posn0, Context)
+	).
+
+:- pred mer_lexer__get_binary(token, io__state, io__state).
+:- mode mer_lexer__get_binary(out, di, uo) is det.
+
+mer_lexer__get_binary(Token) -->
+	io__read_char(Result),
+	( { Result = error(Error) },
+		{ Token = io_error(Error) }
+	; { Result = eof },
+		{ Token = error("unterminated binary constant") }
+	; { Result = ok(Char) },
+		( { char__is_binary_digit(Char) } ->
+			mer_lexer__get_binary_2([Char], Token)
+		;
+			io__putback_char(Char),
+			{ Token = error("unterminated binary constant") }
+		)
+	).
+
+:- pred mer_lexer__string_get_binary(string, int, posn, token, string_token_context,
+		posn, posn).
+:- mode mer_lexer__string_get_binary(in, in, in, out, out, in, out) is det.
+
+mer_lexer__string_get_binary(String, Len, Posn0, Token, Context) -->
+	( mer_lexer__string_read_char(String, Len, Char) ->
+		( { char__is_binary_digit(Char) } ->
+			mer_lexer__string_get_binary_2(String, Len, Posn0,
+				Token, Context)
+		;
+			mer_lexer__string_ungetchar(String),
+			{ Token = error("unterminated binary constant") },
+			mer_lexer__string_get_context(Posn0, Context)
+		)
+	;
+		{ Token = error("unterminated binary constant") },
+		mer_lexer__string_get_context(Posn0, Context)
+	).
+
+:- pred mer_lexer__get_binary_2(list(char), token, io__state, io__state).
+:- mode mer_lexer__get_binary_2(in, out, di, uo) is det.
+
+mer_lexer__get_binary_2(Chars, Token) -->
+	io__read_char(Result),
+	( { Result = error(Error) },
+		{ Token = io_error(Error) }
+	; { Result = eof },
+		{ mer_lexer__rev_char_list_to_int(Chars, 2, Token) }
+	; { Result = ok(Char) },
+		( { char__is_binary_digit(Char) } ->
+			mer_lexer__get_binary_2([Char | Chars], Token)
+		;
+			io__putback_char(Char),
+			{ mer_lexer__rev_char_list_to_int(Chars, 2, Token) }
+		)
+	).
+
+:- pred mer_lexer__string_get_binary_2(string, int, posn, token,
+			string_token_context, posn, posn).
+:- mode mer_lexer__string_get_binary_2(in, in, in, out, out, in, out) is det.
+
+mer_lexer__string_get_binary_2(String, Len, Posn0, Token, Context) -->
+	( mer_lexer__string_read_char(String, Len, Char) ->
+		( { char__is_binary_digit(Char) } ->
+			mer_lexer__string_get_binary_2(String, Len, Posn0,
+				Token, Context)
+		;
+			mer_lexer__string_ungetchar(String),
+			mer_lexer__grab_string(String, Posn0, BinaryString),
+			{ mer_lexer__conv_string_to_int(BinaryString, 2, Token) },
+			mer_lexer__string_get_context(Posn0, Context)
+		)
+	;
+		mer_lexer__grab_string(String, Posn0, BinaryString),
+		{ mer_lexer__conv_string_to_int(BinaryString, 2, Token) },
+		mer_lexer__string_get_context(Posn0, Context)
+	).
+
+:- pred mer_lexer__get_octal(token, io__state, io__state).
+:- mode mer_lexer__get_octal(out, di, uo) is det.
+
+mer_lexer__get_octal(Token) -->
+	io__read_char(Result),
+	( { Result = error(Error) },
+		{ Token = io_error(Error) }
+	; { Result = eof },
+		{ Token = error("unterminated octal constant") }
+	; { Result = ok(Char) },
+		( { char__is_octal_digit(Char) } ->
+			mer_lexer__get_octal_2([Char], Token)
+		;
+			io__putback_char(Char),
+			{ Token = error("unterminated octal constant") }
+		)
+	).
+
+:- pred mer_lexer__string_get_octal(string, int, posn, token, string_token_context,
+				posn, posn).
+:- mode mer_lexer__string_get_octal(in, in, in, out, out, in, out) is det.
+
+mer_lexer__string_get_octal(String, Len, Posn0, Token, Context) -->
+	( mer_lexer__string_read_char(String, Len, Char) ->
+		( { char__is_octal_digit(Char) } ->
+			mer_lexer__string_get_octal_2(String, Len, Posn0,
+				Token, Context)
+		;
+			mer_lexer__string_ungetchar(String),
+			{ Token = error("unterminated octal constant") },
+			mer_lexer__string_get_context(Posn0, Context)
+		)
+	;
+		{ Token = error("unterminated octal constant") },
+		mer_lexer__string_get_context(Posn0, Context)
+	).
+
+:- pred mer_lexer__get_octal_2(list(char), token, io__state, io__state).
+:- mode mer_lexer__get_octal_2(in, out, di, uo) is det.
+
+mer_lexer__get_octal_2(Chars, Token) -->
+	io__read_char(Result),
+	( { Result = error(Error) },
+		{ Token = io_error(Error) }
+	; { Result = eof },
+		{ mer_lexer__rev_char_list_to_int(Chars, 8, Token) }
+	; { Result = ok(Char) },
+		( { char__is_octal_digit(Char) } ->
+			mer_lexer__get_octal_2([Char | Chars], Token)
+		;
+			io__putback_char(Char),
+			{ mer_lexer__rev_char_list_to_int(Chars, 8, Token) }
+		)
+	).
+
+:- pred mer_lexer__string_get_octal_2(string, int, posn, token,
+				string_token_context, posn, posn).
+:- mode mer_lexer__string_get_octal_2(in, in, in, out, out, in, out) is det.
+
+mer_lexer__string_get_octal_2(String, Len, Posn0, Token, Context) -->
+	( mer_lexer__string_read_char(String, Len, Char) ->
+		( { char__is_octal_digit(Char) } ->
+			mer_lexer__string_get_octal_2(String, Len, Posn0,
+				Token, Context)
+		;
+			mer_lexer__string_ungetchar(String),
+			mer_lexer__grab_string(String, Posn0, BinaryString),
+			{ mer_lexer__conv_string_to_int(BinaryString, 8, Token) },
+			mer_lexer__string_get_context(Posn0, Context)
+		)
+	;
+		mer_lexer__grab_string(String, Posn0, BinaryString),
+		{ mer_lexer__conv_string_to_int(BinaryString, 8, Token) },
+		mer_lexer__string_get_context(Posn0, Context)
+	).
+
+:- pred mer_lexer__get_hex(token, io__state, io__state).
+:- mode mer_lexer__get_hex(out, di, uo) is det.
+
+mer_lexer__get_hex(Token) -->
+	io__read_char(Result),
+	( { Result = error(Error) },
+		{ Token = io_error(Error) }
+	; { Result = eof },
+		{ Token = error("unterminated hex constant") }
+	; { Result = ok(Char) },
+		( { char__is_hex_digit(Char) } ->
+			mer_lexer__get_hex_2([Char], Token)
+		;
+			io__putback_char(Char),
+			{ Token = error("unterminated hex constant") }
+		)
+	).
+
+:- pred mer_lexer__string_get_hex(string, int, posn, token, string_token_context,
+				posn, posn).
+:- mode mer_lexer__string_get_hex(in, in, in, out, out, in, out) is det.
+
+mer_lexer__string_get_hex(String, Len, Posn0, Token, Context) -->
+	( mer_lexer__string_read_char(String, Len, Char) ->
+		( { char__is_hex_digit(Char) } ->
+			mer_lexer__string_get_hex_2(String, Len, Posn0,
+				Token, Context)
+		;
+			mer_lexer__string_ungetchar(String),
+			{ Token = error("unterminated hex constant") },
+			mer_lexer__string_get_context(Posn0, Context)
+		)
+	;
+		{ Token = error("unterminated hex constant") },
+		mer_lexer__string_get_context(Posn0, Context)
+	).
+
+
+:- pred mer_lexer__get_hex_2(list(char), token, io__state, io__state).
+:- mode mer_lexer__get_hex_2(in, out, di, uo) is det.
+
+mer_lexer__get_hex_2(Chars, Token) -->
+	io__read_char(Result),
+	( { Result = error(Error) },
+		{ Token = io_error(Error) }
+	; { Result = eof },
+		{ mer_lexer__rev_char_list_to_int(Chars, 16, Token) }
+	; { Result = ok(Char) },
+		( { char__is_hex_digit(Char) } ->
+			mer_lexer__get_hex_2([Char | Chars], Token)
+		;
+			io__putback_char(Char),
+			{ mer_lexer__rev_char_list_to_int(Chars, 16, Token) }
+		)
+	).
+
+:- pred mer_lexer__string_get_hex_2(string, int, posn, token,
+				string_token_context, posn, posn).
+:- mode mer_lexer__string_get_hex_2(in, in, in, out, out, in, out) is det.
+
+mer_lexer__string_get_hex_2(String, Len, Posn0, Token, Context) -->
+	( mer_lexer__string_read_char(String, Len, Char) ->
+		( { char__is_hex_digit(Char) } ->
+			mer_lexer__string_get_hex_2(String, Len, Posn0,
+				Token, Context)
+		;
+			mer_lexer__string_ungetchar(String),
+			mer_lexer__grab_string(String, Posn0, BinaryString),
+			{ mer_lexer__conv_string_to_int(BinaryString, 16, Token) },
+			mer_lexer__string_get_context(Posn0, Context)
+		)
+	;
+		mer_lexer__grab_string(String, Posn0, BinaryString),
+		{ mer_lexer__conv_string_to_int(BinaryString, 16, Token) },
+		mer_lexer__string_get_context(Posn0, Context)
+	).
+
+
+:- pred mer_lexer__get_number(list(char), token, io__state, io__state).
+:- mode mer_lexer__get_number(in, out, di, uo) is det.
+
+mer_lexer__get_number(Chars, Token) -->
+	io__read_char(Result),
+	( { Result = error(Error) },
+		{ Token = io_error(Error) }
+	; { Result = eof },
+		{ mer_lexer__rev_char_list_to_int(Chars, 10, Token) }
+	; { Result = ok(Char) },
+		( { char__is_digit(Char) } ->
+			mer_lexer__get_number([Char | Chars], Token)
+		; { Char = ('.') } ->
+			mer_lexer__get_int_dot(Chars, Token)
+		; { Char = 'e' ; Char = 'E' } ->
+			mer_lexer__get_float_exponent([Char | Chars], Token)
+		;
+			io__putback_char(Char),
+			{ mer_lexer__rev_char_list_to_int(Chars, 10, Token) }
+		)
+	).
+
+:- pred mer_lexer__string_get_number(string, int, posn, token,
+					string_token_context, posn, posn).
+:- mode mer_lexer__string_get_number(in, in, in, out, out, in, out) is det.
+
+mer_lexer__string_get_number(String, Len, Posn0, Token, Context) -->
+	( mer_lexer__string_read_char(String, Len, Char) ->
+		( { char__is_digit(Char) } ->
+			mer_lexer__string_get_number(String, Len, Posn0,
+				Token, Context)
+		; { Char = ('.') } ->
+			mer_lexer__string_get_int_dot(String, Len, Posn0,
+				Token, Context)
+		; { Char = 'e' ; Char = 'E' } ->
+			mer_lexer__string_get_float_exponent(String, Len, Posn0,
+				Token, Context)
+		;
+			mer_lexer__string_ungetchar(String),
+			mer_lexer__grab_string(String, Posn0, NumberString),
+			{ mer_lexer__conv_string_to_int(NumberString, 10, Token) },
+			mer_lexer__string_get_context(Posn0, Context)
+		)
+	;
+		mer_lexer__grab_string(String, Posn0, NumberString),
+		{ mer_lexer__conv_string_to_int(NumberString, 10, Token) },
+		mer_lexer__string_get_context(Posn0, Context)
+	).
+
+	% XXX the float literal syntax doesn't match ISO Prolog
+
+:- pred mer_lexer__get_int_dot(list(char), token, io__state, io__state).
+:- mode mer_lexer__get_int_dot(in, out, di, uo) is det.
+
+mer_lexer__get_int_dot(Chars, Token) -->
+	io__read_char(Result),
+	( { Result = error(Error) },
+		{ Token = io_error(Error) }
+	; { Result = eof },
+		io__putback_char('.'),
+		{ mer_lexer__rev_char_list_to_int(Chars, 10, Token) }
+	; { Result = ok(Char) },
+		( { char__is_digit(Char) } ->
+			mer_lexer__get_float_decimals([Char, '.' | Chars], Token)
+		;
+			io__putback_char(Char),
+			io__putback_char('.'),
+			{ mer_lexer__rev_char_list_to_int(Chars, 10, Token) }
+		)
+	).
+
+:- pred mer_lexer__string_get_int_dot(string, int, posn, token,
+					string_token_context, posn, posn).
+:- mode mer_lexer__string_get_int_dot(in, in, in, out, out, in, out) is det.
+
+mer_lexer__string_get_int_dot(String, Len, Posn0, Token, Context) -->
+	( mer_lexer__string_read_char(String, Len, Char) ->
+		( { char__is_digit(Char) } ->
+			mer_lexer__string_get_float_decimals(String, Len, Posn0,
+				Token, Context)
+		;
+			mer_lexer__string_ungetchar(String),
+			mer_lexer__string_ungetchar(String),
+			mer_lexer__grab_string(String, Posn0, NumberString),
+			{ mer_lexer__conv_string_to_int(NumberString, 10, Token) },
+			mer_lexer__string_get_context(Posn0, Context)
+		)
+	;
+		mer_lexer__string_ungetchar(String),
+		mer_lexer__grab_string(String, Posn0, NumberString),
+		{ mer_lexer__conv_string_to_int(NumberString, 10, Token) },
+		mer_lexer__string_get_context(Posn0, Context)
+	).
+
+:- pred mer_lexer__get_float_decimals(list(char), token, io__state, io__state).
+:- mode mer_lexer__get_float_decimals(in, out, di, uo) is det.
+
+	% we've read past the decimal point, so now get the decimals
+
+mer_lexer__get_float_decimals(Chars, Token) -->
+	io__read_char(Result),
+	( { Result = error(Error) },
+		{ Token = io_error(Error) }
+	; { Result = eof },
+		{ mer_lexer__rev_char_list_to_float(Chars, Token) }
+	; { Result = ok(Char) },
+		( { char__is_digit(Char) } ->
+			mer_lexer__get_float_decimals([Char | Chars], Token)
+		; { Char = 'e' ; Char = 'E' } ->
+			mer_lexer__get_float_exponent([Char | Chars], Token)
+		;
+			io__putback_char(Char),
+			{ mer_lexer__rev_char_list_to_float(Chars, Token) }
+		)
+	).
+
+:- pred mer_lexer__string_get_float_decimals(string, int, posn, token,
+					string_token_context, posn, posn).
+:- mode mer_lexer__string_get_float_decimals(in, in, in, out, out, in, out) is det.
+
+mer_lexer__string_get_float_decimals(String, Len, Posn0, Token, Context) -->
+	( mer_lexer__string_read_char(String, Len, Char) ->
+		( { char__is_digit(Char) } ->
+			mer_lexer__string_get_float_decimals(String, Len, Posn0,
+				Token, Context)
+		; { Char = 'e' ; Char = 'E' } ->
+			mer_lexer__string_get_float_exponent(String, Len, Posn0,
+				Token, Context)
+		;
+			mer_lexer__string_ungetchar(String),
+			mer_lexer__grab_string(String, Posn0, FloatString),
+			{ mer_lexer__conv_to_float(FloatString, Token) },
+			mer_lexer__string_get_context(Posn0, Context)
+		)
+	;
+		mer_lexer__grab_string(String, Posn0, FloatString),
+		{ mer_lexer__conv_to_float(FloatString, Token) },
+		mer_lexer__string_get_context(Posn0, Context)
+	).
+
+:- pred mer_lexer__get_float_exponent(list(char), token, io__state, io__state).
+:- mode mer_lexer__get_float_exponent(in, out, di, uo) is det.
+
+mer_lexer__get_float_exponent(Chars, Token) -->
+	io__read_char(Result),
+	( { Result = error(Error) },
+		{ Token = io_error(Error) }
+	; { Result = eof },
+		{ mer_lexer__rev_char_list_to_float(Chars, Token) }
+	; { Result = ok(Char) },
+		( { Char = ('+') ; Char = ('-') } ->
+			mer_lexer__get_float_exponent_2([Char | Chars], Token)
+		; { char__is_digit(Char) } ->
+			mer_lexer__get_float_exponent_3([Char | Chars], Token)
+		;
+			io__putback_char(Char),
+			{ Token =
+			  error("unterminated exponent in float token") }
+		)
+	).
+
+:- pred mer_lexer__string_get_float_exponent(string, int, posn, token, 	
+				string_token_context, posn, posn).
+:- mode mer_lexer__string_get_float_exponent(in, in, in, out, out, in, out) is det.
+
+mer_lexer__string_get_float_exponent(String, Len, Posn0, Token, Context) -->
+	( mer_lexer__string_read_char(String, Len, Char) ->
+		( { Char = ('+') ; Char = ('-') } ->
+			mer_lexer__string_get_float_exponent_2(String, Len, Posn0,
+				Token, Context)
+		; { char__is_digit(Char) } ->
+			mer_lexer__string_get_float_exponent_3(String, Len, Posn0,
+				Token, Context)
+		;
+			mer_lexer__string_ungetchar(String),
+			{ Token =
+			  error("unterminated exponent in float token") },
+			mer_lexer__string_get_context(Posn0, Context)
+		)
+	;
+		mer_lexer__grab_string(String, Posn0, FloatString),
+		{ mer_lexer__conv_to_float(FloatString, Token) },
+		mer_lexer__string_get_context(Posn0, Context)
+	).
+
+:- pred mer_lexer__get_float_exponent_2(list(char), token,
+				io__state, io__state).
+:- mode mer_lexer__get_float_exponent_2(in, out, di, uo) is det.
+
+	% we've read past the E signalling the start of the exponent -
+	% make sure that there's at least one digit following,
+	% and then get the remaining digits
+
+mer_lexer__get_float_exponent_2(Chars, Token) -->
+	io__read_char(Result),
+	( { Result = error(Error) },
+		{ Token = io_error(Error) }
+	; { Result = eof },
+		{ Token = error("unterminated exponent in float token") }
+	; { Result = ok(Char) },
+		( { char__is_digit(Char) } ->
+			mer_lexer__get_float_exponent_3([Char | Chars], Token)
+		;
+			io__putback_char(Char),
+			{ Token =
+			  error("unterminated exponent in float token") }
+		)
+	).
+
+:- pred mer_lexer__string_get_float_exponent_2(string, int, posn, token,
+				string_token_context, posn, posn).
+:- mode mer_lexer__string_get_float_exponent_2(in, in, in, out, out, in, out)
+				is det.
+
+	% we've read past the E signalling the start of the exponent -
+	% make sure that there's at least one digit following,
+	% and then get the remaining digits
+
+mer_lexer__string_get_float_exponent_2(String, Len, Posn0, Token, Context) -->
+	( mer_lexer__string_read_char(String, Len, Char) ->
+		( { char__is_digit(Char) } ->
+			mer_lexer__string_get_float_exponent_3(String, Len, Posn0,
+				Token, Context)
+		;
+			mer_lexer__string_ungetchar(String),
+			{ Token =
+			  error("unterminated exponent in float token") },
+			mer_lexer__string_get_context(Posn0, Context)
+		)
+	;
+		{ Token = error("unterminated exponent in float token") },
+		mer_lexer__string_get_context(Posn0, Context)
+	).
+
+:- pred mer_lexer__get_float_exponent_3(list(char), token,
+					io__state, io__state).
+:- mode mer_lexer__get_float_exponent_3(in, out, di, uo) is det.
+
+	% we've read past the first digit of the exponent -
+	% now get the remaining digits
+
+mer_lexer__get_float_exponent_3(Chars, Token) -->
+	io__read_char(Result),
+	( { Result = error(Error) },
+		{ Token = io_error(Error) }
+	; { Result = eof },
+		{ mer_lexer__rev_char_list_to_float(Chars, Token) }
+	; { Result = ok(Char) },
+		( { char__is_digit(Char) } ->
+			mer_lexer__get_float_exponent_3([Char | Chars], Token)
+		;
+			io__putback_char(Char),
+			{ mer_lexer__rev_char_list_to_float(Chars, Token) }
+		)
+	).
+
+:- pred mer_lexer__string_get_float_exponent_3(string, int, posn, token,
+				string_token_context, posn, posn).
+:- mode mer_lexer__string_get_float_exponent_3(in, in, in, out, out, in, out)
+				is det.
+
+mer_lexer__string_get_float_exponent_3(String, Len, Posn0, Token, Context) -->
+	( mer_lexer__string_read_char(String, Len, Char) ->
+		( { char__is_digit(Char) } ->
+			mer_lexer__string_get_float_exponent_3(String, Len, Posn0,
+				Token, Context)
+		;
+			mer_lexer__string_ungetchar(String),
+			mer_lexer__grab_string(String, Posn0, FloatString),
+			{ mer_lexer__conv_to_float(FloatString, Token) },
+			mer_lexer__string_get_context(Posn0, Context)
+		)
+	;
+		mer_lexer__grab_string(String, Posn0, FloatString),
+		{ mer_lexer__conv_to_float(FloatString, Token) },
+		mer_lexer__string_get_context(Posn0, Context)
+	).
+
+%-----------------------------------------------------------------------------%
+
+	% Utility routines
+
+:- pred mer_lexer__rev_char_list_to_int(list(char), int, token).
+:- mode mer_lexer__rev_char_list_to_int(in, in, out) is det.
+
+mer_lexer__rev_char_list_to_int(RevChars, Base, Token) :-
+	mer_lexer__rev_char_list_to_string(RevChars, String),
+	mer_lexer__conv_string_to_int(String, Base, Token).
+
+:- pred mer_lexer__conv_string_to_int(string, int, token).
+:- mode mer_lexer__conv_string_to_int(in, in, out) is det.
+
+mer_lexer__conv_string_to_int(String, Base, Token) :-
+	( string__base_string_to_int(Base, String, Int) ->
+		Token = integer(Int)
+	;
+		Token = error("invalid integer token")
+	).
+
+:- pred mer_lexer__rev_char_list_to_float(list(char), token).
+:- mode mer_lexer__rev_char_list_to_float(in, out) is det.
+
+mer_lexer__rev_char_list_to_float(RevChars, Token) :-
+	mer_lexer__rev_char_list_to_string(RevChars, String),
+	mer_lexer__conv_to_float(String, Token).
+
+:- pred mer_lexer__conv_to_float(string, token).
+:- mode mer_lexer__conv_to_float(in, out) is det.
+
+mer_lexer__conv_to_float(String, Token) :-
+	( string__to_float(String, Float) ->
+		Token = float(Float)
+	;
+		Token = error("invalid float token")
+	).
+
+:- pred mer_lexer__rev_char_list_to_string(list(char), string).
+:- mode mer_lexer__rev_char_list_to_string(in, out) is det.
+
+mer_lexer__rev_char_list_to_string(RevChars, String) :-
+       string__from_rev_char_list(RevChars, String).
+
+%-----------------------------------------------------------------------------%
Index: library/mer_parser.m
===================================================================
RCS file: mer_parser.m
diff -N mer_parser.m
--- /dev/null	Thu Sep  2 15:00:04 1999
+++ mer_parser.m	Tue Aug 22 12:17:41 2000
@@ -0,0 +1,877 @@
+%---------------------------------------------------------------------------%
+% Copyright (C) 1995-2000 The University of Melbourne.
+% This file may only be copied under the terms of the GNU Library General
+% Public License - see the file COPYING.LIB in the Mercury distribution.
+%---------------------------------------------------------------------------%
+%
+% file: mer_parser.m.
+% main author: fjh.
+% stability: high.
+%
+% This file exports the predicate mer_parser__read_term, which reads
+% a term from the current input stream.
+% The mer_parser__read_term_from_string predicates are the same as the
+% read_term predicates, except that the term is read from
+% a string rather than from the current input stream.
+% The mer_parser__parse_token_list predicate is similar,
+% but it takes a list of tokens rather than a string.
+%
+% The parser and lexer are intended to exactly follow ISO Prolog
+% syntax, but there are some departures from that for three reasons:
+%
+%	(1) I wrote some of the code at home when the ISO Prolog draft
+%	    was at uni - so in some places I just guessed.
+%	(2) In some places the lexer reports an error when it shouldn't.
+%	(3) There are a couple of hacks to make it compatible with NU-Prolog
+%	    syntax.
+%
+% The parser is a relatively straight-forward top-down recursive descent
+% parser, made somewhat complicated by the need to handle operator
+% precedences.  It uses `mer_lexer__get_token_list' to read a list of tokens.
+% It uses the routines in module `ops' to look up operator precedences.
+%
+%-----------------------------------------------------------------------------%
+
+:- module mer_parser.
+:- interface.
+:- import_module io, term_io, mer_lexer.
+
+%-----------------------------------------------------------------------------%
+%
+% mer_parser__read_term/{3,4}:
+%	Reads in (and parses) terms from the current input stream.
+
+:- pred mer_parser__read_term(read_term(T), io__state, io__state).
+:- mode mer_parser__read_term(out, di, uo) is det.
+% mer_parser__read_term(Result):
+%	Reads a term from the current input stream.
+
+:- pred mer_parser__read_term(string, read_term(T), io__state, io__state).
+:- mode mer_parser__read_term(in, out, di, uo) is det.
+% mer_parser__read_term(FileName, Result):
+%	Reads a term from the current input stream.
+%	The string is the filename to use for the current input stream;
+%	this is used in constructing the term__contexts in the read term.
+%	This interface is used to support the `:- pragma source_file'
+%	directive.
+
+%-----------------------------------------------------------------------------%
+%
+% mer_parser__read_term_from_string/{4,6}:
+%	Parses terms from a string.
+
+	% The read_term_from_string predicates are the same as the
+	% read_term predicates, except that the term is read from
+	% a string rather than from the current input stream.
+	% The returned value `EndPos' is the position one character
+	% past the end of the term read.
+	% The arguments `MaxOffset' and `StartPos' in the six-argument version
+	% specify the length of the string and the position within the
+	% string at which to start parsing.
+
+:- pred mer_parser__read_term_from_string(string, string, posn, read_term(T)).
+:- mode mer_parser__read_term_from_string(in, in, out, out) is det.
+%	mer_parser__read_term_from_string(FileName, String, EndPos, Term).
+
+:- pred mer_parser__read_term_from_string(string, string, int, posn, posn,
+					read_term(T)).
+:- mode mer_parser__read_term_from_string(in, in, in, in, out, out) is det.
+%	mer_parser__read_term_from_string(FileName, String, MaxOffset, StartPos,
+%				    EndPos, Term).
+
+%-----------------------------------------------------------------------------%
+%
+% mer_parser__parse_tokens/3:
+%	Parses a list of tokens.
+
+:- pred mer_parser__parse_tokens(string, token_list, read_term(T)).
+:- mode mer_parser__parse_tokens(in, in, out) is det.
+	% mer_parser__parse_tokens(FileName, TokenList, Result):
+
+%-----------------------------------------------------------------------------%
+
+:- implementation.
+:- import_module string, char, int, float, bool, list, std_util, require.
+:- import_module map, term, varset.
+:- import_module ops.
+
+:- type parse(T)
+	--->	ok(T)
+	;	error(string, token_list).
+
+%-----------------------------------------------------------------------------%
+
+mer_parser__read_term(Result) -->
+	io__input_stream_name(FileName),
+	mer_parser__read_term(FileName, Result).
+
+mer_parser__read_term(FileName, Result) -->
+	mer_lexer__get_token_list(Tokens),
+	{ mer_parser__parse_tokens(FileName, Tokens, Result) }.
+
+mer_parser__read_term_from_string(FileName, String, EndPos, Result) :-
+	string__length(String, Len),
+	StartPos = posn(1, 0, 0),
+	mer_parser__read_term_from_string(FileName, String, Len, StartPos, EndPos,
+			Result).
+
+mer_parser__read_term_from_string(FileName, String, Len, StartPos, EndPos,
+		Result) :-
+	mer_lexer__string_get_token_list(String, Len, Tokens, StartPos, EndPos),
+	mer_parser__parse_tokens(FileName, Tokens, Result).
+
+%-----------------------------------------------------------------------------%
+
+mer_parser__parse_tokens(FileName, Tokens, Result) :-
+	( Tokens = token_nil ->
+		Result = eof
+	;
+		mer_parser__init_state(FileName, Tokens, ParserState0),
+		mer_parser__parse_whole_term(Term, ParserState0, ParserState),
+		mer_parser__final_state(ParserState, VarSet, LeftOverTokens),
+		mer_parser__check_for_errors(Term, VarSet,
+			Tokens, LeftOverTokens, Result)
+	).
+
+:- pred mer_parser__check_for_errors(parse(term(T)), varset(T),
+		token_list, token_list, read_term(T)).
+:- mode mer_parser__check_for_errors(in, in, in, in, out) is det.
+
+mer_parser__check_for_errors(error(ErrorMessage, ErrorTokens), _VarSet, Tokens,
+		_LeftOverTokens, Result) :-
+	% check if the error was caused by a bad token
+	(
+		mer_parser__check_for_bad_token(Tokens,
+			BadTokenMessage, BadTokenLineNum)
+	->
+		Message = BadTokenMessage,
+		LineNum = BadTokenLineNum
+	;
+		% find the token that caused the error
+		(
+			ErrorTokens = token_cons(ErrorTok, ErrorTokLineNum, _)
+		->
+			mer_lexer__token_to_string(ErrorTok, TokString),
+			string__append_list( ["Syntax error at ", TokString,
+						": ", ErrorMessage], Message),
+			LineNum = ErrorTokLineNum
+		;
+			(
+				Tokens = token_cons(_, FirstTokLineNum, _)
+			->
+				LineNum = FirstTokLineNum
+			;
+				error("mer_parser__check_for_errors")
+			),
+			string__append("Syntax error: ", ErrorMessage, Message)
+		)
+	),
+	Result = error(Message, LineNum).
+
+mer_parser__check_for_errors(ok(Term), VarSet, Tokens, LeftOverTokens, Result) :-
+	(
+		mer_parser__check_for_bad_token(Tokens, Message, LineNum)
+	->
+		Result = error(Message, LineNum)
+	;
+		LeftOverTokens = token_cons(Token, LineNum, _)
+	->
+		mer_lexer__token_to_string(Token, TokString),
+		string__append("Syntax error: unexpected ", TokString,
+			Message),
+		Result = error(Message, LineNum)
+	;
+		Result = term(VarSet, Term)
+	).
+
+:- pred mer_parser__check_for_bad_token(token_list, string, int).
+:- mode mer_parser__check_for_bad_token(in, out, out) is semidet.
+
+mer_parser__check_for_bad_token(token_cons(Token, LineNum, Tokens),
+		Message, LineNum) :-
+	( Token = io_error(IO_Error) ->
+		io__error_message(IO_Error, IO_ErrorMessage),
+		string__append("I/O error: ", IO_ErrorMessage, Message)
+	; Token = junk(Char) ->
+		char__to_int(Char, Code),
+		string__int_to_base_string(Code, 10, Decimal),
+		string__int_to_base_string(Code, 16, Hex),
+		string__append_list(["Syntax error: Illegal character 0x",
+			Hex, " (", Decimal, ") in input"], Message)
+	; Token = error(ErrorMessage) ->
+		string__append("Syntax error: ", ErrorMessage, Message)
+	;
+		mer_parser__check_for_bad_token(Tokens, Message, LineNum)
+	).
+
+:- pred mer_parser__parse_whole_term(parse(term(T)),
+		mer_parser__state(T), mer_parser__state(T)).
+:- mode mer_parser__parse_whole_term(out, in, out) is det.
+
+mer_parser__parse_whole_term(Term) -->
+	mer_parser__parse_term(Term0),
+	( { Term0 = ok(_) } ->
+		( mer_parser__get_token(end) ->
+			{ Term = Term0 }
+		;
+			mer_parser__unexpected("operator or `.' expected", Term)
+		)
+	;
+		% propagate error upwards
+		{ Term = Term0 }
+	).
+
+
+
+:- pred mer_parser__parse_term(parse(term(T)), mer_parser__state(T), mer_parser__state(T)).
+:- mode mer_parser__parse_term(out, in, out) is det.
+
+mer_parser__parse_term(Term) -->
+	mer_parser__parse_term_2(1201, no, Term).
+
+:- pred mer_parser__parse_arg(parse(term(T)), mer_parser__state(T), mer_parser__state(T)).
+:- mode mer_parser__parse_arg(out, in, out) is det.
+
+mer_parser__parse_arg(Term) -->
+	mer_parser__parse_term_2(1201, yes, Term).
+
+	% XXX I think ISO prolog syntax would require us to
+	% change that to  
+	%	mer_parser__parse_term(999, no, Term).
+	% The above is because we need bug-for-bug compatibility
+	% with the NU-Prolog parser in order to support e.g. `::' in args.
+
+:- pred mer_parser__parse_term_2(int, bool, parse(term(T)),
+				mer_parser__state(T), mer_parser__state(T)).
+:- mode mer_parser__parse_term_2(in, in, out, in, out) is det.
+
+mer_parser__parse_term_2(MaxPriority, IsArg, Term) -->
+	mer_parser__parse_left_term(MaxPriority, IsArg, LeftPriority, LeftTerm0),
+	( { LeftTerm0 = ok(LeftTerm) } ->
+		mer_parser__parse_rest(MaxPriority, IsArg, LeftPriority, LeftTerm,
+			Term)
+	;
+		% propagate error upwards
+		{ Term = LeftTerm0 }
+	).
+
+:- pred mer_parser__parse_left_term(int, bool, int, parse(term(T)),
+				mer_parser__state(T), mer_parser__state(T)).
+:- mode mer_parser__parse_left_term(in, in, out, out, in, out) is det.
+
+mer_parser__parse_left_term(MaxPriority, IsArg, OpPriority, Term) -->
+	( mer_parser__get_token(Token, Context) ->
+		(
+			% check for unary minus of integer
+			{ Token = name("-") },
+			mer_parser__get_token(integer(X), _IntContext)
+		->
+			mer_parser__get_term_context(Context, TermContext),
+			{ NegX is 0 - X },
+			{ Term = ok(term__functor(term__integer(NegX), [],
+						TermContext)) },
+			{ OpPriority = 0 }
+		;
+			% check for unary minus of float
+			{ Token = name("-") },
+			mer_parser__get_token(float(F), _FloatContext)
+		->
+			mer_parser__get_term_context(Context, TermContext),
+			{ NegF is 0.0 - F },
+			{ Term = ok(term__functor(term__float(NegF), [],
+				TermContext)) },
+			{ OpPriority = 0 }
+		;
+			% check for binary prefix op
+			{ Token = name(Op) },
+			\+ mer_parser__peek_token(open_ct),
+			mer_parser__get_ops_table(OpTable),
+			{ ops__lookup_binary_prefix_op(OpTable, Op,
+				BinOpPriority, RightAssoc, RightRightAssoc) },
+			{ BinOpPriority =< MaxPriority },
+			mer_parser__peek_token(NextToken),
+			{ mer_parser__could_start_term(NextToken, yes) }
+		->
+			{ mer_parser__adjust_priority(RightAssoc, BinOpPriority,
+							RightPriority) },
+			{ mer_parser__adjust_priority(RightRightAssoc,
+					BinOpPriority, RightRightPriority) },
+			{ OpPriority = BinOpPriority },
+			mer_parser__parse_term_2(RightPriority, IsArg, RightResult),
+			( { RightResult = ok(RightTerm) } ->
+				mer_parser__parse_term_2(RightRightPriority, IsArg,
+							RightRightResult),
+				( { RightRightResult = ok(RightRightTerm) } ->
+					mer_parser__get_term_context(Context,
+						TermContext),
+					{ Term = ok(term__functor(
+						term__atom(Op),
+						[RightTerm, RightRightTerm],
+						TermContext)) }
+				;
+					% propagate error upwards
+					{ Term = RightRightResult }
+				)
+			;
+				% propagate error upwards
+				{ Term = RightResult }
+			)
+		;
+			% check for unary prefix op
+			{ Token = name(Op) },
+			\+ mer_parser__peek_token(open_ct),
+			mer_parser__get_ops_table(OpTable),
+			{ ops__lookup_prefix_op(OpTable, Op, UnOpPriority,
+							RightAssoc) },
+			{ UnOpPriority =< MaxPriority },
+			mer_parser__peek_token(NextToken),
+			{ mer_parser__could_start_term(NextToken, yes) }
+		->
+			{ mer_parser__adjust_priority(RightAssoc, UnOpPriority,
+							RightPriority) },
+			mer_parser__parse_term_2(RightPriority, IsArg, RightResult),
+			{ OpPriority = UnOpPriority },
+			( { RightResult = ok(RightTerm) } ->
+				mer_parser__get_term_context(Context, TermContext),
+				{ Term = ok(term__functor(term__atom(Op),
+						[RightTerm], TermContext)) }
+			;
+				% propagate error upwards
+				{ Term = RightResult }
+			)
+		;
+			mer_parser__parse_simple_term(Token, Context, MaxPriority,
+				Term),
+			{ OpPriority = 0 }
+		)
+	;
+		mer_parser__error("unexpected end-of-file at start of sub-term",
+			Term),
+		{ OpPriority = 0 }
+	).
+
+:- pred mer_parser__parse_rest(int, bool, int, term(T), parse(term(T)),
+				mer_parser__state(T), mer_parser__state(T)).
+:- mode mer_parser__parse_rest(in, in, in, in, out, in, out) is det.
+
+mer_parser__parse_rest(MaxPriority, IsArg, LeftPriority, LeftTerm, Term) -->
+	(
+		% infix op
+		mer_parser__get_token(Token, Context),
+		{ Token = comma, IsArg = no ->
+			Op0 = ","
+		;
+			Token = name(Op0)
+		},
+		(
+				% A token surrounded by backquotes is a
+				% prefix token being using in an
+				% infix manner.
+			{ Op0 = "`" }
+		->
+			mer_parser__get_token(OpToken, _),
+			(
+				{ OpToken = name(NameOp) }
+			->
+				{ Op = NameOp },
+				{ VariableTerm = [] }
+			;
+				{ OpToken = variable(VariableOp) },
+				{ Op = "" },
+				mer_parser__add_var(VariableOp, Var),
+				{ VariableTerm = [term__variable(Var)] }
+			),
+			mer_parser__get_token(name("`"), _),
+
+			{ OpPriority = 100 },
+			{ LeftAssoc = y },
+			{ RightAssoc = x }
+		;
+			{ Op = Op0 },
+			{ VariableTerm = [] },
+			mer_parser__get_ops_table(OpTable),
+			{ ops__lookup_infix_op(OpTable, Op,
+					OpPriority, LeftAssoc, RightAssoc) }
+		),
+		{ OpPriority =< MaxPriority },
+		{ mer_parser__check_priority(LeftAssoc, OpPriority, LeftPriority) }
+	->
+		{ mer_parser__adjust_priority(RightAssoc, OpPriority,
+					RightPriority) },
+		mer_parser__parse_term_2(RightPriority, IsArg, RightTerm0),
+		( { RightTerm0 = ok(RightTerm) } ->
+			mer_parser__get_term_context(Context, TermContext),
+			{ OpTerm = term__functor(term__atom(Op),
+				list__append(VariableTerm,
+					[LeftTerm, RightTerm]),
+				TermContext) },
+			mer_parser__parse_rest(MaxPriority, IsArg, OpPriority,
+				OpTerm, Term)
+		;
+			% propagate error upwards
+			{ Term = RightTerm0 }
+		)
+	;
+		% postfix op
+		mer_parser__get_token(name(Op), Context),
+		mer_parser__get_ops_table(OpTable),
+		{ ops__lookup_postfix_op(OpTable, Op, OpPriority, LeftAssoc) },
+		{ OpPriority =< MaxPriority },
+		{ mer_parser__check_priority(LeftAssoc, OpPriority, LeftPriority) }
+	->
+		mer_parser__get_term_context(Context, TermContext),
+		{ OpTerm = term__functor(term__atom(Op), [LeftTerm],
+			TermContext) },
+		mer_parser__parse_rest(MaxPriority, IsArg, OpPriority, OpTerm, Term)
+	;
+		{ Term = ok(LeftTerm) }
+	).
+
+%-----------------------------------------------------------------------------%
+
+:- pred mer_parser__parse_simple_term(token, token_context, int, parse(term(T)),
+				mer_parser__state(T), mer_parser__state(T)).
+:- mode mer_parser__parse_simple_term(in, in, in, out, in, out) is det.
+
+mer_parser__parse_simple_term(Token, Context, Priority, Term) -->
+    ( mer_parser__parse_simple_term_2(Token, Context, Priority, Term0) ->
+	mer_parser__check_for_higher_order_term(Term0, Context, Term)
+    ;
+	mer_parser__unexpected_tok(Token, Context,
+		"unexpected token at start of (sub)term", Term)
+    ).
+
+	% term --> integer		% priority 0
+	% term --> float		% priority 0
+	% term --> name("-") integer	% priority 0
+	% term --> name("-") float	% priority 0
+	% term --> atom(NonOp)		% priority 0
+	% term --> atom(Op)		% priority 1201
+	%	atom --> name
+	%	atom --> open_list, close_list
+	%	atom --> open_curly, close_curly
+	% term --> variable		% priority 0
+	% term --> atom, open_ct, arg_list, close
+	%	arg_list --> arg
+	%	arg_list --> arg, comma, arg_list
+	% term --> open, term, close
+	% term --> open_ct, term, close
+	% term --> term, op, term	% with various conditions
+	% term --> op, term		% with various conditions
+	% term --> term, op		% with various conditions
+
+:- pred mer_parser__parse_simple_term_2(token, token_context, int, parse(term(T)),
+				mer_parser__state(T), mer_parser__state(T)).
+:- mode mer_parser__parse_simple_term_2(in, in, in, out, in, out) is semidet.
+
+mer_parser__parse_simple_term_2(name(Atom), Context, Prec, Term) -->
+	mer_parser__get_term_context(Context, TermContext),
+	( mer_parser__get_token(open_ct) ->
+		mer_parser__parse_args(Args0),
+		(	{ Args0 = ok(Args) },
+			{ Term = ok(term__functor(term__atom(Atom), Args,
+				TermContext)) }
+		;
+			% propagate error upwards
+			{ Args0 = error(Message, Tokens) },
+			{ Term = error(Message, Tokens) }
+		)
+	;
+		mer_parser__get_ops_table(OpTable),
+		{ ops__lookup_op(OpTable, Atom) ->
+			Prec >= 1201
+		;
+			true
+		},
+		{ Term = ok(term__functor(term__atom(Atom), [], TermContext)) }
+	).
+
+mer_parser__parse_simple_term_2(variable(VarName), _, _, Term) -->
+	mer_parser__add_var(VarName, Var),
+	{ Term = ok(term__variable(Var)) }.
+
+mer_parser__parse_simple_term_2(integer(Int), Context, _, Term) -->
+	mer_parser__get_term_context(Context, TermContext),
+	{ Term = ok(term__functor(term__integer(Int), [], TermContext)) }.
+
+mer_parser__parse_simple_term_2(float(Float), Context, _, Term) -->
+	mer_parser__get_term_context(Context, TermContext),
+	{ Term = ok(term__functor(term__float(Float), [], TermContext)) }.
+
+mer_parser__parse_simple_term_2(string(String), Context, _, Term) -->
+	mer_parser__get_term_context(Context, TermContext),
+	{ Term = ok(term__functor(term__string(String), [], TermContext)) }.
+
+mer_parser__parse_simple_term_2(open, _, _, Term) -->
+	mer_parser__parse_term(Term0),
+	( { Term0 = ok(_) } ->
+		( mer_parser__get_token(close) ->
+			{ Term = Term0 }
+		;
+			mer_parser__unexpected("expecting `)' or operator", Term)
+		)
+	;
+		% propagate error upwards
+		{ Term = Term0 }
+	).
+
+mer_parser__parse_simple_term_2(open_ct, Context, Prec, Term) -->
+	mer_parser__parse_simple_term_2(open, Context, Prec, Term).
+
+mer_parser__parse_simple_term_2(open_list, Context, _, Term) -->
+	mer_parser__get_term_context(Context, TermContext),
+	( mer_parser__get_token(close_list) ->
+		mer_parser__parse_special_atom("[]", TermContext, Term)
+	;
+		mer_parser__parse_list(Term)
+	).
+
+mer_parser__parse_simple_term_2(open_curly, Context, _, Term) -->
+	mer_parser__get_term_context(Context, TermContext),
+	( mer_parser__get_token(close_curly) ->
+		mer_parser__parse_special_atom("{}", TermContext, Term)
+	;
+		mer_parser__parse_term(SubTerm0),
+		( { SubTerm0 = ok(SubTerm) } ->
+			( mer_parser__get_token(close_curly) ->
+				{ Term = ok(term__functor(term__atom("{}"), 
+					[SubTerm], TermContext)) }
+			;
+				mer_parser__unexpected("expecting `}' or operator",
+					Term)
+			)
+		;
+			% propagate error upwards
+			{ Term = SubTerm0 }
+		)
+	).
+
+:- pred mer_parser__check_for_higher_order_term(parse(term(T)), token_context,
+		parse(term(T)), mer_parser__state(T), mer_parser__state(T)).
+:- mode mer_parser__check_for_higher_order_term(in, in, out, in, out) is det.
+
+mer_parser__check_for_higher_order_term(Term0, Context, Term) -->
+	%
+	% As an extension to ISO Prolog syntax,
+	% we check for the syntax "Term(Args)", and parse it
+	% as the term ''(Term, Args).  The aim of this extension
+	% is to provide a nicer syntax for higher-order stuff.
+	%
+	( { Term0 = ok(Term1) }, mer_parser__get_token(open_ct) ->
+		mer_parser__get_term_context(Context, TermContext),
+		mer_parser__parse_args(Args0),
+		(	{ Args0 = ok(Args) },
+			{ Term2 = ok(term__functor(term__atom(""),
+				[Term1 | Args],
+				TermContext)) },
+			mer_parser__check_for_higher_order_term(Term2, Context, Term)
+		;
+			% propagate error upwards
+			{ Args0 = error(Message, Tokens) },
+			{ Term = error(Message, Tokens) }
+		)
+	;
+		{ Term = Term0 }
+	).
+
+:- pred mer_parser__parse_special_atom(string, term__context, parse(term(T)),
+		mer_parser__state(T), mer_parser__state(T)).
+:- mode mer_parser__parse_special_atom(in, in, out, in, out) is det.
+
+mer_parser__parse_special_atom(Atom, TermContext, Term) -->
+	( mer_parser__get_token(open_ct) ->
+		mer_parser__parse_args(Args0),
+		(	{ Args0 = ok(Args) },
+			{ Term = ok(term__functor(term__atom(Atom),
+				Args, TermContext)) }
+		;
+			% propagate error upwards
+			{ Args0 = error(Message, Tokens) },
+			{ Term = error(Message, Tokens) }
+		)
+	;
+		{ Term = ok(term__functor(term__atom(Atom), [], TermContext)) }
+	).
+
+:- pred mer_parser__parse_list(parse(term(T)), mer_parser__state(T), mer_parser__state(T)).
+:- mode mer_parser__parse_list(out, in, out) is det.
+
+mer_parser__parse_list(List) -->
+	mer_parser__parse_arg(Arg0),
+	( { Arg0 = ok(Arg) } ->
+	    ( mer_parser__get_token(Token, Context) ->
+		mer_parser__get_term_context(Context, TermContext),
+		( { Token = comma } ->
+		    mer_parser__parse_list(Tail0),
+		    ( { Tail0 = ok(Tail) } ->
+		        { List = ok(term__functor(term__atom("."), [Arg, Tail],
+						TermContext)) }
+		    ;
+			% propagate error
+			{ List = Tail0 }
+		    )
+		; { Token = ht_sep } ->
+		    mer_parser__parse_arg(Tail0),
+		    ( { Tail0 = ok(Tail) } ->
+			( mer_parser__get_token(close_list) ->
+		            { List = ok(term__functor(term__atom("."),
+					[Arg, Tail], TermContext)) }
+			;
+			    mer_parser__unexpected("expecting ']' or operator",
+				List)
+			)
+		    ;
+			% propagate error
+			{ List = Tail0 }
+		    )
+		; { Token = close_list } ->
+		    { Tail = term__functor(term__atom("[]"), [], TermContext) },
+		    { List = ok(term__functor(term__atom("."), [Arg, Tail],
+				TermContext)) }
+		;
+		    mer_parser__unexpected_tok(Token, Context,
+			"expected comma, `|', `]', or operator", List)
+		)
+	    ;
+		% XXX error message should state the line that the
+		% list started on
+		mer_parser__error("unexpected end-of-file in list", List)
+	    )
+	;
+	    % propagate error
+	    { List = Arg0 }
+	).
+
+:- pred mer_parser__parse_args(parse(list(term(T))),
+		mer_parser__state(T), mer_parser__state(T)).
+:- mode mer_parser__parse_args(out, in, out) is det.
+
+mer_parser__parse_args(List) -->
+	mer_parser__parse_arg(Arg0),
+	(   { Arg0 = ok(Arg) },
+	    ( mer_parser__get_token(Token, Context) ->
+		( { Token = comma } ->
+		    mer_parser__parse_args(Tail0),
+		    ( { Tail0 = ok(Tail) } ->
+			{ List = ok([Arg|Tail]) }
+		    ;
+			% propagate error upwards
+		        { List = Tail0 }
+		    )
+		; { Token = close } ->
+		    { List = ok([Arg]) }
+		;
+		    mer_parser__unexpected_tok(Token, Context,
+				"expected `,', `)', or operator", List)
+		)
+	    ;
+		mer_parser__error("unexpected end-of-file in argument list", List)
+	    )
+	;
+	    { Arg0 = error(Message, Tokens) },
+	    % propagate error upwards
+	    { List = error(Message, Tokens) }
+	).
+
+%-----------------------------------------------------------------------------%
+%-----------------------------------------------------------------------------%
+
+	% Routines that manipulate the parser state.
+
+:- type mer_parser__state(T)
+	--->	mer_parser__state(
+			string,		% the name of the stream being parsed
+			ops__table,	% the current set of operators
+			varset(T),	% the names of the variables in the
+					% term being parsed
+			token_list,	% the remaining tokens
+			map(string, var(T))
+					% a map from variable name to variable
+					% so we know when to make a fresh var
+		).
+
+%-----------------------------------------------------------------------------%
+
+	% We encountered an error.  See if the next token
+	% was an infix or postfix operator.  If so, it would
+	% normally form part of the term, so the error must
+	% have been an operator precedence error.  Otherwise,
+	% it was some other sort of error, so issue the usual
+	% error message.
+
+:- pred mer_parser__unexpected(string, parse(U), mer_parser__state(T), mer_parser__state(T)).
+:- mode mer_parser__unexpected(in, out, in, out) is det.
+
+mer_parser__unexpected(UsualMessage, Error) -->
+	( mer_parser__get_token(Token, Context) ->
+		mer_parser__unexpected_tok(Token, Context, UsualMessage, Error)
+	;
+		mer_parser__error(UsualMessage, Error)
+	).
+
+:- pred mer_parser__unexpected_tok(token, token_context, string, parse(U),
+				mer_parser__state(T), mer_parser__state(T)).
+:- mode mer_parser__unexpected_tok(in, in, in, out, in, out) is det.
+
+mer_parser__unexpected_tok(Token, Context, UsualMessage, Error) -->
+	% push the token back, so that the error message
+	% points at it rather than at the following token
+	mer_parser__unget_token(Token, Context),
+	(
+		{ Token = name(Op)
+		; Token = comma, Op = ","
+		},
+		mer_parser__get_ops_table(OpTable),
+		{ ops__lookup_infix_op(OpTable, Op, _, _, _)
+		; ops__lookup_postfix_op(OpTable, Op, _, _)
+		}
+	->
+		mer_parser__error("operator precedence error", Error)
+	;
+		mer_parser__error(UsualMessage, Error)
+	).
+
+%-----------------------------------------------------------------------------%
+
+:- pred mer_parser__error(string, parse(U), mer_parser__state(T), mer_parser__state(T)).
+:- mode mer_parser__error(in, out, in, out) is det.
+
+mer_parser__error(Message, error(Message, Tokens), ParserState, ParserState) :-
+	ParserState = mer_parser__state(_, _, _, Tokens, _).
+
+%-----------------------------------------------------------------------------%
+
+:- pred mer_parser__could_start_term(token, bool).
+:- mode mer_parser__could_start_term(in, out) is det.
+
+mer_parser__could_start_term(name(_), yes).
+mer_parser__could_start_term(variable(_), yes).
+mer_parser__could_start_term(integer(_), yes).
+mer_parser__could_start_term(float(_), yes).
+mer_parser__could_start_term(string(_), yes).
+mer_parser__could_start_term(open, yes).
+mer_parser__could_start_term(open_ct, yes).
+mer_parser__could_start_term(close, no).
+mer_parser__could_start_term(open_list, yes).
+mer_parser__could_start_term(close_list, no).
+mer_parser__could_start_term(open_curly, yes).
+mer_parser__could_start_term(close_curly, no).
+mer_parser__could_start_term(ht_sep, no).
+mer_parser__could_start_term(comma, no).
+mer_parser__could_start_term(end, no).
+mer_parser__could_start_term(junk(_), no).
+mer_parser__could_start_term(error(_), no).
+mer_parser__could_start_term(io_error(_), no).
+mer_parser__could_start_term(eof, no).
+
+%-----------------------------------------------------------------------------%
+
+:- pred mer_parser__init_state(string, token_list, mer_parser__state(T)).
+:- mode mer_parser__init_state(in, in, out) is det.
+
+mer_parser__init_state(FileName, Tokens, ParserState) :-
+	ops__init_op_table(OpTable),
+	varset__init(VarSet),
+	map__init(Names),
+	ParserState = mer_parser__state(FileName, OpTable, VarSet, Tokens, Names).
+
+:- pred mer_parser__final_state(mer_parser__state(T), varset(T), token_list).
+:- mode mer_parser__final_state(in, out, out) is det.
+
+mer_parser__final_state(mer_parser__state(_FileName, _OpTable, VarSet, TokenList,
+		_Names), VarSet, TokenList).
+
+%-----------------------------------------------------------------------------%
+
+:- pred mer_parser__get_token(token, mer_parser__state(T), mer_parser__state(T)).
+:- mode mer_parser__get_token(out, in, out) is semidet.
+
+mer_parser__get_token(Token) -->
+	mer_parser__get_token(Token, _Context).
+
+:- pred mer_parser__get_token(token, token_context,
+		mer_parser__state(T), mer_parser__state(T)).
+:- mode mer_parser__get_token(out, out, in, out) is semidet.
+:- mode mer_parser__get_token(in, in, out, in) is det.
+
+mer_parser__get_token(Token, Context,
+		mer_parser__state(FileName, OpTable, VarSet, Tokens0, Names),
+		mer_parser__state(FileName, OpTable, VarSet, Tokens, Names)) :-
+	Tokens0 = token_cons(Token, Context, Tokens).
+
+:- pred mer_parser__unget_token(token, token_context,
+		mer_parser__state(T), mer_parser__state(T)).
+:- mode mer_parser__unget_token(in, in, in, out) is det.
+:- mode mer_parser__unget_token(out, out, out, in) is semidet.
+
+mer_parser__unget_token(Token, Context, ParseState0, ParseState) :-
+	mer_parser__get_token(Token, Context, ParseState, ParseState0).
+
+:- pred mer_parser__peek_token(token, mer_parser__state(T), mer_parser__state(T)).
+:- mode mer_parser__peek_token(out, in, out) is semidet.
+
+mer_parser__peek_token(Token) -->
+	mer_parser__peek_token(Token, _Context).
+
+:- pred mer_parser__peek_token(token, token_context,
+		mer_parser__state(T), mer_parser__state(T)).
+:- mode mer_parser__peek_token(out, out, in, out) is semidet.
+
+mer_parser__peek_token(Token, Context) -->
+	=(mer_parser__state(_, _, _, Tokens, _)),
+	{ Tokens = token_cons(Token, Context, _) }.
+
+%-----------------------------------------------------------------------------%
+
+:- pred mer_parser__add_var(string, var(T), mer_parser__state(T), mer_parser__state(T)).
+:- mode mer_parser__add_var(in, out, in, out) is det.
+
+mer_parser__add_var(VarName, Var,
+		mer_parser__state(FileName, OpTable, VarSet0, Tokens, Names0),
+		mer_parser__state(FileName, OpTable, VarSet, Tokens, Names)) :-
+	( VarName = "_" ->
+		varset__new_var(VarSet0, Var, VarSet),
+		Names = Names0
+	; map__search(Names0, VarName, Var0) ->
+		Var = Var0,
+		VarSet = VarSet0,
+		Names = Names0
+	;
+		varset__new_named_var(VarSet0, VarName, Var, VarSet),
+		map__det_insert(Names0, VarName, Var, Names)
+	).
+
+:- pred mer_parser__get_ops_table(ops__table, mer_parser__state(T), mer_parser__state(T)).
+:- mode mer_parser__get_ops_table(out, in, out) is det.
+
+mer_parser__get_ops_table(OpTable) -->
+	=(mer_parser__state(_, OpTable, _, _, _)).
+
+%-----------------------------------------------------------------------------%
+%-----------------------------------------------------------------------------%
+
+:- pred mer_parser__adjust_priority(ops__assoc, int, int).
+:- mode mer_parser__adjust_priority(in, in, out) is det.
+
+mer_parser__adjust_priority(y, Priority, Priority).
+mer_parser__adjust_priority(x, OldPriority, NewPriority) :-
+	NewPriority is OldPriority - 1.
+
+:- pred mer_parser__check_priority(ops__assoc, int, int).
+:- mode mer_parser__check_priority(in, in, in) is semidet.
+
+mer_parser__check_priority(y, MaxPriority, Priority) :-
+	Priority =< MaxPriority.
+mer_parser__check_priority(x, MaxPriority, Priority) :-
+	Priority < MaxPriority.
+
+:- pred mer_parser__get_term_context(token_context, term__context,
+				mer_parser__state(T), mer_parser__state(T)).
+:- mode mer_parser__get_term_context(in, out, in, out) is det.
+
+mer_parser__get_term_context(TokenContext, TermContext) -->
+	=(mer_parser__state(FileName, _Ops, _VarSet, _Tokens, _Names)),
+	{ term__context_init(FileName, TokenContext, TermContext) }.
+
+%-----------------------------------------------------------------------------%
+%-----------------------------------------------------------------------------%
Index: library/parser.m
===================================================================
RCS file: parser.m
diff -N parser.m
--- /tmp/cvsYl5FBV	Tue Aug 22 14:19:31 2000
+++ /dev/null	Thu Sep  2 15:00:04 1999
@@ -1,877 +0,0 @@
-%---------------------------------------------------------------------------%
-% Copyright (C) 1995-2000 The University of Melbourne.
-% This file may only be copied under the terms of the GNU Library General
-% Public License - see the file COPYING.LIB in the Mercury distribution.
-%---------------------------------------------------------------------------%
-%
-% file: parser.m.
-% main author: fjh.
-% stability: high.
-%
-% This file exports the predicate parser__read_term, which reads
-% a term from the current input stream.
-% The parser__read_term_from_string predicates are the same as the
-% read_term predicates, except that the term is read from
-% a string rather than from the current input stream.
-% The parser__parse_token_list predicate is similar,
-% but it takes a list of tokens rather than a string.
-%
-% The parser and lexer are intended to exactly follow ISO Prolog
-% syntax, but there are some departures from that for three reasons:
-%
-%	(1) I wrote some of the code at home when the ISO Prolog draft
-%	    was at uni - so in some places I just guessed.
-%	(2) In some places the lexer reports an error when it shouldn't.
-%	(3) There are a couple of hacks to make it compatible with NU-Prolog
-%	    syntax.
-%
-% The parser is a relatively straight-forward top-down recursive descent
-% parser, made somewhat complicated by the need to handle operator
-% precedences.  It uses `lexer__get_token_list' to read a list of tokens.
-% It uses the routines in module `ops' to look up operator precedences.
-%
-%-----------------------------------------------------------------------------%
-
-:- module parser.
-:- interface.
-:- import_module io, term_io, lexer.
-
-%-----------------------------------------------------------------------------%
-%
-% parser__read_term/{3,4}:
-%	Reads in (and parses) terms from the current input stream.
-
-:- pred parser__read_term(read_term(T), io__state, io__state).
-:- mode parser__read_term(out, di, uo) is det.
-% parser__read_term(Result):
-%	Reads a term from the current input stream.
-
-:- pred parser__read_term(string, read_term(T), io__state, io__state).
-:- mode parser__read_term(in, out, di, uo) is det.
-% parser__read_term(FileName, Result):
-%	Reads a term from the current input stream.
-%	The string is the filename to use for the current input stream;
-%	this is used in constructing the term__contexts in the read term.
-%	This interface is used to support the `:- pragma source_file'
-%	directive.
-
-%-----------------------------------------------------------------------------%
-%
-% parser__read_term_from_string/{4,6}:
-%	Parses terms from a string.
-
-	% The read_term_from_string predicates are the same as the
-	% read_term predicates, except that the term is read from
-	% a string rather than from the current input stream.
-	% The returned value `EndPos' is the position one character
-	% past the end of the term read.
-	% The arguments `MaxOffset' and `StartPos' in the six-argument version
-	% specify the length of the string and the position within the
-	% string at which to start parsing.
-
-:- pred parser__read_term_from_string(string, string, posn, read_term(T)).
-:- mode parser__read_term_from_string(in, in, out, out) is det.
-%	parser__read_term_from_string(FileName, String, EndPos, Term).
-
-:- pred parser__read_term_from_string(string, string, int, posn, posn,
-					read_term(T)).
-:- mode parser__read_term_from_string(in, in, in, in, out, out) is det.
-%	parser__read_term_from_string(FileName, String, MaxOffset, StartPos,
-%				    EndPos, Term).
-
-%-----------------------------------------------------------------------------%
-%
-% parser__parse_tokens/3:
-%	Parses a list of tokens.
-
-:- pred parser__parse_tokens(string, token_list, read_term(T)).
-:- mode parser__parse_tokens(in, in, out) is det.
-	% parser__parse_tokens(FileName, TokenList, Result):
-
-%-----------------------------------------------------------------------------%
-
-:- implementation.
-:- import_module string, char, int, float, bool, list, std_util, require.
-:- import_module map, term, varset.
-:- import_module lexer, ops.
-
-:- type parse(T)
-	--->	ok(T)
-	;	error(string, token_list).
-
-%-----------------------------------------------------------------------------%
-
-parser__read_term(Result) -->
-	io__input_stream_name(FileName),
-	parser__read_term(FileName, Result).
-
-parser__read_term(FileName, Result) -->
-	lexer__get_token_list(Tokens),
-	{ parser__parse_tokens(FileName, Tokens, Result) }.
-
-parser__read_term_from_string(FileName, String, EndPos, Result) :-
-	string__length(String, Len),
-	StartPos = posn(1, 0, 0),
-	parser__read_term_from_string(FileName, String, Len, StartPos, EndPos,
-			Result).
-
-parser__read_term_from_string(FileName, String, Len, StartPos, EndPos,
-		Result) :-
-	lexer__string_get_token_list(String, Len, Tokens, StartPos, EndPos),
-	parser__parse_tokens(FileName, Tokens, Result).
-
-%-----------------------------------------------------------------------------%
-
-parser__parse_tokens(FileName, Tokens, Result) :-
-	( Tokens = token_nil ->
-		Result = eof
-	;
-		parser__init_state(FileName, Tokens, ParserState0),
-		parser__parse_whole_term(Term, ParserState0, ParserState),
-		parser__final_state(ParserState, VarSet, LeftOverTokens),
-		parser__check_for_errors(Term, VarSet,
-			Tokens, LeftOverTokens, Result)
-	).
-
-:- pred parser__check_for_errors(parse(term(T)), varset(T),
-		token_list, token_list, read_term(T)).
-:- mode parser__check_for_errors(in, in, in, in, out) is det.
-
-parser__check_for_errors(error(ErrorMessage, ErrorTokens), _VarSet, Tokens,
-		_LeftOverTokens, Result) :-
-	% check if the error was caused by a bad token
-	(
-		parser__check_for_bad_token(Tokens,
-			BadTokenMessage, BadTokenLineNum)
-	->
-		Message = BadTokenMessage,
-		LineNum = BadTokenLineNum
-	;
-		% find the token that caused the error
-		(
-			ErrorTokens = token_cons(ErrorTok, ErrorTokLineNum, _)
-		->
-			lexer__token_to_string(ErrorTok, TokString),
-			string__append_list( ["Syntax error at ", TokString,
-						": ", ErrorMessage], Message),
-			LineNum = ErrorTokLineNum
-		;
-			(
-				Tokens = token_cons(_, FirstTokLineNum, _)
-			->
-				LineNum = FirstTokLineNum
-			;
-				error("parser__check_for_errors")
-			),
-			string__append("Syntax error: ", ErrorMessage, Message)
-		)
-	),
-	Result = error(Message, LineNum).
-
-parser__check_for_errors(ok(Term), VarSet, Tokens, LeftOverTokens, Result) :-
-	(
-		parser__check_for_bad_token(Tokens, Message, LineNum)
-	->
-		Result = error(Message, LineNum)
-	;
-		LeftOverTokens = token_cons(Token, LineNum, _)
-	->
-		lexer__token_to_string(Token, TokString),
-		string__append("Syntax error: unexpected ", TokString,
-			Message),
-		Result = error(Message, LineNum)
-	;
-		Result = term(VarSet, Term)
-	).
-
-:- pred parser__check_for_bad_token(token_list, string, int).
-:- mode parser__check_for_bad_token(in, out, out) is semidet.
-
-parser__check_for_bad_token(token_cons(Token, LineNum, Tokens),
-		Message, LineNum) :-
-	( Token = io_error(IO_Error) ->
-		io__error_message(IO_Error, IO_ErrorMessage),
-		string__append("I/O error: ", IO_ErrorMessage, Message)
-	; Token = junk(Char) ->
-		char__to_int(Char, Code),
-		string__int_to_base_string(Code, 10, Decimal),
-		string__int_to_base_string(Code, 16, Hex),
-		string__append_list(["Syntax error: Illegal character 0x",
-			Hex, " (", Decimal, ") in input"], Message)
-	; Token = error(ErrorMessage) ->
-		string__append("Syntax error: ", ErrorMessage, Message)
-	;
-		parser__check_for_bad_token(Tokens, Message, LineNum)
-	).
-
-:- pred parser__parse_whole_term(parse(term(T)),
-		parser__state(T), parser__state(T)).
-:- mode parser__parse_whole_term(out, in, out) is det.
-
-parser__parse_whole_term(Term) -->
-	parser__parse_term(Term0),
-	( { Term0 = ok(_) } ->
-		( parser__get_token(end) ->
-			{ Term = Term0 }
-		;
-			parser__unexpected("operator or `.' expected", Term)
-		)
-	;
-		% propagate error upwards
-		{ Term = Term0 }
-	).
-
-
-
-:- pred parser__parse_term(parse(term(T)), parser__state(T), parser__state(T)).
-:- mode parser__parse_term(out, in, out) is det.
-
-parser__parse_term(Term) -->
-	parser__parse_term_2(1201, no, Term).
-
-:- pred parser__parse_arg(parse(term(T)), parser__state(T), parser__state(T)).
-:- mode parser__parse_arg(out, in, out) is det.
-
-parser__parse_arg(Term) -->
-	parser__parse_term_2(1201, yes, Term).
-
-	% XXX I think ISO prolog syntax would require us to
-	% change that to  
-	%	parser__parse_term(999, no, Term).
-	% The above is because we need bug-for-bug compatibility
-	% with the NU-Prolog parser in order to support e.g. `::' in args.
-
-:- pred parser__parse_term_2(int, bool, parse(term(T)),
-				parser__state(T), parser__state(T)).
-:- mode parser__parse_term_2(in, in, out, in, out) is det.
-
-parser__parse_term_2(MaxPriority, IsArg, Term) -->
-	parser__parse_left_term(MaxPriority, IsArg, LeftPriority, LeftTerm0),
-	( { LeftTerm0 = ok(LeftTerm) } ->
-		parser__parse_rest(MaxPriority, IsArg, LeftPriority, LeftTerm,
-			Term)
-	;
-		% propagate error upwards
-		{ Term = LeftTerm0 }
-	).
-
-:- pred parser__parse_left_term(int, bool, int, parse(term(T)),
-				parser__state(T), parser__state(T)).
-:- mode parser__parse_left_term(in, in, out, out, in, out) is det.
-
-parser__parse_left_term(MaxPriority, IsArg, OpPriority, Term) -->
-	( parser__get_token(Token, Context) ->
-		(
-			% check for unary minus of integer
-			{ Token = name("-") },
-			parser__get_token(integer(X), _IntContext)
-		->
-			parser__get_term_context(Context, TermContext),
-			{ NegX is 0 - X },
-			{ Term = ok(term__functor(term__integer(NegX), [],
-						TermContext)) },
-			{ OpPriority = 0 }
-		;
-			% check for unary minus of float
-			{ Token = name("-") },
-			parser__get_token(float(F), _FloatContext)
-		->
-			parser__get_term_context(Context, TermContext),
-			{ NegF is 0.0 - F },
-			{ Term = ok(term__functor(term__float(NegF), [],
-				TermContext)) },
-			{ OpPriority = 0 }
-		;
-			% check for binary prefix op
-			{ Token = name(Op) },
-			\+ parser__peek_token(open_ct),
-			parser__get_ops_table(OpTable),
-			{ ops__lookup_binary_prefix_op(OpTable, Op,
-				BinOpPriority, RightAssoc, RightRightAssoc) },
-			{ BinOpPriority =< MaxPriority },
-			parser__peek_token(NextToken),
-			{ parser__could_start_term(NextToken, yes) }
-		->
-			{ parser__adjust_priority(RightAssoc, BinOpPriority,
-							RightPriority) },
-			{ parser__adjust_priority(RightRightAssoc,
-					BinOpPriority, RightRightPriority) },
-			{ OpPriority = BinOpPriority },
-			parser__parse_term_2(RightPriority, IsArg, RightResult),
-			( { RightResult = ok(RightTerm) } ->
-				parser__parse_term_2(RightRightPriority, IsArg,
-							RightRightResult),
-				( { RightRightResult = ok(RightRightTerm) } ->
-					parser__get_term_context(Context,
-						TermContext),
-					{ Term = ok(term__functor(
-						term__atom(Op),
-						[RightTerm, RightRightTerm],
-						TermContext)) }
-				;
-					% propagate error upwards
-					{ Term = RightRightResult }
-				)
-			;
-				% propagate error upwards
-				{ Term = RightResult }
-			)
-		;
-			% check for unary prefix op
-			{ Token = name(Op) },
-			\+ parser__peek_token(open_ct),
-			parser__get_ops_table(OpTable),
-			{ ops__lookup_prefix_op(OpTable, Op, UnOpPriority,
-							RightAssoc) },
-			{ UnOpPriority =< MaxPriority },
-			parser__peek_token(NextToken),
-			{ parser__could_start_term(NextToken, yes) }
-		->
-			{ parser__adjust_priority(RightAssoc, UnOpPriority,
-							RightPriority) },
-			parser__parse_term_2(RightPriority, IsArg, RightResult),
-			{ OpPriority = UnOpPriority },
-			( { RightResult = ok(RightTerm) } ->
-				parser__get_term_context(Context, TermContext),
-				{ Term = ok(term__functor(term__atom(Op),
-						[RightTerm], TermContext)) }
-			;
-				% propagate error upwards
-				{ Term = RightResult }
-			)
-		;
-			parser__parse_simple_term(Token, Context, MaxPriority,
-				Term),
-			{ OpPriority = 0 }
-		)
-	;
-		parser__error("unexpected end-of-file at start of sub-term",
-			Term),
-		{ OpPriority = 0 }
-	).
-
-:- pred parser__parse_rest(int, bool, int, term(T), parse(term(T)),
-				parser__state(T), parser__state(T)).
-:- mode parser__parse_rest(in, in, in, in, out, in, out) is det.
-
-parser__parse_rest(MaxPriority, IsArg, LeftPriority, LeftTerm, Term) -->
-	(
-		% infix op
-		parser__get_token(Token, Context),
-		{ Token = comma, IsArg = no ->
-			Op0 = ","
-		;
-			Token = name(Op0)
-		},
-		(
-				% A token surrounded by backquotes is a
-				% prefix token being using in an
-				% infix manner.
-			{ Op0 = "`" }
-		->
-			parser__get_token(OpToken, _),
-			(
-				{ OpToken = name(NameOp) }
-			->
-				{ Op = NameOp },
-				{ VariableTerm = [] }
-			;
-				{ OpToken = variable(VariableOp) },
-				{ Op = "" },
-				parser__add_var(VariableOp, Var),
-				{ VariableTerm = [term__variable(Var)] }
-			),
-			parser__get_token(name("`"), _),
-
-			{ OpPriority = 100 },
-			{ LeftAssoc = y },
-			{ RightAssoc = x }
-		;
-			{ Op = Op0 },
-			{ VariableTerm = [] },
-			parser__get_ops_table(OpTable),
-			{ ops__lookup_infix_op(OpTable, Op,
-					OpPriority, LeftAssoc, RightAssoc) }
-		),
-		{ OpPriority =< MaxPriority },
-		{ parser__check_priority(LeftAssoc, OpPriority, LeftPriority) }
-	->
-		{ parser__adjust_priority(RightAssoc, OpPriority,
-					RightPriority) },
-		parser__parse_term_2(RightPriority, IsArg, RightTerm0),
-		( { RightTerm0 = ok(RightTerm) } ->
-			parser__get_term_context(Context, TermContext),
-			{ OpTerm = term__functor(term__atom(Op),
-				list__append(VariableTerm,
-					[LeftTerm, RightTerm]),
-				TermContext) },
-			parser__parse_rest(MaxPriority, IsArg, OpPriority,
-				OpTerm, Term)
-		;
-			% propagate error upwards
-			{ Term = RightTerm0 }
-		)
-	;
-		% postfix op
-		parser__get_token(name(Op), Context),
-		parser__get_ops_table(OpTable),
-		{ ops__lookup_postfix_op(OpTable, Op, OpPriority, LeftAssoc) },
-		{ OpPriority =< MaxPriority },
-		{ parser__check_priority(LeftAssoc, OpPriority, LeftPriority) }
-	->
-		parser__get_term_context(Context, TermContext),
-		{ OpTerm = term__functor(term__atom(Op), [LeftTerm],
-			TermContext) },
-		parser__parse_rest(MaxPriority, IsArg, OpPriority, OpTerm, Term)
-	;
-		{ Term = ok(LeftTerm) }
-	).
-
-%-----------------------------------------------------------------------------%
-
-:- pred parser__parse_simple_term(token, token_context, int, parse(term(T)),
-				parser__state(T), parser__state(T)).
-:- mode parser__parse_simple_term(in, in, in, out, in, out) is det.
-
-parser__parse_simple_term(Token, Context, Priority, Term) -->
-    ( parser__parse_simple_term_2(Token, Context, Priority, Term0) ->
-	parser__check_for_higher_order_term(Term0, Context, Term)
-    ;
-	parser__unexpected_tok(Token, Context,
-		"unexpected token at start of (sub)term", Term)
-    ).
-
-	% term --> integer		% priority 0
-	% term --> float		% priority 0
-	% term --> name("-") integer	% priority 0
-	% term --> name("-") float	% priority 0
-	% term --> atom(NonOp)		% priority 0
-	% term --> atom(Op)		% priority 1201
-	%	atom --> name
-	%	atom --> open_list, close_list
-	%	atom --> open_curly, close_curly
-	% term --> variable		% priority 0
-	% term --> atom, open_ct, arg_list, close
-	%	arg_list --> arg
-	%	arg_list --> arg, comma, arg_list
-	% term --> open, term, close
-	% term --> open_ct, term, close
-	% term --> term, op, term	% with various conditions
-	% term --> op, term		% with various conditions
-	% term --> term, op		% with various conditions
-
-:- pred parser__parse_simple_term_2(token, token_context, int, parse(term(T)),
-				parser__state(T), parser__state(T)).
-:- mode parser__parse_simple_term_2(in, in, in, out, in, out) is semidet.
-
-parser__parse_simple_term_2(name(Atom), Context, Prec, Term) -->
-	parser__get_term_context(Context, TermContext),
-	( parser__get_token(open_ct) ->
-		parser__parse_args(Args0),
-		(	{ Args0 = ok(Args) },
-			{ Term = ok(term__functor(term__atom(Atom), Args,
-				TermContext)) }
-		;
-			% propagate error upwards
-			{ Args0 = error(Message, Tokens) },
-			{ Term = error(Message, Tokens) }
-		)
-	;
-		parser__get_ops_table(OpTable),
-		{ ops__lookup_op(OpTable, Atom) ->
-			Prec >= 1201
-		;
-			true
-		},
-		{ Term = ok(term__functor(term__atom(Atom), [], TermContext)) }
-	).
-
-parser__parse_simple_term_2(variable(VarName), _, _, Term) -->
-	parser__add_var(VarName, Var),
-	{ Term = ok(term__variable(Var)) }.
-
-parser__parse_simple_term_2(integer(Int), Context, _, Term) -->
-	parser__get_term_context(Context, TermContext),
-	{ Term = ok(term__functor(term__integer(Int), [], TermContext)) }.
-
-parser__parse_simple_term_2(float(Float), Context, _, Term) -->
-	parser__get_term_context(Context, TermContext),
-	{ Term = ok(term__functor(term__float(Float), [], TermContext)) }.
-
-parser__parse_simple_term_2(string(String), Context, _, Term) -->
-	parser__get_term_context(Context, TermContext),
-	{ Term = ok(term__functor(term__string(String), [], TermContext)) }.
-
-parser__parse_simple_term_2(open, _, _, Term) -->
-	parser__parse_term(Term0),
-	( { Term0 = ok(_) } ->
-		( parser__get_token(close) ->
-			{ Term = Term0 }
-		;
-			parser__unexpected("expecting `)' or operator", Term)
-		)
-	;
-		% propagate error upwards
-		{ Term = Term0 }
-	).
-
-parser__parse_simple_term_2(open_ct, Context, Prec, Term) -->
-	parser__parse_simple_term_2(open, Context, Prec, Term).
-
-parser__parse_simple_term_2(open_list, Context, _, Term) -->
-	parser__get_term_context(Context, TermContext),
-	( parser__get_token(close_list) ->
-		parser__parse_special_atom("[]", TermContext, Term)
-	;
-		parser__parse_list(Term)
-	).
-
-parser__parse_simple_term_2(open_curly, Context, _, Term) -->
-	parser__get_term_context(Context, TermContext),
-	( parser__get_token(close_curly) ->
-		parser__parse_special_atom("{}", TermContext, Term)
-	;
-		parser__parse_term(SubTerm0),
-		( { SubTerm0 = ok(SubTerm) } ->
-			( parser__get_token(close_curly) ->
-				{ Term = ok(term__functor(term__atom("{}"), 
-					[SubTerm], TermContext)) }
-			;
-				parser__unexpected("expecting `}' or operator",
-					Term)
-			)
-		;
-			% propagate error upwards
-			{ Term = SubTerm0 }
-		)
-	).
-
-:- pred parser__check_for_higher_order_term(parse(term(T)), token_context,
-		parse(term(T)), parser__state(T), parser__state(T)).
-:- mode parser__check_for_higher_order_term(in, in, out, in, out) is det.
-
-parser__check_for_higher_order_term(Term0, Context, Term) -->
-	%
-	% As an extension to ISO Prolog syntax,
-	% we check for the syntax "Term(Args)", and parse it
-	% as the term ''(Term, Args).  The aim of this extension
-	% is to provide a nicer syntax for higher-order stuff.
-	%
-	( { Term0 = ok(Term1) }, parser__get_token(open_ct) ->
-		parser__get_term_context(Context, TermContext),
-		parser__parse_args(Args0),
-		(	{ Args0 = ok(Args) },
-			{ Term2 = ok(term__functor(term__atom(""),
-				[Term1 | Args],
-				TermContext)) },
-			parser__check_for_higher_order_term(Term2, Context, Term)
-		;
-			% propagate error upwards
-			{ Args0 = error(Message, Tokens) },
-			{ Term = error(Message, Tokens) }
-		)
-	;
-		{ Term = Term0 }
-	).
-
-:- pred parser__parse_special_atom(string, term__context, parse(term(T)),
-		parser__state(T), parser__state(T)).
-:- mode parser__parse_special_atom(in, in, out, in, out) is det.
-
-parser__parse_special_atom(Atom, TermContext, Term) -->
-	( parser__get_token(open_ct) ->
-		parser__parse_args(Args0),
-		(	{ Args0 = ok(Args) },
-			{ Term = ok(term__functor(term__atom(Atom),
-				Args, TermContext)) }
-		;
-			% propagate error upwards
-			{ Args0 = error(Message, Tokens) },
-			{ Term = error(Message, Tokens) }
-		)
-	;
-		{ Term = ok(term__functor(term__atom(Atom), [], TermContext)) }
-	).
-
-:- pred parser__parse_list(parse(term(T)), parser__state(T), parser__state(T)).
-:- mode parser__parse_list(out, in, out) is det.
-
-parser__parse_list(List) -->
-	parser__parse_arg(Arg0),
-	( { Arg0 = ok(Arg) } ->
-	    ( parser__get_token(Token, Context) ->
-		parser__get_term_context(Context, TermContext),
-		( { Token = comma } ->
-		    parser__parse_list(Tail0),
-		    ( { Tail0 = ok(Tail) } ->
-		        { List = ok(term__functor(term__atom("."), [Arg, Tail],
-						TermContext)) }
-		    ;
-			% propagate error
-			{ List = Tail0 }
-		    )
-		; { Token = ht_sep } ->
-		    parser__parse_arg(Tail0),
-		    ( { Tail0 = ok(Tail) } ->
-			( parser__get_token(close_list) ->
-		            { List = ok(term__functor(term__atom("."),
-					[Arg, Tail], TermContext)) }
-			;
-			    parser__unexpected("expecting ']' or operator",
-				List)
-			)
-		    ;
-			% propagate error
-			{ List = Tail0 }
-		    )
-		; { Token = close_list } ->
-		    { Tail = term__functor(term__atom("[]"), [], TermContext) },
-		    { List = ok(term__functor(term__atom("."), [Arg, Tail],
-				TermContext)) }
-		;
-		    parser__unexpected_tok(Token, Context,
-			"expected comma, `|', `]', or operator", List)
-		)
-	    ;
-		% XXX error message should state the line that the
-		% list started on
-		parser__error("unexpected end-of-file in list", List)
-	    )
-	;
-	    % propagate error
-	    { List = Arg0 }
-	).
-
-:- pred parser__parse_args(parse(list(term(T))),
-		parser__state(T), parser__state(T)).
-:- mode parser__parse_args(out, in, out) is det.
-
-parser__parse_args(List) -->
-	parser__parse_arg(Arg0),
-	(   { Arg0 = ok(Arg) },
-	    ( parser__get_token(Token, Context) ->
-		( { Token = comma } ->
-		    parser__parse_args(Tail0),
-		    ( { Tail0 = ok(Tail) } ->
-			{ List = ok([Arg|Tail]) }
-		    ;
-			% propagate error upwards
-		        { List = Tail0 }
-		    )
-		; { Token = close } ->
-		    { List = ok([Arg]) }
-		;
-		    parser__unexpected_tok(Token, Context,
-				"expected `,', `)', or operator", List)
-		)
-	    ;
-		parser__error("unexpected end-of-file in argument list", List)
-	    )
-	;
-	    { Arg0 = error(Message, Tokens) },
-	    % propagate error upwards
-	    { List = error(Message, Tokens) }
-	).
-
-%-----------------------------------------------------------------------------%
-%-----------------------------------------------------------------------------%
-
-	% Routines that manipulate the parser state.
-
-:- type parser__state(T)
-	--->	parser__state(
-			string,		% the name of the stream being parsed
-			ops__table,	% the current set of operators
-			varset(T),	% the names of the variables in the
-					% term being parsed
-			token_list,	% the remaining tokens
-			map(string, var(T))
-					% a map from variable name to variable
-					% so we know when to make a fresh var
-		).
-
-%-----------------------------------------------------------------------------%
-
-	% We encountered an error.  See if the next token
-	% was an infix or postfix operator.  If so, it would
-	% normally form part of the term, so the error must
-	% have been an operator precedence error.  Otherwise,
-	% it was some other sort of error, so issue the usual
-	% error message.
-
-:- pred parser__unexpected(string, parse(U), parser__state(T), parser__state(T)).
-:- mode parser__unexpected(in, out, in, out) is det.
-
-parser__unexpected(UsualMessage, Error) -->
-	( parser__get_token(Token, Context) ->
-		parser__unexpected_tok(Token, Context, UsualMessage, Error)
-	;
-		parser__error(UsualMessage, Error)
-	).
-
-:- pred parser__unexpected_tok(token, token_context, string, parse(U),
-				parser__state(T), parser__state(T)).
-:- mode parser__unexpected_tok(in, in, in, out, in, out) is det.
-
-parser__unexpected_tok(Token, Context, UsualMessage, Error) -->
-	% push the token back, so that the error message
-	% points at it rather than at the following token
-	parser__unget_token(Token, Context),
-	(
-		{ Token = name(Op)
-		; Token = comma, Op = ","
-		},
-		parser__get_ops_table(OpTable),
-		{ ops__lookup_infix_op(OpTable, Op, _, _, _)
-		; ops__lookup_postfix_op(OpTable, Op, _, _)
-		}
-	->
-		parser__error("operator precedence error", Error)
-	;
-		parser__error(UsualMessage, Error)
-	).
-
-%-----------------------------------------------------------------------------%
-
-:- pred parser__error(string, parse(U), parser__state(T), parser__state(T)).
-:- mode parser__error(in, out, in, out) is det.
-
-parser__error(Message, error(Message, Tokens), ParserState, ParserState) :-
-	ParserState = parser__state(_, _, _, Tokens, _).
-
-%-----------------------------------------------------------------------------%
-
-:- pred parser__could_start_term(token, bool).
-:- mode parser__could_start_term(in, out) is det.
-
-parser__could_start_term(name(_), yes).
-parser__could_start_term(variable(_), yes).
-parser__could_start_term(integer(_), yes).
-parser__could_start_term(float(_), yes).
-parser__could_start_term(string(_), yes).
-parser__could_start_term(open, yes).
-parser__could_start_term(open_ct, yes).
-parser__could_start_term(close, no).
-parser__could_start_term(open_list, yes).
-parser__could_start_term(close_list, no).
-parser__could_start_term(open_curly, yes).
-parser__could_start_term(close_curly, no).
-parser__could_start_term(ht_sep, no).
-parser__could_start_term(comma, no).
-parser__could_start_term(end, no).
-parser__could_start_term(junk(_), no).
-parser__could_start_term(error(_), no).
-parser__could_start_term(io_error(_), no).
-parser__could_start_term(eof, no).
-
-%-----------------------------------------------------------------------------%
-
-:- pred parser__init_state(string, token_list, parser__state(T)).
-:- mode parser__init_state(in, in, out) is det.
-
-parser__init_state(FileName, Tokens, ParserState) :-
-	ops__init_op_table(OpTable),
-	varset__init(VarSet),
-	map__init(Names),
-	ParserState = parser__state(FileName, OpTable, VarSet, Tokens, Names).
-
-:- pred parser__final_state(parser__state(T), varset(T), token_list).
-:- mode parser__final_state(in, out, out) is det.
-
-parser__final_state(parser__state(_FileName, _OpTable, VarSet, TokenList,
-		_Names), VarSet, TokenList).
-
-%-----------------------------------------------------------------------------%
-
-:- pred parser__get_token(token, parser__state(T), parser__state(T)).
-:- mode parser__get_token(out, in, out) is semidet.
-
-parser__get_token(Token) -->
-	parser__get_token(Token, _Context).
-
-:- pred parser__get_token(token, token_context,
-		parser__state(T), parser__state(T)).
-:- mode parser__get_token(out, out, in, out) is semidet.
-:- mode parser__get_token(in, in, out, in) is det.
-
-parser__get_token(Token, Context,
-		parser__state(FileName, OpTable, VarSet, Tokens0, Names),
-		parser__state(FileName, OpTable, VarSet, Tokens, Names)) :-
-	Tokens0 = token_cons(Token, Context, Tokens).
-
-:- pred parser__unget_token(token, token_context,
-		parser__state(T), parser__state(T)).
-:- mode parser__unget_token(in, in, in, out) is det.
-:- mode parser__unget_token(out, out, out, in) is semidet.
-
-parser__unget_token(Token, Context, ParseState0, ParseState) :-
-	parser__get_token(Token, Context, ParseState, ParseState0).
-
-:- pred parser__peek_token(token, parser__state(T), parser__state(T)).
-:- mode parser__peek_token(out, in, out) is semidet.
-
-parser__peek_token(Token) -->
-	parser__peek_token(Token, _Context).
-
-:- pred parser__peek_token(token, token_context,
-		parser__state(T), parser__state(T)).
-:- mode parser__peek_token(out, out, in, out) is semidet.
-
-parser__peek_token(Token, Context) -->
-	=(parser__state(_, _, _, Tokens, _)),
-	{ Tokens = token_cons(Token, Context, _) }.
-
-%-----------------------------------------------------------------------------%
-
-:- pred parser__add_var(string, var(T), parser__state(T), parser__state(T)).
-:- mode parser__add_var(in, out, in, out) is det.
-
-parser__add_var(VarName, Var,
-		parser__state(FileName, OpTable, VarSet0, Tokens, Names0),
-		parser__state(FileName, OpTable, VarSet, Tokens, Names)) :-
-	( VarName = "_" ->
-		varset__new_var(VarSet0, Var, VarSet),
-		Names = Names0
-	; map__search(Names0, VarName, Var0) ->
-		Var = Var0,
-		VarSet = VarSet0,
-		Names = Names0
-	;
-		varset__new_named_var(VarSet0, VarName, Var, VarSet),
-		map__det_insert(Names0, VarName, Var, Names)
-	).
-
-:- pred parser__get_ops_table(ops__table, parser__state(T), parser__state(T)).
-:- mode parser__get_ops_table(out, in, out) is det.
-
-parser__get_ops_table(OpTable) -->
-	=(parser__state(_, OpTable, _, _, _)).
-
-%-----------------------------------------------------------------------------%
-%-----------------------------------------------------------------------------%
-
-:- pred parser__adjust_priority(ops__assoc, int, int).
-:- mode parser__adjust_priority(in, in, out) is det.
-
-parser__adjust_priority(y, Priority, Priority).
-parser__adjust_priority(x, OldPriority, NewPriority) :-
-	NewPriority is OldPriority - 1.
-
-:- pred parser__check_priority(ops__assoc, int, int).
-:- mode parser__check_priority(in, in, in) is semidet.
-
-parser__check_priority(y, MaxPriority, Priority) :-
-	Priority =< MaxPriority.
-parser__check_priority(x, MaxPriority, Priority) :-
-	Priority < MaxPriority.
-
-:- pred parser__get_term_context(token_context, term__context,
-				parser__state(T), parser__state(T)).
-:- mode parser__get_term_context(in, out, in, out) is det.
-
-parser__get_term_context(TokenContext, TermContext) -->
-	=(parser__state(FileName, _Ops, _VarSet, _Tokens, _Names)),
-	{ term__context_init(FileName, TokenContext, TermContext) }.
-
-%-----------------------------------------------------------------------------%
-%-----------------------------------------------------------------------------%
Index: library/term_io.m
===================================================================
RCS file: /home/mercury1/repository/mercury/library/term_io.m,v
retrieving revision 1.58
diff -u -r1.58 term_io.m
--- library/term_io.m	1999/06/09 14:19:09	1.58
+++ library/term_io.m	2000/08/22 02:14:41
@@ -121,10 +121,10 @@
 
 :- implementation.
 :- import_module std_util, require, list, string, int, char.
-:- import_module lexer, parser, ops.
+:- import_module mer_lexer, mer_parser, ops.
 
 term_io__read_term(Result) -->
-	parser__read_term(Result).
+	mer_parser__read_term(Result).
 
 %-----------------------------------------------------------------------------%
 %-----------------------------------------------------------------------------%
@@ -441,7 +441,7 @@
 			% graphic token (6.4.2)
 			{ string__to_char_list(S, Chars) },
 			{ \+ (  list__member(Char, Chars),
-				\+ lexer__graphic_token_char(Char)) },
+				\+ mer_lexer__graphic_token_char(Char)) },
 			{ Chars \= [] },
 			%
 			% We need to quote tokens starting with '#',
cvs diff: Diffing profiler
cvs diff: Diffing robdd
cvs diff: Diffing runtime
cvs diff: Diffing runtime/GETOPT
cvs diff: Diffing runtime/machdeps
cvs diff: Diffing samples
cvs diff: Diffing samples/c_interface
cvs diff: Diffing samples/c_interface/c_calls_mercury
cvs diff: Diffing samples/c_interface/cplusplus_calls_mercury
cvs diff: Diffing samples/c_interface/mercury_calls_c
cvs diff: Diffing samples/c_interface/mercury_calls_cplusplus
cvs diff: Diffing samples/c_interface/mercury_calls_fortran
cvs diff: Diffing samples/c_interface/simpler_c_calls_mercury
cvs diff: Diffing samples/c_interface/simpler_cplusplus_calls_mercury
cvs diff: Diffing samples/diff
cvs diff: Diffing samples/muz
cvs diff: Diffing samples/rot13
cvs diff: Diffing samples/solutions
cvs diff: Diffing samples/tests
cvs diff: Diffing samples/tests/c_interface
cvs diff: Diffing samples/tests/c_interface/c_calls_mercury
cvs diff: Diffing samples/tests/c_interface/cplusplus_calls_mercury
cvs diff: Diffing samples/tests/c_interface/mercury_calls_c
cvs diff: Diffing samples/tests/c_interface/mercury_calls_cplusplus
cvs diff: Diffing samples/tests/c_interface/mercury_calls_fortran
cvs diff: Diffing samples/tests/c_interface/simpler_c_calls_mercury
cvs diff: Diffing samples/tests/c_interface/simpler_cplusplus_calls_mercury
cvs diff: Diffing samples/tests/diff
cvs diff: Diffing samples/tests/muz
cvs diff: Diffing samples/tests/rot13
cvs diff: Diffing samples/tests/solutions
cvs diff: Diffing samples/tests/toplevel
cvs diff: Diffing scripts
cvs diff: Diffing tests
cvs diff: Diffing tests/benchmarks
cvs diff: Diffing tests/debugger
cvs diff: Diffing tests/debugger/declarative
cvs diff: Diffing tests/dppd
cvs diff: Diffing tests/general
cvs diff: Diffing tests/general/accumulator
cvs diff: Diffing tests/hard_coded
cvs diff: Diffing tests/hard_coded/exceptions
cvs diff: Diffing tests/hard_coded/purity
cvs diff: Diffing tests/hard_coded/sub-modules
cvs diff: Diffing tests/hard_coded/typeclasses
cvs diff: Diffing tests/invalid
cvs diff: Diffing tests/invalid/purity
cvs diff: Diffing tests/misc_tests
cvs diff: Diffing tests/tabling
cvs diff: Diffing tests/term
cvs diff: Diffing tests/valid
cvs diff: Diffing tests/warnings
cvs diff: Diffing tools
cvs diff: Diffing trace
cvs diff: Diffing trial
cvs diff: Diffing util
--------------------------------------------------------------------------
mercury-developers mailing list
Post messages to:       mercury-developers at cs.mu.oz.au
Administrative Queries: owner-mercury-developers at cs.mu.oz.au
Subscriptions:          mercury-developers-request at cs.mu.oz.au
--------------------------------------------------------------------------