[mercury-users] need help: Mercury+C

[Tyson Dowd]

On 02-Sep-1998, Peter Ross <petdr at cs.mu.OZ.AU> wrote:
> On 02-Sep-1998, Tyson Dowd <trd at cs.mu.OZ.AU> wrote:
> > On 01-Sep-1998, tcklnbrg <at at ingenuity-sw.com> wrote:
> > 
> > > 
> > > The tokenize(..) c code in lexer.c does open and "read-only" a file.  
> > > Does this mean that the Mercury function that corresponds to c
> > > tokenize(..)
> > > in file lexer.c, must be declared with the 2 extra io__state args?  
> > 
> > Maybe.  You will need to have some sort of state arguments to represent
> > the state of the lexer, so io__state is a good choice to start with.
> > 
> You should have a pair of arguments which are (di, uo) and that
> represent the "state of the world".  These are needed to ensure that the
> code is never backtracked over.

This is true, but you can in fact use a different approach than
io__states representing the state of the world.

You can model tokenize as a non-deterministic tokenizer, and
create some non-det C code around it to give it this interface.
Depending on the application, the fact that it does I/O might be
irrelevant for modelling its behaviour, or only relevant when the
tokenizer is first initialized, and not thereafter.

You can also use a tokenizer state, which is a separate pair of (di, uo)
arguments, to represent the "token stream" as opposed to any other I/O.
This is handy if you have multiple token streams you need to handle.

But 95% of the time, io__states will be fine.