[m-dev.] Re: [mercury-users] lex updates?
Holger Krug
hkrug at rationalizer.com
Mon Sep 24 17:52:15 AEST 2001
On Mon, Sep 24, 2001 at 05:24:46PM +1000, Michael Day wrote:
>
> > Concerning lex please come back if there is no check-in by Ralph
> > within the next couple of days or if you or anybody else needs the
> > changes immediately.
>
> If you've got a spare moment I wouldn't mind having a look at the changes,
> as I'm just trying to write a scanner/parser for cascading style sheets
> and the old lex interface is frustrating (sorry Ralph :)
Here is the diff.
--
Holger Krug
hkrug at rationalizer.com
-------------- next part --------------
Index: COPYING.DOC
===================================================================
RCS file: COPYING.DOC
diff -N COPYING.DOC
--- /dev/null Wed Jul 5 19:49:51 2000
+++ /tmp/cvsFjuzNF Mon Sep 24 09:49:08 2001
@@ -0,0 +1,355 @@
+ GNU Free Documentation License
+ Version 1.1, March 2000
+
+ Copyright (C) 2000 Free Software Foundation, Inc.
+ 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+
+0. PREAMBLE
+
+The purpose of this License is to make a manual, textbook, or other
+written document "free" in the sense of freedom: to assure everyone
+the effective freedom to copy and redistribute it, with or without
+modifying it, either commercially or noncommercially. Secondarily,
+this License preserves for the author and publisher a way to get
+credit for their work, while not being considered responsible for
+modifications made by others.
+
+This License is a kind of "copyleft", which means that derivative
+works of the document must themselves be free in the same sense. It
+complements the GNU General Public License, which is a copyleft
+license designed for free software.
+
+We have designed this License in order to use it for manuals for free
+software, because free software needs free documentation: a free
+program should come with manuals providing the same freedoms that the
+software does. But this License is not limited to software manuals;
+it can be used for any textual work, regardless of subject matter or
+whether it is published as a printed book. We recommend this License
+principally for works whose purpose is instruction or reference.
+
+
+1. APPLICABILITY AND DEFINITIONS
+
+This License applies to any manual or other work that contains a
+notice placed by the copyright holder saying it can be distributed
+under the terms of this License. The "Document", below, refers to any
+such manual or work. Any member of the public is a licensee, and is
+addressed as "you".
+
+A "Modified Version" of the Document means any work containing the
+Document or a portion of it, either copied verbatim, or with
+modifications and/or translated into another language.
+
+A "Secondary Section" is a named appendix or a front-matter section of
+the Document that deals exclusively with the relationship of the
+publishers or authors of the Document to the Document's overall subject
+(or to related matters) and contains nothing that could fall directly
+within that overall subject. (For example, if the Document is in part a
+textbook of mathematics, a Secondary Section may not explain any
+mathematics.) The relationship could be a matter of historical
+connection with the subject or with related matters, or of legal,
+commercial, philosophical, ethical or political position regarding
+them.
+
+The "Invariant Sections" are certain Secondary Sections whose titles
+are designated, as being those of Invariant Sections, in the notice
+that says that the Document is released under this License.
+
+The "Cover Texts" are certain short passages of text that are listed,
+as Front-Cover Texts or Back-Cover Texts, in the notice that says that
+the Document is released under this License.
+
+A "Transparent" copy of the Document means a machine-readable copy,
+represented in a format whose specification is available to the
+general public, whose contents can be viewed and edited directly and
+straightforwardly with generic text editors or (for images composed of
+pixels) generic paint programs or (for drawings) some widely available
+drawing editor, and that is suitable for input to text formatters or
+for automatic translation to a variety of formats suitable for input
+to text formatters. A copy made in an otherwise Transparent file
+format whose markup has been designed to thwart or discourage
+subsequent modification by readers is not Transparent. A copy that is
+not "Transparent" is called "Opaque".
+
+Examples of suitable formats for Transparent copies include plain
+ASCII without markup, Texinfo input format, LaTeX input format, SGML
+or XML using a publicly available DTD, and standard-conforming simple
+HTML designed for human modification. Opaque formats include
+PostScript, PDF, proprietary formats that can be read and edited only
+by proprietary word processors, SGML or XML for which the DTD and/or
+processing tools are not generally available, and the
+machine-generated HTML produced by some word processors for output
+purposes only.
+
+The "Title Page" means, for a printed book, the title page itself,
+plus such following pages as are needed to hold, legibly, the material
+this License requires to appear in the title page. For works in
+formats which do not have any title page as such, "Title Page" means
+the text near the most prominent appearance of the work's title,
+preceding the beginning of the body of the text.
+
+
+2. VERBATIM COPYING
+
+You may copy and distribute the Document in any medium, either
+commercially or noncommercially, provided that this License, the
+copyright notices, and the license notice saying this License applies
+to the Document are reproduced in all copies, and that you add no other
+conditions whatsoever to those of this License. You may not use
+technical measures to obstruct or control the reading or further
+copying of the copies you make or distribute. However, you may accept
+compensation in exchange for copies. If you distribute a large enough
+number of copies you must also follow the conditions in section 3.
+
+You may also lend copies, under the same conditions stated above, and
+you may publicly display copies.
+
+
+3. COPYING IN QUANTITY
+
+If you publish printed copies of the Document numbering more than 100,
+and the Document's license notice requires Cover Texts, you must enclose
+the copies in covers that carry, clearly and legibly, all these Cover
+Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on
+the back cover. Both covers must also clearly and legibly identify
+you as the publisher of these copies. The front cover must present
+the full title with all words of the title equally prominent and
+visible. You may add other material on the covers in addition.
+Copying with changes limited to the covers, as long as they preserve
+the title of the Document and satisfy these conditions, can be treated
+as verbatim copying in other respects.
+
+If the required texts for either cover are too voluminous to fit
+legibly, you should put the first ones listed (as many as fit
+reasonably) on the actual cover, and continue the rest onto adjacent
+pages.
+
+If you publish or distribute Opaque copies of the Document numbering
+more than 100, you must either include a machine-readable Transparent
+copy along with each Opaque copy, or state in or with each Opaque copy
+a publicly-accessible computer-network location containing a complete
+Transparent copy of the Document, free of added material, which the
+general network-using public has access to download anonymously at no
+charge using public-standard network protocols. If you use the latter
+option, you must take reasonably prudent steps, when you begin
+distribution of Opaque copies in quantity, to ensure that this
+Transparent copy will remain thus accessible at the stated location
+until at least one year after the last time you distribute an Opaque
+copy (directly or through your agents or retailers) of that edition to
+the public.
+
+It is requested, but not required, that you contact the authors of the
+Document well before redistributing any large number of copies, to give
+them a chance to provide you with an updated version of the Document.
+
+
+4. MODIFICATIONS
+
+You may copy and distribute a Modified Version of the Document under
+the conditions of sections 2 and 3 above, provided that you release
+the Modified Version under precisely this License, with the Modified
+Version filling the role of the Document, thus licensing distribution
+and modification of the Modified Version to whoever possesses a copy
+of it. In addition, you must do these things in the Modified Version:
+
+A. Use in the Title Page (and on the covers, if any) a title distinct
+ from that of the Document, and from those of previous versions
+ (which should, if there were any, be listed in the History section
+ of the Document). You may use the same title as a previous version
+ if the original publisher of that version gives permission.
+B. List on the Title Page, as authors, one or more persons or entities
+ responsible for authorship of the modifications in the Modified
+ Version, together with at least five of the principal authors of the
+ Document (all of its principal authors, if it has less than five).
+C. State on the Title page the name of the publisher of the
+ Modified Version, as the publisher.
+D. Preserve all the copyright notices of the Document.
+E. Add an appropriate copyright notice for your modifications
+ adjacent to the other copyright notices.
+F. Include, immediately after the copyright notices, a license notice
+ giving the public permission to use the Modified Version under the
+ terms of this License, in the form shown in the Addendum below.
+G. Preserve in that license notice the full lists of Invariant Sections
+ and required Cover Texts given in the Document's license notice.
+H. Include an unaltered copy of this License.
+I. Preserve the section entitled "History", and its title, and add to
+ it an item stating at least the title, year, new authors, and
+ publisher of the Modified Version as given on the Title Page. If
+ there is no section entitled "History" in the Document, create one
+ stating the title, year, authors, and publisher of the Document as
+ given on its Title Page, then add an item describing the Modified
+ Version as stated in the previous sentence.
+J. Preserve the network location, if any, given in the Document for
+ public access to a Transparent copy of the Document, and likewise
+ the network locations given in the Document for previous versions
+ it was based on. These may be placed in the "History" section.
+ You may omit a network location for a work that was published at
+ least four years before the Document itself, or if the original
+ publisher of the version it refers to gives permission.
+K. In any section entitled "Acknowledgements" or "Dedications",
+ preserve the section's title, and preserve in the section all the
+ substance and tone of each of the contributor acknowledgements
+ and/or dedications given therein.
+L. Preserve all the Invariant Sections of the Document,
+ unaltered in their text and in their titles. Section numbers
+ or the equivalent are not considered part of the section titles.
+M. Delete any section entitled "Endorsements". Such a section
+ may not be included in the Modified Version.
+N. Do not retitle any existing section as "Endorsements"
+ or to conflict in title with any Invariant Section.
+
+If the Modified Version includes new front-matter sections or
+appendices that qualify as Secondary Sections and contain no material
+copied from the Document, you may at your option designate some or all
+of these sections as invariant. To do this, add their titles to the
+list of Invariant Sections in the Modified Version's license notice.
+These titles must be distinct from any other section titles.
+
+You may add a section entitled "Endorsements", provided it contains
+nothing but endorsements of your Modified Version by various
+parties--for example, statements of peer review or that the text has
+been approved by an organization as the authoritative definition of a
+standard.
+
+You may add a passage of up to five words as a Front-Cover Text, and a
+passage of up to 25 words as a Back-Cover Text, to the end of the list
+of Cover Texts in the Modified Version. Only one passage of
+Front-Cover Text and one of Back-Cover Text may be added by (or
+through arrangements made by) any one entity. If the Document already
+includes a cover text for the same cover, previously added by you or
+by arrangement made by the same entity you are acting on behalf of,
+you may not add another; but you may replace the old one, on explicit
+permission from the previous publisher that added the old one.
+
+The author(s) and publisher(s) of the Document do not by this License
+give permission to use their names for publicity for or to assert or
+imply endorsement of any Modified Version.
+
+
+5. COMBINING DOCUMENTS
+
+You may combine the Document with other documents released under this
+License, under the terms defined in section 4 above for modified
+versions, provided that you include in the combination all of the
+Invariant Sections of all of the original documents, unmodified, and
+list them all as Invariant Sections of your combined work in its
+license notice.
+
+The combined work need only contain one copy of this License, and
+multiple identical Invariant Sections may be replaced with a single
+copy. If there are multiple Invariant Sections with the same name but
+different contents, make the title of each such section unique by
+adding at the end of it, in parentheses, the name of the original
+author or publisher of that section if known, or else a unique number.
+Make the same adjustment to the section titles in the list of
+Invariant Sections in the license notice of the combined work.
+
+In the combination, you must combine any sections entitled "History"
+in the various original documents, forming one section entitled
+"History"; likewise combine any sections entitled "Acknowledgements",
+and any sections entitled "Dedications". You must delete all sections
+entitled "Endorsements."
+
+
+6. COLLECTIONS OF DOCUMENTS
+
+You may make a collection consisting of the Document and other documents
+released under this License, and replace the individual copies of this
+License in the various documents with a single copy that is included in
+the collection, provided that you follow the rules of this License for
+verbatim copying of each of the documents in all other respects.
+
+You may extract a single document from such a collection, and distribute
+it individually under this License, provided you insert a copy of this
+License into the extracted document, and follow this License in all
+other respects regarding verbatim copying of that document.
+
+
+7. AGGREGATION WITH INDEPENDENT WORKS
+
+A compilation of the Document or its derivatives with other separate
+and independent documents or works, in or on a volume of a storage or
+distribution medium, does not as a whole count as a Modified Version
+of the Document, provided no compilation copyright is claimed for the
+compilation. Such a compilation is called an "aggregate", and this
+License does not apply to the other self-contained works thus compiled
+with the Document, on account of their being thus compiled, if they
+are not themselves derivative works of the Document.
+
+If the Cover Text requirement of section 3 is applicable to these
+copies of the Document, then if the Document is less than one quarter
+of the entire aggregate, the Document's Cover Texts may be placed on
+covers that surround only the Document within the aggregate.
+Otherwise they must appear on covers around the whole aggregate.
+
+
+8. TRANSLATION
+
+Translation is considered a kind of modification, so you may
+distribute translations of the Document under the terms of section 4.
+Replacing Invariant Sections with translations requires special
+permission from their copyright holders, but you may include
+translations of some or all Invariant Sections in addition to the
+original versions of these Invariant Sections. You may include a
+translation of this License provided that you also include the
+original English version of this License. In case of a disagreement
+between the translation and the original English version of this
+License, the original English version will prevail.
+
+
+9. TERMINATION
+
+You may not copy, modify, sublicense, or distribute the Document except
+as expressly provided for under this License. Any other attempt to
+copy, modify, sublicense or distribute the Document is void, and will
+automatically terminate your rights under this License. However,
+parties who have received copies, or rights, from you under this
+License will not have their licenses terminated so long as such
+parties remain in full compliance.
+
+
+10. FUTURE REVISIONS OF THIS LICENSE
+
+The Free Software Foundation may publish new, revised versions
+of the GNU Free Documentation License from time to time. Such new
+versions will be similar in spirit to the present version, but may
+differ in detail to address new problems or concerns. See
+http://www.gnu.org/copyleft/.
+
+Each version of the License is given a distinguishing version number.
+If the Document specifies that a particular numbered version of this
+License "or any later version" applies to it, you have the option of
+following the terms and conditions either of that specified version or
+of any later version that has been published (not as a draft) by the
+Free Software Foundation. If the Document does not specify a version
+number of this License, you may choose any version ever published (not
+as a draft) by the Free Software Foundation.
+
+
+ADDENDUM: How to use this License for your documents
+
+To use this License in a document you have written, include a copy of
+the License in the document and put the following copyright and
+license notices just after the title page:
+
+ Copyright (c) YEAR YOUR NAME.
+ Permission is granted to copy, distribute and/or modify this document
+ under the terms of the GNU Free Documentation License, Version 1.1
+ or any later version published by the Free Software Foundation;
+ with the Invariant Sections being LIST THEIR TITLES, with the
+ Front-Cover Texts being LIST, and with the Back-Cover Texts being LIST.
+ A copy of the license is included in the section entitled "GNU
+ Free Documentation License".
+
+If you have no Invariant Sections, write "with no Invariant Sections"
+instead of saying which ones are invariant. If you have no
+Front-Cover Texts, write "no Front-Cover Texts" instead of
+"Front-Cover Texts being LIST"; likewise for Back-Cover Texts.
+
+If your document contains nontrivial examples of program code, we
+recommend releasing these examples in parallel under your choice of
+free software license, such as the GNU General Public License,
+to permit their use in free software.
Index: COPYING.LGPL
===================================================================
RCS file: COPYING.LGPL
diff -N COPYING.LGPL
--- /dev/null Wed Jul 5 19:49:51 2000
+++ /tmp/cvsC7vCBa Mon Sep 24 09:49:08 2001
@@ -0,0 +1,504 @@
+ GNU LESSER GENERAL PUBLIC LICENSE
+ Version 2.1, February 1999
+
+ Copyright (C) 1991, 1999 Free Software Foundation, Inc.
+ 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+[This is the first released version of the Lesser GPL. It also counts
+ as the successor of the GNU Library Public License, version 2, hence
+ the version number 2.1.]
+
+ Preamble
+
+ The licenses for most software are designed to take away your
+freedom to share and change it. By contrast, the GNU General Public
+Licenses are intended to guarantee your freedom to share and change
+free software--to make sure the software is free for all its users.
+
+ This license, the Lesser General Public License, applies to some
+specially designated software packages--typically libraries--of the
+Free Software Foundation and other authors who decide to use it. You
+can use it too, but we suggest you first think carefully about whether
+this license or the ordinary General Public License is the better
+strategy to use in any particular case, based on the explanations below.
+
+ When we speak of free software, we are referring to freedom of use,
+not price. Our General Public Licenses are designed to make sure that
+you have the freedom to distribute copies of free software (and charge
+for this service if you wish); that you receive source code or can get
+it if you want it; that you can change the software and use pieces of
+it in new free programs; and that you are informed that you can do
+these things.
+
+ To protect your rights, we need to make restrictions that forbid
+distributors to deny you these rights or to ask you to surrender these
+rights. These restrictions translate to certain responsibilities for
+you if you distribute copies of the library or if you modify it.
+
+ For example, if you distribute copies of the library, whether gratis
+or for a fee, you must give the recipients all the rights that we gave
+you. You must make sure that they, too, receive or can get the source
+code. If you link other code with the library, you must provide
+complete object files to the recipients, so that they can relink them
+with the library after making changes to the library and recompiling
+it. And you must show them these terms so they know their rights.
+
+ We protect your rights with a two-step method: (1) we copyright the
+library, and (2) we offer you this license, which gives you legal
+permission to copy, distribute and/or modify the library.
+
+ To protect each distributor, we want to make it very clear that
+there is no warranty for the free library. Also, if the library is
+modified by someone else and passed on, the recipients should know
+that what they have is not the original version, so that the original
+author's reputation will not be affected by problems that might be
+introduced by others.
+
+ Finally, software patents pose a constant threat to the existence of
+any free program. We wish to make sure that a company cannot
+effectively restrict the users of a free program by obtaining a
+restrictive license from a patent holder. Therefore, we insist that
+any patent license obtained for a version of the library must be
+consistent with the full freedom of use specified in this license.
+
+ Most GNU software, including some libraries, is covered by the
+ordinary GNU General Public License. This license, the GNU Lesser
+General Public License, applies to certain designated libraries, and
+is quite different from the ordinary General Public License. We use
+this license for certain libraries in order to permit linking those
+libraries into non-free programs.
+
+ When a program is linked with a library, whether statically or using
+a shared library, the combination of the two is legally speaking a
+combined work, a derivative of the original library. The ordinary
+General Public License therefore permits such linking only if the
+entire combination fits its criteria of freedom. The Lesser General
+Public License permits more lax criteria for linking other code with
+the library.
+
+ We call this license the "Lesser" General Public License because it
+does Less to protect the user's freedom than the ordinary General
+Public License. It also provides other free software developers Less
+of an advantage over competing non-free programs. These disadvantages
+are the reason we use the ordinary General Public License for many
+libraries. However, the Lesser license provides advantages in certain
+special circumstances.
+
+ For example, on rare occasions, there may be a special need to
+encourage the widest possible use of a certain library, so that it becomes
+a de-facto standard. To achieve this, non-free programs must be
+allowed to use the library. A more frequent case is that a free
+library does the same job as widely used non-free libraries. In this
+case, there is little to gain by limiting the free library to free
+software only, so we use the Lesser General Public License.
+
+ In other cases, permission to use a particular library in non-free
+programs enables a greater number of people to use a large body of
+free software. For example, permission to use the GNU C Library in
+non-free programs enables many more people to use the whole GNU
+operating system, as well as its variant, the GNU/Linux operating
+system.
+
+ Although the Lesser General Public License is Less protective of the
+users' freedom, it does ensure that the user of a program that is
+linked with the Library has the freedom and the wherewithal to run
+that program using a modified version of the Library.
+
+ The precise terms and conditions for copying, distribution and
+modification follow. Pay close attention to the difference between a
+"work based on the library" and a "work that uses the library". The
+former contains code derived from the library, whereas the latter must
+be combined with the library in order to run.
+
+ GNU LESSER GENERAL PUBLIC LICENSE
+ TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
+
+ 0. This License Agreement applies to any software library or other
+program which contains a notice placed by the copyright holder or
+other authorized party saying it may be distributed under the terms of
+this Lesser General Public License (also called "this License").
+Each licensee is addressed as "you".
+
+ A "library" means a collection of software functions and/or data
+prepared so as to be conveniently linked with application programs
+(which use some of those functions and data) to form executables.
+
+ The "Library", below, refers to any such software library or work
+which has been distributed under these terms. A "work based on the
+Library" means either the Library or any derivative work under
+copyright law: that is to say, a work containing the Library or a
+portion of it, either verbatim or with modifications and/or translated
+straightforwardly into another language. (Hereinafter, translation is
+included without limitation in the term "modification".)
+
+ "Source code" for a work means the preferred form of the work for
+making modifications to it. For a library, complete source code means
+all the source code for all modules it contains, plus any associated
+interface definition files, plus the scripts used to control compilation
+and installation of the library.
+
+ Activities other than copying, distribution and modification are not
+covered by this License; they are outside its scope. The act of
+running a program using the Library is not restricted, and output from
+such a program is covered only if its contents constitute a work based
+on the Library (independent of the use of the Library in a tool for
+writing it). Whether that is true depends on what the Library does
+and what the program that uses the Library does.
+
+ 1. You may copy and distribute verbatim copies of the Library's
+complete source code as you receive it, in any medium, provided that
+you conspicuously and appropriately publish on each copy an
+appropriate copyright notice and disclaimer of warranty; keep intact
+all the notices that refer to this License and to the absence of any
+warranty; and distribute a copy of this License along with the
+Library.
+
+ You may charge a fee for the physical act of transferring a copy,
+and you may at your option offer warranty protection in exchange for a
+fee.
+
+ 2. You may modify your copy or copies of the Library or any portion
+of it, thus forming a work based on the Library, and copy and
+distribute such modifications or work under the terms of Section 1
+above, provided that you also meet all of these conditions:
+
+ a) The modified work must itself be a software library.
+
+ b) You must cause the files modified to carry prominent notices
+ stating that you changed the files and the date of any change.
+
+ c) You must cause the whole of the work to be licensed at no
+ charge to all third parties under the terms of this License.
+
+ d) If a facility in the modified Library refers to a function or a
+ table of data to be supplied by an application program that uses
+ the facility, other than as an argument passed when the facility
+ is invoked, then you must make a good faith effort to ensure that,
+ in the event an application does not supply such function or
+ table, the facility still operates, and performs whatever part of
+ its purpose remains meaningful.
+
+ (For example, a function in a library to compute square roots has
+ a purpose that is entirely well-defined independent of the
+ application. Therefore, Subsection 2d requires that any
+ application-supplied function or table used by this function must
+ be optional: if the application does not supply it, the square
+ root function must still compute square roots.)
+
+These requirements apply to the modified work as a whole. If
+identifiable sections of that work are not derived from the Library,
+and can be reasonably considered independent and separate works in
+themselves, then this License, and its terms, do not apply to those
+sections when you distribute them as separate works. But when you
+distribute the same sections as part of a whole which is a work based
+on the Library, the distribution of the whole must be on the terms of
+this License, whose permissions for other licensees extend to the
+entire whole, and thus to each and every part regardless of who wrote
+it.
+
+Thus, it is not the intent of this section to claim rights or contest
+your rights to work written entirely by you; rather, the intent is to
+exercise the right to control the distribution of derivative or
+collective works based on the Library.
+
+In addition, mere aggregation of another work not based on the Library
+with the Library (or with a work based on the Library) on a volume of
+a storage or distribution medium does not bring the other work under
+the scope of this License.
+
+ 3. You may opt to apply the terms of the ordinary GNU General Public
+License instead of this License to a given copy of the Library. To do
+this, you must alter all the notices that refer to this License, so
+that they refer to the ordinary GNU General Public License, version 2,
+instead of to this License. (If a newer version than version 2 of the
+ordinary GNU General Public License has appeared, then you can specify
+that version instead if you wish.) Do not make any other change in
+these notices.
+
+ Once this change is made in a given copy, it is irreversible for
+that copy, so the ordinary GNU General Public License applies to all
+subsequent copies and derivative works made from that copy.
+
+ This option is useful when you wish to copy part of the code of
+the Library into a program that is not a library.
+
+ 4. You may copy and distribute the Library (or a portion or
+derivative of it, under Section 2) in object code or executable form
+under the terms of Sections 1 and 2 above provided that you accompany
+it with the complete corresponding machine-readable source code, which
+must be distributed under the terms of Sections 1 and 2 above on a
+medium customarily used for software interchange.
+
+ If distribution of object code is made by offering access to copy
+from a designated place, then offering equivalent access to copy the
+source code from the same place satisfies the requirement to
+distribute the source code, even though third parties are not
+compelled to copy the source along with the object code.
+
+ 5. A program that contains no derivative of any portion of the
+Library, but is designed to work with the Library by being compiled or
+linked with it, is called a "work that uses the Library". Such a
+work, in isolation, is not a derivative work of the Library, and
+therefore falls outside the scope of this License.
+
+ However, linking a "work that uses the Library" with the Library
+creates an executable that is a derivative of the Library (because it
+contains portions of the Library), rather than a "work that uses the
+library". The executable is therefore covered by this License.
+Section 6 states terms for distribution of such executables.
+
+ When a "work that uses the Library" uses material from a header file
+that is part of the Library, the object code for the work may be a
+derivative work of the Library even though the source code is not.
+Whether this is true is especially significant if the work can be
+linked without the Library, or if the work is itself a library. The
+threshold for this to be true is not precisely defined by law.
+
+ If such an object file uses only numerical parameters, data
+structure layouts and accessors, and small macros and small inline
+functions (ten lines or less in length), then the use of the object
+file is unrestricted, regardless of whether it is legally a derivative
+work. (Executables containing this object code plus portions of the
+Library will still fall under Section 6.)
+
+ Otherwise, if the work is a derivative of the Library, you may
+distribute the object code for the work under the terms of Section 6.
+Any executables containing that work also fall under Section 6,
+whether or not they are linked directly with the Library itself.
+
+ 6. As an exception to the Sections above, you may also combine or
+link a "work that uses the Library" with the Library to produce a
+work containing portions of the Library, and distribute that work
+under terms of your choice, provided that the terms permit
+modification of the work for the customer's own use and reverse
+engineering for debugging such modifications.
+
+ You must give prominent notice with each copy of the work that the
+Library is used in it and that the Library and its use are covered by
+this License. You must supply a copy of this License. If the work
+during execution displays copyright notices, you must include the
+copyright notice for the Library among them, as well as a reference
+directing the user to the copy of this License. Also, you must do one
+of these things:
+
+ a) Accompany the work with the complete corresponding
+ machine-readable source code for the Library including whatever
+ changes were used in the work (which must be distributed under
+ Sections 1 and 2 above); and, if the work is an executable linked
+ with the Library, with the complete machine-readable "work that
+ uses the Library", as object code and/or source code, so that the
+ user can modify the Library and then relink to produce a modified
+ executable containing the modified Library. (It is understood
+ that the user who changes the contents of definitions files in the
+ Library will not necessarily be able to recompile the application
+ to use the modified definitions.)
+
+ b) Use a suitable shared library mechanism for linking with the
+ Library. A suitable mechanism is one that (1) uses at run time a
+ copy of the library already present on the user's computer system,
+ rather than copying library functions into the executable, and (2)
+ will operate properly with a modified version of the library, if
+ the user installs one, as long as the modified version is
+ interface-compatible with the version that the work was made with.
+
+ c) Accompany the work with a written offer, valid for at
+ least three years, to give the same user the materials
+ specified in Subsection 6a, above, for a charge no more
+ than the cost of performing this distribution.
+
+ d) If distribution of the work is made by offering access to copy
+ from a designated place, offer equivalent access to copy the above
+ specified materials from the same place.
+
+ e) Verify that the user has already received a copy of these
+ materials or that you have already sent this user a copy.
+
+ For an executable, the required form of the "work that uses the
+Library" must include any data and utility programs needed for
+reproducing the executable from it. However, as a special exception,
+the materials to be distributed need not include anything that is
+normally distributed (in either source or binary form) with the major
+components (compiler, kernel, and so on) of the operating system on
+which the executable runs, unless that component itself accompanies
+the executable.
+
+ It may happen that this requirement contradicts the license
+restrictions of other proprietary libraries that do not normally
+accompany the operating system. Such a contradiction means you cannot
+use both them and the Library together in an executable that you
+distribute.
+
+ 7. You may place library facilities that are a work based on the
+Library side-by-side in a single library together with other library
+facilities not covered by this License, and distribute such a combined
+library, provided that the separate distribution of the work based on
+the Library and of the other library facilities is otherwise
+permitted, and provided that you do these two things:
+
+ a) Accompany the combined library with a copy of the same work
+ based on the Library, uncombined with any other library
+ facilities. This must be distributed under the terms of the
+ Sections above.
+
+ b) Give prominent notice with the combined library of the fact
+ that part of it is a work based on the Library, and explaining
+ where to find the accompanying uncombined form of the same work.
+
+ 8. You may not copy, modify, sublicense, link with, or distribute
+the Library except as expressly provided under this License. Any
+attempt otherwise to copy, modify, sublicense, link with, or
+distribute the Library is void, and will automatically terminate your
+rights under this License. However, parties who have received copies,
+or rights, from you under this License will not have their licenses
+terminated so long as such parties remain in full compliance.
+
+ 9. You are not required to accept this License, since you have not
+signed it. However, nothing else grants you permission to modify or
+distribute the Library or its derivative works. These actions are
+prohibited by law if you do not accept this License. Therefore, by
+modifying or distributing the Library (or any work based on the
+Library), you indicate your acceptance of this License to do so, and
+all its terms and conditions for copying, distributing or modifying
+the Library or works based on it.
+
+ 10. Each time you redistribute the Library (or any work based on the
+Library), the recipient automatically receives a license from the
+original licensor to copy, distribute, link with or modify the Library
+subject to these terms and conditions. You may not impose any further
+restrictions on the recipients' exercise of the rights granted herein.
+You are not responsible for enforcing compliance by third parties with
+this License.
+
+ 11. If, as a consequence of a court judgment or allegation of patent
+infringement or for any other reason (not limited to patent issues),
+conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License. If you cannot
+distribute so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you
+may not distribute the Library at all. For example, if a patent
+license would not permit royalty-free redistribution of the Library by
+all those who receive copies directly or indirectly through you, then
+the only way you could satisfy both it and this License would be to
+refrain entirely from distribution of the Library.
+
+If any portion of this section is held invalid or unenforceable under any
+particular circumstance, the balance of the section is intended to apply,
+and the section as a whole is intended to apply in other circumstances.
+
+It is not the purpose of this section to induce you to infringe any
+patents or other property right claims or to contest validity of any
+such claims; this section has the sole purpose of protecting the
+integrity of the free software distribution system which is
+implemented by public license practices. Many people have made
+generous contributions to the wide range of software distributed
+through that system in reliance on consistent application of that
+system; it is up to the author/donor to decide if he or she is willing
+to distribute software through any other system and a licensee cannot
+impose that choice.
+
+This section is intended to make thoroughly clear what is believed to
+be a consequence of the rest of this License.
+
+ 12. If the distribution and/or use of the Library is restricted in
+certain countries either by patents or by copyrighted interfaces, the
+original copyright holder who places the Library under this License may add
+an explicit geographical distribution limitation excluding those countries,
+so that distribution is permitted only in or among countries not thus
+excluded. In such case, this License incorporates the limitation as if
+written in the body of this License.
+
+ 13. The Free Software Foundation may publish revised and/or new
+versions of the Lesser General Public License from time to time.
+Such new versions will be similar in spirit to the present version,
+but may differ in detail to address new problems or concerns.
+
+Each version is given a distinguishing version number. If the Library
+specifies a version number of this License which applies to it and
+"any later version", you have the option of following the terms and
+conditions either of that version or of any later version published by
+the Free Software Foundation. If the Library does not specify a
+license version number, you may choose any version ever published by
+the Free Software Foundation.
+
+ 14. If you wish to incorporate parts of the Library into other free
+programs whose distribution conditions are incompatible with these,
+write to the author to ask for permission. For software which is
+copyrighted by the Free Software Foundation, write to the Free
+Software Foundation; we sometimes make exceptions for this. Our
+decision will be guided by the two goals of preserving the free status
+of all derivatives of our free software and of promoting the sharing
+and reuse of software generally.
+
+ NO WARRANTY
+
+ 15. BECAUSE THE LIBRARY IS LICENSED FREE OF CHARGE, THERE IS NO
+WARRANTY FOR THE LIBRARY, TO THE EXTENT PERMITTED BY APPLICABLE LAW.
+EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR
+OTHER PARTIES PROVIDE THE LIBRARY "AS IS" WITHOUT WARRANTY OF ANY
+KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE
+LIBRARY IS WITH YOU. SHOULD THE LIBRARY PROVE DEFECTIVE, YOU ASSUME
+THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
+
+ 16. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN
+WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY
+AND/OR REDISTRIBUTE THE LIBRARY AS PERMITTED ABOVE, BE LIABLE TO YOU
+FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR
+CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE
+LIBRARY (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING
+RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A
+FAILURE OF THE LIBRARY TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF
+SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH
+DAMAGES.
+
+ END OF TERMS AND CONDITIONS
+
+ How to Apply These Terms to Your New Libraries
+
+ If you develop a new library, and you want it to be of the greatest
+possible use to the public, we recommend making it free software that
+everyone can redistribute and change. You can do so by permitting
+redistribution under these terms (or, alternatively, under the terms of the
+ordinary General Public License).
+
+ To apply these terms, attach the following notices to the library. It is
+safest to attach them to the start of each source file to most effectively
+convey the exclusion of warranty; and each file should have at least the
+"copyright" line and a pointer to where the full notice is found.
+
+ <one line to give the library's name and a brief idea of what it does.>
+ Copyright (C) <year> <name of author>
+
+ This library is free software; you can redistribute it and/or
+ modify it under the terms of the GNU Lesser General Public
+ License as published by the Free Software Foundation; either
+ version 2.1 of the License, or (at your option) any later version.
+
+ This library is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ Lesser General Public License for more details.
+
+ You should have received a copy of the GNU Lesser General Public
+ License along with this library; if not, write to the Free Software
+ Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+Also add information on how to contact you by electronic and paper mail.
+
+You should also get your employer (if you work as a programmer) or your
+school, if any, to sign a "copyright disclaimer" for the library, if
+necessary. Here is a sample; alter the names:
+
+ Yoyodyne, Inc., hereby disclaims all copyright interest in the
+ library `Frob' (a library for tweaking knobs) written by James Random Hacker.
+
+ <signature of Ty Coon>, 1 April 1990
+ Ty Coon, President of Vice
+
+That's all there is to it!
+
+
Index: Mmakefile
===================================================================
RCS file: /var/cvs/rat/rationality/logic/mercury/extras/lex/Mmakefile,v
retrieving revision 1.1.1.1
retrieving revision 1.2
diff -u -r1.1.1.1 -r1.2
--- Mmakefile 2001/07/13 09:33:50 1.1.1.1
+++ Mmakefile 2001/08/14 18:16:42 1.2
@@ -18,7 +18,7 @@
# A directory $(INSTALL_PREFIX)/lib/mercury will be created, if
# necessary, and everything put there.
#
-#INSTALL_PREFIX = $(HOME)/mercury
+INSTALL_PREFIX = ${MERCURY_HOME}-extras
# Omit this line if you want to install the default grades.
# Edit this line if you want to install with different grades.
Index: README
===================================================================
RCS file: /var/cvs/rat/rationality/logic/mercury/extras/lex/README,v
retrieving revision 1.1.1.1
retrieving revision 1.14
diff -u -r1.1.1.1 -r1.14
--- README 2001/07/13 09:33:50 1.1.1.1
+++ README 2001/08/14 18:16:42 1.14
@@ -1,12 +1,14 @@
lex 1.0 (very alpha)
Fri Aug 25 17:54:28 2000
Copyright (C) 2001 Ralph Becket <rbeck at microsoft.com>
-
THIS FILE IS HEREBY CONTRIBUTED TO THE MERCURY PROJECT TO
BE RELEASED UNDER WHATEVER LICENCE IS DEEMED APPROPRIATE
BY THE ADMINISTRATORS OF THE MERCURY PROJECT.
-
-
+Sun Aug 5 16:15:27 UTC 2001
+Copyright (C) 2001 The Rationalizer Intelligent Software AG
+ The changes made by Rationalizer are contributed under the terms
+ of the GNU Free Documentation License - see the file COPYING.DOC
+ in this directory.
This package defines a lexer for Mercury. There is plenty of scope
for optimization, however it is reasonably efficient and does provide
@@ -23,25 +25,32 @@
:- type token
---> comment
- ; id
- ; num.
+ ; id(string)
+ ; num(int)
+ ; space.
3. Set up a list of annotated_lexemes.
Lexemes = [
- lexeme(noval(comment), (atom('%'), star(dot))),
- lexeme(value(id), identifier),
- lexeme(ignore, whitespace)
+ ( "%" ++ *(dot) -> return(comment) ),
+ ( identifier -> (func(Id) = id(Id)) ),
+ ( signed_int -> (func(N) = num(string__det_to_int(N))) ),
+ ( whitespace -> return(space) )
]
-noval tokens are simply identified;
-value tokens are identified and returned with the string matched;
-ignore regexps are simply passed over.
+A lexeme is a pair (RegExp - TokFn) where RegExp is a
+regular expression and TokFn is a token_creator function mapping
+the string matched by RegExp to a token value.
+
4. Set up a lexer with an appropriate read predicate (see the buf module).
Lexer = lex__init(Lexemes, lex__read_from_stdin)
+ or:
+
+ Lexer = lex__init(Lexemes, lex__read_from_stdin, ignore(space))
+
5. Obtain a live lexer state.
State0 = lex__start(Lexer, IO0)
@@ -49,12 +58,21 @@
6. Use it to lex the input stream.
lex__read(Result, State0, State1),
- ( Result = ok(NoValToken), ...
- ; Result = ok(ValueToken, String), ...
- ; Result = error(OffsetInInputStream), ...
+ ( Result = ok(Token), ...
+ ; Result = error(Message, OffsetInInputStream), ...
; Result = eof, ...
)
+ NOTE: The result type of lex__read is io__read_result(token).
+ io__read_result is documented in the library file io.m as:
+ :- type io__read_result(T) ---> ok(T)
+ ; eof
+ ; error(string, int).
+ % error message, line number
+ In contrast to this the `int' lex returns in the case of an error
+ does not correspond to the line number but to the character offset.
+ Hence be careful when processing lex errors.
+
7. If you need to manipulate the source object, you can.
lex__manipulate_source(io__print("Not finished yet?"), State1, State2)
@@ -69,7 +87,12 @@
and the option to write out a compilable source file for the lexer.
+OPPORTUNITIES FOR MODULARIZATION
+1. Remove regexp functionality from lex.m and lex.regexp.m and put it into
+ a distinct regexp library.
+
+
OPPORTUNITIES FOR OPTIMIZATION
1. Move from chars to bytes.
@@ -78,4 +101,16 @@
3. Implement the first-byte optimization whereby the set of `live lexemes'
is decided by the first byte read in on a lexing pass.
4. Implement state machine minimization (may or may not be worthwhile.)
+
+
+FEATURES TO ADD:
+
+1. Symbol table management (additional parameters for the user-defined
+ predicates containing the symbol table before and after processing
+ a lexeme)
+2. func (string) = regexp, where the function parameter contains a
+ regexp definition in a form like used in languages in Perl, awk etc.
+3. line# as part of the offset
+4. extend the lexer interface somehow to get more detailed information
+ about the token resp. error position
Index: lex.buf.m
===================================================================
RCS file: /var/cvs/rat/rationality/logic/mercury/extras/lex/lex.buf.m,v
retrieving revision 1.1.1.1
retrieving revision 1.3
diff -u -r1.1.1.1 -r1.3
--- lex.buf.m 2001/07/13 09:33:50 1.1.1.1
+++ lex.buf.m 2001/08/06 20:06:17 1.3
@@ -1,8 +1,9 @@
% ---------------------------------------------------------------------------- %
+% vim: ts=4 sw=4 et tw=0 wm=0 ff=unix
+%
% lex.buf.m
% Copyright (C) 2001 Ralph Becket <rbeck at microsoft.com>
% Sat Aug 19 16:56:30 BST 2000
-% vim: ts=4 sw=4 et tw=0 wm=0 ff=unix
%
% THIS FILE IS HEREBY CONTRIBUTED TO THE MERCURY PROJECT TO
% BE RELEASED UNDER WHATEVER LICENCE IS DEEMED APPROPRIATE
@@ -94,7 +95,7 @@
:- interface.
-:- import_module int, array, char, bool, string, io.
+:- import_module array, char, bool, string.
@@ -171,7 +172,6 @@
:- implementation.
:- import_module exception.
-
% The amount the buffer is grown by if (a) more space is
Index: lex.convert_NFA_to_DFA.m
===================================================================
RCS file: /var/cvs/rat/rationality/logic/mercury/extras/lex/lex.convert_NFA_to_DFA.m,v
retrieving revision 1.1.1.1
retrieving revision 1.2
diff -u -r1.1.1.1 -r1.2
--- lex.convert_NFA_to_DFA.m 2001/07/13 09:33:53 1.1.1.1
+++ lex.convert_NFA_to_DFA.m 2001/08/06 20:06:18 1.2
@@ -1,8 +1,10 @@
%-----------------------------------------------------------------------------
+%
+% vim: ts=4 sw=4 et tw=0 wm=0 ff=unix
+%
% lex.convert_NFA_to_DFA.m
% Copyright (C) 2001 Ralph Becket <rbeck at microsoft.com>
% Fri Aug 18 12:30:25 BST 2000
-% vim: ts=4 sw=4 et tw=0 wm=0 ff=unix
%
% Powerset construction used to transform NFAs into DFAs.
%
Index: lex.lexeme.m
===================================================================
RCS file: /var/cvs/rat/rationality/logic/mercury/extras/lex/lex.lexeme.m,v
retrieving revision 1.1.1.1
retrieving revision 1.10
diff -u -r1.1.1.1 -r1.10
--- lex.lexeme.m 2001/07/13 09:33:53 1.1.1.1
+++ lex.lexeme.m 2001/08/14 18:16:42 1.10
@@ -1,16 +1,23 @@
%-----------------------------------------------------------------------------
+%
+% vim: ts=4 sw=4 et tw=0 wm=0 ff=unix
+%
% lex.lexeme.m
-% Copyright (C) 2001 Ralph Becket <rbeck at microsoft.com>
% Sat Aug 19 08:22:32 BST 2000
-% vim: ts=4 sw=4 et tw=0 wm=0 ff=unix
+% Copyright (C) 2001 Ralph Becket <rbeck at microsoft.com>
+% THIS FILE IS HEREBY CONTRIBUTED TO THE MERCURY PROJECT TO
+% BE RELEASED UNDER WHATEVER LICENCE IS DEEMED APPROPRIATE
+% BY THE ADMINISTRATORS OF THE MERCURY PROJECT.
+% Thu Jul 26 07:45:47 UTC 2001
+% Copyright (C) 2001 The Rationalizer Intelligent Software AG
+% The changes made by Rationalizer are contributed under the terms
+% of the GNU Lesser General Public License, see the file COPYING.LGPL
+% in this directory.
%
% A lexeme combines a token with a regexp. The lexer compiles
% lexemes and returns the longest successul parse in the input
% stream or an error if no match occurs.
%
-% THIS FILE IS HEREBY CONTRIBUTED TO THE MERCURY PROJECT TO
-% BE RELEASED UNDER WHATEVER LICENCE IS DEEMED APPROPRIATE
-% BY THE ADMINISTRATORS OF THE MERCURY PROJECT.
%
%----------------------------------------------------------------------------- %
@@ -22,15 +29,17 @@
:- type compiled_lexeme(T)
---> compiled_lexeme(
- clxm_token :: T,
- clxm_state :: state_no,
- clxm_transition_map :: transition_map
+ token :: token_creator(T),
+ state :: state_no,
+ transition_map :: transition_map
).
+:- inst compiled_lexeme(Inst)
+ ---> compiled_lexeme(Inst, ground, ground).
:- type transition_map
---> transition_map(
- trm_accepting_states :: bitmap,
- trm_rows :: array(row)
+ accepting_states :: bitmap,
+ rows :: array(row)
).
% A transition row is an array of byte_transitions.
@@ -59,12 +68,12 @@
% an accepting state_no.
%
:- pred next_state(compiled_lexeme(T), state_no, char, state_no, bool).
-:- mode next_state(in, in, in, out, out) is semidet.
+:- mode next_state(in(live_lexeme), in, in, out, out) is semidet.
% Succeeds iff a compiled_lexeme is in an accepting state_no.
%
-:- pred in_accepting_state(compiled_lexeme(T)).
-:- mode in_accepting_state(in) is semidet.
+:- pred in_accepting_state(live_lexeme(T)).
+:- mode in_accepting_state(in(live_lexeme)) is semidet.
%----------------------------------------------------------------------------- %
%----------------------------------------------------------------------------- %
@@ -77,7 +86,7 @@
%------------------------------------------------------------------------------%
compile_lexeme(Lexeme) = CompiledLexeme :-
- Lexeme = lexeme(Token, RegExp),
+ Lexeme = (RegExp - TokenCreator),
NFA = remove_null_transitions(regexp_to_NFA(RegExp)),
DFA = convert_NFA_to_DFA(NFA),
StartState = DFA ^ smc_start_state,
@@ -87,7 +96,7 @@
Accepting = set_accepting_states(StopStates, bitmap__new(N, no)),
Rows = array(set_up_rows(0, N, Transitions)),
TransitionMap = transition_map(Accepting, Rows),
- CompiledLexeme = compiled_lexeme(Token, StartState, TransitionMap).
+ CompiledLexeme = compiled_lexeme(TokenCreator, StartState, TransitionMap).
%------------------------------------------------------------------------------%
@@ -157,9 +166,11 @@
%------------------------------------------------------------------------------%
next_state(CLXM, CurrentState, Char, NextState, IsAccepting) :-
- Rows = CLXM ^ clxm_transition_map ^ trm_rows,
- AcceptingStates = CLXM ^ clxm_transition_map ^ trm_accepting_states,
- find_next_state(char__to_int(Char), Rows ^ elem(CurrentState), NextState),
+ Rows = CLXM ^ transition_map ^ rows,
+ AcceptingStates = CLXM ^ transition_map ^ accepting_states,
+ find_next_state(char__to_int(Char),
+ Rows ^ elem(CurrentState),
+ NextState),
IsAccepting = bitmap__get(AcceptingStates, NextState).
%------------------------------------------------------------------------------%
@@ -189,7 +200,7 @@
in_accepting_state(CLXM) :-
bitmap__is_set(
- CLXM ^ clxm_transition_map ^ trm_accepting_states, CLXM ^ clxm_state
+ CLXM ^ transition_map ^ accepting_states, CLXM ^ state
).
%------------------------------------------------------------------------------%
Index: lex.m
===================================================================
RCS file: /var/cvs/rat/rationality/logic/mercury/extras/lex/lex.m,v
retrieving revision 1.1.1.1
retrieving revision 1.13
diff -u -r1.1.1.1 -r1.13
--- lex.m 2001/07/13 09:33:53 1.1.1.1
+++ lex.m 2001/08/14 18:16:42 1.13
@@ -1,15 +1,20 @@
%----------------------------------------------------------------------------- %
+% vim: ts=4 sw=4 et tw=0 wm=0 ff=unix
+%
% lex.m
% Copyright (C) 2001 Ralph Becket <rbeck at microsoft.com>
% Sun Aug 20 09:08:46 BST 2000
-% vim: ts=4 sw=4 et tw=0 wm=0 ff=unix
-%
-% This module puts everything together, compiling a list of lexemes
-% into state machines and turning the input stream into a token stream.
-%
% THIS FILE IS HEREBY CONTRIBUTED TO THE MERCURY PROJECT TO
% BE RELEASED UNDER WHATEVER LICENCE IS DEEMED APPROPRIATE
% BY THE ADMINISTRATORS OF THE MERCURY PROJECT.
+% Thu Jul 26 07:45:47 UTC 2001
+% Copyright (C) 2001 The Rationalizer Intelligent Software AG
+% The changes made by Rationalizer are contributed under the terms
+% of the GNU Lesser General Public License, see the file COPYING.LGPL
+% in this directory.
+%
+% This module puts everything together, compiling a list of lexemes
+% into state machines and turning the input stream into a token stream.
%
%----------------------------------------------------------------------------- %
@@ -19,36 +24,23 @@
:- import_module std_util, string, char, list, io.
+:- type token_creator(Token)
+ == (func(string) = Token).
+:- inst token_creator
+ == (func(in) = out is det).
-
-:- type annotated_lexeme(Token)
- == lexeme(annotated_token(Token)).
-
:- type lexeme(Token)
- ---> lexeme(
- lxm_token :: Token,
- lxm_regexp :: regexp
- ).
-
-:- type annotated_token(Token)
- ---> noval(Token) % Just return ok(Token) on success.
- ; value(Token) % Return ok(Token, String) on success.
- ; ignore. % Just skip over these tokens.
+ == pair(regexp, token_creator(Token)).
+
+:- inst lexeme(Inst)
+ ---> (ground - Inst).
:- type lexer(Token, Source).
:- inst lexer
- == bound(lexer(ground, read_pred)).
+ ---> lexer(ground, ignore_pred, read_pred).
:- type lexer_state(Token, Source).
-:- type lexer_result(Token)
- ---> ok(Token) % Noval token matched.
- ; ok(Token, string) % Value token matched.
- ; eof % End of input.
- ; error(int). % No matches for string at this offset.
-
-
-
:- type offset
== int. % Byte offset into the source data.
@@ -65,28 +57,46 @@
== pred(offset, read_result, T, T).
:- inst read_pred
== ( pred(in, out, di, uo) is det ).
-
+ % ignore_pred(Token): if it does not fail, Token must be ignored
+ %
+:- type ignore_pred(Tok)
+ == pred(Tok).
+:- inst ignore_pred
+ == ( pred(in) is semidet ).
% The type of regular expressions.
%
-:- type regexp
- ---> null % The empty regexp
- ; atom(char) % Match a single char
- ; (regexp >> regexp) % Concatenation
- ; (regexp \/ regexp) % Alternation
- ; star(regexp) % Kleene closure
- .
+:- type regexp.
+ % The typeclass for types having a natural converter to regexp's
+ %
+:- typeclass regexp(T) where [
+ func re(T) = regexp
+].
+ % Handling regexp's based on the typeclass regexp(T)
+ %
+:- func null = regexp.
+:- func T1 ++ T2 = regexp <= (regexp(T1), regexp(T2)).
+ % one of the following two functions will declared be deprecated
+ % later on, we still only have to decide which one
+:- func T1 \/ T2 = regexp <= (regexp(T1), regexp(T2)).
+:- func (T1 or T2) = regexp <= (regexp(T1), regexp(T2)).
+:- func *(T) = regexp <= regexp(T).
+
+ % Some instances of typeclass regexp(T)
+ %
+:- instance regexp(regexp).
+:- instance regexp(char).
+:- instance regexp(string).
% Some basic non-primitive regexps.
%
-:- func str(string) = regexp. % str("abc") = atom(a) >> atom(b) >> atom(c)
-:- func any(string) = regexp. % any("abc") = atom(a) \/ atom(b) \/ atom(c)
-:- func anybut(string) = regexp.% anybut("abc") is complement of any("abc")
-:- func opt(regexp) = regexp. % opt(R) = R \/ null
-:- func plus(regexp) = regexp. % plus(R) = R \/ star(R)
+:- func any(string) = regexp. % any("abc") = ('a') \/ ('b') \/ ('c')
+:- func anybut(string) = regexp. % anybut("abc") is complement of any("abc")
+:- func ?(T) = regexp <= regexp(T). % ?(R) = R \/ null
+:- func +(T) = regexp <= regexp(T). % +(R) = R \/ *(R)
% Some useful single-char regexps.
%
@@ -95,34 +105,54 @@
:- func upper = regexp. % upper = any("ABC...Z")
:- func alpha = regexp. % alpha = lower \/ upper
:- func alphanum = regexp. % alphanum = alpha \/ digit
-:- func identstart = regexp. % identstart = alpha \/ str("_")
-:- func ident = regexp. % ident = alphanum \/ str("_")
-:- func nl = regexp. % nl = str("\n")
-:- func tab = regexp. % tab = str("\t")
-:- func spc = regexp. % spc = str(" ")
+:- func identstart = regexp. % identstart = alpha \/ "_"
+:- func ident = regexp. % ident = alphanum \/ "_"
+:- func nl = regexp. % nl = re("\n")
+:- func tab = regexp. % tab = re("\t")
+:- func spc = regexp. % spc = re(" ")
:- func wspc = regexp. % wspc = any(" \t\n\r\f\v")
-:- func dot = regexp. % dot = any("<except \n>")
+:- func dot = regexp. % dot = anybut("\n")
% Some useful compound regexps.
%
-:- func nat = regexp. % nat = plus(digit)
-:- func signed_int = regexp. % signed_int = opt(any("+-")) >> nat
+:- func nat = regexp. % nat = +(digit)
+:- func signed_int = regexp. % signed_int = ?(any("+-")) ++ nat
:- func real = regexp. % real = \d+((.\d+([eE]int)?)|[eE]int)
-:- func identifier = regexp. % identifier = identstart >> star(ident)
-:- func whitespace = regexp. % whitespace = star(wspc)
-:- func junk = regexp. % junk = star(dot)
+:- func identifier = regexp. % identifier = identstart ++ *(ident)
+:- func whitespace = regexp. % whitespace = *(wspc)
+:- func junk = regexp. % junk = *(dot)
+
+ % Utility predicate to create ignore_pred's.
+ % Use it in the form `ignore(my_token)' to ignore just `my_token'.
+ %
+:- pred ignore(Token::in, Token::in) is semidet.
+
+ % Utility function to return noval tokens.
+ % Use it in the form `return(my_token) inside a lexeme definition.
+ %
+:- func return(T,string) = T.
+
+ % Utility operator to create lexemes.
+:- func (T1 -> token_creator(Tok)) = pair(regexp,token_creator(Tok))
+ <= regexp(T1).
-
-
% Construct a lexer from which we can generate running
% instances.
%
-:- func init(list(annotated_lexeme(Tok)), read_pred(Src)) = lexer(Tok, Src).
+:- func init(list(lexeme(Tok)), read_pred(Src)) = lexer(Tok, Src).
:- mode init(in, in(read_pred)) = out(lexer) is det.
+
+ % Construct a lexer from which we can generate running
+ % instances. If we construct a lexer with init/4, we
+ % can additionally ignore specific tokens.
+ %
+:- func init(list(lexeme(Tok)), read_pred(Src), ignore_pred(Tok)) =
+ lexer(Tok, Src).
+:- mode init(in, in(read_pred), in(ignore_pred)) = out(lexer) is det.
- % Handy read predicates.
+ % Handy read predicates.
%
-:- pred read_from_stdin(offset, read_result, io__state, io__state).
+:- pred read_from_stdin(offset, read_result, io, io).
:- mode read_from_stdin(in, out, di, uo) is det.
:- pred read_from_string(offset, read_result, string, string).
@@ -135,7 +165,8 @@
:- func start(lexer(Tok, Src), Src) = lexer_state(Tok, Src).
:- mode start(in(lexer), di) = uo is det.
-:- pred read(lexer_result(Tok), lexer_state(Tok, Src), lexer_state(Tok, Src)).
+:- pred read(io__read_result(Tok), lexer_state(Tok, Src),
+ lexer_state(Tok, Src)).
:- mode read(out, di, uo) is det.
% Stop a running instance of a lexer and retrieve the input source.
@@ -143,7 +174,7 @@
:- func stop(lexer_state(_Tok, Src)) = Src.
:- mode stop(di) = uo is det.
- % Sometimes (e.g. when lexing the io__state) you want access to the
+ % Sometimes (e.g. when lexing the io__io) you want access to the
% input stream without interrupting the lexing process. This pred
% provides that sort of access.
%
@@ -170,39 +201,67 @@
:- type lexer(Token, Source)
---> lexer(
- lex_compiled_lexemes :: list(live_lexeme(Token)),
- lex_buf_read_pred :: read_pred(Source)
- ).
-
-
+ lex_compiled_lexemes :: list(live_lexeme(Token)),
+ lex_ignore_pred :: ignore_pred(Token),
+ lex_buf_read_pred :: read_pred(Source)
+ ).
:- type lexer_instance(Token, Source)
---> lexer_instance(
- lexi_init_lexemes :: list(live_lexeme(Token)),
- lexi_live_lexemes :: list(live_lexeme(Token)),
- lexi_current_winner :: winner(Token),
- lexi_buf_state :: buf_state(Source)
- ).
-:- inst lexer_instance
- == bound(lexer_instance(ground, ground, ground, buf_state)).
-
+ init_lexemes :: list(live_lexeme(Token)),
+ live_lexemes :: list(live_lexeme(Token)),
+ current_winner :: winner(Token),
+ buf_state :: buf_state(Source),
+ ignore_pred :: ignore_pred(Token)
+ ).
+:- inst lexer_instance
+ ---> lexer_instance(
+ live_lexeme_list,
+ live_lexeme_list,
+ winner,
+ buf__buf_state,
+ ignore_pred
+ ).
:- type live_lexeme(Token)
- == compiled_lexeme(annotated_token(Token)).
+ == compiled_lexeme(Token).
+:- inst live_lexeme
+ == compiled_lexeme(token_creator).
+:- inst live_lexeme_list
+ == list__list_skel(live_lexeme).
:- type winner(Token)
- == maybe(pair(annotated_token(Token), offset)).
+ == maybe(pair(token_creator(Token), offset)).
+:- inst winner
+ ---> yes(pair(token_creator, ground))
+ ; no.
+
+%----------------------------------------------------------------------------- %
+
+ignore(Tok,Tok).
%----------------------------------------------------------------------------- %
-init(Lexemes, BufReadPred) = lexer(CompiledLexemes, BufReadPred) :-
- CompiledLexemes = list__map(lexeme__compile_lexeme, Lexemes).
+return(Token, _) = Token.
%----------------------------------------------------------------------------- %
+(R1 -> TC) = (re(R1) - TC).
+
+%----------------------------------------------------------------------------- %
+
+init(Lexemes, BufReadPred) = init(Lexemes, BufReadPred, DontIgnoreAnything) :-
+ DontIgnoreAnything = ( pred(_::in) is semidet :- semidet_fail ).
+
+init(Lexemes, BufReadPred, IgnorePred) =
+ lexer(CompiledLexemes, IgnorePred, BufReadPred) :-
+ CompiledLexemes = list__map(compile_lexeme, Lexemes).
+
+%----------------------------------------------------------------------------- %
+
start(Lexer, Src) = LexerState :-
init_lexer_instance(Lexer, Instance, Buf),
LexerState = args_lexer_state(Instance, Buf, Src).
@@ -215,7 +274,9 @@
init_lexer_instance(Lexer, Instance, Buf) :-
buf__init(Lexer ^ lex_buf_read_pred, BufState, Buf),
InitLexemes = Lexer ^ lex_compiled_lexemes,
- Instance = lexer_instance(InitLexemes, InitLexemes, no, BufState).
+ IgnorePred = Lexer ^ lex_ignore_pred,
+ Instance = lexer_instance(InitLexemes, InitLexemes, no,
+ BufState, IgnorePred).
%----------------------------------------------------------------------------- %
@@ -231,7 +292,7 @@
-:- pred read_0(lexer_result(Tok),
+:- pred read_0(io__read_result(Tok),
lexer_instance(Tok, Src), lexer_instance(Tok, Src),
buf, buf, Src, Src).
:- mode read_0(out,
@@ -243,7 +304,7 @@
%
read_0(Result, Instance0, Instance, Buf0, Buf, Src0, Src) :-
- BufState0 = Instance0 ^ lexi_buf_state,
+ BufState0 = Instance0 ^ buf_state,
buf__read(BufReadResult, BufState0, BufState1, Buf0, Buf1, Src0, Src1),
(
@@ -259,7 +320,7 @@
%----------------------------------------------------------------------------- %
-:- pred process_char(lexer_result(Tok), char,
+:- pred process_char(io__read_result(Tok), char,
lexer_instance(Tok, Src), lexer_instance(Tok, Src),
buf_state(Src), buf, buf, Src, Src).
:- mode process_char(out, in, in(lexer_instance), out(lexer_instance),
@@ -268,8 +329,8 @@
process_char(Result, Char, Instance0, Instance,
BufState, Buf0, Buf, Src0, Src) :-
- LiveLexemes0 = Instance0 ^ lexi_live_lexemes,
- Winner0 = Instance0 ^ lexi_current_winner,
+ LiveLexemes0 = Instance0 ^ live_lexemes,
+ Winner0 = Instance0 ^ current_winner,
advance_live_lexemes(Char, buf__cursor_offset(BufState),
LiveLexemes0, LiveLexemes, Winner0, Winner),
@@ -282,44 +343,44 @@
LiveLexemes = [_ | _], % Still some open possibilities.
Instance1 = (((Instance0
- ^ lexi_live_lexemes := LiveLexemes)
- ^ lexi_current_winner := Winner)
- ^ lexi_buf_state := BufState),
+ ^ live_lexemes := LiveLexemes)
+ ^ current_winner := Winner)
+ ^ buf_state := BufState),
read_0(Result, Instance1, Instance, Buf0, Buf, Src0, Src)
).
%----------------------------------------------------------------------------- %
-:- pred process_any_winner(lexer_result(Tok), winner(Tok),
+:- pred process_any_winner(io__read_result(Tok), winner(Tok),
lexer_instance(Tok, Src), lexer_instance(Tok, Src),
buf_state(Src), buf, buf, Src, Src).
-:- mode process_any_winner(out, in,
+:- mode process_any_winner(out, in(winner),
in(lexer_instance), out(lexer_instance),
in(buf_state), array_di, array_uo, di, uo) is det.
-process_any_winner(Result, yes(ATok - Offset), Instance0, Instance,
+process_any_winner(Result, yes(TokenCreator - Offset), Instance0, Instance,
BufState0, Buf0, Buf, Src0, Src) :-
BufState1 = buf__rewind_cursor(Offset, BufState0),
Instance1 = ((( Instance0
- ^ lexi_live_lexemes := Instance0 ^ lexi_init_lexemes)
- ^ lexi_current_winner := no)
- ^ lexi_buf_state := buf__commit(BufState1)),
+ ^ live_lexemes := Instance0 ^ init_lexemes)
+ ^ current_winner := no)
+ ^ buf_state := buf__commit(BufState1)),
(
- ATok = noval(Token),
- Result = ok(Token),
- Instance = Instance1,
- Buf = Buf0,
- Src = Src0
- ;
- ATok = value(Token),
- Result = ok(Token, buf__string_to_cursor(BufState1, Buf)),
- Instance = Instance1,
- Buf = Buf0,
- Src = Src0
- ;
- ATok = ignore,
- read_0(Result, Instance1, Instance, Buf0, Buf, Src0, Src)
+ if
+
+ get_token_from_buffer(BufState1, Buf0, Instance0,
+ TokenCreator, Token)
+ then
+
+ Result = ok(Token),
+ Instance = Instance1,
+ Buf = Buf0,
+ Src = Src0
+
+ else
+
+ read_0(Result, Instance1, Instance, Buf0, Buf, Src0, Src)
).
process_any_winner(Result, no, Instance0, Instance,
@@ -327,16 +388,16 @@
Start = buf__start_offset(BufState0),
BufState1 = buf__rewind_cursor(Start + 1, BufState0),
- Result = error(Start),
+ Result = error("input not matched by any regexp", Start),
Instance = ((( Instance0
- ^ lexi_live_lexemes :=
- Instance0 ^ lexi_init_lexemes)
- ^ lexi_current_winner := no)
- ^ lexi_buf_state := buf__commit(BufState1)).
+ ^ live_lexemes :=
+ Instance0 ^ init_lexemes)
+ ^ current_winner := no)
+ ^ buf_state := buf__commit(BufState1)).
%----------------------------------------------------------------------------- %
-:- pred process_eof(lexer_result(Tok),
+:- pred process_eof(io__read_result(Tok),
lexer_instance(Tok, Src), lexer_instance(Tok, Src),
buf_state(Src), buf).
:- mode process_eof(out, in(lexer_instance), out(lexer_instance),
@@ -346,44 +407,63 @@
( if
- live_lexeme_in_accepting_state(Instance0 ^ lexi_live_lexemes, ATok)
+ live_lexeme_in_accepting_state(Instance0 ^ live_lexemes, TokenCreator)
then
% Return the token and set things up so that we return
% eof next.
- (
- ATok = noval(Token),
- Result = ok(Token)
- ;
- ATok = value(Token),
- Result = ok(Token, buf__string_to_cursor(BufState, Buf))
- ;
- ATok = ignore,
- Result = eof
- )
+ %
+ (
+ if
+
+ get_token_from_buffer(BufState, Buf, Instance0,
+ TokenCreator, Token)
+ then
+
+ Result = ok(Token)
+
+ else
+ Result = eof
+ )
+
else
Result = eof
),
Instance = ((Instance0
- ^ lexi_live_lexemes := [])
- ^ lexi_buf_state := buf__commit(BufState)).
+ ^ live_lexemes := [])
+ ^ buf_state := buf__commit(BufState)).
+
+%----------------------------------------------------------------------------- %
+
+:- pred get_token_from_buffer(buf_state(Src), buf, lexer_instance(Tok, Src),
+ token_creator(Tok), Tok).
+:- mode get_token_from_buffer(in(buf_state), array_ui, in(lexer_instance),
+ in(token_creator), out) is semidet.
+
+get_token_from_buffer(BufState, Buf, Instance, TokenCreator, Token) :-
+ Match = buf__string_to_cursor(BufState, Buf),
+ Token = TokenCreator(Match),
+ IgnorePred = Instance ^ ignore_pred,
+ not IgnorePred(Token).
%----------------------------------------------------------------------------- %
:- pred advance_live_lexemes(char, offset,
list(live_lexeme(Token)), list(live_lexeme(Token)),
winner(Token), winner(Token)).
-:- mode advance_live_lexemes(in, in, in, out, in, out) is det.
+:- mode advance_live_lexemes(in, in, in(live_lexeme_list),
+ out(live_lexeme_list),
+ in(winner), out(winner)) is det.
advance_live_lexemes(_Char, _Offset, [], [], Winner, Winner).
advance_live_lexemes(Char, Offset, [L0 | Ls0], Ls, Winner0, Winner) :-
- State0 = L0 ^ clxm_state,
- ATok = L0 ^ clxm_token,
+ State0 = L0 ^ state,
+ ATok = L0 ^ token,
( if next_state(L0, State0, Char, State, IsAccepting) then
@@ -395,7 +475,7 @@
Winner1 = yes(ATok - Offset)
),
advance_live_lexemes(Char, Offset, Ls0, Ls1, Winner1, Winner),
- Ls = [( L0 ^ clxm_state := State ) | Ls1]
+ Ls = [( L0 ^ state := State ) | Ls1]
else
@@ -405,15 +485,17 @@
%----------------------------------------------------------------------------- %
:- pred live_lexeme_in_accepting_state(list(live_lexeme(Tok)),
- annotated_token(Tok)).
-:- mode live_lexeme_in_accepting_state(in, out) is semidet.
+ token_creator(Tok)).
+:- mode live_lexeme_in_accepting_state(in(live_lexeme_list),
+ out(token_creator)) is semidet.
live_lexeme_in_accepting_state([L | Ls], Token) :-
( if in_accepting_state(L)
- then Token = L ^ clxm_token
+ then Token = L ^ token
else live_lexeme_in_accepting_state(Ls, Token)
).
+
%----------------------------------------------------------------------------- %
%----------------------------------------------------------------------------- %
@@ -423,10 +505,10 @@
:- type lexer_state(Tok, Src)
---> lexer_state(
- lxr_instance :: lexer_instance(Tok, Src),
- lxr_buf :: buf,
- lxr_src :: Src
- ).
+ run :: lexer_instance(Tok, Src),
+ buf :: buf,
+ src :: Src
+ ).
%------------------------------------------------------------------------------%
@@ -471,17 +553,50 @@
Result = eof
).
+
%----------------------------------------------------------------------------- %
-% Some basic non-primitive regexps.
+% The type of regular expressions.
+:- type regexp
+ ---> eps % The empty regexp
+ ; atom(char) % Match a single char
+ ; conc(regexp,regexp) % Concatenation
+ ; alt(regexp, regexp) % Alternation
+ ; star(regexp) % Kleene closure
+ .
+
+%----------------------------------------------------------------------------- %
+
+% Some instances of typeclass regexp(T)
+:- instance regexp(regexp) where [
+ re(RE) = RE
+].
+
+:- instance regexp(char) where [
+ re(C) = atom(C)
+].
-str(S) = R :-
+:- instance regexp(string) where [
+ re(S) = R :-
( if S = "" then
R = null
else
L = string__length(S),
C = string__index_det(S, L - 1),
- R = str_foldr(func(Cx, Rx) = (atom(Cx) >> Rx), S, atom(C), L - 2)
- ).
+ R = str_foldr(func(Cx, Rx) = (Cx ++ Rx), S, re(C), L - 2)
+ )
+].
+
+
+%----------------------------------------------------------------------------- %
+% Basic primitive regexps.
+null = eps.
+R1 ++ R2 = conc(re(R1), re(R2)).
+R1 \/ R2 = alt(re(R1), re(R2)).
+(R1 or R2) = alt(re(R1), re(R2)).
+*(R1) = star(re(R1)).
+
+%----------------------------------------------------------------------------- %
+% Some basic non-primitive regexps.
any(S) = R :-
( if S = "" then
@@ -489,7 +604,7 @@
else
L = string__length(S),
C = string__index_det(S, L - 1),
- R = str_foldr(func(Cx, Rx) = (atom(Cx) \/ Rx), S, atom(C), L - 2)
+ R = str_foldr(func(Cx, Rx) = (Cx \/ Rx), S, re(C), L - 2)
).
anybut(S0) = R :-
@@ -511,9 +626,9 @@
else str_foldr(Fn, S, Fn(string__index_det(S, I), X), I - 1)
).
-opt(R) = (R \/ null).
+?(R) = (R \/ null).
-plus(R) = (R >> star(R)).
++(R) = (R ++ *(R)).
%----------------------------------------------------------------------------- %
% Some useful single-char regexps.
@@ -535,24 +650,24 @@
dot = anybut("\n").
alpha = (lower \/ upper).
alphanum = (alpha \/ digit).
-identstart = (alpha \/ atom('_')).
-ident = (alphanum \/ atom('_')).
-nl = atom('\n').
-tab = atom('\t').
-spc = atom(' ').
+identstart = (alpha \/ ('_')).
+ident = (alphanum \/ ('_')).
+nl = re('\n').
+tab = re('\t').
+spc = re(' ').
%----------------------------------------------------------------------------- %
% Some useful compound regexps.
-nat = plus(digit).
-signed_int = (opt(any("+-")) >> nat).
-real = (signed_int >> (
- (atom('.') >> nat >> opt(any("eE") >> signed_int)) \/
- (any("eE") >> signed_int)
- )).
-identifier = (identstart >> star(ident)).
-whitespace = star(wspc).
-junk = star(dot).
+nat = +(digit).
+signed_int = ?("+" or "-") ++ nat.
+real = signed_int ++ (
+ ("." ++ nat ++ ?(("e" or "E") ++ signed_int)) or
+ ( ("e" or "E") ++ signed_int)
+ ).
+identifier = (identstart ++ *(ident)).
+whitespace = *(wspc).
+junk = *(dot).
%----------------------------------------------------------------------------- %
%----------------------------------------------------------------------------- %
Index: lex.regexp.m
===================================================================
RCS file: /var/cvs/rat/rationality/logic/mercury/extras/lex/lex.regexp.m,v
retrieving revision 1.1.1.1
retrieving revision 1.6
diff -u -r1.1.1.1 -r1.6
--- lex.regexp.m 2001/07/13 09:33:56 1.1.1.1
+++ lex.regexp.m 2001/08/14 18:16:42 1.6
@@ -1,15 +1,20 @@
%----------------------------------------------------------------------------- %
-% lex.regexp.m
-% Copyright (C) 2001 Ralph Becket <rbeck at microsoft.com>
-% Fri Aug 18 06:43:09 BST 2000
% vim: ts=4 sw=4 et tw=0 wm=0 ff=unix
-%
-% Converts basic regular expressions into non-deterministic finite
-% automata (NFAs).
%
+% lex.regexp.m
+% Fri Aug 18 06:43:09 BST 2000
+% Copyright (C) 2001 Ralph Becket <rbeck at microsoft.com>
% THIS FILE IS HEREBY CONTRIBUTED TO THE MERCURY PROJECT TO
% BE RELEASED UNDER WHATEVER LICENCE IS DEEMED APPROPRIATE
% BY THE ADMINISTRATORS OF THE MERCURY PROJECT.
+% Thu Jul 26 07:45:47 UTC 2001
+% Copyright (C) 2001 The Rationalizer Intelligent Software AG
+% The changes made by Rationalizer are contributed under the terms
+% of the GNU Lesser General Public License, see the file COPYING.LGPL
+% in this directory.
+%
+% Converts basic regular expressions into non-deterministic finite
+% automata (NFAs).
%
%----------------------------------------------------------------------------- %
@@ -53,16 +58,16 @@
% The primitive regexps.
-compile(X, null, Y, [null(X, Y)]) --> [].
+compile(X, eps, Y, [null(X, Y)]) --> [].
compile(X, atom(C), Y, [trans(X, C, Y)]) --> [].
-compile(X, (RA >> RB), Y, TsA ++ TsB) -->
+compile(X, conc(RA,RB), Y, TsA ++ TsB) -->
counter__allocate(Z),
compile(X, RA, Z, TsA),
compile(Z, RB, Y, TsB).
-compile(X, (RA \/ RB), Y, TsA ++ TsB) -->
+compile(X, alt(RA, RB), Y, TsA ++ TsB) -->
compile(X, RA, Y, TsA),
compile(X, RB, Y, TsB).
Index: samples/demo.m
===================================================================
RCS file: /var/cvs/rat/rationality/logic/mercury/extras/lex/samples/demo.m,v
retrieving revision 1.1.1.1
retrieving revision 1.8
diff -u -r1.1.1.1 -r1.8
--- samples/demo.m 2001/07/13 09:33:56 1.1.1.1
+++ samples/demo.m 2001/08/14 18:16:48 1.8
@@ -1,13 +1,18 @@
%----------------------------------------------------------------------------- %
% demo.m
-% Copyright (C) 2001 Ralph Becket <rbeck at microsoft.com>
% Sun Aug 20 18:11:42 BST 2000
-% vim: ts=4 sw=4 et tw=0 wm=0 ff=unix ft=mercury
-%
+% Copyright (C) 2001 Ralph Becket <rbeck at microsoft.com>
% THIS FILE IS HEREBY CONTRIBUTED TO THE MERCURY PROJECT TO
% BE RELEASED UNDER WHATEVER LICENCE IS DEEMED APPROPRIATE
% BY THE ADMINISTRATORS OF THE MERCURY PROJECT.
+% Thu Jul 26 07:45:47 UTC 2001
+% Copyright (C) 2001 The Rationalizer Intelligent Software AG
+% The changes made by Rationalizer are contributed under the terms
+% of the GNU General Public License - see the file COPYING in the
+% Mercury Distribution.
%
+% vim: ts=4 sw=4 et tw=0 wm=0 ff=unix ft=mercury
+%
%----------------------------------------------------------------------------- %
:- module demo.
@@ -23,7 +28,7 @@
:- implementation.
-:- import_module char, string, exception, array, list.
+:- import_module char, string, exception, array, list, std_util, int.
:- import_module lex.
%----------------------------------------------------------------------------- %
@@ -40,7 +45,7 @@
"),
- { Lexer = lex__init(lexemes, lex__read_from_stdin) },
+ { Lexer = lex__init(lexemes, lex__read_from_stdin, ignore(space)) },
call((pred(IO0::di, IO::uo) is det :-
State0 = lex__start(Lexer, IO0),
tokenise_stdin(State0, State),
@@ -66,48 +71,50 @@
%----------------------------------------------------------------------------- %
:- type token
- ---> noun
- ; comment
- ; integer
- ; real
- ; verb
- ; conj
- ; prep
+ ---> noun(string)
+ ; comment(string)
+ ; integer(int)
+ ; real(float)
+ ; verb(string)
+ ; conj(string)
+ ; prep(string)
; punc
+ ; space
.
-:- func lexemes = list(annotated_lexeme(token)).
+:- func lexemes = list(lexeme(token)).
lexemes = [
- lexeme(value(comment),
- (atom('%') >> junk)),
-
- lexeme(value(integer),
- (signed_int)),
-
- lexeme(value(real),
- (real)),
-
- lexeme(value(noun), str("cat")),
- lexeme(value(noun), str("dog")),
- lexeme(value(noun), str("rat")),
- lexeme(value(noun), str("mat")),
-
- lexeme(value(verb),
- (str("sat") \/ str("caught") \/ str("chased"))),
-
- lexeme(value(conj),
- (str("and") \/ str("then"))),
+ ( "%" ++ junk -> (func(Match) = comment(Match)) ),
+ ( signed_int -> (func(Match) = integer(string__det_to_int(Match))) ),
+ ( real -> (func(Match) = real(det_string_to_float(Match))) ),
+ ( "cat" -> (func(Match) = noun(Match)) ),
+ ( "dog" -> (func(Match) = noun(Match)) ),
+ ( "rat" -> (func(Match) = noun(Match)) ),
+ ( "mat" -> (func(Match) = noun(Match)) ),
+ % Here we use `or'
+ ( "sat" or "caught" or "chased" ->
+ (func(Match) = verb(Match)) ),
+ ( "and" or "then" ->
+ (func(Match) = conj(Match)) ),
+ % Now we use `\/', it's the same as `or'. We would like to
+ % know from you, which one looks nicer.
+ ( "the" \/ "it" \/ "them" \/ "to" \/ "on" ->
+ (func(Match) = prep(Match)) ),
+ ( any("~!@#$%^&*()_+`-={}|[]\\:"";'<>?,./") ->
+ return(punc) ),
+ ( whitespace -> return(space) )
+].
- lexeme(value(prep),
- (str("the") \/ str("it") \/ str("them") \/ str("to") \/ str("on"))),
+:- func det_string_to_float(string) = float.
- lexeme(noval(punc),
- (any("~!@#$%^&*()_+`-={}|[]\\:"";'<>?,./"))),
+:- import_module require.
- lexeme(ignore,
- whitespace)
-].
+det_string_to_float(String) = Float :-
+ ( if string__to_float(String,Float0)
+ then Float = Float0
+ else error("Floating point number overflow")
+ ).
%----------------------------------------------------------------------------- %
%----------------------------------------------------------------------------- %
More information about the developers
mailing list