[m-dev.] 0.13 release: op/3 syntax

[Mark Brown]

Hi Doug,

First of all, thanks for taking the time to contribute to the Mercury
implementation.  We always try to write our code in a way that makes it
possible for people outside the group to read and modify it, so we're
pleased to see that you're taking that opportunity.

I hope you don't get too discouraged by our objections to your change.
For the record, we're applying the same standards that we apply to
ourselves, and before each release we do indeed tend to have a few
in-house battles over what new features should be allowed in before the
branch and what should wait until after.

On 01-Mar-2006, doug.auclair at logicaltypes.com <doug.auclair at logicaltypes.com> wrote:
> Dear Ralph, thank you for your reply.  You wrote:
> 
> >We've discussed your op/3 extension in the office and we have some
> >reservations that make us reluctant to add it to the repository, at
> >least for 0.13.
> 
> If your reservations are unshakeable, then, there it is.

They are quite firm.  But that doesn't release us from the obligation to
justify them, of course.  And since you've put quite a lot of effort into
this, I think we owe it to you.

> Otherwise,
> if your objections have reasonable, acceptable, answers, then please
> consider op/3 declarations once more.  To wit:
> 
> >The main objections are:
> >- lack of demand (just one request, so far!), coupled with unfortunate
> >   experiences using Prolog code in the past that has used op/3;
> 
> These are two objections.  The first, lack of demand, is already
> answered (yes, I'm the only one calling for this addition, but that
> does not mean that others would find it useful. Micheal at
> yeslogic.com may have wished to use op/3 declarations, but, not having
> the ability pushed forward using what he had available -- just a
> conjecture.  If it helps, I can have two other companies request op/3
> declaration functionality for Mercury ... would that make it a
> majority of commercial users requesting this feature?); but, I walked
> through at least two revisions of dead code in the compiler attempting
> to add op-like declaration syntax, so, actually most of my revisions
> will be helpful in that it cuts away this dead code.  This code points
> to a need being addressed by the Mercury team.

I'm not sure exactly which existing code you are referring to, but it's
probably only there for historical reasons.  As you may be aware, Mercury
started life by being bootstrapped from Prolog, which explains to a large
extent why it uses Prolog syntax.  Any support for op/3 is quite likely
to be left over from that era.

What I _can_ say is that the idea of providing op/3 in Mercury was
abandoned a long time ago.  There was a majority view (not unanimous,
though) of people in the Mercury group that user defined operators almost
always make code harder to read for anyone other than the person who wrote
it.  This would particularly be the case with op/3's relationship with the
module system -- effectively each file has its own syntax, so in a large
project (e.g., hundreds or thousands of files) that's either a lot of
different syntaxes to understand, or else a lot of duplicated op/3
declarations.

We decided long ago that backticks (e.g., `op`) had none of the disadvantages
of user defined operators, most of the advantages, and were almost as concise.

> I had that very need,
> so I completed the implementation.
> 
> The bad experience from Prolog using op/3 was what?  If it was caused by
> the declaration's global extent, then my module-only extent implementation
> solves this problem.

The global extent was only part of the problem.  The main difficulty is
the one I mention above, and the one Ralph alludes to below: to understand
someone's code you first have to parse it in your mind, and to do that you
need to know exactly which user-defined operators apply to the file you are
looking at.

Note that if you use vim or emacs tags (which I find to be indispensable
when looking at compiler code) then you will commonly jump around from
one file to another.  You needn't even be aware of exactly which file
you are looking at.

> This ties into the next objection.
> 
> >- lack of orthogonality and code maintenance problems, since the same
> >   syntax in different modules now has to be carefully checked to see
> >   that it parses the same way (e.g., if the modules have differing op
> >   declarations for the same symbol); and
> 
> As this implementation of op/3 is module-only extent this "problem" is 
> permitted, but only insofar as the operator is not defined already in
> the operator syntax from Mercury itself (i.e. an op/3 declaration CANNOT
> override syntax defined by Mercury (for example, syntax already defined
> by library/ops.m)).

I was going to ask about this, because the documentation you posted earlier
didn't make that clear (unless I missed something).  From briefly looking
at your code, it seems that it won't report a meaningful error message
if this rule is violated, or if you try to redefine an operator you have
previously defined.  This definitely needs to be addressed: the compiler
throwing an exception is always considered a compiler bug.

> This op/3 implementation was intentionally designed
> this way -- we have modules that define a set of syntax to address a certain
> problem set, but in another module, dealing with another problem set, we use
> some of the same operators differently because the semantics of the problem
> itself is different.

Personally, I'd hate to have to understand such code.  Having two predicates
with the same name but different semantics is bad enough.  Having two
operators with the same name but different precedence is truly diabolical,
IMHO.

> In Prolog, depending on the implementation, this
> causes issues, even though the problems are clearly separated across
> modules.  This implementation of op/3 declarations for Mercury
> addresses that problem in a safe manner.

Well, I think you've done a better job than Prolog.  But I'm not convinced
by your assertion that the scope of an op/3 declaration should be from its
position until the end of the file.  Assuming I am coerced into accepting
the idea that the same operator may have different precedence in different
files, I still think that others who use this feature would expect a more
comprehensive interaction with the module system.  For example, that
operators would be able to be exported (but when imported they should only
affect the code in the module they are imported into, not the code imported
from other modules as Prolog does).

Another reason why we have never proceeded with user defined operators
is because, to my knowledge, no one has addressed the interactions with
the module system adequately.

> 
> >- we are planning on moving the compiler to use a new yacc-style parser
> >   in the not-too- distant future and it's very unclear how we'd make
> >   that change if we also support dynamic syntax changes, a la op/3.
> 
> Hm! So, does this mean that library/ops.m is deprecated?  That the
> declarations there will be ignored in future compiler releases?  That
> library/parser.m/read_term_(from_string_)with_op_table will become
> obsolete?
> 
> If so, then, yes, op/3 declarations could gum the works.  If not, then
> there may lie an misunderstanding of what the op/3 declarations do:
> the syntax /ADDITIONS/ (neither deletions nor alterations) to a /MODULE/
> (not globally) occur /AT COMPILE TIME/ (not 'dynamically').

What is compile-time for the code using op/3 is run-time for the parser
that parses that code.  I think this is what is usually meant by a
dynamic grammar, and what Ralph was referring to.

> The
> underlying internal representation of parsed op/3-enhanced syntax is
> equivalent (if not identical) to the internal representation of
> standard Mercury.  If library/ops.m remains in force after the planned
> parser change, then the changes to allow op/3-declarations should not
> be an issue.
> 
> -----
> 
> The above are my answers to the stated objections.  If not
> satifactory, then, of course, op/3 declarations are off the table for
> release 0.13.  But I urge otherwise for two reasons: 1) the dead code
> supporting hooks for a planned op/3-like declaration set are already
> gumming up the compiler works (since nobody else uses those hooks then
> replacing them with mine: first causes no harm -- primus non nocere,
> and second actually replaces void functionality with useful
> functionality),

As should be apparent by now, I don't agree that this feature does no harm.

> and 2) funding: once our contract gets turned 
> full on, we are going to need commercial support.  As Zoltan discussed
> with me, that company providing the support is probably going to be
> very closely affiliated with the Mercury team, which means, that since
> part of the commercial support for Logical Types will be the maintenance
> of the op/3 declaration functionality, two versions of Mercury must be
> maintained (one for Logical Types, one for all other users) which I can
> relate from personal experience is tedious at best.  Rolling in op/3
> eliminates the disconnect that will occur when assisting the
> supporting company address issues we raise.

An alternative approach to dealing with this issue is to implement a
preprocessor that exists entirely separately of the compiler.  It would
parse the module items exactly as your modified version of the parser does,
and then spit out items with all user defined operators in non-operator
form (and make liberal use of `:- pragma source_file' and `#line'
directives so that error messages are reported correctly).

This would only need to be updated when Mercury's syntax changes, which is
relatively rare.

> 
> So, back to my original request.  If my answers to the objections are
> satisfactory, please include op/3 in release 0.13.

Our release criteria, which we have been using successfully for many years,
include the following:

	- No language features get added to the repository until their
	  desirability is broadly accepted, the semantics is completely
	  and precisely defined, and their implementation is well tested
	  and passes peer review.

	- Significant language features must be added early in the release
	  cycle.  Not late in the release cycle, and definitely not just
	  before release.  Such changes can only be validated effectively
	  if they have been put to use for a reasonable period of time, and
	  it is important that this validation has happened prior to the
	  feature being made available in a numbered release. (*)

If these criteria aren't met, there is too much risk that the numbered
releases will be unstable and/or that we will be forced to make backwards
incompatible changes to the language.

We can continue the debate on the best design for this new feature, but it
doesn't meet our release criteria and won't be included in the 0.13 release.

Cheers,
Mark.

(*) I think it's worth pointing out that the change to add functional
dependencies was started just after the 0.12 branch was created.  This was
a feature that had strong support and was requested by a significant
number of people.  Between then and now they have been used in at least two
projects, and it is only recently, just before the imminent release, that
some language design issues have been raised.  As a result, fixing the
language issue is scheduled to be done before the 0.13 branch is created.

--------------------------------------------------------------------------
mercury-developers mailing list
Post messages to:       mercury-developers at cs.mu.oz.au
Administrative Queries: owner-mercury-developers at cs.mu.oz.au
Subscriptions:          mercury-developers-request at cs.mu.oz.au
--------------------------------------------------------------------------