packages proposal

[Fergus Henderson]

On 11-Feb-1998, Tyson Dowd <trd at stimpy.cs.mu.oz.au> wrote:
> 
> I'm a little unsure about the intended relationship between packages
> and programs. What if a package includes a module that defines main?
> What is a program imports a package that defines main? I think I know
> what should happen, but your definition by analogy doesn't really
> make it clear.

Previous a program was a set of modules, and the rule was that only
one module could define a predicate called "main" with arity 2.

This proposal would make a program a set of packages and/or modules,
with the rule still being that only one module (including modules
contained within packages) can define a predicate called "main" with
arity two.

> Also, you're not very clear about the meaning of `:- implementation'
> in pacakges.

Any modules included in the implementation section of a package
are private to that package; they may be used or imported only by
other modules in the same package.

> Should you have to include the dependencies in the `:- implementation'?

What do you mean by the package's dependencies -- do you mean the packages
which this package depends on, or do you mean the modules which are
included in this package?  Your question below makes sense for both:

> Why can't they be generated for you like they are for programs?

There are several reasons why the modules which comprise a package must
be explicitly declared.  One reason is that you need to declare whether
they are public or private (by including them in either the interface
or the implementation).  Another reason is readability: this information
is good documentation.  But the biggest reason is just that while
for a program you can just name the module containing main/2
and the program can be implicitly defined as that module plus
the set of modules and packages that are used, directly or indirectly,
by that module, whereas for a package there is no simple equivalent.

One possibility would be that for a package you could just name a
directory and then the package would be implicitly defined as all
the modules in that directory.  Private modules are relatively uncommon,
so we could default to making them all public.
To avoid losing the readability advantages of an explicit package
declaration, the implementation could automatically generate the package file.

Allowing implicit package definitions in this manner would provide a small
increase in convenience and maintainability in the common case where
a package contains no private modules.

I suppose even the private module case could be handled by
allowing

	:- private module foo.
	...
	:- end_module.

instead of

	:- module foo.
	...
	:- end_module.

but that would be a bit non-orthogonal.  For preds, types, etc.,
Mercury uses the location of a declaration (whether it is in the
interface section or the implementation section) to specify whether it
is public or private, rather than using `:- private ...' on the
declaration.  So if we want implicit package definitions even for
packages containing private modules, then the orthogonal solution would
be to somehow use the location in the filesystem to specify whether a
module is public or private -- for example, a package might correspond
to a directory containing a pair of subdirectories named `interface' and
`implementation'.  However, this is starting to get a bit complicated.

Oh, this brings to mind another minor point: for the modules included
in the interface of a package, only the interfaces of those modules
are part of the package's interface; the implementations are part
of the package's implementation.  So arguably it might be more
consistent to use something like

	:- package foo.
	:- interface.
	:- include_module_interface m1, m2, m3.
	:- implementation.
	:- include_module_implementation m1, m2, m3.
	:- include_module m4, m5.
	:- end_package foo.

instead of

	:- package foo.
	:- interface.
	:- include_module m1, m2, m3.
	:- implementation.
	:- include_module m4, m5.
	:- end_package foo.

However, the additional complexity of `include_module_interface'
and `include_module_implementation' would outweight the advantages
of the simpler/more consistent definition of what constitutes
a package's interface, so I'm not seriously proposing this.
Instead, we just say that if `include_module' occurs in the
interface section of a package, then the interface of that
package includes the interface of that module
and the implementation of the package includes the
implementation of that module.

> > 	An `:- import_package foo' declaration declares that this
> > 	module or package makes use of modules defined in package
> > 	`foo'.  Implementations could use this information to make
> > 	it easier to use packages.  For example, for our current
> > 	implementation, we could modify `mmc --generate-dependencies'
> > 	to check for `import_package' declarations and would spit out
> > 	appropropriate things in the `.dep' file which Mmake would use
> > 	to include appropriate options when invoking `ml'. 
> > 
> > 	The Mercury implementation could encourage the use of
> > 	standard paths for a package based on the package name,
> > 	just as the current implementation strongly encourages the use
> > 	of the same module name and file name.
> > 	Command-line options would specify the directory
> 
> Options for what command? Should mmc search the libraries, or should it
> just know about the -I option, and get mmake (or --generate
> dependencies) to do the work?

mmc would get two new options, namely `--in-package'
(or perhaps just `--package') and `--use-package'.

Compiling a module with `--use-package bar' would be the same
as inserting `:- use_package bar' at the top of the module.
If a package definition has a `:- use_package' declaration,
then `mmc --generate-dependencies' and `mmake' would ensure
that mmc is invoked with the appropriate `--use-package' option
when compiling modules in that package.

`--in-package' specifies which package a module is part of.
`mmc --generate-dependencies' and `mmake' would ensure
that modules in a package are compiled with the appropriate
`--in-package' option.

The effect of these two options is to allow proper scoping
for package names, which still preserving separate compilation
of individual modules.  For example, you can write

	:- package foo.
	:- import_package bar.
	:- include_module m1, m2.

and then if you use Mmake (or if do it manually by invoking mmc
with `--in-package foo --use-package bar') to compile module `m1',
then inside modules `m1' and `m2', all definitions are implicitly
package-qualified with `foo.', and the `:- import_package bar'
declaration scopes over `m1' and `m2', so there's no need
to duplicate the `use_package' declarations in each module.

In addition, mmc --generate-depencies and mmake would collaborate to
add a corresponding `-l' option to the link line for each `import_package'
declaration.

OK, that solves everything except the directories and search paths.

There's still a few details here which I haven't nailed down.
In particular, do we want the `.int's, `.a's, `.so's, and `.init's
for a package to all be in the same directory, or do we want them do
be in separate subdirectories (e.g. `ints', `libs/<FULLARCH>', and `modules')?
Do we want to allow both possibilities?

So following is still quite tentative.  But, since you asked...

In addition to the above-mentioned options,
mmc should get a new option `-P' (`--package-path'),
or perhaps two similar options, for the two possibilities (separate
ints, libs, modules subdirectories, or everything in one directory).
Each `-P <dir>' option would specify a directory to search for packages.
`-I' would still be used for modules which are not part of a package.
The search path used to look for module interfaces would include
<dir>/<package-name> and/or <dir>/ints/<package-name>
for each <dir> in the package-path and for each <package-name>
for which there was an `import_package' declaration,
plus the directories specified by `-I'.

Or perhaps we should have a `--package-dir <package>=<dir>' option,
which would specify the directory for a given package.
For example `--package-dir std=/usr/local/mercury-0.8/lib/mercury'.
We could have both `--package-dir' and `--package-install-dir'
options to cater for the two cases (single directory or subdirs).

Then again, maybe we should change the current build environment
so that it builds interfaces in ./ints, objects in ./objs/<FULLARCH>,
libs in ./libs/<FULLARCH>, etc., to (a) avoid very big directories
with lots of clutter, (b) support compiling for different architectures
in the same directory, and (c) match the structure for installed
packages, so that we need only one of `--package-dir' and
`--package-install-dir'.

We should definitely solve the problem of too many files in one
directory, but I'm not sure whether it would be best to use a separate
subdirectory for each different type of file (i.e. `ints' contains
`foo.int' and `bar.int' while `objs' contains foo.' and `bar.o'),
or whether we should use a separate subdirectory for each module
(i.e. `foo.dir' contains `foo.int' and `foo.o', while `bar.dir'
contains `bar.int' and `bar.o').

> Are packages more of a language feature or a build environment feature?
> What is the division of responsibiliy?

Packages are a language feature.
The mapping from packages and modules to files and directories is
a build environment feature.
Whether or not you have to write the package definition explicitly
or whether the implementation will create it for you is a build
environment feature.

> > Because this proposal does not allow modules to contain other modules,
> > and because packages can contain only other modules, not types or
> > insts, etc., this proposal avoids much of the complexity that would
> > be associated with having.
> 
> Having ... lunch? A baby? Infinite loops? Nested modules?

Nested modules ;-)

> > One thing that this proposal does not address is meta-packages
> > (packages of packages) and/or hierarchical partitioning of the
> > package namespace (allowing `.' in package names).
> > 
> > I think meta-packages would probably be a bad idea (too much
> > complexity for too little gain), but I think you can probably
> > get most of the same effect using a hierarchical partitioning
> > of the package namespace.
> 
> I don't think meta-packages are worth implementing right now (or
> in the near future) but it might be worth leaving the syntax or
> implementation open. 
> 
> (if only because I'm sure they once said the same thing about
> subdirectories).

I thought about it a bit more, and it's actually very easy.
Anytime you import a module from a package, you need to look
up the package definition to check whether the module is private.
Meta-packages are the same; if a package name is of the form
`foo.bar', then when using or importing that package, the only
additional thing the compiler needs to do is to look up the
package definition for package `foo' and check whether its
subpackage `bar' is private.

-- 
Fergus Henderson <fjh at cs.mu.oz.au>   |  "I have always known that the pursuit
WWW: <http://www.cs.mu.oz.au/~fjh>   |  of excellence is a lethal habit"
PGP: finger fjh at 128.250.37.3         |     -- the last words of T. S. Garp.