[m-dev.] Determining which grades an installed Mercury compiler supports

[Keri Harris]

On 18/07/2018 20:53, Zoltan Somogyi wrote:
> 
> 
> On Wed, 18 Jul 2018 18:28:48 +0200, Keri Harris <keri at gentoo.org> wrote:
>> I've run into a couple of interesting issues in how Mercury determines
>> which grades it supports.
> 
> The Mercury compiler can generate code for any of its target languages
> without any outside help, so I presume you are talking about how
> it determines which grades have their libraries installed in the locations
> in which it expects, i.e. how it figures what to report when given
> --output-libgrades.

Yes, that's correct.

>> The erlang grade can have problems with this approach since init files
>> can be installed into the mercury/module/erlang directory. In
>> particular, a problem can arise when a 3rd-party library installs an
>> erlang init file, and then Mercury is later reinstalled without support
>> for the erlang grade; these residual 3rd-party init files can confuse
>> Mercury into believing that it still does support the erlang grade. The
>> following sequence of events leads to this:
>>
>> * compile and install Mercury package such that grade erlang is installed
>>     - /usr/lib/mercury/modules/erlang/mer_std.init is created
>>
>> * compile and install mercury_fixed library from mercury/extras/fixed
>>     - /usr/lib/mercury/modules/erlang/mercury_fixed.init is created
>>
>> * recompile and install Mercury such that grade erlang is no longer
>> installed
>>     - /usr/lib/mercury/modules/erlang/ still exists
> 
> Why does the package manager allow the installation of something that is
> *already* installed, without *uninstalling* first?
> 
>  From what I can see, if one installation overwrites another, the results
> are inherently fragile. Mercury installs are tested *only* in the context of a fresh,
> clean install; using any other kind of install will work only by luck. One could
> get around this by expending specific engineering effort to design (and later test)
> an install scheme that could survive specific kinds of partial installs, but
> in Mercury's case (and in most other cases) I don't think this would be
> worthwhile.

Package managers do uninstall old instances of packages whenever a 
package is upgraded/reinstalled. Paradoxically the uninstall phase 
always of an old package occurs *after* the install phase of the new 
package.

Gentoo Linux is a bit particular in that it is a source distribution, so 
in addition to installing the package it must first compile the package. 
Installing mercury on Gentoo Linux involves the following steps:

1. compile mercury in a sandbox. If anything during the compile attempts 
to write anywhere outside the sandbox, the compile is aborted.

2. install mercury in a sandbox. The install uses DESTDIR so that files 
are installed into a safe location still within the sandbox. If the 
install attempts to write anywhere outside the sandbox, the compile is 
aborted.

(At this stage the old instance of mercury still resides on the live 
filesystem and is still fully functional; the new instance of mercury 
now resides at DESTDIR in the sandbox).

3. package manager copies mercury installation from sandbox to live 
filesystem.

(At this stage the live filesystem contains a mix of both old and new 
versions. Many files from the old instance will have been overwritten by 
copies from the new instance; some cruft from the old instance may still 
remain).

4. package manager deletes any stray files from old instance of package 
that have not been overwritten by newer versions. All files within the 
sandbox are then deleted.

(At this stage just the new instance of mercury exists on the live 
filesystem; all files from the old instance have either been overwritten 
in step #3 or deleted in step #4).

Package managers that are not source based (rpm, dpkg etc) essentially 
just run steps #2, #3 and #4 where they extract files into a sandbox, 
copy them to the live filesystem, and then clean up old files.

Installing the new instance of a package before uninstalling the old 
instance is done for a number of reasons:

* there is (obviously) a gap between uninstalling a package and then 
reinstalling that package. During this gap files from the package no 
longer exist on the filesystem. If you are upgrading an important 
package (e.g some system package like coreutils) then uninstalling the 
package before the new version is installed will likely leave the system 
in a non-functional state.

* if steps #1 or #2 of a package upgrade fail, you still have the old 
version of the package in a functional state. (This is particularly 
important for source-based distributions since compile-time failures are 
possible).

So in the case of upgrading mercury via a package manager (or possibly 
just reinstalling mercury to change the supported std library grades) 
you're never left with non-directory files from an old instance of 
mercury. Thus an install of mercury using a package manager gives you 
the same result as if you had performed a clean install. (And behind the 
scenes you actually are performing a clean install - into a safe 
location in a sandbox).

The problem with the erlang grade is simply that when mercury is 
reinstalled without the erlang grade, the package manager may decide not 
to delete the mercury/modules/erlang directory since other packages 
(e.g. mercury-extras) have placed their own files into this directory. 
And since the mere existence of mercury/modules/erlang currently implies 
that the erlang std library is installed we can eventually get into 
trouble when compiling Mercury code.

I'm in agreement that Mercury shouldn't be expected to re-engineer 
anything in order to follow some special install scheme. Handling cruft 
from old installs is already part of the package manager's 
responsibility. As an aside, the existing Mercury build system is very 
good. Two strengths in particular stand out:

1. it uses sane defaults for things (CFLAGS, install locations etc)
2. so many things can be overridden via environment variables or 
mmc/mmake arguments. (This is important since each OS or distribution 
tends to have their own special policies that need to be followed).

>> I can work around the empty-directory problem easily enough by adding
>> zero-byte files into the empty modules/$grade directories when the
>> mercury compiler is installed. Then uninstalling any other package that
>> also owns the modules/$grade directories won't result in these
>> directories being deleted.
> 
> That would fix this particular instance of this problem, but not other instances.
> I don't think anything would be guaranted to work short of
> 
> - checking that every single file that is supposed to be installed is present,
>     and has the expected cryptographic hash, and
> - checking that *only* the expected files are present in the install directories.

Package managers do keep track of which installed packages own which 
files, and package managers prevent different packages from attempting 
to install the same (non-directory) file. So upgrading or reinstalling 
packages tends to go remarkably smoothly. Handling empty directories on 
package uninstall is a corner case that some package managers have 
trouble with.

>> Strictly speaking, these issues are more related to bundling Mercury
>> with package manager than being inherent to Mercury itself. But both of
>> these issues make me wonder if there is a more resilient way of
>> determining which grades the compiler supports.
> 
> We could bake into the compiler the list of the grades that were configured
> to be installed at the time the compiler executable was itself created.
> However, that would still be vulnerable to parts of the install directory
> being overwritten later.

That would certainly work. Another option would be to follow the same 
logic as that used for C grades - look for a uniquely identifiable file 
belonging to the std library for each non-C grade.


Thanks

Keri