[m-dev.] issues with installing libraries

[Peter Wang]

On Sat, 31 Aug 2024 02:32:10 +0200 "Zoltan Somogyi" <zoltan.somogyi at runbox.com> wrote:
> While I was working (very part-time) on the just-posted DIFF.mli4,
> much of my time was spent trying to understand the issues below.
> I would like to ask you guys to think about them too, and see what answers,
> if any, you can give to the questions below.
> 
> Zoltan.
> 
> Issue 1:
> 
>     We currently install .mh files, which are not grade-specific,
>     to *two* sets of non-grade-specific directories. These are
> 
>     set 1:   Prefix/lib/mercury/inc
>     set 2:   Prefix/lib/mercury/ints and Prefix/lib/mercury/ints/Mercury/mhs
> 
>     I believe that set 1 is intended for use by hand-written user code
>     that uses Mercury predicates and/or functions exported to C.
> 
>     QUESTION Does anyone know the purpose of set 2? What purpose does it serve
>     that set 1 cannot satisfy?
> 

When it comes to (seemingly) redundant copies of files, I always
remember something about satisfying the VPATH mechanism of make.

Is it this?

            % This is needed so that the file will be found in Mmake's VPATH.
            install_subdir_file(ProgressStream, Globals, NgsLibDirMap,
                LibDir / "ints", ModuleName, {ExtMh, "mhs"}, !Succeeded, !IO)

> Issue 2:
> 
>     We currently install .mih and .opt files, which are grade-specific,
>     into grade-specific directories. That is fine. However, we *also* install
>     them into non-grade-specific directories, which is ... not fine.
>     We do so in two separate ways.
> 
>     - For .opt files, we install them into a non-grade-specific directory
>       for the *current* grade only. This is the grade in which the installed
>       directory is compiled.
> 
>     - For .mih files, we install them into a non-grade-specific directory
>       for *all* libgrades, starting with the current grade. In the usual
>       case where the current grade is not the only libgrade, this means that
>       each install of a .mih file for a non-first libgrade in the
>       non-grade-specific directory will overwrite the install of
>       that same .mih file for the previous libgrade, leaving only the
>       .mih files of the last libgrade in that non-grade-specific directory.
> 
>     QUESTION Does anyone know of any reason why the install of these
>     grade-specific files in a non-grade-specific directory would be useful?
> 

I don't know.

>     QUESTION Does anyone recall any problem that you suspect *could* have
>     been caused by getting a .mih or .opt file *for the wrong grade*
>     from a non-grade-specific directory?
> 

I don't remember it being a problem, for the reasons you state.

>     For .mih files, I expect such problems to be quite rare, because
>     they exist only for MLDS grades targeting C, and I expect that for
>     most modules, their .mih files will be either identical or
>     mostly-identical for all such grades.

Right.

>     For .opt files, such problems may be somewhat more frequent, but
>     since probably the most frequent symptom of such problems is that
>     the compiler does not perform an optimization that it *should*
>     be able to do, it would be difficult to say for sure.

Right.

> Issue 3:
> 
>     We can install a library in multiple grades
> 
>     - either via mmake (the "lib%.install_grades" target
>       in scripts/Mmake.rules, which delegates part of its work
>       to two mmake rules in auto-gemerated .dep files),
> 
>     - or via "mmc --make lib%.install" (so to speak).
> 
>     These two routes differ in several ways.
> 
>     - The set of grades to install is determined differently by the
>       two routes. The mmake route install the librades specified by the
>       ALL_LIBGRADES make variable, whose value can be specified via
>       a library-specific make variable. On the other hand the set of grades
>       to install is the same for every library when using mmc --make;
>       it will be the set of libgrades found to be installed by
>       compiler/check_libgrades.m, if this is not overridden by the
>       --libgrade accumulating option.
> 
>     - The mmake route installs .trans_opt files (though the code that does
>       this also handles .int* files but NOT .opt files, which is strange),
>       while the mmc --make route never does.
> 
>     I believe there are two related root causes of this, and an unrelated
>     third root cause.
> 
>     Cause 1 is that there is no central documentation of what files
>     should be installed where, and more important, *why* are they installed
>     in those places. This would require the full documentation of the search
>     methods we use to find things in install directories. (See Issue 4.)
> 
>     Note; the EXT file I sent to m-rev recently documents what *does*
>     get installed where, which can be a starting point for the first half
>     of this documentation. However, I have not yet found all the places
>     that *set up* search paths, and I suspect that the most important ones
>     are not in the Mercury implementation at all, but in individual users'
>     Mmakefiles and/or Mercury.options files.
> 
>     Cause 2 is that the places in the system that deal with installing files
>     do not even have links to each other, so there is no reminder to people
>     working on one part (e.g. mmc --make) to update the corrresponding parts
>     (in e.g. Mmake.rules). I plan to add those links in a future diff.
> 
>     Cause 3 is that our requirements changed over time. When we worked
>     on termination analysis in the late 1990s, we installed .trans_opt files
>     to give users access to its results. By the time Simon implemented
>     the initial version of mmc --make in 2002, this became less important,
>     so we willingly tolerated the difference in the treatment of .trans_opt
>     files.
> 
>     QUESTION Ideally, both install routes should install the same files
>     in the same directories. In practice, making that happen will take
>     a significant amount of work. For each of the differences above,
>     can you please say whether you have been impacted by the difference.

I use mmake to install the Mercury system. Thereafter, I never install
any Mercury libraries using either mmake or mmc --make, so I have not
been impacted. It's just easier to build any Mercury modules (other than
the standard library) as part of the same application.

> Issue 4:
> 
>     One constraint on the structure of the install directory (and the
>     *main* constraint on the structure of its non-user-facing parts,
>     which make its bulk) are the compiler's search methods. Those methods
>     need to find the .int, .opt etc files of
> 
>     - the modules that are part of the current program, and
>     - the modules that are not part of the current program, and are instead
>       parts of external libraries.
> 
>     For modules that are not part of the current program, both mmake
>     and mmc --make will install all grade-specific files into grade-specific
>     directories, and all non-grade-specific files into non-grade-specific
>     directories (with some exceptions, such as Issue 2 above).
> 
>     For modules that are part of the current program, we cannot assume that;
>     with --no-use-grade-subdirs, grade-specific files may be in
>     non-grade-specific directories, whose names may, or may not,
>     have a suffix such as Mercury/int3s (for .int3 files).
> 
>     We currently handle such uncertainty in two different ways, depending
>     on whether the extension of the file want to search for contains "max"
>     in the names of its extension category in file_names.m.
> 
>     Consider searching for module_x.int3. The name of the extension category
>     for .int3 files does NOT contain "max". This means that e.g. with
>     --use-subdirs and --no-use-grade-subdirs, both module_name_to_file_name
>     and module_name_to_search_file_name will return Mercury/int3s/module_x.int3
>     when given module_x and .int3 as inputs. If the search_directories
>     option (which contains the accumulated strings specified for the
>     --search-directory option) contains ["sd1", "sd2", "sd3"], then our
>     search will look for
> 
>         sd1/Mercury/int3s/module_x.int3
>         sd2/Mercury/int3s/module_x.int3
>         sd3/Mercury/int3s/module_x.int3
> 
>     Consider searching for module_x.opt, whose extension is in the
>     ext_cur_ngs_gs_max_ngs category, which DOES contain "max".
>     This means that e.g. with --use-grade-subdirs, when given module_x
>     and .opt as inputs,
> 
>         module_name_to_file_name will return
>             Mercury/<grade>/<arch>/Mercury/opts/module_x.opt
> 
>         but module_name_to_search_file_name will return
>             Mercury/opts/module_x.opt
> 
>         so we would search
>             sd1/Mercury/opts/module_x.opt
>             sd2/Mercury/opts/module_x.opt
>             sd3/Mercury/opts/module_x.opt
> 
>     I think it is this search strategy that was the motivation
>     for the current setup of install directories in a way that causes
>     the second "Mercury" in the name of grade-specific directories.
>     Specifically, the correspondence between
> 
>     Mercury/<grade>/<arch>/Mercury/opts/module_x.opt and
>                            Mercury/opts/module_x.opt
> 
>     allows InstallDir/Mercury/<grade>/<arch>/Mercury/opts/module_x.opt
>     to be found by
>     
>     - BOTH a search for the full grade-specific name in InstallDir,
>     - AND a search for the non-grade-specific name, Mercury/opts/module_x.opt,
>       in InstallDir/<grade>/<arch>.
> 
>     QUESTION What do you guys think of this theory? Does anyone have an
>     alternative theory?

That's how I've understood it.

>     QUESTION Does anyone know, or have reason to believe, that there is
>     still a need to find .opt files using both kinds of searches?
>     If there is, or may be, such a need, would it go away if we implemented
>     one of the mechanisms in the next question?
> 
>     QUESTION Would a special option that says "here is opts at sdN,
>     add the directories
> 
>             sdN/Mercury/<grade>/<arch>/Mercury/opts
>             sdN/Mercury/opts
>             sdN
> 
>     to the search_directories accumulating string option" be sufficiently
>     useful to implement?
> 
>     How about an option that says "here is @sdN, add the directories
> 
>             sdN/Mercury/<grade>/<arch>/Mercury/ExtDir
>             sdN/Mercury/ExtDir
>             sdN
> 
>     to the search_directories accumulating string option, *but only when
>     searching for a file with an extension for which ext_to_dir_path
>     in compiler/file_names.m returns ExtDir*"?
> 
>     Both of these would work both on files in installed libraries *and*
>     on files in workspaces.

I haven't needed something like that (obviously, given the above).

> Issue 5:
> 
>     For .mih files, which are in the ext_cur_ngs_gs_max_cur,
>     the results would be
> 
>         module_name_to_file_name will return
>             Mercury/<grade>/<arch>/Mercury/mihs/module_x.mih
> 
>         but module_name_to_search_file_name will return
>             module_x.mih
> 
>         so we would search
>             cid1/module_x.mih
>             cid2/module_x.mih
>             cid3/module_x.mih
> 
>     The difference is that it is mmc that would search for .opt files,
>     but the search for .mih files is done by the C compiler, with the
>     directories to be searched (which are accumulated from -c-include-directory
>     options, hence cidN above). This is why e.g. compiler/COMP_FLAGS.in
>     specifies BOTH --c-include-directory ../library and --c-include-directory
>     ../library/Mercury/mihs, and would have to include the grade-specific
>     subdir name as well, if we ever wanted mmc to be built with mmc --make
>     and --use-grade-subdirs. I presume that programs that do use mmc --make
>     and --use-grade-subdirs already *do* have to specify the full pathname
>     of the directories that contain the .mih files of a library they want
>     to use.

I believe the --c-include-directory options in *_FLAGS files are
required because mmake is invoked in separate subdirs. The C compiler
needs to be told about header files that were created in other subdirs.

Most projects using mmc --make are probably NOT built in pieces like
that. That's my guess.

>     QUESTION Do we give any guidance to Mercury programmers (who are not
>     on the Mercury team) about what directory names they need to give to
>     --c-include-directory? The user guide's section on that option, 9.11,
>     does not give any. We could add some, describing the present complicated
>     setup, ... or we could try to simplify the setup, and describe *that*
>     setup. Maybe we could add a sepcial option that would effectively add three
>     directories to the --c-include-directory accumulating string option.
>     If its argument is e.g. mihs at sd1, it would add the directories
> 
>             sd1/Mercury/<grade>/<arch>/Mercury/mihs
>             sd1/Mercury/mihs
>             sd1
> 
>     to the accumulating list.
> 
>     QUESTION What would people prefer? Some of the above, or the status quo?
> 

If I'm right, it isn't really a problem that most users will face.

> Issue 6:
> 
>     The directory paths we install grade-specific files in
>     include one directory name component that represents the grade,
>     and another that represents the target architecture. The latter
>     is clearly useful for execution and library files that contain
>     machine code, in that they detect and report attempts to use
>     e.g. an x86-64 library on an Arm machine. However, not *all*
>     grade-specific files contain machine code. For these, the directory
>     name component that identifies the target architecture is unnecessary,
>     though it is not harmful. It can even be very slightly helpful in that
> 
>     - installing a library in e.g. /path/to/install_dir on one architecture,
>     - and then installing a slightly different version of that same library,
>       also to /path/to/install_dir, for a different architecture,
> 
>     will not overwrite the existing, old versions of those grade-specific
>     files, and each grade-and-architecture-specific directory will contain
>     grade-specific non-machine-code files and grade-specific machine-code
>     files that are from the same version and should thus be consistent
>     with each other.
> 
>     However, this helpfulness is useless, because the second install
>     with mmc -mmake *will* overwrite the old version's installed
>     *non*-grade-specific files, leaving them out of sync with the first
>     install's grade-specific files. When installing with mmake, nothing
>     should be out of sync, because mmake's install rules will *delete*
>     the target directories before recreating and populating them.
> 
>     QUESTION Does anyone remember any occasion where we said or promised
>     anything to users about the outcome of any attempt to install to an
>     already-existing directory?
> 

No.

>     If the answer is "no", which I think it is, then we don't need
>     to include the <arch> component in the grade-specific relative path name
>     of the grade specific files that are not themselves architecture dependent.
>     (For the files that contain machine code, we want to keep it
>     for the help it gives in detecting attempts to link e.g. a .so file
>     with .o files of a different architecture.)

Peter