[m-dev.] issues with installing libraries

[Zoltan Somogyi]

On Thu, 12 Sep 2024 15:04:06 +1000, Peter Wang <novalazy at gmail.com> wrote:
> >     We currently install .mh files, which are not grade-specific,
> >     to *two* sets of non-grade-specific directories. These are
> > 
> >     set 1:   Prefix/lib/mercury/inc
> >     set 2:   Prefix/lib/mercury/ints and Prefix/lib/mercury/ints/Mercury/mhs
> > 
> >     I believe that set 1 is intended for use by hand-written user code
> >     that uses Mercury predicates and/or functions exported to C.
> > 
> >     QUESTION Does anyone know the purpose of set 2? What purpose does it serve
> >     that set 1 cannot satisfy?
> > 
> 
> When it comes to (seemingly) redundant copies of files, I always
> remember something about satisfying the VPATH mechanism of make.

So do I, but "satisfying VPATH" is only a clue about the actual motivation
of this setup; it does not say what the motivation actually was.

> Is it this?
> 
>             % This is needed so that the file will be found in Mmake's VPATH.
>             install_subdir_file(ProgressStream, Globals, NgsLibDirMap,
>                 LibDir / "ints", ModuleName, {ExtMh, "mhs"}, !Succeeded, !IO)

The same point applies here as well.

> >     QUESTION Does anyone know of any reason why the install of these
> >     grade-specific files in a non-grade-specific directory would be useful?
> > 
> 
> I don't know.
> 
> >     QUESTION Does anyone recall any problem that you suspect *could* have
> >     been caused by getting a .mih or .opt file *for the wrong grade*
> >     from a non-grade-specific directory?
> 
> I don't remember it being a problem, for the reasons you state.

I figured that people wouldn't know the answers to those questions,
but I had to ask.

> > Issue 3:
> > 
> >     We can install a library in multiple grades
> > 
> >     - either via mmake (the "lib%.install_grades" target
> >       in scripts/Mmake.rules, which delegates part of its work
> >       to two mmake rules in auto-gemerated .dep files),
> > 
> >     - or via "mmc --make lib%.install" (so to speak).
> > 
> >     These two routes differ in several ways.
> > 
> >     - The set of grades to install is determined differently by the
> >       two routes. The mmake route install the librades specified by the
> >       ALL_LIBGRADES make variable, whose value can be specified via
> >       a library-specific make variable. On the other hand the set of grades
> >       to install is the same for every library when using mmc --make;
> >       it will be the set of libgrades found to be installed by
> >       compiler/check_libgrades.m, if this is not overridden by the
> >       --libgrade accumulating option.
> > 
> >     - The mmake route installs .trans_opt files (though the code that does
> >       this also handles .int* files but NOT .opt files, which is strange),
> >       while the mmc --make route never does.
> > 
> >     I believe there are two related root causes of this, and an unrelated
> >     third root cause.
> > 
> >     Cause 1 is that there is no central documentation of what files
> >     should be installed where, and more important, *why* are they installed
> >     in those places. This would require the full documentation of the search
> >     methods we use to find things in install directories. (See Issue 4.)
> > 
> >     Note; the EXT file I sent to m-rev recently documents what *does*
> >     get installed where, which can be a starting point for the first half
> >     of this documentation. However, I have not yet found all the places
> >     that *set up* search paths, and I suspect that the most important ones
> >     are not in the Mercury implementation at all, but in individual users'
> >     Mmakefiles and/or Mercury.options files.
> > 
> >     Cause 2 is that the places in the system that deal with installing files
> >     do not even have links to each other, so there is no reminder to people
> >     working on one part (e.g. mmc --make) to update the corrresponding parts
> >     (in e.g. Mmake.rules). I plan to add those links in a future diff.
> > 
> >     Cause 3 is that our requirements changed over time. When we worked
> >     on termination analysis in the late 1990s, we installed .trans_opt files
> >     to give users access to its results. By the time Simon implemented
> >     the initial version of mmc --make in 2002, this became less important,
> >     so we willingly tolerated the difference in the treatment of .trans_opt
> >     files.
> > 
> >     QUESTION Ideally, both install routes should install the same files
> >     in the same directories. In practice, making that happen will take
> >     a significant amount of work. For each of the differences above,
> >     can you please say whether you have been impacted by the difference.
> 
> I use mmake to install the Mercury system. Thereafter, I never install
> any Mercury libraries using either mmake or mmc --make, so I have not
> been impacted. It's just easier to build any Mercury modules (other than
> the standard library) as part of the same application.

OK, I will not invest any effort into fixing this difference.

> > Issue 4:
> > 
> >     One constraint on the structure of the install directory (and the
> >     *main* constraint on the structure of its non-user-facing parts,
> >     which make its bulk) are the compiler's search methods. Those methods
> >     need to find the .int, .opt etc files of
> > 
> >     - the modules that are part of the current program, and
> >     - the modules that are not part of the current program, and are instead
> >       parts of external libraries.
> > 
> >     For modules that are not part of the current program, both mmake
> >     and mmc --make will install all grade-specific files into grade-specific
> >     directories, and all non-grade-specific files into non-grade-specific
> >     directories (with some exceptions, such as Issue 2 above).
> > 
> >     For modules that are part of the current program, we cannot assume that;
> >     with --no-use-grade-subdirs, grade-specific files may be in
> >     non-grade-specific directories, whose names may, or may not,
> >     have a suffix such as Mercury/int3s (for .int3 files).
> > 
> >     We currently handle such uncertainty in two different ways, depending
> >     on whether the extension of the file want to search for contains "max"
> >     in the names of its extension category in file_names.m.
> > 
> >     Consider searching for module_x.int3. The name of the extension category
> >     for .int3 files does NOT contain "max". This means that e.g. with
> >     --use-subdirs and --no-use-grade-subdirs, both module_name_to_file_name
> >     and module_name_to_search_file_name will return Mercury/int3s/module_x.int3
> >     when given module_x and .int3 as inputs. If the search_directories
> >     option (which contains the accumulated strings specified for the
> >     --search-directory option) contains ["sd1", "sd2", "sd3"], then our
> >     search will look for
> > 
> >         sd1/Mercury/int3s/module_x.int3
> >         sd2/Mercury/int3s/module_x.int3
> >         sd3/Mercury/int3s/module_x.int3
> > 
> >     Consider searching for module_x.opt, whose extension is in the
> >     ext_cur_ngs_gs_max_ngs category, which DOES contain "max".
> >     This means that e.g. with --use-grade-subdirs, when given module_x
> >     and .opt as inputs,
> > 
> >         module_name_to_file_name will return
> >             Mercury/<grade>/<arch>/Mercury/opts/module_x.opt
> > 
> >         but module_name_to_search_file_name will return
> >             Mercury/opts/module_x.opt
> > 
> >         so we would search
> >             sd1/Mercury/opts/module_x.opt
> >             sd2/Mercury/opts/module_x.opt
> >             sd3/Mercury/opts/module_x.opt
> > 
> >     I think it is this search strategy that was the motivation
> >     for the current setup of install directories in a way that causes
> >     the second "Mercury" in the name of grade-specific directories.
> >     Specifically, the correspondence between
> > 
> >     Mercury/<grade>/<arch>/Mercury/opts/module_x.opt and
> >                            Mercury/opts/module_x.opt
> > 
> >     allows InstallDir/Mercury/<grade>/<arch>/Mercury/opts/module_x.opt
> >     to be found by
> >     
> >     - BOTH a search for the full grade-specific name in InstallDir,
> >     - AND a search for the non-grade-specific name, Mercury/opts/module_x.opt,
> >       in InstallDir/<grade>/<arch>.
> > 
> >     QUESTION What do you guys think of this theory? Does anyone have an
> >     alternative theory?
> 
> That's how I've understood it.
> 
> >     QUESTION Does anyone know, or have reason to believe, that there is
> >     still a need to find .opt files using both kinds of searches?
> >     If there is, or may be, such a need, would it go away if we implemented
> >     one of the mechanisms in the next question?
> > 
> >     QUESTION Would a special option that says "here is opts at sdN,
> >     add the directories
> > 
> >             sdN/Mercury/<grade>/<arch>/Mercury/opts
> >             sdN/Mercury/opts
> >             sdN
> > 
> >     to the search_directories accumulating string option" be sufficiently
> >     useful to implement?
> > 
> >     How about an option that says "here is @sdN, add the directories
> > 
> >             sdN/Mercury/<grade>/<arch>/Mercury/ExtDir
> >             sdN/Mercury/ExtDir
> >             sdN
> > 
> >     to the search_directories accumulating string option, *but only when
> >     searching for a file with an extension for which ext_to_dir_path
> >     in compiler/file_names.m returns ExtDir*"?
> > 
> >     Both of these would work both on files in installed libraries *and*
> >     on files in workspaces.
> 
> I haven't needed something like that (obviously, given the above).

OK.

> > Issue 5:
> > 
> >     For .mih files, which are in the ext_cur_ngs_gs_max_cur,
> >     the results would be
> > 
> >         module_name_to_file_name will return
> >             Mercury/<grade>/<arch>/Mercury/mihs/module_x.mih
> > 
> >         but module_name_to_search_file_name will return
> >             module_x.mih
> > 
> >         so we would search
> >             cid1/module_x.mih
> >             cid2/module_x.mih
> >             cid3/module_x.mih
> > 
> >     The difference is that it is mmc that would search for .opt files,
> >     but the search for .mih files is done by the C compiler, with the
> >     directories to be searched (which are accumulated from -c-include-directory
> >     options, hence cidN above). This is why e.g. compiler/COMP_FLAGS.in
> >     specifies BOTH --c-include-directory ../library and --c-include-directory
> >     ../library/Mercury/mihs, and would have to include the grade-specific
> >     subdir name as well, if we ever wanted mmc to be built with mmc --make
> >     and --use-grade-subdirs. I presume that programs that do use mmc --make
> >     and --use-grade-subdirs already *do* have to specify the full pathname
> >     of the directories that contain the .mih files of a library they want
> >     to use.
> 
> I believe the --c-include-directory options in *_FLAGS files are
> required because mmake is invoked in separate subdirs. The C compiler
> needs to be told about header files that were created in other subdirs.

Agreed.

> Most projects using mmc --make are probably NOT built in pieces like
> that. That's my guess.

Also agreed.

> >     QUESTION Do we give any guidance to Mercury programmers (who are not
> >     on the Mercury team) about what directory names they need to give to
> >     --c-include-directory? The user guide's section on that option, 9.11,
> >     does not give any. We could add some, describing the present complicated
> >     setup, ... or we could try to simplify the setup, and describe *that*
> >     setup. Maybe we could add a sepcial option that would effectively add three
> >     directories to the --c-include-directory accumulating string option.
> >     If its argument is e.g. mihs at sd1, it would add the directories
> > 
> >             sd1/Mercury/<grade>/<arch>/Mercury/mihs
> >             sd1/Mercury/mihs
> >             sd1
> > 
> >     to the accumulating list.
> > 
> >     QUESTION What would people prefer? Some of the above, or the status quo?
> 
> If I'm right, it isn't really a problem that most users will face.

No, but we do.

With the current status quo, we cannot implement --use-grade-subdirs for mmake
and then use it ourselves without specifying, for each directory we want to search for
e.g. .mih files not just sd1 and sd1/Mercury/mihs, but also an instance of sd1/Mercury/
<grade>/><arch>/Mercury/mihs *for every grade* we want to compile the system in*.

> > Issue 6:
> > 
> >     The directory paths we install grade-specific files in
> >     include one directory name component that represents the grade,
> >     and another that represents the target architecture. The latter
> >     is clearly useful for execution and library files that contain
> >     machine code, in that they detect and report attempts to use
> >     e.g. an x86-64 library on an Arm machine. However, not *all*
> >     grade-specific files contain machine code. For these, the directory
> >     name component that identifies the target architecture is unnecessary,
> >     though it is not harmful. It can even be very slightly helpful in that
> > 
> >     - installing a library in e.g. /path/to/install_dir on one architecture,
> >     - and then installing a slightly different version of that same library,
> >       also to /path/to/install_dir, for a different architecture,
> > 
> >     will not overwrite the existing, old versions of those grade-specific
> >     files, and each grade-and-architecture-specific directory will contain
> >     grade-specific non-machine-code files and grade-specific machine-code
> >     files that are from the same version and should thus be consistent
> >     with each other.
> > 
> >     However, this helpfulness is useless, because the second install
> >     with mmc -mmake *will* overwrite the old version's installed
> >     *non*-grade-specific files, leaving them out of sync with the first
> >     install's grade-specific files. When installing with mmake, nothing
> >     should be out of sync, because mmake's install rules will *delete*
> >     the target directories before recreating and populating them.
> > 
> >     QUESTION Does anyone remember any occasion where we said or promised
> >     anything to users about the outcome of any attempt to install to an
> >     already-existing directory?
> 
> No.

Thanks. If Julien does not remember any such promise either, then
I can rationalise the install directory structure.

> >     If the answer is "no", which I think it is, then we don't need
> >     to include the <arch> component in the grade-specific relative path name
> >     of the grade specific files that are not themselves architecture dependent.
> >     (For the files that contain machine code, we want to keep it
> >     for the help it gives in detecting attempts to link e.g. a .so file
> >     with .o files of a different architecture.)
> 
> Peter

Thank you.

Julien, do you plan to reply at some point?

Zoltan.