[m-rev.] for review: --generate-dependencies-ints

[Zoltan Somogyi]

On 2023-10-11 15:03 +11:00 AEDT, "Peter Wang" <novalazy at gmail.com> wrote:
> On Wed, 11 Oct 2023 02:42:15 +1100 "Zoltan Somogyi" <zoltan.somogyi at runbox.com> wrote:
>> For review by anyone. The benchmarking script, and the summary
>> of its results that the log message references, are attached.
>> The two arenas (one built by the benchmarking script
>> the conventional way, one built using the new option)
>> end up with identical contents, so the new option works.
>> 
>> The attached diff just adds a new capability, but does not add
>> any uses of it. That is because using it requires decisions
>> that I would like your feedback on.
>> 
>> One decision is: should the new option --generate-dependencies-ints
>> replace the existing --generate-dependencies option? By this I mean
>> actually *deleting* the new option, and making its implementation
>> replace the old implementation of --generate-dependencies.
>> This case would require an update of the documentation of
>> the old option, instead of documentation of the new option.
>> Any announcement of the new capability would also depend
>> on this decision.
> 
> No to deleting --generate-dependencies completely, see below.

OK.
 
>> 
>> If we want to keep both options, the next question is: how do we
>> get mmake to use the new option? We would need some mmake target suffix
>> that Mmake.rules would recognize as calling for the new option
>> --generate-dependencies-ints, instead of just --generate-dependencies.
>> What should that suffix be? I think .depend-ints would be a
>> logical choice, but I am not sure it would look right to users.
> 
> Seems okay.

Will do that then, unless Julien comes up with a better name.

>> If we did implement --generate-dependecies-ints-targets,
>> that would turn mmc into a whole-of-program compiler.
>> (Optionally, since separate compilation would of course
>> still be available.) However, it would do so only for
>> programs that don't need intermodule optimization.
>> Adding intermodule optimization to this framework
>> would actually be the hardest part. We *could* generate
>> .opt files using worker processes and a work queue,
>> but in that case, recording the parse trees of the .opt files
>> being written out would be a problem. While recording
>> the parse tree in the *worker process that created it*
>> would be trivial, *getting that parse tree back to the
>> main process* would be far from trivial. And without that,
>> the process of generating target language code would
>> be required to open, read and parse .opt files, thus
>> leaving some speedup opportunities on the table.
> 
> Right.
> 
> Also it would be replicating much of mmc --make.

That was one reason why I didn't think this worth doing just now.
However, one wouldn't have to duplicate make.build.m if
one could just generalize it sufficiently to work well
for both use cases. Whether that is possible or not, I haven't even
started thinking about.

>> Above, I talked about integrating the new capability
>> with mmake. We should of course integrate it with mmc --make
>> as well, but mmc --make *already* uses worker processes,
>> so I expect that the issue discussed in the above paragraph
>> would need to be tackled even with --generate-dependecies-ints.
> 
> mmc --make doesn't use --generate-dependencies,
> and can generate interface files in parallel (to an extent).

It seems I didn't express myself clearly. I was talking about the fact
that mmc --make's implementation of parallelism, in make.build.m,
currently allows for only one form of commucation between the
actions that build different targets: the contents of the target files.
I meant that integrating mmc --make with *the approach now taken
by --generate-dependencies-ints* of communicating via in-memory
parse trees would require solving the problem of how to share
those parse trees between processes.

>> Add new option --generate-dependencies-ints.
>> 
>> This new option extends --generate-dependencies to take advantage of the
>> opportunity afforded by the fact that "mmc --generate-dependencies prog.m"
>> reads in every Mercury source file in the current directory that is
>> part of "prog". It does this by
>> 
>> - generating the .int3 file of all local-directory modules of the program;
>> - generating the .int0 file for each these modules that has submodules;
>> - generating the .int and .int2 files of all local-directory modules
>>   of the program.
>> 
>> Normally, the process of creating .int0, .int and .int3 files would
>> need to read in .int0 and .int3 files, but in every one of these cases,
>> we have just written out those files, so simply holding onto their
>> parse trees, we can skip this step. On my laptop, on a directory
>> containing library/*.m, mdbcomp/*.m and compiler/*.m, generating
>> the dependencies and generating all the .int3/.int0/.int/.int2 files
>> takes just over 25 seconds. Using the new option, the same process
>> takes less than 10 seconds.
> 
> --generate-dependencies-ints doesn't take advantage of parallelism
> (yet). Generating interface files in parallel might still be faster,
> despite --generate-dependencies-ints doing less work.

Agreed. However, some simple optimizations could allow the current code
to approach the speed of fully parallel creation of all .intN files.

We could optimize the creation of .int3 files by generating their parse trees
in the main process sequentially, thus making it trivial to keep hold of those
parse trees,  but hand over those parse trees to worker processes for
writing out. I don't know what the relative costs of parse tree generation
and parse tree output are, but the fact that only the latter task needs
I/O indicates that this should get a significant fraction of the speedup
available from parallelism.

The creation of .int0 files does not really need to be optimized, since there are
so relatively few of them. Doing them sequentially should therefore not lose much.

And as for .int and .int2 files, they can be generated fully in parallel,
because their inputs consist only of .m, .int3 and .int0 files, and the previous phases
have kept all their parse trees. Solving the problem of how to gather the parse trees
of the .int/.int2 files they generate in an in-memory (as opposed to on-disk) form
is required only if want to proceed to code generation.

> The diff looks okay.

Thanks for that; I will apply the changes you pointed out.
I want to install this diff on my laptop and make at least a start on the next
step before I commit this, so that should happen tomorrow.

Zoltan.