[m-rev.] for review: --generate-dependencies-ints

[Zoltan Somogyi]

For review by anyone. The benchmarking script, and the summary
of its results that the log message references, are attached.
The two arenas (one built by the benchmarking script
the conventional way, one built using the new option)
end up with identical contents, so the new option works.

The attached diff just adds a new capability, but does not add
any uses of it. That is because using it requires decisions
that I would like your feedback on.

One decision is: should the new option --generate-dependencies-ints
replace the existing --generate-dependencies option? By this I mean
actually *deleting* the new option, and making its implementation
replace the old implementation of --generate-dependencies.
This case would require an update of the documentation of
the old option, instead of documentation of the new option.
Any announcement of the new capability would also depend
on this decision.

If we want to keep both options, the next question is: how do we
get mmake to use the new option? We would need some mmake target suffix
that Mmake.rules would recognize as calling for the new option
--generate-dependencies-ints, instead of just --generate-dependencies.
What should that suffix be? I think .depend-ints would be a
logical choice, but I am not sure it would look right to users.

The speedup demonstrated by the benchmark results comes from
avoiding unnecessary opening, reading and parsing of both
source files and interface files. That same saving would
be available if we extended --generate-dependecies-ints
to --generate-dependecies-ints-targets, i.e. we also generated
.c, .cs or .java files. The reason I did not do that
is that compilation is a significantly more expensive process
than the generation of .intN files, and therefore benefits
much more from parallelism. This means that looping over
the modules to be compiled with list.map2_foldl2 would not do;
one would want to set up some worker processes, and feed them
modules to generate target language code for from a work queue.
That is a much bigger change than this one.

If we did implement --generate-dependecies-ints-targets,
that would turn mmc into a whole-of-program compiler.
(Optionally, since separate compilation would of course
still be available.) However, it would do so only for
programs that don't need intermodule optimization.
Adding intermodule optimization to this framework
would actually be the hardest part. We *could* generate
.opt files using worker processes and a work queue,
but in that case, recording the parse trees of the .opt files
being written out would be a problem. While recording
the parse tree in the *worker process that created it*
would be trivial, *getting that parse tree back to the
main process* would be far from trivial. And without that,
the process of generating target language code would
be required to open, read and parse .opt files, thus
leaving some speedup opportunities on the table.

Above, I talked about integrating the new capability
with mmake. We should of course integrate it with mmc --make
as well, but mmc --make *already* uses worker processes,
so I expect that the issue discussed in the above paragraph
would need to be tackled even with --generate-dependecies-ints.

Opinions?

Zoltan.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Log.gdi
Type: application/octet-stream
Size: 4220 bytes
Desc: not available
URL: <http://lists.mercurylang.org/archives/reviews/attachments/20231011/8ac63317/attachment-0005.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: DIFF.gdi
Type: application/octet-stream
Size: 74235 bytes
Desc: not available
URL: <http://lists.mercurylang.org/archives/reviews/attachments/20231011/8ac63317/attachment-0006.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: TIMEZ
Type: application/octet-stream
Size: 603 bytes
Desc: not available
URL: <http://lists.mercurylang.org/archives/reviews/attachments/20231011/8ac63317/attachment-0007.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: INT0s
Type: application/octet-stream
Size: 523 bytes
Desc: not available
URL: <http://lists.mercurylang.org/archives/reviews/attachments/20231011/8ac63317/attachment-0008.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: SPEEDSUM.generate_deps_ints.2023_10_10
Type: application/octet-stream
Size: 290 bytes
Desc: not available
URL: <http://lists.mercurylang.org/archives/reviews/attachments/20231011/8ac63317/attachment-0009.obj>