[m-rev.] for review: start using --generate-dependencies-ints

[Zoltan Somogyi]

On 2023-10-21 15:20 +11:00 AEDT, "Julien Fischer" <jfischer at opturion.com> wrote:
>> On 2023-10-20 22:27 +11:00 AEDT, "Julien Fischer" <jfischer at opturion.com> wrote:
>>> mmc --make (which is used to build the C# library) supports a no-op
>>> depend target, but doesn't recongise depend_ints. So either need
>>> similarly support depend_ints as a no-op target for mmc --make or
>>> make the above target in the Mmakefile dependent on the grade and not
>>> use depend_ints in the non-C grades.
>>
>> I think the former is preferable. Can you test whether just extending the filter
>> on line 108 of make.top_level.m to cover ".depend_ints" would work?
> 
> I have committed a diff that does this and fixes the problem. There's
> another issue with this change: it is orders of magnitude slower on
> Windows.
> 
> On my Linux machine a top-level mmake depend is now ~17 seconds; on
> Windows (using MSYS2) it is ~21 minutes.

That is a LOT bigger multiplier than what I was expecting,
especially given that --generate-dependencies-ints saves
a LOT of process creation compared to the old approach,
and I know process creation is much slower on Windows than on linux.

> The primary culprit for this
> seems to be all the file copying that is going on.

How certain is that assertion? Do you have profiling data? If not,
could you get some without undue effort?

The code for writing all the .intN files initially writes to a tmp file,
so it can compare the new .intN file with its old version, and decide
whether to update the corresponding timestamp file accordingly.
If there is no old version to compare with, then on linux, we just
rename the tmp file to the non-tmp file. Does this rename require
a file copy on windows? If yes, then it would make sense to
avoid this two step by writing to the non-tmp file directly
if it does not exist initially (which should be a common case
when making the dependencies). I think it should be possible
to do this in a way that wouldn't incur extra cost on linux,
and would avoid the copy on windows. The code required
would have to be significantly more complex than the current code,
though.

> While I suspect
> the times are, more or less, equivalent to what was required for
> building all the inteface files under the old approach,

The amount of work required should be less than the work
required under the old approach, since we save having to read
the .int3 and .int0 files we wrote. The amount of time may be more,
since --generate-dependencies-ints generates and writes the
interface files in sequence, while mmake and mmc --make
can do so in parallel.

On linux, the work saved is so much, and the cost of writing
all the interface files is so small, that the loss of parallelism
has no real noticable effect. On windows, with the base cost
being enormously higher, ... you get the effect you saw.

One way to fix this would be to keep using --generate-dependencies
(no -ints at the end) on Windows.

> the main
> problem is that there very little visual feedback on what is happening.
> (I initially though that the compiler had gone into a loop, until I
> turned on --verbose.)

Fixing that would be easy. We could

- print a message at the start about "making dependencies and all interface files",
- print a message "about to make all .intN files" at the start of each phase, and/or
-print "made/about to make x.intN" for each interface file.

I would be fine with any combination, and we could even pick different combinations
on different OSs. What do other people prefer?

Zoltan,