[m-rev.] for review: parallel mmc make

[Julien Fischer]

On Mon, 12 Feb 2007, Peter Wang wrote:

> On 2007-02-09, Julien Fischer <juliensf at csse.unimelb.edu.au> wrote:
>>

>> Do you have any performance figures for this change yet?
>> asm_fast.gc vs. asm_fast.par.gc with one job vs. asm_fast.par.gc with
>> multiple jobs.
>
> I was testing mainly with hlc.par.gc, which works as expected, i.e. the
> C -> object code step itself is nearly twice as fast with two
> processors, but the rest of the build process is slower due to the .par
> grade.
>
> It turns out asm_fast.par.gc isn't making use of extra processors
> properly.  I won't go into the details (which I'm not entirely sure of
> yet), but I think we might have to make spawn/3 actually create new
> threads/engines in asm_fast.par.gc (as we do in hlc.par.gc), rather than
> reusing the fixed set of engines that were created at the start of the
> program.  Those would then only be used for executing parallel
> conjunctions.  That makes sense to me anyway.  Any objections?

I don't fully understand the problem - so I can't object or not object.

>>> Currently this is limited to running the
>>> "compile target code to object code" step in parallel.
>>
>> I take it that this because of the way mmc --make currently computes
>> its task list?  (I think we discussed this at once?)
>
> Yes, one problem is that when two threads are compiling different
> modules, they may try to simultaneously create the same interface file
> of one of the modules both of them import.

I don't think there's actually much, if anything, to be gained from
using multiple jobs to build the interface files.  The .opt and
.analysis files are probably the other main thing that needs to be
parallelised.

>>> @@ -578,7 +581,7 @@
>>>
>>> io_set_globals(Globals, !IO) :-
>>>    type_to_univ(Globals, UnivGlobals),
>>> -    io.set_globals(UnivGlobals, !IO).
>>> +    globals.set_globals(UnivGlobals, !IO).
>>>
>>
>> Can some of the unsafe_promise_uniques in this module now be deleted?
>
> Yes.

My change to make the I/O globals non-unique got rid of them anyway.

>>> +:- pred run_in_child(child_exits::in,
>>> +    foldl2_pred_with_status(T, Info, io)::in(foldl2_pred_with_status),
>>> +    Info::in, T::in, io::di, io::uo) is det.
>>> +
>>> +run_in_child(ChildExits, P, Info, T, !IO) :-
>>> +    promise_equivalent_solutions [!:IO] (
>>> +        spawn((pred(!.IO::di, !:IO::uo) is cc_multi :-
>>> +            %
>>> +            % Make sure something always gets written to ChildExits when P
>>> +            % finishes, even if terminated by an exception.
>>> +            %
>>> +            try_io((pred(Succ::out, !.IO::di, !:IO::uo) is det :-
>>> +                P(T, Succ, Info, _Info, !IO)
>>> +            ), Result, !IO),
>>> +            (
>>> +                Result = succeeded(Success),
>>> +                channel.put(ChildExits, Success, !IO)
>>> +            ;
>>> +                Result = exception(_),
>>> +                channel.put(ChildExits, no, !IO),
>>> +                rethrow(Result)
>>
>> Rethrowing that exception is almost certain to do something very dodgy
>> if the thread is not the main thread.  It might be better if parent loop
>> were just informed that a child had terminated with an exception via
>> some other mechanism (the channel would be the obvious one) rather
>> than rethrowing the exception.
>
> Good idea.  But what should the parent do with the exception(s)?  I made
> the parent rethrow the first exception it gets (after all the running
> children terminate).

In the case that multiple children throw exceptions couldn't you just
get the parent to rethrow a list of them?

This can't be cleared up properly until we have a mechanism to handle
threads (and parallel conjunction) and exceptions in general.

>>> @@ -5521,6 +5642,7 @@
>>>    mercury_file_init(System.Console.OpenStandardOutput(),
>>>        null, System.Console.Out, ML_file_encoding_kind.ML_raw_binary);
>>>
>>> +// Note: these are set again in io.init_state.
>>
>> Use /* ... */ in preference to //.  (We should try to maintain
>> support for C compilers other than gcc - and some C compilers have not
>> made much headway in supporting C99.)
>
> The // comments appear in C# and Java code.

Sorry, didn't see that.

>
> Partial interdiff follows.

That's fine.

Julien.
--------------------------------------------------------------------------
mercury-reviews mailing list
Post messages to:       mercury-reviews at csse.unimelb.edu.au
Administrative Queries: owner-mercury-reviews at csse.unimelb.edu.au
Subscriptions:          mercury-reviews-request at csse.unimelb.edu.au
--------------------------------------------------------------------------