[m-users.] seg fault, 64 bit

[Paul Bone]

On Sat, Jun 08, 2013 at 01:31:15AM +0200, Tomas By wrote:
> Hi,
> 
> Thanks for the reply. I am using the default grade (asm_fast?)

Well, I hope it's not "asm_fast" and is instead "asm_fast.gc" without
garbage collection memory won't be freed and processes will consume memory
very quickly.  The asm_fast grade is only supported to help us test some
vary trivial programs without garbage collection.  Note also that
"asm_fast.gc" is a grade, as is "asm_fast", but "asm_fast" and "gc" are
grade componenets.  (Yes the whole grades system is too complex when users
just want to write programs.)


> What confuses me is that it is not that much data, and 3GB is a ridiculous
> amount of stack. I have run (what I believe to be) bigger computations on
> 32 bit machines with less than a GB total memory (this machine has 8GB).
> 
> I do not have any tail(or mistakenly non-tail)-recursive functions in my
> code, only foldl's and traversal of recursive data structures.
> 
> The code that causes the problem is this:
> 
> | :- type tnfstats--->
> tnfs(vstats,vstats,vstats,vstats,map(int,int),map(reftype,int)).
> | :- type vstats ---> vs(int,int,int,map(string,int),int,int).
> | :- func new_vs = vstats.
> | new_vs = vs(0,0,0,map.init,0,0).
> |
> | :- pred tnfiles_stats(list(tnfile)::in,tnfstats::out) is det.
> | tnfiles_stats(Fs,Stats) :-
> |   Stats0 = tnfs(new_vs,new_vs,new_vs,new_vs,map.init,map.init),
> |   foldl(tnfile_stats,Fs,Stats0,Stats1),
> |   Stats1 = tnfs(SS,PS,OS,RS,Cmap,Rmap),
> |   map.foldl((pred(_::in,V::in,M0::in,M::out) is det :-
> |     ( map.search(M0,V,N) ->
> |       map.det_update(V,N+1,M0,M)
> |     ; map.det_insert(V,1,M0,M) )),Cmap,map.init,Imap),
> |   Stats = tnfs(SS,PS,OS,RS,Imap,Rmap).
> |
> | :- pred tnfile_stats(tnfile::in,tnfstats::in,tnfstats::out) is det.
> | tnfile_stats(file(_,Chunks),!Stats) :-
> |   foldl(chunk_stats,Chunks,!Stats).
> |
> | :- pred chunk_stats(chunk::in,tnfstats::in,tnfstats::out) is det.
> | chunk_stats(chunk(_,_,_,_,_,Triples),!Stats) :-
> |   foldl(triple_stats,Triples,!Stats).
> 
> where triple_stats/3 updates the tnfstats (no recursive calls).
> 
> Should I use some other data structure than map's?
> 
> The tnfile's vary a bit in size (by a factor ten or so), and there are
> around a thousand of them.
> 
> Shouldn't the stack be back to `zero' at each iteration of
> `foldl(tnfile_stats,Fs,Stats0,Stats1)' ?

Yes, it should.  But it depends on how things are compiled.  In particular
some grades don't handle recursion well at all (usually debugging and
profiling grades).

> If not, how does your suggestion to split the loops help?

Before you try it you must identify which loops are problems and work out
how deep to split them.  It'd also be a good idea to prove that this is a
stack overflow, and not some other problem.

Foldl is tail recursive anyway, but I've found that doing the following can
avoid crashes in grades that do not support tail recursion.  What this does
is limit the inner loop to a depth of 100,000 in this case (test to find a
number that works most optimally, usually by measuring how many frames will
fit on the stack and dividing by two.  After processing the first 100,000
elements, foldl2 will return unwinding all it's stack frames and in the
next iteration of foldl it will start again abeit with slightly less stack
space because of foldl's frame.  If foldl and foldl2 consume the same amount
of stack size. and previously your program crashed after 200,000 iterations
of normal foldl. then this alternative version of foldl should handle up to
about 100,000 * 100,000 recursions.

    % Outer loop.
foldl(_, [], !Acc).
foldl(P, L, !Acc) :-
    foldl2(100000, P, L, LRest, !Acc),
    foldl(P, LRest, !Acc).

    % Inner loop.
foldl2(_, [], [], !Acc).
foldl2(P, [X | Xs], Rest, !Acc) :-
    ( N > 0 ->
        P(X, !Acc),
        foldl2(N - 1, P, Xs, Rest, !Acc)
    ;
        Rest = [X | Xs]
    ).

Good luck.

-- 
Paul Bone
http://www.bone.id.au