Performance

[Fergus Henderson]

On 23-Sep-1998, William Donovan <wdonovan at webtv.net> wrote:
> You claim that Mercury is about 5x faster than
> SICStus native mode.

The figure varies from benchmark to benchmark.
It also varies depending on what options you give the Mercury compiler.

> SICStus claims that
> their Fastcode is also about 5x faster than 
> their native code.

I don't know where you got that idea from.
SICStus "fastcode" _is_ their native code.

I think they probably meant that their
"fastcode" (native code) is about 5x faster
than their "compactcode" (bytecode).

However, for the set of small benchmarks that we tried, SICStus "fastcode"
was only about 3.4 times faster than SICStus "compactcode".
See <http://www.cs.mu.oz.au/research/mercury/information/benchmarks.html>
for detailed results of those benchmarks.  But the most relevant
numbers from those tests are the following:

				 h. average (of 10 benchmarks)
	sicstuscomp_seq		 2.02
	sicstusfast_seq		 6.83
	mercury_asm_fast.gc	 9.24
	mercury_asm_fast	36.52

So for *these* benchmarks, SICStus fastcode is about 3.4 times faster than
SICStus compactcode; Mercury with conservative GC is about 35% faster
(1.35 times faster) than SICStus fastcode; and Mercury without GC is
about 5.3 times faster than SICStus fastcode.

However, these are mostly tiny benchmarks, run many times over in a loop;
on all the systems except for Mercury with conservative GC, this results
in a very small memory footprint and hence very good cache performance.
For larger applications, the figures for Mercury with conservative GC
are likely to be closer to the figures for Mercury without GC.

We're currently working on a different garbage collector for Mercury
which we hope will give performance very close to Mercury without GC
even for small benchmarks.

The only large benchmark I've done is the Pseudoknot benchmark.
That benchmark is about 3500 lines of source.  This benchmark has
been used to compare functional programming languages, but actually
it is a searching application, so it is quite appropriate as a
benchmark for logic programming languages too. 
I got the following figures.

			run time	runtime memory usage
			(user+sys)	
Mercury 0.6
	asm_fast	3.4+1.4		21M	
	asm_fast.gc	6.8+1.3		<1M	

SICStus 2.1#9
	fastcode	69+2s		15M	  

BinProlog 3.4.5		N/A		>300M

NU-Prolog 1.6.4		N/A		N/A

So for *this* benchmark, Mercury with GC was about 8.8 times faster than
SICStus, and used way less memory; without GC it was 15 times faster
than SICStus, but used a bit more memory.  (BinProlog and NU-Prolog
couldn't even run the benchmark.)

I think it's safe to say that Mercury is very likely to be significantly
faster than SICStus on most applications.  However, because the results
vary quite significantly from benchmark to benchmark, and because we haven't
really done much benchmarking on large applications, it's difficult to
say exactly how much the difference will be in any given case.

Cheers,
	Fergus.

-- 
Fergus Henderson <fjh at cs.mu.oz.au>  |  "I have always known that the pursuit
WWW: <http://www.cs.mu.oz.au/~fjh>  |  of excellence is a lethal habit"
PGP: finger fjh at 128.250.37.3        |     -- the last words of T. S. Garp.