[m-users.] explaining index use

[Tomas By]

Hello,

As nobody else has replied, I offer my suggestion: forget about fact 
tables and write your own indexing code.

I have some Mercury programs that use millions of data items, indexed in 
multiple ways, and have no problems running it on normal PC hardware 
now. At one point, 10-15 years ago, I had to buy another 128 Mb memory 
chip I think it was. It is nothing like your estimates.

Basically you have to do three things: define a type that describes you 
data (in the file); write a `read' procedure that repeatedly calls 
io.read with this type as out parameter; then create mappings (eg using 
the `map' library) from key to data for all your needed access modes.

/Tomas



On 2016-07-21 05:31, Daniels, Marcus G wrote:
> Hi,
> 
> I have a large fact table (millions of entries) and several modes for
> its use.   I'm not econfident I am getting optimal use of the indices.
> 
> 
> Is there a technique like with most SQL databases where one can
> EXPLAIN the query to see what plan is used?
> 
> The way I read the goals it would happen in an order I expect, but the
> performance isn't good.
> 
> I can't afford to make modes for all possible combinations.  The fact
> table would take weeks to compile and hundreds of gigabytes of RAM
> (for the C compiler).
> 
> Because of the size of the table I put it in a shared library, but
> then I have trouble with deep profiling:
> 
> #0  0x00002aad782b33c1 in __strlen_sse2_pminub () from
> /usr/lib64/libc.so.6
> 
> #1  0x00002aaaaaaf9028 in MR_write_string (fp=fp at entry=0x5ece40,
> ptr=0x0) at mercury_deep_profiling.c:1631
> 
> #2  0x00002aaaaaaf970a in MR_write_out_module_proc_reps_start
> (procrep_fp=0x5ece40,
> 
>     module_layout=0x2aabc2eac6e0
> <mercury__sharedVertex__sharedVertex_fact_table_hash_table_3_4825_data+3360>)
> 
> 
>     at mercury_deep_profiling.c:817
> 
> #3  0x00002aab490a68bf in
> mercury__sharedVertex__write_out_proc_statics () from
> ./libsharedVertex.so
> 
> #4  0x0000000000410da1 in ?? ()
> 
> #5  0x00002aaaaaafa00e in MR_write_out_profiling_tree () at
> mercury_deep_profiling.c:479
> 
> Thanks for any suggestions,
> 
> Marcus
> _______________________________________________
> users mailing list
> users at lists.mercurylang.org
> https://lists.mercurylang.org/listinfo/users