[m-users.] explaining index use

[Peter Wang]

On Fri, 29 Jul 2016 16:00:42 +0000, "Daniels, Marcus G" <mdaniels at lanl.gov> wrote:
> Tomas & Dirk, thanks.   I implemented what Tomas suggested and the load is about 5 minutes, even at 160 million records.  The indexing is more painful, though at about 30 minutes.  I tried to io.read_binary and io.write_binary for both the data and the indices but the files end up to be about 50 GB and it takes hours to load that and seems to have quite a bit of overhead.   Really what would be nice in this situation would be some sort of an attach to a fixed  virtual memory address via a mmap'ed region.   Like the effect of a fact table in a module in a shared library but without the expensive setup costs.  I haven't used logic databases before, but I imagine the latency per lookup goes up by quite a bit compared to an in-memory 2-3-4 tree?

Hi,

You might be better off with a database after all.

You could write a tool that generates a file containing the records and
indices from a table of facts (without invoking a C compiler) then,
as you suggest, mmap that in and do your lookups on that.
Perhaps such a thing exists?  It doesn't need to be Mercury specific.
On the other hand, it's no different from a read-only database.

Peter