htdig using too much memory

Peter Ross petdr at cs.mu.OZ.AU
Fri Feb 5 08:45:49 AEDT 1999


On 05-Feb-1999, Fergus Henderson <fjh at cs.mu.OZ.AU> wrote:
> Hi,
> 
> The mercury nightly tests have been failing on hydra for the past
> couple of weeks.  The reason seems to be that it is running out of
> memory, due to `htmerge' taking up all of the physical memory
> and most of the virual memory on the machine:
> 
>   PID USER     PRI  NI  SIZE  RSS SHARE STAT  LIB %CPU %MEM   TIME COMMAND
>  3943 petdr     11   0  290M 202M   104 R       0  7.3 80.2  14:58 htmerge
> 
> So, Peter Ross, can you please stop running htmerge as part of your
> cron script to install the Mercury WWW pages?
> 
Done.  All that needed to be done is to check out the bin directory of
the w3 module and comment out the appropiate lines in install_webpages,
for future reference.

> And does anyone have any ideas as to why htmerge is using so much memory,
> or how we can avoid this problem?
> 
The problem appears to be in indexing the mailing lists, not the rest of
the web site.  It may be better just to construct the databases from
scratch, rather then merging the results from the current index with the
old index.  This may take longer but not require the whole wordlist
database (about 200M) not to be in memory (I am assuming that this is
the problem).

I will investigate this fix today.

I guess the simple fix would be to arrange the htmerge and nightly tests
to occur at different times.  How long do the nightly tests usually
take?

Pete.



More information about the developers mailing list