htdig using too much memory

[Peter Ross]

On 05-Feb-1999, Peter Ross at UNI <petdr at cs.mu.OZ.AU> wrote:
> On 05-Feb-1999, Fergus Henderson <fjh at cs.mu.OZ.AU> wrote:
> > Hi,
> > 
> > The mercury nightly tests have been failing on hydra for the past
> > couple of weeks.  The reason seems to be that it is running out of
> > memory, due to `htmerge' taking up all of the physical memory
> > and most of the virual memory on the machine:
> > 
> >   PID USER     PRI  NI  SIZE  RSS SHARE STAT  LIB %CPU %MEM   TIME COMMAND
> >  3943 petdr     11   0  290M 202M   104 R       0  7.3 80.2  14:58 htmerge
> > 
> > So, Peter Ross, can you please stop running htmerge as part of your
> > cron script to install the Mercury WWW pages?
> > 
> Done.  All that needed to be done is to check out the bin directory of
> the w3 module and comment out the appropiate lines in install_webpages,
> for future reference.
> 
> > And does anyone have any ideas as to why htmerge is using so much memory,
> > or how we can avoid this problem?
> > 
> The problem appears to be in indexing the mailing lists, not the rest of
> the web site.  It may be better just to construct the databases from
> scratch, rather then merging the results from the current index with the
> old index.  This may take longer but not require the whole wordlist
> database (about 200M) not to be in memory (I am assuming that this is
> the problem).
> 
> I will investigate this fix today.
> 
No you always need to run htmerge.

Pete.