The mailing list server is a very busy box. The problem is mailman's terrible
implementation of an archiver called pipermail.

The problem is mostly in how pipermail builds the date and thread views of an
archive.

For all the ugly details, http://www.mn-linux.org/search/ and search for
"mailman slow"

A somewhat brief explaination:

First mailman hands off the archiving responsibility to pipermail.

Pipermail then appends every message to a mbox file in
$MAILMANHOME/archives/private/tcwug-list.mbox, this alone is nasty. Since many
of the mailing list have Gb sized mbox files. (crunchy disks)

Next, a cron job runs, that takes the huge mbox file above and breaks it out
into "time slices", which for tcwug is monthly and shoves the the current
month's postings into $MAILMANHOME/archives/private/tcwug-list/YYYY-MMM.txt.

Another cron job runs, which takes the
$MAILMANHOME/archives/private/tcwug-list/YYYY-MMMM.txt and gzip -9's the file.
(heavy cpu load).

Yet another cron job runs with takes the
$MAILMANHOME/archives/private/tcwug-list.mbox file and breaks it out into the
individual messages in
$MAILMANHOME/archives/private/tcwug-list/YYYY-MMMM/xxxxxx.html. (crunch disk and
heavy cpu load).

The design is poor, the mailman group knows it. They plan a re-write of
pipermail some time in the future. Even the original author of pipermail calls
it crap and abandoned it. It's only supported in mailman
(http://www.amk.ca/python/unmaintained/pipermail.html).

Like always in the open source world, if you are a python guru, look at the code
and contribute. Help fix the problem. Fixing a problem is hard. Complaining
about a problem is easy. :-)

*** OR ****

If you are like me and think the archiving of mail message should be left to an
external package and you know Java, join the Mail Portable Online Storage System
on sourceforge (https://sourceforge.net/projects/mposs/). It's goal is to
have functionality like Geocrawler, but open source. 

Random stats:

The box is under load, it runs around 3.5 to 4.0 almost 24hrs a day.

The box also processes a lot of mail (IMHO)

May 12 - May 31: 314,013
Jun 1  - Jun 5 : 27,267

vendor_id       : GenuineIntel
cpu family      : 6
model           : 1
model name      : Pentium Pro
stepping        : 9
cpu MHz         : 199.437
cache size      : 256 KB

             total       used       free     shared    buffers     cached
Mem:        126704     123888       2816         12      10552      25348
-/+ buffers/cache:      87988      38716
Swap:       262136      23116     239020

(Yes, it could use more RAM)


-- 
Bob Tanner <tanner at real-time.com>         | Phone : (952)943-8700
http://www.mn-linux.org, Minnesota, Linux | Fax   : (952)943-8500
http://www.tcwug.org, Minnesota, Wireless | Coding isn't a crime. 
Key fingerprint =  6C E9 51 4F D5 3E 4C 66 62 A9 10 E5 35 85 39 D9