Worklog

[Research] Sendmail, Cyrus, and LMTP

It appears that when you tell Sendmail to use LMTP to deliver to mailboxes, it doesn’t reject unknown recipients, instead letting LMTP do that.

Understandable, since with Cyrus, at least, Sendmail may have no idea who is an actual user. But this means that we’re suddenly having to bounce nonexistent users rather than SMTP-rejecting them. In my architecture, this isn’t a big deal, but I’d rather avoid it.

Apparently, though, there’s a work-around for this. This page has instructions for using the cyrus smmapd process and Sendmail 8.13.x’s socketmap feature to verify that a user exists and their mailbox is writable before letting Sendmail let go of the message to LMTP.

I’m compiling 8.13.1 on the test box now to see how well it works.

—

Some more info: This post has some useful tips.

FreeBSD’s 8.13.1 port includes patches for Cyrus and socket map, if you enable them. The doco is sparse all around, but we’ll get this working.

—

Actually, the FreeBSD doco and the Cyrus patches to the port are quite simple. I don’t know how well that’ll carry over into Solaris, if we go that route.

[Firefighting] PACO and a slightly unhappy disk

PACO seems to have a slightly unhappy disk, but probably nothing to worry about yet.

Got this in the error log today:

As far as I can tell, it’s just found a new bad block. Keep an eye on it, but don’t call Sun for replacement just yet.

[Upgrades] Mailman close to ready

I’ve got the Mailman upgrade procedure worked out and I’m having some other people take a look at the new lists.

It turns out that the standard upgrade procedure is actually the best thing to do here. We lose real name information for all list members, but that’s the breaks when you deal with unsupported home-brew software. The basic procedure is, for each list:

1. tar up the list settings and archives.
2. untar the tarball on the new system in the new Mailman directory

Then run update -f to force an update on all lists. Run check_perms to fix permissions problems. Then configure the lists with new settings: web_page_url and which_archiver (to make sure we’re using MHonArc). Then wipe and rebuild the archives. Finally, run htdig to generate the search indices.

[Firefighting] LDAP Indexing

Yesterday I rebuilt the LDAP indices.

It takes about 30 to 45 minutes for a full rebuild, assuming slapd doesn’t crash in the middle. Be sure to turn off the heavy hitters for LDAP: Samba and RADIUS (small amounts of LDAP traffic is ok).

I’m not sure it really solved the problem, though. And I’m not sure what the problem was, other than ASHTI showing a load average of around 1 for long periods of time and Samba complaining about not being able to connect to LDAP a lot. It may be time to upsize the LDAP server.

[Upgrades] Mailman upgrade

Spent most of the day working on a Mailman upgrade, at long last.

It’s been a while in coming, but we’re finally getting to it. In looking through the FreeBSD port for Mailman, I also found reference to this site with a bunch of Mailman patches. In particular, there are patches to integrate Mailman and ht://Dig and easily add MHonArc into Mailman (which is what we’re using already). The ht://Dig part gives us searchable archives, even for private lists.

So today I managed to get most of this working. The main things left at this point are to figure out a decent migration strategy from our in-house-hacked version of Mailman to the new install, while keeping all the lists, their options, their passwords, and the membership. Shouldn’t be too hard with the help of dbdump and some interesting parsing.

[Research] Cyrus SQUAT

Running the squatter indexing process on the Cyrus IMAP mailboxes seems to significantly improve some of the tests run yesterday.

In particular, the second and third SquirrelMail tests have their times cut by about a third. Most of the other times didn’t change significantly.

[Research] Cyrus Benchmarking

I got Cyrus working to a good approximation and then went about comparing it on the same system with both UW-IMAP and Dovecot. It doesn’t completely blow the others out of the water, but it is distinctly faster in most cases.

The server is a VMware instance on a Dell PE 2650 running FreeBSD 4.7 with 256MB of RAM and a 3.1GHz CPU. The UFS filesystem has softupdates turned on. As before, I ran clearcache prior to each test to clear the RAM cache. The numbers follow (and again, the zeroes in the Dovecot tests represent the ORDEREDSUBJECT thread capability that it does not have):

And again, with thanks to gnuplot:

Follow the red line for Cyrus. It’s the fastest in all but four tests:

• 1.capability, which simply logs in and requests the CAPABILITY for the server.
• 1.sort, which selects INBOX and then does a SORT (REVERSE DATE) where UW-IMAP just barely nudges it out.
• 2.squirrelmail, which does some EXAMINEs, EXPUNGEs, UID SORT (ARRIVAL), a FETCH of some header info for the first twenty UIDs, an EXAMINE, and then disconnect.
• 3.squirrelmail, which does a LIST, LSUB, LIST, EXAMINE, and STATUS.

Both Dovecot in maildir mode and UW-IMAP beat it out by a significant amount on the second SquirrelMail test. Interestingly, the 1.squirrelmail test, which is identical to 2.squirrelmail with the substitution of a UID THREAD REFERENCES for the UID SORT shows Cyrus winning hands down. The second SquirrelMail test is also the only one where Cyrus took longer than 1 second to complete.

I think this shows Cyrus in a pretty good light. It’s fastest at most tasks. I’d worry more about it not being the fastest in two of the three SuirrelMail tests except that the third test is so close as to not necessarily be relevant.

I’m also pretty impressed with its administration. We’ve all heard how nasty and complicated Cyrus is to manage, but that’s not at all what I found: setting it up was a matter of installing OpenLDAP, Berkeley DB 3, Cyrus SASL, and Cyrus IMAP, and then doing a little careful reading for the three config files: imapd.conf, cyrus.conf, and saslauthd.conf. With some intelligent values in there, it just goes right along. I can’t say it’s easier than UW-IMAP, but that’s hardly saying anything. It’s not a whole lot worse than either Dovecot or Courier.

[Research] Cyrus IMAP

Spent most of the day after this morning’s explorations playing with Cyrus IMAP.

I’m not entirely sure what all the fuss is about. Yes, Cyrus is a little more complex than any of the others, and yes, SASL is a slight pain, but I was able to get a test instance of Cyrus up and running on a fairly fresh FreeBSD VMWare instance in a couple of hours (I already had OpenLDAP installed). I’ve even got it authenticating to our LDAP server, and the great part is that the whole system, Sendmail and all, doesn’t care about Unix users — if the Cyrus mailbox exists, that’s all it cares about. So a Cyrus box doesn’t have to be in NIS or have the passwd file synced.

The setup I’ve been testing has the alternate path separator, so that folders are “foo/bar/baz” rather than “foo.bar.baz”, which may make converting from UW-IMAP a little easier.

The main glitch will be transferring all the mail. I don’t know that there’s any way to directly copy messages into the Cyrus mail store and preserve the message flags (copying them in is easy, if followed by a reconstruct call, but they all appear as new messages). The approach at this point would probably be to temporarily reset the user’s password, do an IMAP to IMAP copy from the old server to the new (including all subfolders), and then set the password back to the original.

There are some IMAP copy/sync tools out there, but they all seem to have problems or at least areas where they wouldn’t fit our mentality. I’ll probably end up coding something in perl if we go this way.

First off, though, is to get UW-IMAP (and possibly Dovecot or Courier) installed on this test server to compare some real numbers (running the tests referenced earlier produced extremely good looking numbers, but we’re also running on a much more powerful server and I wasn’t clearing the memory cache between each test).

[Research] IMAP Performance

I’ve been looking at IMAP performance over the last few days, and I’ve come up with some pretty heretical results.

The platform for these tests was FreeBSD 4.5/i386, running on a Dell PE 2400. The filesystem was UFS with softupdates enabled. RAM was 512MB, and clearcache was run prior to every test. I tested UW-IMAP using mbox format, Dovecot using both mbox and maildir formats, and Courier-IMAP using maildir.

The results (the two tests that have 0 times are for Dovecot on the THREAD ORDEREDSUBJECT command, which is not implemented in Dovecot):

Or, as a graph:

The PINE and SquirrelMail tests are taken from a typical short session dump of the corresponding mail clients. The PINE test is a simple opening of the inbox, fetching the header information for a range of messages, fetching the header information for another range, and finally fetching a message. The first two SquirrelMail tests are variants on the form of open mailbox, expunge, sort by some key, and then fetch the header information for a message range. The third just lists subscribed mailboxes and gets a status for INBOX.

This seems to suggest that while maildirs are faster for certain operations, mbox is quite acceptable overall, in particular the UW-IMAP implementation (the Dovecot version seems to fall over). This is counter to the prevailing wisdom that maildirs scream. On the other hand, it seems that the functions used most often by mail clients typically involve a fair amount of sorting, something that maildirs are particularly bad at (except sorting by arrival).

Not included in this set of tests were numbers for creating a new mailbox, copying a message from INBOX to the new mailbox, expunging, copying the message back, expunging, and then deleting the new mailbox. Preliminary tests indicate that Dovecot’s maildir and UW-IMAP’s mbox implementations are about the same speed, Courier is slightly slower, and Dovecot’s mbox implementation completely falls over. This is perhaps a slightly overblown scenario, but without the mailbox creation and deletion it’s not far off from actual usage: there are a number of mail clients that will do a search of the INBOX for certain things, copy those messages to another mailbox (like, say, the spam box), and then expunge — possibly over and over.

I am still working on testing Cyrus, and I’d like to test Sun ONE as well, if I can get a server up and running for it.

[Research] Greylisting

Ran across some possibly decent greylisting sources last night.

milter-greylist looks like a good, fast greylisting implementation. The main question there is whether I can make it fit into the right spot with respect to MIMEDefang. Also, libspf2 would be useful in tuning the greylist. Ought to look into that even if I want to implement a MIMEDefang-internal greylist.

[Research] Sympa and Mailman

E-mail system upgrades are on the slate for this spring, and part of that involves upgrading the mailing list software. I’ve been reading up on the latest versions of Sympa and Mailman as the top contenders.

Sympa has some nice features over Mailman v2.0, but Mailman v2.1 (which is current) gains a lot of those. I still somewhat dislike Mailman’s binary-everything worldview (making command line fixes difficult), but they’ve added a bunch of utility programs for dealing with binary files. Sympa has some interesting LDAP integration features, but I suspect that those aren’t terribly high priority.

At this point, Mailman is in the lead, but we still need to finish looking at Sympa.

[Upgrades] NTP Changes

We’re deprecating MIR as an NTP server in preference to EIRENE as we retire MIR.

I pointed EIRENE at the same two NTP servers that MIR uses, as well as adding the *.pool.ntp.org round robin DNS servers. I’ve changed all the servers to talk to EIRENE, although in some cases we need an ntpd restart at the time that the new CNAME record is published.

[Firefighting] PacketShaper Replaced

The replacement for our dead PacketShaper arrived today, and with Aaron and Kevan’s help, I got it online.

Aaron and Kevan got it rackmounted and connected to the appropriate cables. At that point I ran through the guided basic setup to give it its network identity. With that, I was able to connect over FTP, transfer the config.ldi file, and then load the current configuration as the previous shaper had it. All in all, quite straightforward. The new unit has 512MB of RAM, which bodes well for NetFlow reports.