Recovering from IMAP crash

One morning I connect to my IMAP server and it tells me there is an I/O error. Terrified, I connect using SSH and find out the hard drive is full of backup files. This in turn seems to have caught Cyrus in the worst possible moment, and I struggle to restart it. Whenever I do, I get weird error messages like this:
Feb 21 04:35:15 invent lmtpunix[9573]: DBERROR db4: PANIC: fatal region error detected; run recovery
Feb 21 04:35:15 invent lmtpunix[9573]: DBERROR: critical database situation
Of course, I have not the slightest idea what to do. At least there is a hint to a recovery tool, no?

What To Look At

First things first: the Cyrus documentation is not as good as it should be, but not as bad as it could. Fortunately, a lot of people seem to have had similar problems, and I could garner a lot of information.

There are four places that store information relevant to this process:

1. /var/spool/imap the sacred location of the mail store (your email)
2. /usr/lib/cyrus where the tools for cyrus are located
3. /etc with the configuration files
4. /var/lib/imap the location of the corrupted db files

I had no idea what was corrupt, but got good vibes running ‘file’ on most of the db files in the last directory. Only the mailboxes.db file and the db directory seemed out of whack. That’s good, you can rebuild both easily (if you know how).

First Things First

Back up the directories listed above, at the very least the /var/spool/imap and /var/lib/imap directories! You are going to do some really invasive surgery around here, and you want to be sure you don’t lose anything permanently!

The Script

There is a trick – you have to run a script that does most of the work for you. It is listed under “More…”. Copy and paste it into an editor, save it under a convenient name, chmod +x it, and run. It should do all its work without much complaining. I got it from a Google source, that had a bunch of things wrong. After fixing paths and adding the right secret sauce, it worked just fine.

To run the script, first shut down the IMAP server, then execute. Once the script is done (which took about 10 minutes on my machine), you can restart the server.

Ensure you read the output of /var/log/mail and /var/log/messages to ensure everything is fine! The first few times, I had gotten it wrong, and the log files quickly grew again into making my system run out of space.

#!/bin/sh
#

# backup existing mailboxes.db
mv /var/lib/imap/mailboxes.db /var/lib/imap/mailboxes.db.$$
find /var/spool/imap/user -maxdepth 1 -mindepth 1 | \
while read i; do
export name=$(basename $i)
echo “user.$name default $i lrswipcda cyrus lrswipcda” >> /tmp/newmboxlist.txt
done

# we have everyone’s base directory in /tmp/newmboxlist.txt, now we
# import the new list
/usr/lib/cyrus/bin/ctl_mboxlist -u </tmp/newmboxlist.txt

# and reconstruct to get subfolders into the list
find /var/spool/imap/user -maxdepth 1 -mindepth 1 | \
while read i; do
export name=$(basename $i)
/usr/lib/cyrus/bin/reconstruct -rf user.$name
done