Mutt, Procmail (or ssmtp) & Fetchmail for personal email

Evolution is feature-heavy and terribly slow. Thunderbird is fast but but being a GUI is no good for using remotely (i.e. over a VPN).

Solution - use mutt.

Make sure you customise the below config files to reflect your usernames, email addresses, passwords etc!!!

Set up Procmail rules

This is an example ruleset that I (Oliver) have tailored for my personal filtering tastes, mainly based on Jamie's .procmailrc:

~/.procmailrc

PATH=/bin:/usr/bin
SHELL=/bin/sh
MAILDIR=$HOME/Maildir
DEFAULT=$MAILDIR/

LOGFILE=$MAILDIR/procmail.log
#VERBOSE=on


# backup
#:0 c
#backup/

####
###
## DELETE SPAM FAST!!!
#
SPAM=.Spam/

# if it's already got a spam: yes header from my upstream, then bye bye
:0:
* ^X-Spam-Status: Yes
$SPAM

# spamassassin is dog slow when doing bayesian analysis, so let's use
# bogofilter instead
# -u for update wordlists (autolearn based on classification) - not a great idea to do automatically.
# -e for embedded operation
# -p for pass-through
:0fw
| bogofilter -e -p

# put back in the queue if bogofilter fails
:0e
{ EXITCODE=75 HOST }

# put the spam away
:0
* ^X-Bogosity: Spam, tests=bogofilter
$SPAM

####

# Subversion
:0
* ^Subject:.*svn commit r
* ^TO_devel-staff
.Subversion/

# barracuda
:0
* ^Subject:.*(Daily Traffic Report|av1.puremail.com.au)
.Barracuda/


# Cron
:0
* ^Subject:.*(errors rotating logs|Auto Catchall Script Changelist)
.Cron/

:0
* ^From:.*(Cron Daemon|Anacron)
.Cron/


# Certwatch
:0
* ^Subject:.*The certificate for
* ^From:.*root
.Certwatch/


# Ubuntu
:0
* ^Subject:.*\[USN
.Ubuntu/


# Amanda
:0
* ^Subject:.*(AMANDA (MAIL|VERIFY) REPORT|AMFLUSH MAIL REPORT|AMANDA PROBLEM)
.Amanda/

# Logwatch
:0
* ^Subject:.*Logwatch for
* ^From:.*(root|logwatch)
.Logwatch/

# Osiris
:0
* ^From:.*Osiris Host Integrity System
.Osiris/

# RHN
:0
* ^Subject:.*RHN (Daily Status|Errata Alert)
.RHN/

# CFEngine
:0
* ^From:.*cfengine
.CFEngine/

# Disk related
:0
* ^From:.*Disk Usage Monitor
* ^Subject:.*Low disk space warning
.DiskStuff/

:0
* ^From:.*mdadm monitoring
.DiskStuff/


### system mail above here


### fetch addresses
#:0hc
#| lbdb-fetchaddr

# nscplus
:0
* ^Subject:.*nscplus
.Lists.nscplus/

# LARTC
:0
* ^Subject:.*LARTC
.Lists.LARTC/

# Quagga
:0
* ^Subject:.*quagga-users
.Lists.Quagga/

# Osiris
:0
* ^To:.*osiris@lists.shmoo.com
.Lists.Osiris/

# Fedora DS
:0
* ^Subject:.*Fedora-directory-users
.Lists.FDS/

# Fedora
:0
* ^From:.*fedora-legacy-announce@redhat.com
.Fedora/

# Microsoft
:0
* ^Subject:.*(Microsoft Security Bulletin|Microsoft Security Advisory|Microsoft Security Response Center Bulletin Notification|Microsoft Windows Platform News|Business Insight Newsletter|Technet Flash|Microsoft Security Newsletter|Microsoft at a Glance)
.Microsoft/

# default
:0
$DEFAULT

Set up Mutt

Again, this is mostly based off Jamie's mutt configuration, with a couple of additions notably:

  • header caching to speed up changing of directories with lots of messages
  • a couple of macros to allow you to delete messages as spam or non-spam for enhancing bogofilter's operation

~/.muttrc

#####
####
###  Anchor .muttrc
##
#    You credit for mine idea:
#    Tom Gilbert - http://www.linuxbrit.co.uk/downloads/dot.muttrc
#    Jeff Waugh - http://perkypants.org/dotfiles/muttrc.html
#

####
###
##  Boolean options
#

unset allow_8bit         # use QP or Base64 if unset.
unset arrow_cursor       # "->" as cursor (faster on slow networks).
  set ascii_chars        # Use ASCII chars for threading arrows.
unset askbcc             # ask to fill out Bcc on mesg-send?
unset askcc              # ask to fill out CC on mesg-send?
  set attach_split       # handle every single attachment instead of concat
  set autoedit           # start editor without asking for to, subject,...
  set auto_tag           # if something is tagged, run commands on tagged msgs.
unset beep               # beep on certain user errors and other errors
unset beep_new           # beep on new mail
unset bounce_delivered   # unset => remove Delivered-To: when bouncing.
  set check_new          # search for new messages in maildir/MH folders.
unset collapse_unread    # don't collapse threads with unread mails
unset confirmappend      # prompt for confirm when appending message to ex. mbox  set confirmcreate      # prompt for confirm when saving to non existing mbox
  set copy               # always keep a copy of my mails
  set delete_untag       # untag messages when marking them for deletion.
  set edit_headers       # put headers in editor when composing messages
  set encode_from        # use Quoted-Printable, if ^From in a mail.
  set fast_reply         # do not ask for to, subject,... when replying.
  set fcc_attach         # store attachments too when doing a Fcc
unset fcc_clear          # FCCs will be stored unencrypted and unsigned
  set followup_to        # Add Mail-Followup-To header.
unset force_name         # outgoing mail to be saved in folder of "login name"
  set forward_decode     # decode messages when forwarding without MIME
unset forward_decrypt    # decrypt PGP message, when mime_forward and !mime_fw_dunset forward_quote      # quote forwarded message, if not $mime_forward
  set hdrs               # nohdrs implies not to write my_hdr into mail.
unset header             # quote full header on reply
  set help               # show help in first line of the display
unset hidden_host        # when set, skip the host name part of hostname
  set honor_followup_to  # Mail-Followup-To header is honored?
unset ignore_list_reply_to    # ignore Reply-To pointing back to mailinglist
  set implicit_autoview  # use autoview, if a "copiousoutput" is in mailcap.
  set mailcap_sanitize   # remove ugly chars from mailcap % expands.
unset mark_old           # mark new articles as old, when leaving mbox
unset markers            # put a '+' at the beginning of wrapped lines.
  set menu_scroll        # scroll on last line of menu (set) or do a nextpage?
unset meta_key           # Don't interpret Alt-x as ESC x
unset metoo              # should we keep ourselves in group replies?
  set mh_purge           # remove messages marked as deleted from MH folders.
unset mime_forward_decode     # decode messages when forwarding with MIME
unset pager_stop        # Don't jump to next message at end of message.
unset pgp_replysign     # always sign reply to signed message
unset pgp_show_unusable # Show non-usable keys on the PGP key selection menu?
unset pipe_decode       # Weed headers and PGP-decode message before piping
  set pipe_split        # split tagged messages (or don't) when using "|"
unset prompt_after      # display prompt after exit from pager
unset read_only         # folders not read only
unset reply_self        # if unset, a reply to yourself implies reply to other
  set resolve           # go to next undeleted message after modifying actual
unset reverse_alias     # use realname from reverse alias lookup
  set reverse_name      # Set "From" to address, where replied mail was sent to
unset save_address      # Use full mail adress (with domain) as default folder.
unset save_empty        # remove mailbox when last message is removed.
unset save_name         # Fcc, if ~/Mail/<to> exists.
unset score             # don't do scoring
  set sig_dashes        # Don't prepend "-- \n" to the signature
  set smart_wrap        # wrap long lines at word boundary.
  set sort_re           # set: only put a child in a thread, if starts with Re:
unset status_on_top     # status line at the top or button?
unset strict_threads    # thread only by In-Reply-To/References or by Subject.
unset suspend           # Ctrl-Z is able to suspend mutt.
  set thorough_search   # MIME decode before searching.
  set tilde             # Internal Pager: ~~~~ at and of message?
unset uncollapse_jump   # don't move pointer, when uncollapsing thread
unset use_8bitmime      # run sendmail 8.8.x with -B8BITMIME to enable 8bit
unset use_domain        # add $domain to all addresses without @host
unset use_from          # generate a "From:" header (unless my_hdr From:)
  set user_agent        # Create User-Agent: Mutt/<Version> header.
unset wait_key          # Only "press a key" if return value non-zero
  set wrap_search       # Wrap search to top/bottom.
unset write_bcc         # send Bcc: line to $sendmail


####
###
##  Quadoptions
#

set abort_nosubject=no     # Abort, if message has no subject.
set abort_unmodified=yes   # Abort, if message wasn't edited.
set delete=yes             # ask for confirmation when deleting messages?
set honor_followup_to=yes  # Ask whether to honor Mail-Followup-To header
set include=yes            # quote message, when replying
set mime_forward=ask-no    # use MIME when forwarding mails.
set move=no                # Move mail-spool to $mbox?
set postpone=ask-yes       # ask for postponing when doing an Abort.
set print=ask-yes          # ask whether you really want to print.
set quit=yes               # ask before quitting.
set recall=no              # ask for using postponed message.
set reply_to=yes           # Use or ignore Reply-To.


####
###
##  String options
#

#set charset="iso-8859-1"
set default_hook="~f %s | ~t %s"
set editor="vim -c 'set tw=76 noautoindent expandtab notitle ft=mail' +"
set folder=~/Maildir/
set header_cache=/dev/shm/.headercache
set history=50
set indent_string=">"
set mail_check=5
set mbox=~/Maildir/
set mbox_type=Maildir
set pager_context=1
set pager_index_lines=0
set pgp_timeout=1000
set postponed=~/Maildir/.drafts
set query_command="lbdbq %s"
set quote_regexp="^( {0,4}-?[>|:]| {0,4}[a-z0-9]+[>|]+)+"
set read_inc=25
set realname="Oliver Hookins"
set record=~/Maildir/.sent
set reply_regexp="^((re|RE|SV|Re|r e|aw):[ \t]*)*"
#set signature="~/bin/signature |"
set simple_search="~f %s | ~C %s | ~s %s"
set sort=threads
set sort_alias=alias
set sort_aux=date-received
set sort_browser=alpha
set spoolfile=~/Maildir/
set timeout=30
set tmpdir="/tmp"
set write_inc=25
set mask="^\\.[^.]"



####
###
##  Format strings
#

set alias_format="%4n %t %-12a  %r"
set date_format="!%a %b %d, %Y at %T %Z"
set folder_format=" %N  %f  %> %d  %8s "
set forward_format="fwd: %s"
set hdr_format="%5C %{%b %d} %-15.15n  %s"
set index_format="%5C %Z %[!%d/%m] [%-15.15n] %s"
#set index_format="%Z %[!%d/%m] | %-18.18F | %4c | %s"
set pager_format="%S %n"
set status_format="-%r-Mutt: %f [Msgs:%?M?%M/?%m%?n? New:%n?%?o? Old:%o?%?d? Del:%d?%?F? Flag:%F?%?t? Tag:%t?%?p? Post:%p?%?b? Inc:%b? %?l? %l?]--%>-(%P)---"


####
###
##  Colour settings
#

#  match ftp addresses:
color body    cyan black "ftp://[^ ]*"
#  match http addresses:
color body    brightgreen black "[[:alnum:]][-+.#_[:alnum:]]*@[-+.[:alnum:]]*[[:alnum:]]"

color normal  white black # pager body

#  Coloring quoted text - coloring the first 7 levels:
color quoted    cyan black
color quoted1   yellow black
color quoted2   red black
color quoted3   green black
color quoted4   cyan black
color quoted5   yellow black
color quoted6   red black
color quoted7   green black

color signature brightred black
color status    white blue
color tilde     blue black
color tree      brightmagenta black
color underline yellow black

#  Colorize smileys:  :-)  ;-)  :-/  :-(
color body  yellow black "[;:]-[)/(|]"
color body  yellow black "[;:][)/(|]"
color body  green black "[[:alpha:]]\+://[^ ]*"

color header red black '^Subject.*'
color header yellow black '^X-Debian-PR-Package.*'
color header green black 'From:.*'

#  Remember:  Order counts!  The last matching rule "wins".
#  So we'll start with some general rules first:
#
color index yellow black ~N      # New
color index yellow black ~O      # Old

# mail to myself (personal mail):
#color index brightgreen black '~p'

color index magenta black ~F
color index blue black ~T
color index red black ~D


####
###
##
#

unignore *
ignore content deliver errors-to importance in-reply list mail message-id
ignore mime precedence priority received references resent return-path sender
ignore x-accept x-authentication x-beenthere x-cid x-complaints x-dispatcher
ignore x-egroups x-envelope x-fingerprint x-juno x-keywords x-loop
ignore x-mailing x-mailman x-mime x-ms x-news x-origina x-orcpt envelope
ignore x-priority x-sender x-smtp x-trace x-uid x-unsent x-ftn
ignore old-x-envelope x-gateway old-return-path x-virus-scanned
ignore x-spam x-return-path x-authenticated-sender x-md x-list x-scanner
ignore "from " status lines x-status >from x-received x-bogosity
# awesome barracuda spam headers
ignore x-asg x-barracuda
ignore domainkey-signature
ignore x-brightmail x-ironport

hdr_order Date From: Subject: Reply-To: To: Cc: Bcc:

unmy_hdr *

#my_hdr X-Message-Flag: Magic 8-Ball says Outlook Not Good.
my_hdr X-Mailer: anchor v0.0
#my_hdr Reply-By: `date --date="-30 days" -R`
my_hdr X-No-CC: Please respect the setting of my Mail-Followup-To header



####
###
##  Autoview settings
#

auto_view text/html text/enriched application/x-gunzip
auto_view application/postscript image/gif application/x-tar-gz
auto_view application/pgp-signature

alternative_order text/enriched text/plain text


####
###
##   Keybindings
#

macro   index   <left>  ':mailboxes `find ~/Maildir -maxdepth 1 -type d ! -regex '.*draft.*' -printf "=%P "`<enter>c?<toggle-mailboxes>'        "View Mailboxes"bind    index   <right> display-message
macro   index   I       "c!\n"                  "Go to Inbox"
bind    index   <space> next-unread
#macro  index   r       "<enter-command>unset wait_key\n<pipe-entry>bogofilter -n\n<enter-command>set wait_key\n<reply>" "Reply to message and mark as non-spam"#macro  index   g       "<enter-command>unset wait_key\n<pipe-entry>bogofilter -n\n<enter-command>set wait_key\n<group-reply>" "Group reply to message and mark as non-spam"
#macro  index   L       "<enter-command>unset wait_key\n<pipe-entry>bogofilter -n\n<enter-command>set wait_key\n<list-reply>" "List reply to message and mark as non-spam"
#macro  index   S       "<enter-command>unset wait_key\n<pipe-entry>bogofilter -s\n<enter-command>set wait_key\n<save-message>=.spambox\n" "Mark as spam and save to spambox"
#macro   index   U       "<enter-command>unset wait_key\n<pipe-entry>bogofilter -Sn\n<pipe-entry>formail -I X-Bogosity|procmail\n<enter-command>set wait_key\n<delete-message>" "Mark as non-spam and refilter to correct mailbox"
macro   index   M       "<tag-pattern>.*\r<tag-prefix><clear-flag>N<untag-pattern>.*\r" "Mark all messages in this mailbox as read"
#macro  index   \Cr     <read-thread><next-new-then-unread> "Mark thread as read and skip to next message"

macro   query   <left>  "q"                     "Leave"
macro   query   i       "q"                     "Leave"
macro   query   I       "q"                     "Leave"
macro   query   <enter> "<mail>\n"              "Mail"

macro   pager   I       "c!\n"          "Inbox"
bind    pager   <up>    previous-entry
bind    pager   <down>  next-entry
bind    pager   <left>  exit
bind    pager   <right> view-attachments
macro   pager   S       "<enter-command>unset wait_key\n<pipe-entry>bogofilter -s -l\n<enter-command>set wait_key\n<save-message>=.spambox\n" "Mark as spam and save to spambox"
macro   pager   U       "<enter-command>unset wait_key\n<pipe-entry>bogofilter -n -l\n<enter-command>set wait_key\n<save-message>=\n" "Mark as non-spam and save to INBOX"

#macro  pager   K       <pipe-message>~/bin/plonk.pl<enter> "*plonk*"
macro   pager   \Cr     <read-thread><next-new-then-unread><display-message>
macro   pager   \Ct     <pipe-message>"festival --tts"<enter>

bind    attach  <left>  exit
bind    attach  <right> view-attach

bind    browser <left>  exit
bind    browser <right> select-entry

# start fetchmail
macro   pager   <esc>f  "<enter-command>unset wait_key\n<shell-escape>fetchmail -q\n<shell-escape>fetchmail\n<enter-command>set wait_key\n" "Restart fetchmail"
macro   browser <esc>f  "<enter-command>unset wait_key\n<shell-escape>fetchmail -q\n<shell-escape>fetchmail\n<enter-command>set wait_key\n" "Restart fetchmail"
macro   index   <esc>f  "<enter-command>unset wait_key\n<shell-escape>fetchmail -q\n<shell-escape>fetchmail\n<enter-command>set wait_key\n" "Restart fetchmail"

# Deletions thru bogofilter
macro index d "<enter-command>unset wait_key\n\<pipe-entry>bogofilter -n\n\<enter-command>set wait_key\n\<delete-message>" "delete message as non-spam"
macro index \ed "<enter-command>unset wait_key\n\<pipe-entry>bogofilter -s\n\<enter-command>set wait_key\n\<delete-message>" "delete message as spam"

# view X-Face: headers
macro   pager   <esc>p  |view-x-face\n
macro   index   <esc>p  |view-x-face\n

####
###
##  Hooks  (YARR! P-)
#

# default hooks, override them below
folder-hook . 'unset strict_threads'
#message-hook . 'set attribution = "This one time, at band camp, %n wrote:"'
folder-hook . 'set signature="~/.signature"'
send-hook   . 'set signature="~/.signature"'

# I want forwards to be saved into the forwards mailbox.  See my
# procmailrc recipe for forwards, as well.
save-hook '~s f(w:\|wd\|orward)' ~/Maildir/.forwards

# Loose threading looks terrible in my inbox, so only do it if
# it's guaranteed to work
folder-hook '!' 'set strict_threads'
folder-hook '.*sent' 'set strict_threads'
folder-hook '.*drafts' 'set strict_threads'

####
###
##  Mailboxes
#

mailboxes `find ~/Maildir/ -maxdepth 2 -type d ! -regex '.*draft.*' ! -regex '.*spambox.*' -printf "=%P "`

#unlists *
#unsubscribe *

#lists `find ~/Maildir -maxdepth 1 -type d -regex ".*list.*" -printf "%P " | sed -e 's/list//g' -e 's/cse//g' | tr -d '.'`
#subscribe `find ~/Maildir -maxdepth 1 -type d -regex ".*list.*" ! -regex ".*cse.*" -printf "%P " | sed 's/list//g' | tr -d '.'`

Set up fetchmail

You will need to set up fetchmail to retrieve your messages. It passes the messages into procmail which then puts them in the maildir directories mutt reads. Previously this page described how to set this up on your workstation using pop3 (which is inherently unsecure). A much better way is to use fetchmail's smarts to grab mail via IMAP over SSH.

~/.fetchmailrc

# Run in the background, waking up every 60 seconds
set daemon 60

defaults protocol IMAP
  # set 5 minute timeout on response from a server
  timeout 600
  # expunge mailboxes every 50 messages retrieved
  expunge 50
  # don't keep mail on the server, retrieve it all once
  fetchall
  # my local username for delivery to the MDA here
  is YOURUSERNAME here
  # go directly to procmail, rather than delviering through the SMTP server locally
  mda procmail

# anchor.net.au mail stored on mail.anchor.net.au
poll mail.anchor.net.au protocol IMAP
  # use an SSH tunnel to starb and use imapd directly
  plugin "ssh YOURUSERNAME@starboard /usr/sbin/imapd" auth ssh
  # login username
  username YOURUSERNAME
  # don't close the connection, get notified of new mail by the imap server
  idle

Things you can change: remove mda procmail to not use procmail, change timeout and expunge limits (make expunge less frequent to put less IO load on server) change your username obviously.

SSH keys are obviously a must so that you don't need to put your password in plaintext in the config file.

To fully automate this, have fetchmail started on reboot. Edit your per-user crontab with crontab -e:

@reboot     /usr/bin/fetchmail

Then just run fetchmail once after setting up your .fetchmailrc to get it running daemonized.

Set up Postfix or SSMTP

You will need an MTA running on the local machine in order to have any outbound email working. Mutt is an MUA only and doesn't handle sending email any further than calling the sendmail binary and forgetting about the email thereafter.

Configure your local MTA to relay out, if applicable. On Postfix this is done as follows in main.cf:

relayhost = mail.anchor.net.au

Start postfix

A nice simple alternative to Postfix is SSMTP. There's an Ubuntu package for it (ssmtp). Edit /etc/ssmtp/ssmtp.conf The following should do the trick..

mailhub=smtp.anchor.net.au
rewriteDomain=anchor.net.au

Set up Bogofilter

You will need bogofilter installed if you want to use spam filtering as I have in the procmail and mutt configs above. You could also use spamassassin but it can be an order of magnitude slower.

The bogofilter database needs to be initialised with at least one sample email, so do this before piping anything through procmail rules that use bogofilter. I would recommend initialising the database with a good sampling of both spam and ham. If you only classify spam messages, the database will be unbalanced and it will classify most emails as "Unsure" and not do anything with them.

bogofilter -B -s ~/Maildir/.Spam/cur/*
bogofilter -B -n ~/Maildir/.Somefolder/cur/*

is a good way to train up bogofilter initially with some spam and ham.

Import your old mail

If you are using Thunderbird, follow these rough steps:

  • Configure your mail accounts and set the incoming mail server to some nonsense value so you don't accidentally retrieve more email during import into mutt.
  • Compact all your folders to purge any deleted messages from the mail store files
  • Run something similar to this script to pipe all your messages through procmail and into mutt:

cd "/home/oliver/.mozilla-thunderbird/scso5nj1.default/Mail/Local Folders"

files="Amanda Barracuda Certwatch CFEngine Cron DiskStuff Fedora Inbox Lists  Logwatch Microsoft Osiris Other RHN SSL Subversion Ubuntu Wiki Lists.sbd/LARTC Lists.sbd/Quagga Lists.sbd/nscplus Lists.sbd/Osiris Lists.sbd/FDS"

cat $files | formail -s procmail
cat "Lists.sbd/RT devel" | formail -s procmail
cat "Cron.sbd/Anchor Updates" | formail -s procmail

I have had to use a couple of separate lines for processing since some folders had spaces in their names. This script does not import the Sent folder.

  • Enter mutt and confirm everything is ok.
  • Then start up fetchmail and grab the most recent emails.
  • You will need to go through all the folders and mark everything as read, as each email will be treated as new. The above .muttrc defines a macro M to do this.

Alternatively you can manually import emails by browsing to the mail store file in mutt (which will open it as an mbox mailbox), then using the copy function to copy the messages into a Maildir in your mutt message store.

Procmail Hints

Troubleshooting Unix username with real email address

If your Unix username differs from the real email username (i.e. mokona vs mokona.apapa@anchor.com.au), you might need to tell postfix to do "canonical mappings", otherwise sent mail might have an erroneous return-path (mokona@anchor.com.au). To overcome this annoyance, follow these steps:

  • Edit /etc/postfix/main.cf and add the following line: sender_canonical_maps = hash:/etc/postfix/canonical
  • Edit /etc/postfix/canonical and add a line with your unix username followed by the real email address (i.e. mokona mokona.apapa@anchor.com.au)

  • As root, convert canonical into a postfix database: sudo postmap /etc/postfix/canonical
  • Restart postfix

More info here: [http://www.postfix.org/rewrite.html]