ISP-Style Mail Server using Postfix, Dovecot and MySQL

Abstract

You surely know the big web hosters that allow you to rent a domain and use it to receive emails. If you have a SMGL computer connected permanently to the internet you can offer the same service. You do not even need to have a fixed IP address thanks to dynamic DNS services like dyndns.org. All you need is this document, a pot of delicious tea and a little time. When you are done with the setup your server will be able to...

• receive and store emails for your users
• let your users retrieve the email through IMAP and POP3 - even with SSL to encrypt to connection
• receive and forward (relay) email for your users if they are authenticated
• offer a webmail interface to read emails in a web browser
• detect most spam emails and filter them out or tag them

This document is not a simple copy-and-paste tutorial. The intention is rather to make you understand the different components that you are using. There is deliberately no script that will do all the setup automatically. But in the end you will be skilled enough to debug problems yourself. If you feel you need help with your setup then read the hints on Troubleshooting. The setup in this tutorial has been tested very thoroughly by several people. Unlike many other Postfix tutorials on the internet this is already the third edition. Writing this tutorial took a month of work so these are not just quick draft notes thrown together but a consistent document guiding you.

The configuration described here is not very complicated but still needs to be done carefully. You are expected to have at least basic knowledge of:

• MySQL (creating a database, granting access to users, basic SQL queries)
• SMTP (what it is and what a basic SMTP dialog looks like)
• POP3, IMAP (what they do and what the differences are)

• basic Postfix configuration (understand the default settings in your main.cf, have read through the basic configuration document and know that your mail log file is at /var/log/mail/current)

• SourceMage GNU/Linux (general system administration, using a text editor, reading log files)

The Components

The configuration described here uses these components:

• Postfix for receiving incoming emails from the internet and storing them to the users' mailboxes on the harddisk. And for receiving emails from your users that are sent out to the internet (relaying)

• Dovecot to allow your users to get their emails into their email client through POP3 and IMAP

• Horde/IMP as a webmail interface (although any IMAP capable webmail interface will do)

• MySQL as the database system that stores information about your domains, the user accounts and email forwardings

• DSPAM for scanning incoming emails for viruses, spam and unwanted attachments

• ClamAV to enable the message virus-scanning capabilities of DSPAM

• Metalog for a syslog if you don't already have one (this document assumes you use metalog, though -- adjust accordingly)

Virtual Domains in a Database

Have you ever wondered how internet service providers (ISPs) maintain their thousands of domains they receive mail for? There is surely nobody entering all these domains and aliases into a *main.cf* text configuration file manually. Postfix offers a better way to handle such domains and accounts by means of *virtual domains* and *virtual users*.

• virtual_mailbox_maps Postfix looks for the mapping of virtual users to virtual mailboxes in the *virtual_mailbox_maps* setting. The left-hand side (LHS) of the mapping is the email address and the right-hand side (RHS) is the location of the mailbox or maildir on your harddisk.

In addition to *local users* (those being listed in your /etc/passwd) Postfix can handle any number of *virtual users* on *virtual domains*. Virtual users cannot log into your computer and they have neither a user ID nor a home directory. They just exist in the database. But if you connect Postfix to the database those users can suddenly receive emails. Postfix can work on a list of those virtual users and deliver emails to any directory you like. This would look like:

You also need to tell Postfix which virtual domains you want to use. If a domain is not used on your system then Postfix will reject emails. This would be a list of domains:

• virtual_mailbox_domains Postfix checks which virtual domains you want to receive mail for by looking at the *virtual_mailbox_domains* list. Since Postfix just has a notion of mappings (two columns) a list is (ab)using a mapping where the right-hand side (RHS) just contains any string.

So far the information could have been written into a text file. But you can imagine that this will become confusing quickly. Fortunately Postfix can also get this information from other sources like LDAP or SQL databases. So I am using SQL database tables here.

You have now seen that a *mapping* assigns one value to another. If you query a database you need to tell Postfix which two columns you mean. This is done through 'cf' files as documented at http://www.postfix.org/MYSQL_README.html or through "man 5 mysql_table".

Example file:

# Information on how to connect to your MySQL server
user = someone
password = some_password
hosts = 127.0.0.1

# The database name on the servers.
dbname = mailserver

# The SQL query template.
query = SELECT destination FROM virtual_aliases WHERE source='%s'

This file defines the way that Postfix can access data from your database. It would be suitable for a *virtual_alias_maps* mapping. Imagine you saved the above lines into a file /etc/postfix/mysql-virtual-alias-maps.cf. Then the following line in your main.cf would make Postfix query the database:

    virtual_alias_maps = mysql:/etc/postfix/mysql-virtual_alias_maps

How does this work? Imagine that Postfix is about to send an email to john@doe.net and wants to check the virtual alias map. Postfix then opens up a connection to the MySQL server at the IP address *127.0.0.1* and authenticates to the MySQL server with the username *someone* and the password *some_password*. It selects the database *mailserver* and finally runs a query:

    SELECT destination FROM virtual_aliases WHERE source='john@doe.net'

Let us assume this query returns several results:

• jack@example.com

• jeff@example.com

• kerstin@example.com

That would be equal if you used a text file with aliases like this:

    john@doe.net jack@example.com, jeff@example.com, kerstin@example.com

We hope you get the idea of mappings in databases so far.

Virtual domains versus local domains

It is important to understand the three different kinds of domains that Postfix knows. Most of the "it does not work" emails result from people mixing virtual and local domains. A domain is either a...

    mydestination = localhost

Step 1: Installing the needed SMGL packages

Now that you hopefully understood the basic concepts of virtual domains, mappings and database accesses it is time to get your hands dirty. Become *root* on your server and make sure that your /etc/hostname contains the host name *without* the domain part. The file /etc/mailname is supposed to contain the fully-qualified host name *with* the domain part.

Your /etc/hosts likely also needs to be fixed. Run hostname --fqdn and see if you get the fully-qualified hostname. If you just get the hostname without the domain please check that your /etc/hosts file has the fully-qualified hostname first in the list.

Wrong:

    20.30.40.50   mailserver42 mailserver42.example.com

Right:

    20.30.40.50   mailserver42.example.com mailserver42

• $> and mysql> During the tutorial I will use the $> notation to indicate commands you are supposed to enter in a shell. Sections that start with mysql> need to be entered as SQL commands.

Then begin by installing Postfix with MySQL map support:

    $> cast postfix

This will also install just postfix, and when asked if you want to support MySQL, answer [y], and MySQL will be cast as well. When the mysql spell is casting, make sure you answer [n] to "Build client only?"

As you probably want to offer POP3 and IMAP services to your users you need to install Dovecot, and answer [y] to imap and pop3:

    $> cast dovecot

Your users may like it if you filter out spam and (windows) viruses for them. ClamAV is doing a good job here:

    $> cast clamav

ClamAV isn't used directly, but rather, as a dependency for DSPAM to support virus scanning. You want to install these packages:

    $> cast dspam

Next you will want to install the OpenSSL package so that you can later create a proper SSL certificate letting your users use your mail server securely:

    $> cast openssl

If you intend to offer a webmail service I can recommend the IMP component of Horde. IMP depends on the framework provided by Horde, so it will be cast. Horde requires a web server, so you will be prompted with a list of servers to install. I recommend choosing the apache22 spell when prompted. Type:

    $> cast imp

As your control information for Postfix will be stored in a MySQL database you may want to install the PhpMyAdmin software that allows you to administer the database and its data in your web browser:

    $> cast phpmyadmin

You may want to test the mail server with the simple "telnet" client later:

    $> cast net-telnet

The console-based *mutt* email client lets you read mail from mailboxes directly from the hard disk. It will be helpful for testing the configuration. And it's even a very powerful IMAP email client that many people user as their main mail program. Maybe you start to like it, too. You should install it:

    $> cast mutt

Step 2: Create the database and user

Now it is time to create the MySQL database and its tables. If you are experienced in using MySQL you can enter SQL statements on the 'mysql' command line. Alternatively you may use *phpmyadmin* by pointing your browser at http://yourmailserver/phpmyadmin

If you just installed your MySQL server you will be able to login as user 'root' with an empty password. Set a new password for that account now. In the shell you need to run:

    $> mysqladmin password root2007

• Replace 'root2007' by a more secure password. Install and use 'pwgen' if you do not feel creative. I will use 'root2007' throughout the tutorial though. Replace it by your own choice where appropriate.

Then create the database. Call it 'mailserver' (you will be asked for the above password):

    $> mysqladmin -p create mailserver

For security reasons you will want to create another less privileged MySQL user account that your mail server will use. Connect to your database:

    $> mysql -p

When you see the mysql> prompt enter the following SQL statement to grant the appropriate privileges:

mysql>
    GRANT SELECT ON mailserver.*
    TO mailuser@localhost
    IDENTIFIED BY 'mailuser2007';
    exit

This will create a user called 'mailuser' that has only the privilege to select/read data from the database but not to alter it. If you want to add or alter data in the database either use the 'root' account or create another account for that purpose. The password 'mailuser2007' is just an example. Please replace it by a more decent password.

Step 3: Create the database tables

Inside the newly created database you will have to create tables that store information about domains, forwardings and the users' mailboxes. Connect to the MySQL again and choose the 'mailserver' database:

    $> mysql -p mailserver

You will see the mysql> prompt again. First create a table for the list of virtual domains that you want to host:

mysql>
    CREATE TABLE `virtual_domains` (
    id INT NOT NULL AUTO_INCREMENT PRIMARY KEY,
    name VARCHAR(50) NOT NULL
    ) ENGINE = InnoDB;

The next table contains information on the actual user accounts. Every user has a username and password. It is used for accessing the mailbox by POP3 or IMAP, logging into the webmail service or to send mail if they are not in your local network. As users tend to easily forget things the user's email address is also used as the login username. Create the users table:

mysql>
    CREATE TABLE `virtual_users` (
    id int(11) NOT NULL AUTO_INCREMENT PRIMARY KEY,
    domain_id INT(11) NOT NULL,
    user VARCHAR(40) NOT NULL,
    password VARCHAR(32) NOT NULL,
    CONSTRAINT UNIQUE_EMAIL UNIQUE (domain_id,user),
    FOREIGN KEY (domain_id) REFERENCES virtual_domains(id) ON DELETE CASCADE
    ) ENGINE = InnoDB;

And a table is needed for aliases (email forwardings) from one account to another:

mysql>
    CREATE TABLE `virtual_aliases` (
    id int(11) NOT NULL AUTO_INCREMENT PRIMARY KEY,
    domain_id INT(11) NOT NULL,
    source VARCHAR(20) NOT NULL,
    destination VARCHAR(80) NOT NULL,
    FOREIGN KEY (domain_id) REFERENCES virtual_domains(id) ON DELETE CASCADE
    ) ENGINE = InnoDB;

And finally a table is needed to store the quota dictionary for Dovecot:

mysql>
    CREATE TABLE quota (
    username VARCHAR(255) NOT NULL,
    path VARCHAR(100) NOT NULL,
    current INT(11) default NULL,
    PRIMARY KEY  (username)
    ) ENGINE=InnoDB;

You wonder about the foreign keys? They express that entries in the virtual_aliases and virtual_users tables are connected to entries in the virtual_domains table. This will keep the data in your database consistent because you cannot create virtual aliases or virtual users that are not connected to a virtual domain. And you avoid redundancy because you just store the domain name once - in the virtual_domains table - and nowhere else. The suffix 'ON DELETE CASCADE' means that if you delete a row from the referenced table that the deletion will also be done on the current table automatically. So you do not leave orphaned entries accidentally. Imagine that you do not host a certain domain any longer. You can remove the domain entry from the virtual_domains table and all dependent/referenced entries in the other tables will also be removed. This approach is also called a *normalized* database.

An example of the data in the tables:

The email addresses of these three users in the database would be:

• john@example.com

• steve@example.com

• kerstin@foobar.org

Let us add a simple alias:

This will make the mail for steve@example.com be forwarded to steve.miller@gmail.com. And the mail for kerstin@foobar.org is forwarded to both kerstin42@yahoo.com and kerstin@mycompany.com

Do not be scared if this way looks incredibly complicated. Such a database is not supposed to be maintained by manual SQL commands. But it is still important that you understand the schema of the database tables. (A web interface for maintaining this information is alreadj in the making.)

Step 4: Create the database mapping files

    user = mailuser
    password = mailuser2007
    hosts = 127.0.0.1
    dbname = mailserver
    query = SELECT 1 FROM virtual_domains WHERE name='%s'

    $> postconf -e virtual_mailbox_domains=mysql:/etc/postfix/mysql-virtual-mailbox-domains.cf

    $> mysql -p mailserver

mysql>
    INSERT INTO virtual_domains (id, name) VALUES (1, 'example.com');
    exit

    $> postmap -q example.com mysql:/etc/postfix/mysql-virtual-mailbox-domains.cf

    $> groupadd -g 5000 vmail
    $> useradd -g vmail -u 5000 vmail -d /home/vmail -m

    $> postconf -e virtual_uid_maps=static:5000
    $> postconf -e virtual_gid_maps=static:5000

mysql>
    INSERT INTO virtual_users (id, domain_id, user, password)
    VALUES (1, 1, 'john', MD5('summersun'));
    exit

mysql>
    SELECT CONCAT(virtual_users.user, '@', virtual_domains.name) AS email
    FROM virtual_users
    LEFT JOIN virtual_domains ON virtual_users.domain_id=virtual_domains.id;

    +------------------+
    | email            |
    +------------------+
    | john@example.com |
    +------------------+

mysql>
    SELECT CONCAT(virtual_users.user, '@', virtual_domains.name) AS email,
           virtual_users.password
    FROM virtual_users
    LEFT JOIN virtual_domains ON virtual_users.domain_id=virtual_domains.id;

    +------------------+----------------------------------+
    | email            | password                         |
    +------------------+----------------------------------+
    | john@example.com | 14cbfb845af1f030e372b1cb9275e6dd |
    +------------------+----------------------------------+

mysql>
  CREATE VIEW view_users AS
  SELECT CONCAT(virtual_users.user, '@', virtual_domains.name) AS email,
    virtual_users.password, virtual_users.quota
  FROM virtual_users
  LEFT JOIN virtual_domains ON virtual_users.domain_id=virtual_domains.id;

mysql>
    SELECT * FROM view_users;

    +------------------+----------------------------------+
    | email            | password                         |
    +------------------+----------------------------------+
    | john@example.com | 14cbfb845af1f030e372b1cb9275e6dd |
    +------------------+----------------------------------+

    user = mailuser
    password = mailuser2007
    hosts = 127.0.0.1
    dbname = mailserver
    query = SELECT 1 FROM view_users WHERE email='%s'

    $> postconf -e virtual_mailbox_maps=mysql:/etc/postfix/mysql-virtual-mailbox-maps.cf

    $> postmap -q john@example.com mysql:/etc/postfix/mysql-virtual-mailbox-maps.cf

mysql>
    INSERT INTO virtual_aliases (id, domain_id, source, destination)
    VALUES (1, 1, 'john', 'john@example.com'),
           (2, 1, 'john', 'jmiller@gmail.com');

mysql>
    CREATE VIEW view_aliases AS
    SELECT CONCAT(virtual_aliases.source, '@', virtual_domains.name) AS email,
           destination
    FROM virtual_aliases
    LEFT JOIN virtual_domains ON virtual_aliases.domain_id=virtual_domains.id;

mysql>
    SELECT * FROM view_aliases;

    +------------------+-------------------+
    | email            | destination       |
    +------------------+-------------------+
    | john@example.com | john@example.com  |
    | john@example.com | jmiller@gmail.com |
    +------------------+-------------------+

    user = mailuser
    password = mailuser2007
    hosts = 127.0.0.1
    dbname = mailserver
    query = SELECT destination FROM view_aliases WHERE email='%s'

    $> postmap -q john@example.com mysql:/etc/postfix/mysql-virtual-alias-maps.cf

    john@example.com,jmiller@gmail.com

    user = mailuser
    password = mailuser2007
    hosts = 127.0.0.1
    dbname = mailserver
    query = SELECT email FROM view_users WHERE email='%s'

    $> postmap -q john@example.com mysql:/etc/postfix/mysql-email2email.cf

    john@example.com

    $> postconf -e virtual_alias_maps=mysql:/etc/postfix/mysql-virtual-alias-maps.cf,mysql:/etc/postfix/mysql-email2email.cf

    chgrp postfix /etc/postfix/mysql-*.cf
    chmod u=rw,g=r,o= /etc/postfix/mysql-*.cf

Step 5: Deliver emails through the Dovecot LDA

Postfix comes with a mail delivery agent (MDA) called "virtual" that will usually deliver emails to virtual mailboxes. But Dovecot comes with an own (local) delivery agent also called "Dovecot LDA" that offers more functionality. To make Postfix use that agent you will have to add a service to your /etc/postfix/master.cf:

    dovecot   unix  -       n       n       -       -       pipe
        flags=DRhu user=vmail:vmail argv=/usr/lib/dovecot/deliver -d ${recipient}

Also make Postfix use that service for virtual delivery lines to your /etc/postfix/main.cf:

    $> postconf -e virtual_transport=dovecot
    $> postconf -e dovecot_destination_recipient_limit=1

So far this will make Postfix pass on incoming emails to virtual users to the /usr/lib/dovecot/deliver program. Now it is time to configure Dovecot.

Step 6: Configure Dovecot

Let us configure Dovecot which provides both a POP3 and an IMAP service. The configuration files for Dovecot is found under /etc/dovecot. Start with the...

    protocols = imap imaps pop3 pop3s

    disable_plaintext_auth = no

    mail_location = maildir:/home/vmail/%d/%n

    namespace private {
        separator = .
        prefix = INBOX.
        inbox = yes
    }

    mechanisms = plain login

    passdb sql {
        args = /etc/dovecot-sql.conf
    }

    userdb sql {
        args = /etc/dovecot-sql.conf
    }

    socket listen {
        master {
            path = /var/run/dovecot/auth-master
            mode = 0600
            user = vmail
        }

        client {
            path = /var/spool/postfix/private/auth
            mode = 0660
            user = postfix
            group = postfix
        }
    }

    protocol lda {
        log_path = /home/vmail/dovecot-deliver.log
        auth_socket_path = /var/run/dovecot/auth-master
        postmaster_address = postmaster@example.com
        mail_plugins = cmusieve
        global_script_path = /home/vmail/globalsieverc
    }

    driver = mysql
    connect = host=127.0.0.1 dbname=mailserver user=mailuser password=mailuser2007
    default_pass_scheme = PLAIN-MD5
    password_query = SELECT email as user, password FROM view_users WHERE email='%u';

    dovecot: Dovecot v1.0.rc15 starting up
    dovecot: auth-worker(default): mysql: Connected to 127.0.0.1 (mymailserver)

    $> chgrp vmail /etc/dovecot.conf
    $> chmod g+r /etc/dovecot.conf

Step 7: Send a test mail through TELNET

    $> telnet localhost smtp

    Trying 127.0.0.1...
    Connected to localhost.
    Escape character is '^]'.
    220 mailtest ESMTP Postfix

    ehlo example.com

    250-my-new-mailserver
    250-PIPELINING
    250-SIZE 10240000
    250-VRFY
    250-ETRN
    250-STARTTLS
    250-ENHANCEDSTATUSCODES
    250-8BITMIME
    250 DSN

    mail from:<steve@example.com>

    250 2.1.0 Ok

    rcpt to:<john@example.com>

    250 2.1.5 Ok

    data

    354 End data with <CR><LF>.<CR><LF>

    Hi John,

    just wanted to drop you a note.
    .

    250 2.0.0 Ok: queued as A9D64379C4

    quit

    postfix/smtpd[...]: connect from localhost[127.0.0.1]
    postfix/smtpd[...]: 5FF712A6: client=localhost[127.0.0.1]
    postfix/cleanup[...]: 5FF712A6: message-id=<...>
    postfix/qmgr[...]: 5FF712A6: from=<steve@example.com>, size=364, nrcpt=1 (queue active)
    postfix/pipe[...]: 5FF712A6: to=<john@example.com>, relay=dovecot, ..., status=sent (delivered via dovecot service)
    postfix/qmgr[...]: 5FF712A6: removed
    postfix/smtpd[...]: disconnect from localhost[127.0.0.1]

    $> cd /home/vmail/example.com/john
    $> find
    .
    ./cur
    ./new
    ./new/1179521979.V801I2bbf7M15352.mailtest
    ./tmp

    $> mutt -f .

    q:Quit  d:Del  u:Undel  s:Save  m:Mail  r:Reply  g:Group  ?:Help
       1 N   May 18 steve@example.c (0.1K)

    From: steve@example.com
    To: undisclosed-recipients: ;

    Hi John,

    just wanted to drop you a note.

Step 8: Test fetching emails with IMAP and POP3

John will surely prefer to read his mail in a comfortable mail program. So he needs a way to get access to his mailbox. Two protocols come to play here:

• POP (Post Office Protocol) is a simple protocol that lets you fetch email from a single mailbox. It is usually used to get all emails and then delete them on the server.
• IMAP (Internet Messaging Application Protocol) is also used to fetch email but you can maintain different mailboxes. The *inbox* is where your incoming emails are stored. But users can move emails to different directories. IMAP is useful when you want to access your email from different locations without losing mail because you fetched it from another location.

    $> telnet localhost pop3

    Trying 127.0.0.1...
    Connected to localhost.
    Escape character is '^]'.
    +OK Dovecot ready.

    user john@example.com

    +OK

    pass summersun

    +OK Logged in.

    list

    +OK 1 messages:
    1 474
    .

    retr 1

    +OK 474 octets
    Return-Path: <steve@example.com>
    X-Original-To: john@example.com
    Delivered-To: john@example.com
    Received: from example.com (localhost [127.0.0.1])
        by ... (Postfix) with ESMTP id 692DF379C7
        for <john@example.com>; Fri, 18 May 2007 22:59:31 +0200 (CEST)
    Message-Id: <...>
    Date: Fri, 18 May 2007 22:59:31 +0200 (CEST)
    From: steve@example.com
    To: undisclosed-recipients:;

    Hi John,

    just wanted to drop you a note.
    .

    quit

    +OK Logging out.
    Connection closed by foreign host.

    $> mutt -f imaps://john@example.com@localhost

    $> telnet localhost imap2

    Trying 127.0.0.1...
    Connected to localhost.
    Escape character is '^]'.
    * OK Dovecot ready.

    1 login john@example.com summersun

    1 OK Logged in.

    2 list "" "*"

    * LIST (\HasNoChildren) "." "INBOX"
    2 OK List completed.

    3 select "INBOX"

    * FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
    * OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted.
    * 1 EXISTS
    * 0 RECENT
    * OK [UIDVALIDITY 1180039205] UIDs valid
    * OK [UIDNEXT 3] Predicted next UID
    3 OK [READ-WRITE] Select completed.

    4 fetch 1 all

    * 1 FETCH (FLAGS (\Seen) INTERNALDATE .........
    4 OK Fetch completed.

    5 fetch 1 body[]

    * 1 FETCH (BODY[] {474}
    Return-Path: <steve@example.com>
    X-Original-To: john@example.com
    Delivered-To: john@example.com
    Received: from example.com (localhost [127.0.0.1])
            by ... (Postfix) with ESMTP id 692DF379C7
            for <john@example.com>; Fri, 18 May 2007 22:59:31 +0200 (CEST)
    Message-Id: <...>
    Date: Fri, 18 May 2007 22:59:31 +0200 (CEST)
    From: steve@example.com
    To: undisclosed-recipients:;

    Hi John,

    just wanted to drop you a note.
    )
    5 OK Fetch completed.

    6 logout

    * BYE Logging out
    6 OK Logout completed.
    Connection closed by foreign host.

    $> mutt -f imaps://john@example.com@localhost

    $> openssl req -new -x509 -days 3650 -nodes -out /etc/ssl/certs/dovecot.pem \
        -keyout /etc/ssl/private/dovecot.pem

    Generating a 1024 bit RSA private key
    .........++++++
    ............................++++++
    writing new private key to '/etc/ssl/certs/dovecot.pem'
    -----
    You are about to be asked to enter information that will be incorporated
    into your certificate request.
    What you are about to enter is what is called a Distinguished Name or a DN.
    There are quite a few fields but you can leave some blank
    For some fields there will be a default value,
    If you enter '.', the field will be left blank.
    -----
    Country Name (2 letter code) [AU]:DE
    State or Province Name (full name) [Some-State]:Hamburg
    Locality Name (eg, city) []:Hamburg
    Organization Name (eg, company) [Internet Widgits Pty Ltd]:workaround.org
    Organizational Unit Name (eg, section) []:
    Common Name (eg, YOUR name) []:mailtest.workaround.org
    Email Address []:postmaster@workaround.org

Step 9: Authenticated SMTP

It is important that you understand the problem of *open relays* first. For security reasons Postfix allows either all mails from IP addresses listed in *mynetworks* or to local or virtual domains to send emails. All other cases like someone sending mail to another person on the internet is called *relaying* and is limited for security reasons. If you would relay mail for everybody then spammers would abuse your mail server to send out emails on their behalf. You would be a so called *open relay*. You would add to the spam problem and probably get blacklisted quickly. That will make it hard to ever send out emails again. Setting the *smtpd_recipient_restrictions* right (as described below) is very important.

Generally you can define the networks that are allowed to relay through your mail server by setting the *mynetworks* parameter in your main.cf. Usually you set it to your local network so local user do not need to authenticate:

    $> postconf -e mynetworks=192.168.50.0/24

One case of relaying is but important in real-life. Imagine you have a user who is currently not connected to your local network but wants to send out an email to the internet through your mail server. According to the above rules the user would not be able to do that because neither are they on your network nor are they sending an email *to* one of your domains. You need to find a way to make your remote user be trusted by your mail server. The users will need to show you theyr username and password so you know relaying should be permitted. This is what *authenticated SMTP* is about. And most email clients have that feature built in.

Authenticated SMTP with Postfix has always been a pain. It was done through the SASL (*Simple Authentication and Security Layer*) library that was part of the Cyrus mail server. It was nearly impossible to debug and threw error messages that were gibberish and misleading. Fortunately starting with Postfix 2.3 we can `make Postfix ask the Dovecot server <http://www.postfix.org/SASL_README.html#server_dovecot>` to verify the username and password. And since you already configured Dovecot this is really easy now. Postfix just needs some extra configuration:

    $> postconf -e smtpd_sasl_type=dovecot
    $> postconf -e smtpd_sasl_path=private/auth
    $> postconf -e smtpd_sasl_auth_enable=yes
    $> postconf -e smtpd_recipient_restrictions=permit_mynetworks,permit_sasl_authenticated,reject_unauth_destination

smtpd_sasl_auth_enable enables SMTP authentication altogether. And smtpd_recipient_restrictions defines rules that are checked after the remote user sends the RCPT TO: line during the SMTP dialog. In this case relaying is allowed if:

• permit_mynetworks: the user is in the local network (mynetworks) or

• permit_sasl_authenticated: if the user is authenticated or

• reject_unauth_destination: the mail is destined to a user of a domain that is a local or virtual domain on this system (mydestination, virtual_alias_domains or virtual_mailbox_domains).

There are further restrictions (smtpd_client_restrictions, smtpd_helo_restrictions, smtpd_sender_restrictions) but unless you really want to customize them it is okay to put all restrictions into the smtpd_recipient_restrictions.

Try to authenticate during an SMTP session:

    $> telnet localhost smtp

The server will let you in:

    Trying 127.0.0.1...
    Connected to localhost.
    Escape character is '^]'.
    220 mailtest ESMTP Postfix

Say hello:

    ehlo example.com

Postfix will present a list of features that are available during the SMTP dialog:

    250-mailtest
    250-PIPELINING
    250-SIZE 10240000
    250-VRFY
    250-ETRN
    250-STARTTLS
    250-ENHANCEDSTATUSCODES
    250-8BITMIME
    250 DSN

Send the authentication string with a Base64-encoded password:

    auth plain am9obkBleGFtcGxlLmNvbQBqb2huQGV4YW1wbGUuY29tAHN1bW1lcnN1bg==

The server should accept that authentication:

    235 2.0.0 Authentication successful

Disconnect from Postfix:

    quit

Good bye:

    221 2.0.0 Bye

Authentication works. Great.

• If you have set John's password to somthing else than 'summersun' you need Base64-encode it yourself. Use:

          perl -MMIME:Base64 -e \
              'print encode_base64("john\@example.com\0john\@example.com\0password")';
   

Now you can test sending email with authentication enabled. To make even your local network untrusted temporarily you can set:

    $> postconf -e mydestination=

temporarily. Fire up your mail program and watch your mail.log while you send an email to a domain on the internet. I recommend you send a test email to devnull@workaround.org which is an email address that will just discard your email. If everything worked well your logfile will show:

    postfix/smtpd[4032]: 1234567890: client=..., sasl_method=PLAIN, sasl_username=john@example.com
    postfix/cleanup[4040]: 2EAE8379CB: message-id=<...>
    postfix/qmgr[3963]: 1234567890: from=<john@example.com>, size=830, nrcpt=1 (queue active)
    postfix/smtpd[4032]: disconnect from ...
    postfix/smtp[4041]: 1234567890: to=<devnull@workaround.org>,
        relay=torf.workaround.org[212.12.58.129]:25, delay=6,
        delays=0.09/0.08/5.6/0.23, dsn=2.0.0, status=sent
        (250 OK id=1HsPC3-0008UJ-O5)
    postfix/qmgr[3963]: 2EAE8379CB: removed

Otherwise in case of an error your logfile would look like:

    postfix/smtpd[4032]: connect from ...[10.20.30.40]
    postfix/smtpd[4032]: warning: ...[10.20.30.40]: SASL PLAIN authentication failed:
    postfix/smtpd[4032]: lost connection after AUTH from ...[10.20.30.40]
    postfix/smtpd[4032]: disconnect from ...[10.20.30.40]

Your email programm may have warned you that the mail server uses an untrusted SSL certificate. The default certificate that comes with Postfix is made out to the funny nonexisting "Office for Complication of Otherwise Simple Affairs" organisation. This is sufficient for testing but just like you did for Dovecot it is advised you create a proper SSL certificate that at least has the correct host name included. The default certificate is stored at /etc/ssl/certs/ssl-cert-snakeoil.pem and the default private key is stored at /etc/ssl/private/ssl-cert-snakeoil.key. Consider creating a new certificate/key pair and store it as postfix.pem:

    $> openssl req -new -x509 -days 3650 -nodes -out /etc/ssl/certs/postfix.pem \
        -keyout /etc/ssl/private/postfix.pem

Same procedure as above when you created a certificate for Dovecot. Just remember to set the "Common Name" to the fully-qualified hostname. And set the permissions on the private key so that no unauthorized people can read it:

    $> chmod o= /etc/ssl/private/postfix.pem

You will just have to tell Postfix where to find your certificate and private key:

    $> postconf -e smtpd_tls_cert_file=/etc/ssl/certs/postfix.pem
    $> postconf -e smtpd_tls_key_file=/etc/ssl/private/postfix.pem

When you relay through Postfix again you should not get that certificate warning any longer.

Step 10: AMaViS: Filtering spam and viruses

Two annoying issues when using email are spam and viruses. Fortunately you can fight both using AMaViS (A Mail Virus Scanner). AMaViS is an interface between Postfix, spamassassin (famous for its bayesian spam filtering capabilities) and optionally a virus scanner. AMaViS contains the spam filtering part but has no virus scanner built in. You should install ClamAV for that purpose. It is a free virus scanner that gets updated frequently.

In Debian Etch the AMaViS configuration has been moved from the /etc/amavis/amavisd.conf to different files in the /etc/amavis/conf.d directory. Fortunately the virus scanner "ClamAV" is already configured by default so you do not need to touch that. However I suggest you take a closer look at /etc/amavis/conf.d/20-debian_defaults:

• $sa_spam_subject_tag: if this line stay uncommented then every email that

  • AMaViS suspects to be spam get this string inserted into the email's subject. If you do not want to alter the subject you should comment this line out.

    Users can still filter out spam by checking the X-Spam-Status header.

• $sa_tag_level_deflt: spam with a score that is greater or equal to this

  • level will get spam headers added. Debugging is easier if you always add the headers so you should consider setting it to some very low score (e.g. "-1000" so that headers are always added.

• $sa_tag2_level_deflt: emails with a spam score greater or equal to this level

  • will be marked as spam. Usually this default is reasonable. Lower it if too much spam gets through. Raise it if you get too many regular mails caught as spam.

• $sa_kill_level_deflt: should be set to the same value as $sa_tag2_level_deflt

• $final_spam_destiny: the default D_BOUNCE setting is plain stupid here.

  • The usual approach is to *tag* email as spam. But if you bounced them you would hit some innocent person because spam mails never contain the correct sender address. Just let the user decide what to do with spam. So it is strongly

    recommended that you set this to D_PASS.

• $banned_filename_re: carefully check this list because these patterns tell

  • AMaViS when to bounce an email because it contains data that you do not like to receive in an email

In the file /etc/amavis/conf.d/15-content_filter_mode you also need to remove the '#' from the @bypass_... lines so that spam and virus filtering gets enabled.

In the /etc/spamassassin/local.cf file you should add a line to disable automatic expiration of learned spam emails because that feature conflicts with AMaViS:

    bayes_auto_expire 0

Restart AMaViS (/etc/init.d/amavis restart) if you have made changes to the config files. Make sure that AMaViS is listening on TCP port 10024:

    $> netstat -nap | grep 10024
    tcp  0   0 127.0.0.1:10024     0.0.0.0:*    LISTEN   12345/amavisd

If you get such a line then AMaViS is running and waiting for incoming SMTP sessions. Otherwise check your /var/log/mail.log file - perhaps you have made a mistake in the configuration files.

A few words on how AMaViS will be plugged into Postfix. If some person sends sends you an email from the internet it will be received by Postfix (the *smtpd* daemon) first on TCP port 25 (SMTP). If Postfix accepts the email it will be forwarded to AMaViS on TCP port 10024 (SMTP). And if AMaViS is happy with the email's contents it will be sent back to Postfix on TCP port 10025 (SMTP). Postfix finally delivers the email to the actual recipient.

Making Postfix forward emails to AMaViS is done by setting the content_filter setting. Also set another option that will be explained later:

    $> postconf -e content_filter=smtp-amavis:[127.0.0.1]:10024
    $> postconf -e receive_override_options=no_address_mappings

But we need to define the smtp-amavis service first in the /etc/postfix/master.cf. And also we need Postfix to listen on TCP port 10025 for emails that get sent back from AMaViS:

    smtp-amavis unix -      -       n     -       2  smtp
        -o smtp_data_done_timeout=1200
        -o smtp_send_xforward_command=yes
        -o disable_dns_lookups=yes
        -o max_use=20

    127.0.0.1:10025 inet n  -       -     -       -  smtpd
        -o content_filter=
        -o local_recipient_maps=
        -o relay_recipient_maps=
        -o smtpd_restriction_classes=
        -o smtpd_delay_reject=no
        -o smtpd_client_restrictions=permit_mynetworks,reject
        -o smtpd_helo_restrictions=
        -o smtpd_sender_restrictions=
        -o smtpd_recipient_restrictions=permit_mynetworks,reject
        -o smtpd_data_restrictions=reject_unauth_pipelining
        -o smtpd_end_of_data_restrictions=
        -o mynetworks=127.0.0.0/8
        -o smtpd_error_sleep_time=0
        -o smtpd_soft_error_limit=1001
        -o smtpd_hard_error_limit=1000
        -o smtpd_client_connection_count_limit=0
        -o smtpd_client_connection_rate_limit=0
        -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks

Restart Postfix (postfix reload) first. Now two settings here need some explanation. First the receive_override_options are set to no_address_mappings. This disables all address mappings. Your virtual aliases for examples are not considered at first. Then the email is sent to smtp-amavis and in the end returned to the 127.0.0.1:10025 service that sets a lot of options. One of those options is the receive_override_options again. But this time the no_address_mappings setting is left out. So at this stage Postfix finally checks your virtual aliases. Sounds complicated? Well, it has to be done like this or otherwise your aliases would be evaluated twice which leads to mails sent twice. The other options are used to disable checks that Postfix has already done during the first stage.

• Setting receive_override_options=no_address_mappings makes Postfix not consider aliases any more. So if you wish to disable AMaViS as a content filter then you must not set this parameter either.

Another tiny caveat is that the user "clamav" must be a member of the system group "amavis" so that the two services are allowed to talk to each other:

    $> adduser clamav amavis
    $> /etc/init.d/clamav-daemon restart

And another issue to take care of: AMaViS tries to find out whether a certain email is incoming (sent from the internet to your domains) or outgoing (sent from your system to the internet) by looking at the @acl_local_domains setting. You need to tell AMaVis where to check if a certain domain is one of your destination domains. Edit the /etc/amavis/conf.d/50-user file and before the "1;" enter these lines:

    @lookup_sql_dsn = (
        ['DBI:mysql:database=mailserver;host=127.0.0.1;port=3306',
         'mailuser',
         'mailuser2007']);

    $sql_select_policy = 'SELECT name FROM virtual_domains WHERE CONCAT("@",name) IN (%k)';

The @lookup_sql_dsn defines how AMaVis can access your database. And the $sql_select_policy sets the SQL query that is run when AMaVis wants to determine if the destination domain of the currently scanned email is one of your virtual domains. The %k is a list of strings that AMaVis expects to find. The actual query will look like this:

    SELECT name
    FROM virtual_domains
    WHERE CONCAT("@",name)
    IN (
        'john@example.com',
        'john',
        '@example.com',
        '@.example.com',
        '@.com',
        '@.')

This may look a bit weird. But in the end the string '@example.com' is searched for. Restart AMaVis (telinit run amavis restart) and try sending an email to john@example.com. If you examine the email headers of that mail you should find lines that got added by AMaVis:

    X-Virus-Scanned: Debian amavisd-new at mymailserver
    X-Spam-Score: 0
    X-Spam-Level: 
    X-Spam-Status: No, score=0 tagged_above=-9999 required=6.31 tests=[none]

Your users will be able to filter out spam depending on this information. The X-Spam-Status line will be set to "Yes" if the spam score is above the required score ($sa_tag2_level_deflt). The X-Spam-Level line contains a number of "*" that tell the spam score. If your email program looked for a X-Spam-Level: ***** header then it would catch spam with at least a score of 5. A real-life example of caught spam:

    X-Spam-Status: Yes, hits=16.0 tagged_above=-9999.0 required=6.31
        tests=BAYES_99, FORGED_MUA_OUTLOOK, MSGID_FROM_MTA_ID,
        RCVD_IN_BL_SPAMCOP_NET, UNDISC_RECIPS, URIBL_OB_SURBL, WORK_AT_HOME
        X-Spam-Level: ***************
        X-Spam-Flag: YES

All incoming emails should now be tested for viruses and spam. Try that. Spamassassin comes with a spam sample containing the *GTUBE* (Generic Test for Unsolicited Bulk Email) that contains a signature that is supposed to trigger spam filters - just like EICAR.COM for virus scanners. Send John that spam email:

    $> sendmail john@example.com < /usr/share/doc/spamassassin/examples/sample-spam.txt

Among other information your /var/log/mail.log will now contain a line being written by AMaViS:

    amavis[13001]: (13001-02) Passed SPAM, <...> -> <john@example.com>, ...

So the email was detected as spam and passed through to John. Very well. Finally fix the permissions of the AMaViS configuration files so that no unauthorized user can read the database password:

    $> chmod o= /etc/amavis/conf.d/50-user

Step 11: Learning spam and ham

SpamAssassin is the software that is supposed to tell spam from regular emails ("ham"). It comes with a large number of tests that are applied to emails. It also queries real-time blacklists on the internet. But in addition to the fixed tests there is the *Bayesian* filter. It uses conditional probabilities to analyse the words used in the email mathematically. So it can derive if an email is likely to be spam. This filter can be trained by telling it what *you* think is ham or spam. The more ham and spam emails you show it the better the spam detection will become. You will need to have trained at least 200 ham mails and 200 spam mails or the Bayes filter will not be taken into account. The configuration described here will not distinguish ham and spam per-user but rather keep a global Bayes database. Results may be better if every user had their own database and that's actually possible but currently not used here.

As the 'amavis' user is the owner of the /var/lib/amavis/.spamassassin directory that is used to store the Bayes database and the 'vmail' user is used to store the users' mailboxes we have a permissions problem here. Unfortunately it cannot be easily solved by adding the 'amavis' user to the 'vmail' group. The permissions on /home/vmail do not allow group write access. So currently the spam learning is done by the 'root' user which may be a security problem.

A script that will teach SpamAssassin ham and spam can be run once a day by the 'root' user. Example for a script that you can run through crontab:

    #!/bin/bash -e

    SADIR=/var/lib/amavis/.spamassassin
    DBPATH=/var/lib/amavis/.spamassassin/bayes
    SPAMFOLDERS="\
        /home/vmail/well-known-customer.com/fred/.spam/cur \
        /home/vmail/spamtrap.org/jeanne/.spam/cur \
        "
    HAMFOLDERS="\
        /home/vmail/spamtrap.org/jeanne/.trash/cur \
        "

    for spamfolder in $SPAMFOLDERS ; do \
        echo Learning spam from $spamfolder ; \
        nice sa-learn --spam --showdots --dbpath $DBPATH $spamfolder
    done

    for hamfolder in $HAMFOLDERS ; do \
        echo Learning ham from $hamfolder ; \
        nice sa-learn --ham --showdots --dbpath $DBPATH $hamfolder
    done

    chown -R amavis:amavis $SADIR

You can see what AMaViS has recorded into its database by running:

    $> sa-learn --dbpath /var/lib/amavis/.spamassassin/bayes --dump magic
    0.000          0        977          0  non-token data: nspam
    0.000          0      27866          0  non-token data: nham
    [...]

The *nspam* value is the number of spam emails SpamAssassin has seen. And *nham* is the number of ham emails being learned.

Note: this way you are training AMaViS' *global* bayesian database. So it applies to all emails that enter your system. Make sure you don't learn spam mails from unreliable or untrusted users. They can ruin your spam detection ratio by putting messages in their 'spam' folder that are not really spam. Just because a user never wants to get email from his ex-girlfriend again does not make it spam for everybody. So be careful about the mails you feed to the global bayes database. Also do not make SpamAssassin learn emails that users just 'forwarded' to you. The header information would be gone that contains much of the information that helps SpamAssassin recognize spam. Make sure they use the 'bounce' feature or send the spam email as an attachment. Your best choice is to take the email messages directly from the user's maildir directory.

Optional features

    $> cast imp

    $> cast gollem turba mnemo nag

    require ["fileinto"];
    # Move spam to spam folder
    if exists "X-Spam-Flag" {
      fileinto "spam";
      # Stop here so that we do not reply on spams
      stop;
    }

    $> sendmail john@example.com < /usr/share/doc/spamassassin/examples/sample-spam.txt

    deliver(john@example.com): "2007-06-18 22:17:46 "Info: msgid=<GTUBE1.1010101@example.net>: saved mail to spam

    smtpd_client_restrictions = 
        permit_mynetworks
        permit_sasl_authenticated
        reject_unauth_pipelining
        reject_rbl_client bl.spamcop.net
        reject_rbl_client dynablock.njabl.org
        reject_rbl_client zen.spamhaus.org
        reject_rbl_client list.dsbl.org
        check_policy_service inet:127.0.0.1:60000

    postfix/smtpd[1234]: NOQUEUE: reject: RCPT from mail.example.com[10.20.30.40]:
        450 4.7.1 <mail.example.com[10.20.30.40]>: Client host rejected:
        Greylisted, see http://isg.ee.ethz.ch/tools/postgrey/help/yourdomain.html.html;
        from=<spammer@example.com> to=<one.of@your.users> proto=ESMTP helo=<[10.20.30.40]>

    find /home/vmail/ -name '*,ST' -ctime +7 | xargs rm -f

Troubleshooting

If you are having trouble receiving or sending email you may try these general steps:

1. Check your /var/log/mail/current. 93.7% of all problems cause a more or less clear error message there.

2. Run postfix check. No output means everything is well.

3. Ask on the #postfix IRC channel on irc.freenode.net. I somestimes attend there as MaffooClock but please just ask and wait for people to help you. Especially read the hints in the channel's topic.

4. Ask on the workaround chitchat mailing list (English please)

5. This tutorial is meant for SourceMage GNU/Linux. If you are trying this setup on any distribution you are completely on your own.

6. Sending an email directly to the author asking for help is the worst choice. Sending an email directly to the author giving suggestions or feedback is a good choice.

Thanks

The original version of this tutorial, as written by Christoph Haas, received contributions from the following

• Alexander Stielau
• Axel Zallich
• Christian Garling
• Christian Kurz
• Christof Gruber
• Daniel Hackenberg
• Daniel Wladow
• Devdas Bhagat
• Eicke Kemm
• Jasper SlitsF
• Jesper Krogh
• Jon Cox
• Kay-Michael Voit
• Lucas Baltes
• Mario Duve
• Michael Weisgerber
• Mikael Tokarev
• Oliver Ladner
• Patrick Blitz
• Patrick Jäger
• Pete Boyd
• Ricardo Arguello
• Simon Kammerer
• Thomas "Balu" Walter
• Tim Weippert
• Tobias Scheeper