Virtualenv Installation

This is a step-by-step installation guide for Mailman Suite, also sometimes referred as Mailman 3 Suite or just Mailman 3. There are more than one ways to have Mailman 3 running on your machine.

The commands in this guide are tailored for Ubuntu/Debian systems.

Dependencies

Python3.7+.
While Mailman supports any version of Python > 3.6, versions 3.7+ is recommended.
MTA Setup

Mailman 3 in theory would support any MTA or mail server which can send emails to Mailman over LMTP. Officially, there are configurations for Postfix, Exim4, qmail and sendmail. Mailman Core has a fairly elaborate documentation on setting up your MTA. Look below at Installing Mailman Core to find out the location of configuration file mailman.cfg which is mentioned in the documentation above.

The Web Front-end is based on a Python web framework called Django. For email verification and sending out error logs, Django also must be configured to send out emails.

This guide uses setup instructions for Postfix.

Sass compiler

A sass compiler. Syntactically Awesome Stylesheets or Sass is an extension of CSS, which is used to style webpages, that adds more power to the old CSS syntax. It allows using variables, nested rules, inline imports etc. which CSS originally doesn’t support. Hyperkitty uses this to generate CSS styles.

You can use the C/C++ implementation. Please look at the installation guide for sass to see how you can get one.

Python development packages
Python3 dev package. This is required for building Postorius
Fulltext search

A full text search engine like Whoosh or Xapian. Whoosh is the easiest to setup and can be installed in the virtualenv.

See also

Virtualenv setup on how to setup the virtualenv.

Lynx
An HTML to plaintext converter like lynx is required by Mailman Core if you have configured it to convert emails to plaintext.

Installing Dependencies

To install Mailman Core, you need to following system packages:

$ sudo apt install python3-dev python3-venv sassc lynx

Setup database

This guide is based off on Postgresql database engine. Mailman also supports MySQL/MariaDB in case you have that already running for other applications.:

$ sudo apt install postgresql

$ sudo -u postgres psql
psql (12.5 (Ubuntu 12.5-0ubuntu0.20.04.1))
Type "help" for help.
postgres=# create database mailman;
CREATE DATABASE
postgres=# create database mailmanweb;
CREATE DATABASE
postgres=# create user mailman with encrypted password 'MYPASSWORD';
CREATE ROLE
postgres=# grant all privileges on database mailman to mailman;
GRANT
postgres=# grant all privileges on database mailmanweb to mailman;
GRANT
postgres=# \q

Note

Replace ‘MYPASSWORD’ with a secret password.

We created two databases named, mailman and mailmanweb for Mailman Core and Mailman Web (Django) respectively. We also created a new user mailman and granted it privileges to both the databases.

Setup Mailman user

Create a new user to run all the Mailman services:

$ sudo useradd -m -d /opt/mailman -s /usr/bin/bash mailman
$ sudo chown mailman:mailman /opt/mailman
$ sudo su mailman

Virtualenv setup

Virtualenv is Python’s mechanism to create isoated runtime environments.

Hint

If you are not familiar with virtualenv, checkout the user guide for virtualenv.

Note

Make sure that you are running commands as mailman user from here forth. Setup Mailman user.

Create the virtualenv for Mailman:

$ cd /opt/mailman
$ python3 -m venv venv

Activate virtualenv:

Activate the created virtualenv:

$ source /opt/mailman/venv/bin/activate

Note

The rest of this documentation assumes that virtualenv is activated. Whether or not virtualenv is activated can be seen by a (venv) before the shell prompt.

You can setup mailman user’s shell to automatically activate the virtualenv when you switch to the user by running:

$ echo 'source /opt/mailman/venv/bin/activate' >> /opt/mailman/.bashrc

Installing Mailman Core

Mailman Core is responsible for sending and receiving emails. It exposes a REST API that different clients can use to interact with over an HTTP protocol. The API itself is an administrative API and it is recommended that you don’t expose it to outside of your host or trusted network. To install Core run:

(venv)$ pip install wheel mailman psycopg2-binary

This will install latest release of Mailman Core, and Python bindings for Postgresql database. After this, create a configuration file at /etc/mailman3/mailman.cfg for Mailman Core:

# /etc/mailman3/mailman.cfg
[paths.here]
var_dir: /opt/mailman/mm/var

[mailman]
layout: here
# This address is the "site owner" address.  Certain messages which must be
# delivered to a human, but which can't be delivered to a list owner (e.g. a
# bounce from a list owner), will be sent to this address.  It should point to
# a human.
site_owner: user@example.com

[database]
class: mailman.database.postgresql.PostgreSQLDatabase
url: postgres://mailman:MYPASSWORD@localhost/mailman

[archiver.prototype]
enable: yes

# For the HyperKitty archiver.
[archiver.hyperkitty]
class: mailman_hyperkitty.Archiver
enable: yes
configuration: /opt/mailman/mm/mailman-hyperkitty.cfg
# And, create the /opt/mailman/mm/mailman-hyperkitty.cfg file containing
# these settings uncommented
#[general]
#base_url: http://127.0.0.1:8000/archives/
#api_key: Secret_Hyperkitty_API_Key

[shell]
history_file: $var_dir/history.py

[mta]
verp_confirmations: yes
verp_personalized_deliveries: yes
verp_delivery_interval: 1

See also

The further configuration setup for Mailman Core Configuring Mailman Core.

Setup MTA

A Mail Transfer Agent (MTA) is responsible for sending and receiving Emails on the server. This guide is based off on Postfix MTA, but Mailman also supports other MTAs like Exim4 etc:

$ sudo apt install postfix

Choose “Internet Site” when prompted during installation for choosing Postfix configuration.

Postfix install configuration

Enter the domain name you have chosen for Mailman in next step.

Postfix choose server mail name

To configure Postfix to relay emails to and from Mailman add the following to Postfix’s configuration at /etc/postfix/main.cf:

unknown_local_recipient_reject_code = 550
owner_request_special = no

transport_maps =
    hash:/opt/mailman/mm/var/data/postfix_lmtp
local_recipient_maps =
    hash:/opt/mailman/mm/var/data/postfix_lmtp
relay_domains =
    hash:/opt/mailman/mm/var/data/postfix_domains

See also

See detailed documentation to setup Postfix with Core and some unusual configuration if you already have Postfix running.

Starting Mailman automatically

To start Mailman Core automatically on boot, you can setup a systemd service. Create a new file /etc/systemd/system/mailman3.service:

[Unit]
Description=GNU Mailing List Manager
After=syslog.target network.target postgresql.service

[Service]
Type=forking
PIDFile=/opt/mailman/mm/var/master.pid
User=mailman
Group=mailman
ExecStart=/opt/mailman/venv/bin/mailman start
ExecReload=/opt/mailman/venv/bin/mailman restart
ExecStop=/opt/mailman/venv/bin/mailman stop

[Install]
WantedBy=multi-user.target

You can load this configuration by running:

$ sudo systemctl daemon-reload
# Check the status of the service.
$ sudo systemctl status mailman3

To start the systemd service and Mailman Core:

$ sudo systemctl start mailman3
# Verify that the service is running.
$ sudo systemctl status mailman3

After this, running mailman info (as mailman user with virtualenv active ) should give you an output which looks something like below:

(venv)$ mailman info
GNU Mailman 3.3.2 (Tom Sawyer)
Python 3.8.5 (default, Jul 28 2020, 12:59:40)
[GCC 9.3.0]
config file: /etc/mailman3/mailman.cfg
db url: postgres://mailman:MYPASSWORD@localhost/mailman
devmode: DISABLED
REST root url: http://localhost:8001/3.1/
REST credentials: restadmin:restpass

Warning

Pay attention to the config file output above and make sure that you see /etc/mailman3/mailman.cfg there. If the config path is different, you should make sure to create a new config file at /etc/mailman3/mailman.cfg and re-run the command.

Setup Cron Jobs

Mailman Core requires some cron jobs for periodic actions. To setup cron jobs for Mailman You can run:

sudo -u mailman crontab -e

Add the following in the editor:

@daily /opt/mailman/venv/bin/mailman digests --periodic
@daily /opt/mailman/venv/bin/mailman notify

This will run the commands at midnight every day, you can configure running them at specific time by replacing @daily with 0 8 * * * for running at 08:00 instead.

Installing Web UI

Postorius and Hyperkitty are Mailman’s official Web UI and Archiver. Mailman-web provides a convenient single package to install both of these.

To install the web components, run the following commands with virtualenv activated:

(venv) $ pip install mailman-web mailman-hyperkitty

Initial Configuration

Then, create a new configuration file at /etc/mailman3/settings.py. A sample configuration looks something like this:

# Mailman Web configuration file.
# /etc/mailman3/settings.py

from mailman_web.settings.base import *
from mailman_web.settings.mailman import *


#: Default list of admins who receive the emails from error logging.
ADMINS = (
    ('Mailman Suite Admin', 'root@localhost'),
)

# Postgresql database setup.
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'mailmanweb',
        'USER': 'mailman',
        # TODO: Replace this with the password.
        'PASSWORD': '$MYPASSWORD',
        'HOST': 'localhost',
        'PORT': '5432',
    }
}

# 'collectstatic' command will copy all the static files here.
# Alias this location from your webserver to `/static`
STATIC_ROOT = '/opt/mailman/web/static'


# Make sure that this directory is created or Django will fail on start.
LOGGING['handlers']['file']['filename'] = '/opt/mailman/web/logs/mailmanweb.log'

#: See https://docs.djangoproject.com/en/dev/ref/settings/#allowed-hosts
ALLOWED_HOSTS = [
    "localhost",  # Archiving API from Mailman, keep it.
    # "lists.your-domain.org",
    # Add here all production domains you have.
]

#: Current Django Site being served. This is used to customize the web host
#: being used to serve the current website. For more details about Django
#: site, see: https://docs.djangoproject.com/en/dev/ref/contrib/sites/
SITE_ID = 1

# Set this to a new secret value.
SECRET_KEY = 'MyVerrySecretKey'

# Set this to match the api_key setting in
# /opt/mailman/mm/mailman-hyperkitty.cfg (quoted here, not there).
MAILMAN_ARCHIVER_KEY = 'Secret_Hyperkitty_API_Key'

In the above configuration, make sure to

  • Change the database password in DATABASES
  • Set one or more admins in ADMINS for all the users who will receive emails about errors and exceptions in Django.
  • Add any domains/IP addresses you want to serve Mailman web from into ALLOWED_HOSTS config.
  • Make sure that the logging path is created. You can run mkdir -p /opt/mailman/web/logs to create the path.
  • Also make sure the STATIC_ROOT directory is created.
  • Set the SECRET_KEY to a random value.

See also

All the default settings in mailman-web can be found here.

Run database migrations

To setup the database schema for Mailman’s web components, run:

(venv)$ mailman-web migrate

Collect static files

To copy all the static files (css, js, images) into the STATIC_ROOT from Initial Configuration run:

(venv)$ mailman-web collectstatic

Compress CSS files

To compress the various CSS files offline run:

(venv)$ mailman-web compress

Compile messages for l10n

To update the message catalogs for supported languages run:

(venv)$ mailman-web compilemessages

Setting up a WSGI server

See also

What is WSGI?

These instructions are to setup your Django website behind a webserver. We are using uwsgi as the wsgi server to communicate between the webserver and Django. To install uwsgi, run:

(venv)$ pip install uwsgi

Note

The configuration below doesn’t serve static files, so if you are just “trying-it-out” and want static files to be served, you need to add some additional configuration and steps. See serving static files with uwsgi.

Create a configuration file for uwsgi at /etc/mailman3/uwsgi.ini:

# /etc/mailman3/uwsgi.ini
#
[uwsgi]
# Port on which uwsgi will be listening.
http-socket = 0.0.0.0:8000
# If running uwsgi from the virtual environment ...
virtualenv = /opt/mailman/venv/

module=mailman_web.wsgi:application
# Add to python import path.
pythonpath = /etc/mailman3/
# The default settings module.
env = DJANGO_SETTINGS_MODULE=settings

# Setup default number of processes and threads per process.
master = true
processes = 2
threads = 2

# Setup the django_q related worker processes.
attach-daemon = /opt/mailman/venv/bin/mailman-web qcluster

# Setup the request log.
req-logger = file:/opt/mailman/web/logs/uwsgi.log

# Log qcluster commands separately.
logger = qcluster file:/opt/mailman/web/logs/uwsgi-qcluster.log
log-route = qcluster uwsgi-daemons

# Last log and it logs the rest of the stuff.
logger = file:/opt/mailman/web/logs/uwsgi-error.log

You can run test run uwsgi using the following command:

(venv)$ uwsgi --ini /etc/mailman3/uwsgi.ini

Have a look at uwsgi documentation to learn more about different configuration options.

Automatically starting Mailman-web

In order to automatically start at startup, you can create a second systemd service, mailmanweb.service. Create a new file /etc/systemd/system/mailmanweb.service:

[Unit]
Description=GNU Mailman Web UI
After=syslog.target network.target postgresql.service mailman3.service

[Service]
Environment="PYTHONPATH=/etc/mailman3/"
User=mailman
Group=mailman
ExecStart=/opt/mailman/venv/bin/uwsgi --ini /etc/mailman3/uwsgi.ini

[Install]
WantedBy=multi-user.target

After creating this file, you can run following commands to start:

$ sudo systemctl daemon-reload
$ sudo systemctl start mailmanweb

Then, you check the status by running:

$ systemctl status mailmanweb

Cron Jobs for Mailman Web

Mailman Web requires some cron jobs for periodic actions. To setup cron jobs for Mailman You can run:

sudo -u mailman crontab -e

Add the following in the editor:

* * * * *          /opt/mailman/venv/bin/mailman-web runjobs minutely
0,15,30,45 * * * * /opt/mailman/venv/bin/mailman-web runjobs quarter_hourly
@hourly            /opt/mailman/venv/bin/mailman-web runjobs hourly
@daily             /opt/mailman/venv/bin/mailman-web runjobs daily
@weekly            /opt/mailman/venv/bin/mailman-web runjobs weekly
@monthly           /opt/mailman/venv/bin/mailman-web runjobs monthly
@yearly            /opt/mailman/venv/bin/mailman-web runjobs yearly

Nginx Configuration

You can reverse proxy the requests to uwsgi server using Nginx. Uwsgi has a special protocol called uwsgi protocol that is available in Nginx via the ngx_http_uwsgi_module module. That and the uwsgi_params file are included in a typical Nginx install. Add or edit the configuration file at /etc/nginx/site-available/default:

server {

   listen 443 ssl default_server;
   listen [::]:443 ssl default_server;

   server_name MY_SERVER_NAME;
   location /static/ {
        alias /opt/mailman/web/static/;
   }

   location / {
           proxy_pass 127.0.0.1:8000;
           proxy_set_header Host $host;
           proxy_set_header X-Forwarded-For $remote_addr;

   }
   ssl_certificate /path-to-ssl-certs/cert.pem;
   ssl_certificate_key /path-to-ssl-certs/privkey.pem;

}

For a more complete configuration, you can check out Nginx documentation for setting up SSL.

Replace MY_SERVER_NAME with the appropriate domain name you intend to serve.

After editing the configuration file, you can restart Nginx to pickup the changes:

$ sudo systemctl restart nginx

After this, you should have Mailman running at https://MY_SERVER_NAME.