The Contributor Guide

Mailman 3 consists of a collection of separate-but-linked projects, each of which has its own development setup guide. This makes sense when you want to focus on a single piece of Mailman, but can make setting up the entire Mailman Suite in one go somewhat confusing. This guide attempts to move all the information currently in the wiki and various package documentation into a single “definitive” guide.

Main package documentation on Readthedocs.io:

Getting prerequisites

For the most part, setup for each project will download any needed packages. However, you will need a few system packages to be sure you’ve got the necessary version of Python and its tools, git (to get the source code), postfix (a mail server), and a few other tools that are used during setup.

On Fedora, you probably want to run:

$ sudo yum install python3-setuptools python3-virtualenv python3-devel git gcc nodejs-less postfix python3-tox

On Debian and Ubuntu, this may be something like:

$ sudo apt install python3-setuptools python3-virtualenv python3-dev git gcc node-less nodejs postfix tox

On macOS, if you have Homebrew installed, you may run:

$ brew install tox python node gcc git
$ npm install --global less
$ # Note: postfix is pre-installed on macOS.

If you prefer, you can substitute Exim4 for Postfix. Postfix is the MTA used by most Mailman developers, but we do support Exim 4. (Sendmail support is very much desired, but the Mailman core developers need contributors with Sendmail expertise to help.) For development purposes it doesn’t matter, since we will mock all interactions to external MTA.

You will need tox to run tests.

HyperKitty also needs sassc. You can install sassc using your OS package manager

For Fedora/CentOS:

$ sudo dnf install sassc

sassc is available in the newer versions of Debian(9)/Ubuntu(18.04):

$ sudo apt install sassc

On macOS with Homebrew, you may run:

$ brew install sassc

You can also install sassc from source as per their build documentation

Gitlab Setup

We use Gitlab for source code hosting and our CI. You can fork any of the projects you want and start working on it. If you don’t already have an account on Gitlab, please create one, you will need that for contributing code or participating in any other way.

We also use Gitlab for code reviews. Our workflow looks very similar to the official Gitlab Workflow. Please remember to enable shared runners on your fork, it will be used to build your code and run unittests on pull requests that you will make. It is mandatory that you have runners enabled before you send any pull requests.

Set up a directory

Setting up the whole Mailman suite means you have to pull code from a bunch of different related repositories. You can put all the code anywhere you want, but you might want to set up a directory to keep all the pieces of mailman together. For example:

$ mkdir ~/mailman
# cd ~/mailman

For the rest of this development guide, we are going to assume you’re using ~/mailman as your directory, but you can use whatever you want.

Set up virtual environments

All parts of Mailman support only Python 3.6+. For your development, it is advised that you create a virtualenv so that the packages you install don’t break any of the system packages using Python.

To create the virtualenv run the following command:

$ python3 -m venv venv3

To activate a virtualenv, you need to run the appropriate activate script:

$ source venv3/bin/activate

You must use source (or . if your shell is a pure POSIX shell) everytime you want to activate your development environment. To make your life easier when managing virtualenvs, see virtualenvwrapper .

Set up and run Mailman Core

First, get the code:

$ cd ~/mailman
$ git clone https://gitlab.com/mailman/mailman.git

To set up Mailman Core, you’ll need to switch to your Python 3 virtualenv:

$ source venv3/bin/activate

Then, go into the mailman directory, run setup, and then run mailman info to be sure everything is set up correctly, and that the right settings are in place:

$ cd mailman
$ pip install -e .
$ mailman info

You can edit your mailman.cfg file to make any necessary changes. By default, during development, it is located at var/etc/mailman.cfg. Then start things up:

$ mailman start
$ mailman status        # Check if Mailman started correctly.
$ cd ..

Note that mailman just makes a var/ directory wherever you start it and uses that to store your data. This is great for the purposes of testing so you can easily make fresh installs, but might be confusing if you restart your instance later from a different location and don’t have your original mailman.db file, or if you start looking around and finding var/ directories everywhere.

Later on, if you need to restart Mailman (i.e. if you get the error “Mailman REST API not available. Please start Mailman core.”) then you can also do that by calling the mailman executable from the venv as follows:

$ ~/mailman/venv3/bin/mailman -C ~/mailman/mailman/var/etc/mailman.cfg start

If you are using different virtualenv paths, your path will be different than one listed above. You can find it using which mailman command.

Note that the mailman executable has several sub-commands. One that is particularly useful for debugging is mailman shell.

Note

If you like IPython shell (like I do!), you add the following to your mailman.cfg:

[shell]
use_ipython: yes

    Also, remember to install ipython using pip::

$ pip install ipython

Testing Mailman Core

You can run tests for Mailman Core (or any Mailman project) using tox <https://tox.readthedocs.io/en/latest/>

$ tox -e py311-nocov

This requires that you have Python3.11 installed. You can change it to py3XX depending on the latest version supported by Core.

Note

Take a look at the tox.ini for the latest version of Python supported by Mailman as this documentation could be out of date.

For contributing to Mailman it is incredibly helpful, time saving and resource-saving, running a specific test case, which only tests your contributions without testing every other component of Mailman. To run test cases from a specific file, for example the Mailman Core’s src/mailman/commands/tests/test_cli_members.py file, run a command like this:

$ tox -e py311 -- mailman.commands.tests.test_cli_members

Note

Please notice that / are replaced with . and the .py extension is cut off

You can omit the -e py311 (Or any environment, not just ‘py311’) option and tox will test your files with every environment specified in tox.ini.

$ tox – mailman.commands.tests.test_cli_members

The option -p auto parallelizes the tests:

$ tox -p auto

Of course, you can mix options to run all environments in parallel on one file:

$ tox -p auto -- mailman.commands.tests.test_cli_members

Set up Mailman Client

Get the code:

$ cd ~/mailman
$ git clone https://gitlab.com/mailman/mailmanclient.git

Then set up mailmanclient:

$ cd mailmanclient
$ pip install -e .
$ cd ..

To run the tests:

$ tox -e py311

Set up Django-mailman3

This package holds the Django libraries and templates used by Postorius and HyperKitty.

Get the code and set it up:

$ cd ~/mailman
$ git clone https://gitlab.com/mailman/django-mailman3.git
$ cd django-mailman3
$ pip install -e .
$ cd ..

To run the tests:

$ tox -e py311-django41

Note

It is possible for the version of Python and Django supported at a later time will be different. To get the right environments to run the tests, please run `tox `

Set up and run Postorius

The Postorius documentation, including a more extensive setup guide, can be found here: http://postorius.readthedocs.org/

Make sure to install mailmanclient and django-mailman3 before setting up Postorius. (If you’re following this guide in order, you’ve just done that.)

Get the code and run setup. Make sure you’re in venv which has Python 3.5+ for Postorius:

$ cd ~/mailman
$ git clone https://gitlab.com/mailman/postorius.git
$ cd postorius
$ pip install -e .
$ cd ..

Postorius and HyperKitty both come with example_project directories with basic configuration so you can try them out. For this tutorial, however, we’ll be using a project that combines both instead.

You can run tests using:

$ tox -e py311-django41

Set up a Fake mail server

To be able to actually receive emails, you need to setup a mail server. Mailman core receives emails over LMTP Protocol, which most of the modern MTAs support. However, setup instructions are provided only for Postfix, Exim4 and qmail. Please refer to the MTA documentation at Mailman Core for the details.

You will also have to add some settings to your django configuration. The setup instructions are provided in django’s email documentation.

For development setup, you don’t _have_ to install a working MTA. You can add the following to your mailman.cfg to make sure that it doesn’t try to send emails out:

[devmode]
enabled: yes
recipient: you@yourdomain.com

[mta]
smtp_port: 9025
lmtp_port: 9024
incoming: mailman.testing.mta.FakeMTA

Also, in Django you can add the following configuration to your settings.py:

EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'

This writes everything to stdout. There are other email backends available to use for testing like django.core.mail.backends.filebased.EmailBackend that one can use to write outgoing emails to a file on disk. Please see the docs for other options.

Set up and run HyperKitty

Complete guide here: https://hyperkitty.readthedocs.org/en/latest/development.html

Make sure to install mailmanclient and django-mailman3 before setting up Hyperkitty. (If you’re following this guide in order, you’ve just done that.)

HyperKitty’s default configuration uses the Whoosh search engine in the backend. Install Whoosh using:

$ pip install whoosh

Get the code and run setup:

$ cd ~/mailman
$ git clone https://gitlab.com/mailman/hyperkitty.git
$ cd hyperkitty
$ pip install -e .
$ cd ..

Postorius and HyperKitty both come with example_project directories with basic configuration so you can try them out. By default, they both use port 8000, so if you do want to run both example projects at the same time, do remember that you’ll need to specify a different port on the command line for one of them.

You can run tests using:

$ tox -e py311-django41

However, we’re going to run them both in a single Django instance at the end of this guide, so don’t worry about ports right now.

Set up mailman-hyperkitty

mailman-hyperkitty is the package that actually sends the incoming emails to HyperKitty for archiving. Note that this is one of the components that uses Python 3.

Setting it up:

$ cd ~/mailman
$ git clone https://gitlab.com/mailman/mailman-hyperkitty.git
$ cd mailman-hyperkitty
$ pip install -e .
$ cd ..

You’ll need to fix the default mailman-hyperkitty.cfg file to use the correct url for HyperKitty. If you’re running it on http://localhost:8002 then you need to change base_url to match that.

You can run tests using:

$ tox -e py311-coverage

Run the Mailman Suite (combined hyperkitty+postorius)

You can run HyperKitty and Postorius as separate applications, but many developers are going to want to run them on a single server. The configuration files for this are in a repository called mailman-suite.

The first time you run the suite, you will want to set up a superuser account. This is the account you will use in the web interface to set up your first domains. Please enter an email address otherwise the database won’t be setup correctly and you will run into errors later:

$ cd ~/mailman
$ git clone https://gitlab.com/mailman/mailman-web.git
$ cd mailman-web
$ pip install -e .
$ export DJANGO_SETTINGS_MODULE='mailman_web.settings.dev'
$ mailman-web migrate
$ mailman-web createsuperuser

You’ll want to run the following commands in a window where you can leave them running, since it dumps all the django logs to the console:

$ mailman-web runserver

At this point, you should be able to see Mailman Suite running! In the default setup, you can go to http://127.0.0.1:8000 and start poking around. You should be able to use the superuser account you created to log in and create a domain and then some lists.

The default config file uses a dummy email backend created by this line in settings.py:

EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'

Using this backend, all emails will be printed to the console (rather than sent as email) so you can get the url to verify your email from the console. You can also use FileBackend to write emails to a file on disk.

Don’t leave the console email backend configured and running once you get to the point where you want to send real emails, though!