Mailing Lists Docs

Mailman3 Installation HOWTO

This tutorial is all about how to install mailman3 in a Debian server with Apache2, and how to migrate existing lists from older mailman versions. It is intended specifically for sysadmins, and users in general who are curious of how machines and networks work. It assumes that postfix (a Mail Transport Agent, aka MTA) and mailutils are already installed in the system and configured, and the system can send emails, e.g the root user is sending admin related emails. It also assumes that python3, postgresql and apache2 are installed in the system too. Postfix is one of the possible MTA to be configured with mailman3. Detailed steps for configuring a fresh postfix install and few other MTA options, are included in the official mailman3 docs.

Install Dependency Libraries

We SSH to the remote server and as root do a system update with apt update && apt upgrade. Then we can install some system-wide dependencies for python development tools, python virtual environment, a CSS compressor and an HTML to plaintext convertor:

sudo apt install python3-dev python3-venv sassc lynx

Then we install rust from source, which is needed for python Cryptography library later on:

# curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# rustc --version

Install some extra dependencies for software compiling, and rust's package manager cargo:

# apt-get install build-essential libssl-dev libffi-dev python3-dev cargo

Ref: https://docs.mailman3.org/en/latest/install/virtualenv.html#installing-dependencies

Also GNU mailman wiki suggests to install few extra tools for memory cache, throttle too many failed server connections, and an internalization library to support multilingual environments (https://docs.mailman3.org/en/latest/translation.html):

# apt install memcached
# apt install fail2ban
# apt install gettext

for the sass installation the easiest for a Debian OS is to download from source and make a symbolic link to /usr/local/bin

# cd /usr/local/lib
# wget https://github.com/sass/dart-sass/releases/download/1.32.5/dart-sass-1.32.5-linux-x64.tar.gz
# tar -xf dart-sass-1.32.5-linux-x64.tar.gz
# chmod -R 755 dart-sass
# ln -s /usr/local/lib/dart-sass/sass /usr/local/bin/sass
# rm -f dart-sass-1.32.5-linux-x64.tar.gz

Ref https://wiki.list.org/DOC/Howto_Install_Mailman3_On_Debian10

Create a postgresql database for mailman

https://docs.mailman3.org/en/latest/install/virtualenv.html#setup-database

Setup mailman user and directory

useradd -m -d /opt/mailman -s /usr/bin/bash mailman

Go to mailman’s dir and create a virtualenv

python3 -m venv venv

To activate the virtualenv when we enter user mailman, we can add the following in mailman's .bashrc:

source /opt/mailman/venv/bin/activate

Ref: https://docs.mailman3.org/en/latest/install/virtualenv.html#virtualenv-setup

Install Mailman and other python libraries

(venv)$ pip install wheel mailman psycopg2-binary mailman-web mailman-hyperkitty

mailman-web provides hyperkitty and postorius for the web interface, as well as shortcuts to django admin commands. We install gunicorn for mailman-web application to be able to talk with apache2 server gateway, and a python client to make use of memory caching:

pip install pylibmc gunicorn

Mailman and hyperkitty configurations

Exit mailman user and as root we create /etc/mailman3/ directory, we make owner of this directory the user mailman, and create under it the files mailman.cfg and settings.py. Then as mailman user again, we create the mailman-hyperkitty.cfg file under the /opt/mailman.

mailman.cfg and setting.py samples:

https://docs.mailman3.org/en/latest/install/virtualenv.html#installing-mailman-core

https://docs.mailman3.org/en/latest/install/virtualenv.html#initial-configuration

In the mailman.cfg edit the archiver directive as following:

[archiver.hyperkitty]
class: mailman_hyperkitty.Archiver
enable: yes
configuration: /path/to/example_project/mailman-hyperkitty.cfg

And in mailman-hyperkitty.cfg:

# The base_url should correspond with the apache2 links we configure later
base_url: http://localhost/archives

# Shared API key, must be the identical to the value the same as in the /etc/mailman3/settings.py 
api_key: SecretArchiverAPIKey

Ref: https://hyperkitty.readthedocs.io/en/latest/install.html#connecting-to-mailman

https://gitlab.com/mailman/mailman-hyperkitty/blob/master/mailman-hyperkitty.cfg

Postfix configuration

Check open ports in the system. Look for the smtpd port 25 if it’s open. Postfix is the MTP which relays incoming and outgoing mails to mailman.

sudo ss -tulpn | grep smtpd

https://serverfault.com/questions/149903/what-ports-to-open-for-mail-server

postfix configurations here:

https://docs.mailman3.org/en/latest/install/virtualenv.html#setup-mta

if postfix is already installed skip that step and just edit the postfix/main.cf with the sample in the above step.

Configure mailman-web UI

Mailman web is a django site, which includes postorius and hyperkitty (that we installed before in our virtualenv). Enter mailman user again and run migrations, collect static files for the mailman-web, and create a django admin superuser, see details.

Run mailman-web locally

First with the local django server

(venv) $ pip install Werkzeug
(venv) $ mailman-web runserver_plus

then with gunicorn, e,g: (venv) $ gunicorn -c /opt/mailman/gunicorn.py mailman_web.wsgi:application

Sample gunicorn.py file:

#!/opt/mailman/venv/bin/python

import sys
sys.path[0:0] = [
  '/opt/mailman/',
  '/etc/mailman3/'
  ]

import os
os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'

import gunicorn.app.wsgiapp

if __name__ == '__main__':
    sys.exit(gunicorn.app.wsgiapp.run())

System services

Run as daemons the django server by adding the following services: /lib/systemd/system/ + mailman3.service + gunicorn + qcluster

Sample mailman3.service

Sample gunicorn unit service

Sample qcluster unit service

Then we reload the services and check their status

# systemctl daemon-reload
# systemctl status mailman3

Check status for gunicorn and qcluster too!

Cron jobs

From mailman user run crontab -e and add the following:

@hourly  /opt/mailman/mm/bin/django-admin runjobs hourly
@daily   /opt/mailman/mm/bin/django-admin runjobs daily
@weekly  /opt/mailman/mm/bin/django-admin runjobs weekly
@monthly /opt/mailman/mm/bin/django-admin runjobs monthly
@yearly  /opt/mailman/mm/bin/django-admin runjobs yearly
0,15,30,45 * * * * /opt/mailman/mm/bin/django-admin runjobs quarter_hourly
* * * * * /opt/mailman/mm/bin/django-admin runjobs minutely
# Send periodic digests.
30 3 * * * /opt/mailman/mm/bin/mailman digests --periodic
# Send request reminder for MM 3. Like the checkdbs job for 2.1
0 8 * * * /opt/mailman/mm/bin/mailman notify

Apache2 configurations

Go to sites-available and create new configuration for the domain name of your lists and check if proxy_http is enabled and use it for the localhost mailman-web application.

Sample here:

https://djangodeployment.readthedocs.io/en/latest/05-static-files.html?highlight=apache2#setting-up-apache

See more details:

https://djangodeployment.readthedocs.io/en/latest/04-web-server.html?highlight=apache2#configuring-apache-for-django

Copy existing list from old mailman server to mailman3

Create ssh keys for server where old lists reside and copy public key to current mailman3 server. A howto here. We can use rsync from the old mailman server to copy files in a tmp directory. E.g

rsync -avz ssh /var/lib/mailman/lists mailman@<new-server>:~/tmp/
rsync -avz ssh /var/lib/mailman/archives mailman@<new-server>:~/tmp

Ref: https://docs.rc.fas.harvard.edu/kb/rsync/

Migrate lists

From the new mailman server, enter mailman user. If we want to continue a migrated list with the new mailman3, we need to create a new list with the same name at the new domain:

(venv) $ mailman create foo-list@<new-lists-domain>

Then we import the old list, the archives, and we do an index update:

(venv) $ mailman import21 foo-list@<new-lists-domain> ~/tmp/lists/foo-list/config.pck
(venv) $ python manage.py hyperkitty_import -l foo-list@<new-lists-domain> ~/tmp/archives/private/<list>.mbox/<list>.mbox
(venv) $ mailman-web update_index_one_list foo-list@<new-lists-domain>

Ref: https://docs.mailman3.org/en/latest/migration.html#upgrade-strategy

Interact with the lists from the web UI

We visit the domain name we configured before we apache2. If the lists have a public view permission, then they are listed on the landing page. The same goes with the archives. Old members of the migrated list, would need to create new accounts via this web UI, in order for them to access the archives and their email and list settings. They need to use the same email with which they had subscribed in the old list.

Ref: https://docs.mailman3.org/en/latest/userguide.html#making-a-mailman-account

The admin of mailman3 has to create a superuser as a mailman user (if you haven't done it already):

(venv) $ mailman-web createsuperuser

With the credentials given above, we can login at the admin area <lists-domain>/admin, where we can see and create/edit lists, users, and change their permissions.

User guide docs at https://docs.mailman3.org/en/latest/userguide.html

Extra configurations for mailman web UI: https://docs.mailman3.org/en/latest/config-web.html