Herald

Mar 12, 2020

This document describes how to build Herald, a multi-function server. Herald runs on commodity router hardware and provides a number of features:

SSH access
a web service with PHP support
an SMTP service
an IMAP service
a Git service

We build Herald on top of OpenWrt because of the distribution’s simplicity and small size. Herald is made up of roughly 120 packages, and its programs and configurations take up less than 50 MB of storage space. Here we assume that Herald will run within the confines of a Xen hypervisor.

Establish the Herald VM

Perform the following steps on the Xen Dom0 host to establish the VM which will host Herald:

Obtain the x86_64 OpenWrt image at https://downloads.lede-project.org/releases/17.01.1/targets/x86/64/lede-17.01.1-x86-64-combined-ext4.img.gz.
Uncompress the image and place it at /var/lib/xen/images/herald-lede-17.01.1-x86-64-combined-ext4.img on the Xen Dom0 host.
Create a disk image to serve as the server’s large data store (see our notes on platform virtualization) and name it /var/lib/xen/images/herald-data.qcow.
Write the following at /etc/xen/vm-herald.cfg on the Xen Dom0 host (replace XX:XX:XX:XX:XX:XX):

name    = "herald"
memory  =  1024
vcpus   =  1
builder = "hvm"
vif     = [ "model=e1000,script=vif-bridge,bridge=xenbr0,mac=XX:XX:XX:XX:XX:XX" ]
disk    = [
  "tap2:tapdisk:aio:/var/lib/xen/images/herald-lede-17.01.1-x86-64-combined-ext4.img,xvda,w",
  "tap2:qcow:/var/lib/xen/images/herald-data.qcow,xvdb,w"
          ]
serial  = "pty"

Software installation

Perform the following steps on Herald:

Set the root password: passwd.
Remove unnecessary packages:

opkg remove \
        dnsmasq \
        kmod-ppp \
        kmod-pppoe \
        kmod-pppox \
        kmod-r8169 \
      logd \
        luci-app-firewall \
        luci-lib-ip \
      luci-lib-jsonc \
        luci-lib-nixio \
        luci-proto-ipv6 \
        luci-proto-ppp \
        luci-theme-bootstrap \
      luci-mod-admin-full \
      luci-base \
      luci \
        mtd \
        odhcpd-ipv6only \
        ppp \
        ppp-mod-pppoe \
        r8169-firmware \
        uhttpd-mod-ubus \
      uhttpd

Configure networking by writing /etc/config/network:

config interface loopback
      option ifname lo
      option proto static
      option ipaddr 127.0.0.1
      option netmask 255.0.0.0

config interface lan
      option ifname eth0
      option proto dhcp

Install the necessary software:

opkg update
opkg install \
        block-mount \
        bogofilter \
        ca-certificates \
        dovecot (with GSSAPI and LDAP modules) \
      dovecot-pigeonhole \
        freifunk-watchdog \
        git \
        lighttpd \
      lighttpd-mod-accesslog \
        lighttpd-mod-auth \
        lighttpd-mod-authn_file \
        lighttpd-mod-authn_gssapi \
        lighttpd-mod-fastcgi \
        lighttpd-mod-redirect \
        lighttpd-mod-setenv  \
        php7 \
        php7-fastcgi \
        php7-mod-ctype \
        php7-mod-curl \
        php7-mod-dom \
        php7-mod-exif \
        php7-mod-fileinfo \
        php7-mod-gd \
        php7-mod-hash \
        php7-mod-iconv \
        php7-mod-json \
        php7-mod-mbstring \
        php7-mod-opcache \
        php7-mod-openssl \
        php7-mod-pdo \
        php7-mod-pdo-sqlite \
        php7-mod-session \
        php7-mod-simplexml \
        php7-mod-sqlite3 \
        php7-mod-xml \
        php7-mod-xmlreader \
        php7-mod-xmlwriter \
        php7-mod-zip \
      php7-pecl-krb5 \
      php7-pecl-ldap \
      php7-pecl-mcrypt \
        postfix \
        rsync \
      syslog-ng \
        zoneinfo-core \
        zoneinfo-northamerica

Install a public SSH key at /etc/dropbear/authorized_keys.

Configuring the lighttpd web server

Here we describe how to configure lighttpd to redirect HTTP to HTTPS; authenticate using passwords or GSSAPI, depending on which network the client connects from; maintain a log using syslog; and support FastCGI.

Create /etc/lighttpd/htpasswd to define non-Kerberos accounts which mirror the accounts defined by the network’s Kerberos server.
Set up lighttpd’s Kerberos principal by running kadmin.local on the network’s Kerberos server, and executing the following commands (replace example.com and EXAMPLE.COM):
1. add_principal -randkey HTTP/www.example.com@EXAMPLE.COM
2. (if needed) purgekeys -all HTTP/www.example.com@EXAMPLE.COM
3. ktadd -k keytab HTTP/www.example.com@EXAMPLE.COM
To configure Firefox to authenticate using Kerberos, visit about:config and set (replace example.com):
1. network.negotiate-auth.trusted-uris = https://
2. network.negotiate-auth.delegation-uris = .example.com
. Ensure the /etc/krb5.conf on each client contains dns_canonicalize_hostname = false.
Copy keytab from the network’s Kerberos server to /etc/lighttpd/ on Herald. Set the ownership and permissions of the file with chgrp www-data keytab and chmod 640 keytab, respectively.
/etc/lighttpd/example.com.pem: Some TLS certificate authorities provide free TLS/X.509 certificates. Run openssl req -out CSR.csr -new -newkey rsa:4096 -nodes -keyout privateKey.key to generate a private key and corresponding certificate signing request. You should submit the request (CSR.csr) to your certificate authority, and they should respond with your new certificate, a root CA certificate, and an immediate certificate. Concatenate the private key and certificate to produce etc/lighttpd/example.com.pem.
/etc/lighttpd/ca.pem: Concatenate the immediate and root certificate to produce etc/lighttpd/ca.pem.
/etc/lighttpd/dh-param.pem: Generate Diffie-Hellman parameters using openssl dhparam -out dh-param.pem -2 2048.
/etc/lighttpd/lighttpd.conf (replace example.com):

server.modules = (
)

server.errorlog-use-syslog  = "enable"
server.document-root        = "/mnt/sda1/var/www/example.com"
server.upload-dirs          = ( "/tmp" )
server.pid-file             = "/var/run/lighttpd.pid"
server.username             = "http"
server.groupname            = "www-data"

index-file.names            = ( "index.php", "index.html",
                                "index.htm", "default.htm",
                                "index.lighttpd.html" )

static-file.exclude-extensions = ( ".php", ".pl", ".fcgi" )

server.modules += ( "mod_openssl" )

$SERVER["socket"] == ":443" {
        ssl.engine = "enable"
        ssl.pemfile = "/etc/lighttpd/example.com.pem"
        ssl.ca-file = "/etc/lighttpd/ca.pem"
}

include       "/etc/lighttpd/mime.conf"
include_shell "cat /etc/lighttpd/conf.d/*.conf"

/etc/lighttpd/conf.d/10-redirect.conf:

server.modules += ( "mod_redirect" )

$HTTP["scheme"] == "http" {
        $HTTP["host"] =~ ".*" {
                url.redirect = (".*" => "https://%0$0")
        }
}

/etc/lighttpd/conf.d/20-auth.conf (replace network-cidr, example.com, EXAMPLE.COM, protected-path and application-name; network-cidr represents the network containing the hosts which have access to Kerberos authentication):

server.modules += ( "mod_auth" )

$HTTP["remoteip"] == "network-cidr" {
      auth.backend                  = "gssapi"
      auth.backend.gssapi.keytab    = "/etc/lighttpd/keytab"
      auth.backend.gssapi.principal = "HTTP/www.example.com@EXAMPLE.COM"

        auth.require = (
              "/protected-path" => (
                      "method"  => "gssapi",
                      "realm"   => "EXAMPLE.COM",
                      "require" => "valid-user"
              ),
      )
} else {
      auth.backend                   = "htpasswd"
      auth.backend.htpasswd.userfile = "/etc/lighttpd/htpasswd"

      auth.require = (
              "/protected-path" => (
                      "method"  => "basic",
                      "realm"   => "application-name",
                      "require" => "valid-user"
              ),
      )
}

/etc/lighttpd/conf.d/30-accesslog.conf:

server.modules += ( "mod_accesslog" )
  
accesslog.use-syslog   = "enable"
accesslog.syslog-level = 6

/etc/lighttpd/conf.d/30-fastcgi.conf:

server.modules += ( "mod_fastcgi" )

fastcgi.server = (
        ".php" => ((
                "bin-path" => "/usr/bin/php-cgi",
                "socket" => "/tmp/php-fastcgi.socket"
        ))
)

Set the ownership of lighttpd’s sensitive files using chown root:www-data /etc/lighttpd/*.pem /etc/lighttpd/htpasswd, and set the permissions on these files with chmod 640 /etc/lighttpd/*.pem /etc/lighttpd/htpasswd.
/etc/php.ini: Set doc_root = "/mnt/sda1/var/www/example.com", error_log = syslog, and date.timezone = America/New_York. 15.

Configuring the Postfix SMTP server and Bogofilter spam filter

/etc/postfix/master.cf (replace host.example.com):

# Do not filter mail from localhost (e.g., from Roundcube or a ssh tunnel).
127.0.0.1:smtp      inet  n       -       n       -       -       smtpd

# Filter all other mail through bogofilter (see below).
host.example.com:smtp      inet  n       -       n       -       -       smtpd
   -o content_filter=filter

pickup    unix  n       -       n       60      1       pickup
cleanup   unix  n       -       n       -       0       cleanup
qmgr      unix  n       -       n       300     1       qmgr
tlsmgr    unix  -       -       n       1000?   1       tlsmgr
rewrite   unix  -       -       n       -       -       trivial-rewrite
bounce    unix  -       -       n       -       0       bounce
defer     unix  -       -       n       -       0       bounce
trace     unix  -       -       n       -       0       bounce
verify    unix  -       -       n       -       1       verify
flush     unix  n       -       n       1000?   0       flush
proxymap  unix  -       -       n       -       -       proxymap
proxywrite unix -       -       n       -       1       proxymap
smtp      unix  -       -       n       -       -       smtp
relay     unix  -       -       n       -       -       smtp
showq     unix  n       -       n       -       -       showq
error     unix  -       -       n       -       -       error
retry     unix  -       -       n       -       -       error
discard   unix  -       -       n       -       -       discard
local     unix  -       n       n       -       -       local
virtual   unix  -       n       n       -       -       virtual
lmtp      unix  -       -       n       -       -       lmtp
anvil     unix  -       -       n       -       1       anvil
scache    unix  -       -       n       -       1       scache
filter          unix  -       n       n       -       -       pipe
   flags=Rq user=bogofilter argv=/usr/sbin/postfix-bogofilter -f ${sender} -- ${recipient}

/etc/postfix/main.cf (replace host.example.com, example.com, and mailrelay.example.com):

mail_owner = postfix
setgid_group = postdrop

myhostname = host.example.com
myorigin = example.com
mynetworks = 127.0.0.0/8 192.168.1.0/24

queue_directory = /mnt/sda1/var/spool/postfix
data_directory = /mnt/sda1/var/lib/postfix
mail_spool_directory = /mnt/sda1/var/spool/mail
virtual_mailbox_base = /mnt/sda1/var/spool/mail

relay_domains = $mydestination

smtpd_delay_reject = yes
smtpd_helo_required = yes
smtpd_helo_restrictions =
      permit_mynetworks,
      reject_non_fqdn_helo_hostname,
      reject_invalid_helo_hostname,
      permit

strict_rfc821_envelopes = yes
disable_vrfy_command = yes

smtpd_relay_restrictions =
      permit_mynetworks,
      permit_sasl_authenticated,
      reject_unauth_destination

# Note, we leave reject_rbl_client-like checks for later processing.
smtpd_recipient_restrictions =
      permit_mynetworks,
      reject_unauth_pipelining,
      reject_invalid_hostname,
      reject_non_fqdn_sender,
      reject_non_fqdn_recipient,
      reject_unknown_sender_domain,
      reject_unknown_recipient_domain,
      reject_unknown_reverse_client_hostname,
      reject_unauth_destination,
      reject_rhsbl_helo dbl.spamhaus.org,
      reject_rhsbl_sender dbl.spamhaus.org,
      reject_rhsbl_reverse_client dbl.spamhaus.org,
      reject_rhsbl_sender fresh.fmb.la=127.2.0.[2;14],
      reject_rbl_client zen.spamhaus.org,
      reject_rbl_client dyna.spamrats.com,
      reject_rbl_client hostkarma.junkemailfilter.com=127.0.0.2,
      reject_rbl_client truncate.gbudb.net,
      reject_rbl_client dnsbl.cobion.com,
      reject_rbl_client bl.fmb.la=127.0.0.2,
      reject_rbl_client b.barracudacentral.org,
      reject_rbl_client bl.spamcop.net,
      check_recipient_access cdb:/etc/postfix/recipient_access,
      check_sender_access cdb:/etc/postfix/sender_access,
      reject

relayhost = [mailrelay.example.com]:587
smtp_tls_security_level = encrypt
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = cdb:/etc/postfix/saslpasswd
smtp_sasl_security_options = noplaintext, noanonymous
smtp_sasl_tls_security_options = noanonymous

virtual_mailbox_domains = $mydomain
virtual_mailbox_maps = ldap:/etc/postfix/ldap-users.cf
virtual_mailbox_lock = fcntl
virtual_uid_maps = ldap:/etc/postfix/ldap-uids.cf
virtual_gid_maps = static:8
virtual_alias_maps = cdb:/etc/postfix/virtual
virtual_transport = lmtp:unix:private/dovecot-lmtp

message_size_limit = 104857600
mailbox_size_limit = 0
virtual_mailbox_limit = 0

unknown_local_recipient_reject_code = 550
unknown_address_reject_code  = 554
unknown_hostname_reject_code = 554
unknown_client_reject_code   = 554

biff = no
html_directory = no
manpage_directory = no
readme_directory = no
inet_protocols = ipv4

mailbox_transport = lmtp:unix:private/dovecot-lmtp

/etc/postfix/virtual (replace recipient@example.com):

root recipient@example.com

/etc/postfix/saslpasswd (replace mailrelay.example.com, user@example.com, and password):

[mailrelay.example.com]:587 user@example.com:password

/etc/postfix/recipient_access (replace unrestricted_recipient@example.com):

unrestricted_recipient@example.com OK

/etc/postfix/sender_access (replace permitted_sender@example.com):

permitted_sender@example.com OK

Use postmap to compile each of virtual, saslpasswd, recipient_access, and sender_access.
/etc/postfix/aliases: Set the recipient of root's mail and run postalias /etc/postfix/aliases.
/etc/postfix/ldap-users.cf (replace ldap-server and dc=example,dc=com):

server_host = ldaps://ldap-server
server_port = 636
search_base = ou=people,dc=example,dc=com
version = 3
query_filter = uid=%u
result_attribute = uid

/etc/postfix/ldap-uids.cf (replace ldap-server and dc=example,dc=com):

server_host = ldaps://ldap-server
server_port = 636
search_base = ou=people,dc=example,dc=com
version = 3
query_filter = uid=%u
result_attribute = uidNumber

Check the ownership of Postfix's spool directories.
Create a bogofilter user in /etc/passwd, /etc/group, and /etc/shadow. Set the user’s shell to /bin/false and his home directory to /mnt/sda1/var/spool/bogofilter.
Create the directory /mnt/sda1/var/spool/bogofilter. Restrict the permissions of this directory so that only the bogofilter user may access it. Configure bogofilter to make use of the directory by modifying /etc/bogofilter.cf:

bogofilter_dir=/mnt/sda1/var/spool/bogofilter

Set the ownership of /etc/bogofilter.cf with chown bogofilter /etc/bogofilter.cf.

Configuring the Dovecot POP3/IMAP server

Set up Dovecot’s Kerberos principal by running kadmin.local on the network’s Kerberos server, and executing the following commands (replace example.com and EXAMPLE.COM):
1. add_principal -randkey imap/www.example.com@EXAMPLE.COM
2. (if needed) purgekeys -all imap/www.example.com@EXAMPLE.COM
3. ktadd -k keytab imap/www.example.com@EXAMPLE.COM
Copy keytab from the network’s Kerberos server to /etc/dovecot on Herald. Set the ownership and permissions of the file with chown dovecot keytab and chmod 600 keytab, respectively.
/etc/dovecot/dovecot.conf (replace example.com):

protocols = imap pop3 lmtp

auth_gssapi_hostname = "$ALL"
auth_mechanisms = plain login gssapi
auth_krb5_keytab = /etc/dovecot/keytab

userdb {
      driver = ldap
      args = /etc/dovecot/dovecot-ldap.conf
}

passdb {
      driver = passwd-file
      args = /etc/dovecot/passwd
}

service imap-login {
        inet_listener imaps {
                port = 0
        }
}

service pop3-login {
        inet_listener pop3s {
                port = 0
        }
}

service lmtp {
      unix_listener /mnt/sda1/var/spool/postfix/private/dovecot-lmtp {
              user = postfix
              group = postfix
              mode = 0600
      }
}

protocol lmtp {
      postmaster_address = postmaster@flyn.org
      hostname = flyn.org
      mail_plugins = $mail_plugins sieve
}

plugin {
        fts_autoindex=yes                                                            
      sieve_default = /etc/dovecot/sieve/default.sieve
}

ssl = required
ssl_cert = </etc/dovecot/example.com.cert
ssl_key = </etc/dovecot/example.com.key
ssl_dh = </etc/dovecot/dh-param.pem
mail_location = \
      mbox:/mnt/sda1/var/spool/mail/%n-folders:INBOX=/mnt/sda1/var/spool/mail/%n
mail_access_groups = mail
default_login_user = nobody
disable_plaintext_auth = yes
auth_username_format = %Ln
mbox_write_locks = fcntl

/etc/dovecot/dovecot-ldap.conf (replace dc=example,dc=com):

hosts = localhost
base = ou=people,dc=example,dc=com
user_attrs = homeDirectory=home,uidNumber=uid,gidNumber=gid
user_filter = (&(objectClass=posixAccount)(uid=%n))

/etc/dovecot/passwd (replace placeholder fields; hope to eventually use Kerberos):

user:{PLAIN}password:uid:gid:Full Name:/var/run/dovecot:/bin/false

/etc/dovecot/sieve/default.sieve:

require [
      "body",
      "fileinto",
      "regex",
      "variables"
];

set "blacklist" "(a v(e)*ry bad word|viagra)";

if header :matches "X-Bogosity" "Spam,*" {
      fileinto "Junk";
}

elsif header :regex "Subject" "${blacklist}" {
      fileinto "Junk";
}

elsif body :regex "${blacklist}" {
      fileinto "Junk";
}

else {
      keep;
}

/etc/dovecot/dh-param.pem: Generate Diffie-Hellman parameters using openssl dhparam -out dh-param.pem -2 2048.
Run sievec /etc/dovecot/sieve.
Copy certificate and private key to example.com.cert and example.com.key, respectively.
Set the ownership of Dovecot’s files using chown -R dovecot /etc/dovecot, and set the permissions on the most sensitive files with chmod 600 /etc/dovecot/example.com.key /etc/dovecot/passwd.

Configure Git

Add /usr/bin/git-shell to /etc/shells.
Create a git user in /etc/passwd, /etc/group, and /etc/shadow. Set the user’s shell to /usr/bin/git-shell and his home directory to /mnt/sda1/var/git.
Install the authorized users’ public SSH keys at /mnt/sda1/var/git/.ssh/authorized_keys.

Configure the host firewall

/etc/config/firewall:

config defaults
      option drop_invalid 1
      option input ACCEPT
      option output ACCEPT
      option forward ACCEPT

config zone
      option name lan
      option network lan
      option input DROP
      option output ACCEPT
      option forward DROP

# Allow SSH connections from LAN.
config rule
      option target ACCEPT
      option src lan
      option proto tcp
      option dest_port 22

# Allow SMTP connections from LAN.
config rule                 
      option target ACCEPT
      option src lan
      option proto tcp
      option dest_port 25

# Allow IMAP connections from LAN.
config rule                 
      option target ACCEPT
      option src lan
      option proto tcp
      option dest_port 143

# Allow HTTP connections from LAN.
config rule
      option target ACCEPT
      option src lan
      option proto tcp
      option dest_port 80

# Allow HTTPS connections from LAN.
config rule
      option target ACCEPT
      option src lan
      option proto tcp
      option dest_port 443

Configure basic system settings

/etc/config/fstab:

config mount
        option device   /dev/sda1
        option target   /mnt/sda1
      option fstype   ext4
        option options  rw
        option enabled  1
        option enabled_fsck 0

/etc/config/php7-fastcgi:

config php7-fastcgi
      option enabled 1

/etc/config/system:

config system
      option hostname herald.flyn.org
      option timezone EST5EDT,M3.2.0,M11.1.0

config timeserver ntp
        list server     0.openwrt.pool.ntp.org
        list server     1.openwrt.pool.ntp.org
        list server     2.openwrt.pool.ntp.org
        list server     3.openwrt.pool.ntp.org
        option enabled 1
        option enable_server 0

/etc/config/freifunk-watchdog:

config process
      option process dropbear 
      option initscript /etc/init.d/dropbear

config process
        option process crond
        option initscript '/etc/init.d/cron'
      
config process
      option process lighttpd
      option initscript /etc/init.d/lighttpd

/etc/config/dropbear:

config dropbear
      option PasswordAuth 'off'
      option RootPasswordAuth 'off'
      option Port         '22'