Home
Download

Open Source

Projects
Patches

System Integration

Notes
SRPMs

The Herald multi-function server

This document describes how to build Herald, a multi-function server. Herald runs on commodity router hardware and provides a number of features:

We build Herald on top of OpenWrt because of the distribution's simplicity and small size. Herald is made up of roughly 120 packages, and its programs and configurations take up less than 50 MB of storage space. Here we assume that Herald will run within the confines of a Xen hypervisor.

Establish the Herald VM

Perform the following steps on the Xen Dom0 host to establish the VM which will host Herald:

  1. Obtain the x86_64 OpenWrt image at https://downloads.lede-project.org/releases/17.01.1/targets/x86/64/lede-17.01.1-x86-64-combined-ext4.img.gz.
  2. Uncompress the image and place it at /var/lib/xen/images/herald-lede-17.01.1-x86-64-combined-ext4.img on the Xen Dom0 host.
  3. Create a disk image to serve as the server's large data store (see our notes on platform virtualization) and name it /var/lib/xen/images/herald-data.qcow.
  4. Write the following at /etc/xen/vm-herald.cfg on the Xen Dom0 host (replace XX:XX:XX:XX:XX:XX):
    name    = "herald"
    memory  =  1024
    vcpus   =  1
    builder = "hvm"
    vif     = [ "model=e1000,script=vif-bridge,bridge=xenbr0,mac=XX:XX:XX:XX:XX:XX" ]
    disk    = [
      "tap2:tapdisk:aio:/var/lib/xen/images/herald-lede-17.01.1-x86-64-combined-ext4.img ,xvda,w",
      "tap2:qcow:/var/lib/xen/images/herald-data.qcow,xvdb,w"
              ]
    serial  = "pty"
    

Software installation

Perform the following steps on Herald:

  1. Set the root password: passwd.
  2. Remove unnecessary packages:
    opkg remove \
            dnsmasq \
            kmod-ppp \
            kmod-pppoe \
            kmod-pppox \
            kmod-r8169 \
    	logd \
            luci-app-firewall \
            luci-lib-ip \
    	luci-lib-jsonc \
            luci-lib-nixio \
            luci-proto-ipv6 \
            luci-proto-ppp \
            luci-theme-bootstrap \
    	luci-mod-admin-full \
    	luci-base \
    	luci \
            mtd \
            odhcpd \
            ppp \
            ppp-mod-pppoe \
            r8169-firmware \
            uhttpd-mod-ubus \
    	uhttpd
    
  3. Configure networking by writing /etc/config/network:
    config interface loopback
    	option ifname lo
    	option proto static
    	option ipaddr 127.0.0.1
    	option netmask 255.0.0.0
    
    config interface lan
    	option ifname eth0
    	option proto dhcp
    
  4. Install the necessary software:
    opkg update
    opkg install \
            block-mount \
            bogofilter \
            ca-certificates \
            dovecot (with GSSAPI and LDAP modules) \
    	dovecot-pigeonhole \
            freifunk-watchdog \
            git \
            lighttpd \
    	lighttpd-mod-accesslog \
            lighttpd-mod-auth \
            lighttpd-mod-authn_file \
            lighttpd-mod-authn_gssapi \
            lighttpd-mod-fastcgi \
            lighttpd-mod-redirect \
            lighttpd-mod-setenv  \
            php7 \
            php7-fastcgi \
            php7-mod-ctype \
            php7-mod-curl \
            php7-mod-dom \
            php7-mod-exif \
            php7-mod-fileinfo \
            php7-mod-gd \
            php7-mod-hash \
            php7-mod-iconv \
            php7-mod-json \
            php7-mod-mbstring \
            php7-mod-opcache \
            php7-mod-openssl \
            php7-mod-pdo \
            php7-mod-pdo-sqlite \
            php7-mod-session \
            php7-mod-simplexml \
            php7-mod-sqlite3 \
            php7-mod-xml \
            php7-mod-xmlreader \
            php7-mod-xmlwriter \
            php7-mod-zip \
    	php7-pecl-krb5 \
    	php7-pecl-ldap \
    	php7-pecl-mcrypt \
            postfix \
            prosody \
            rsync \
    	syslog-ng \
            zoneinfo-core \
            zoneinfo-northamerica
    
  5. Install a public SSH key at /etc/dropbear/authorized_keys.

Configuring the lighttpd web server

Here we describe how to configure lighttpd to redirect HTTP to HTTPS; authenticate using passwords or GSSAPI, depending on which network the client connects from; maintain a log using syslog; and support FastCGI.

  1. Create /etc/lighttpd/htpasswd to define non-Kerberos accounts which mirror the accounts defined by the network's Kerberos server.
  2. Set up lighttpd's Kerberos principal by running kadmin.local on the network's Kerberos server, and executing the following commands (replace example.com and EXAMPLE.COM):
    1. add_principal -randkey HTTP/www.example.com@EXAMPLE.COM
    2. (if needed) purgekeys -all HTTP/www.example.com@EXAMPLE.COM
    3. ktadd -k keytab HTTP/www.example.com@EXAMPLE.COM
  3. To configure Firefox to authenticate using Kerberos, visit about:config and set (replace example.com):
    1. network.negotiate-auth.trusted-uris = https://
    2. network.negotiate-auth.delegation-uris = .example.com
  4. Ensure the /etc/krb5.conf on each client contains dns_canonicalize_hostname = false.
  5. Copy keytab from the network's Kerberos server to /etc/lighttpd/ on Herald. Set the ownership and permissions of the file with chgrp www-data keytab and chmod 640 keytab, respectively.
  6. /etc/lighttpd/example.com.pem: Some TLS certificate authorities provide free TLS/X.509 certificates. Run openssl req -out CSR.csr -new -newkey rsa:4096 -nodes -keyout privateKey.key to generate a private key and corresponding certificate signing request. You should submit the request (CSR.csr) to your certificate authority, and they should respond with your new certificate, a root CA certificate, and an immediate certificate. Concatenate the private key and certificate to produce etc/lighttpd/example.com.pem.
  7. /etc/lighttpd/ca.pem: Concatenate the immediate and root certificate to produce etc/lighttpd/ca.pem.
  8. /etc/lighttpd/dh-param.pem: Generate Diffie-Hellman parameters using openssl dhparam -out dh-param.pem -2 2048.
  9. /etc/lighttpd/lighttpd.conf (replace example.com):
    server.modules = (
    )
    
    server.errorlog-use-syslog  = "enable"
    server.document-root        = "/mnt/sda1/var/www/example.com"
    server.upload-dirs          = ( "/tmp" )
    server.pid-file             = "/var/run/lighttpd.pid"
    server.username             = "http"
    server.groupname            = "www-data"
    
    index-file.names            = ( "index.php", "index.html",
                                    "index.htm", "default.htm",
                                    "index.lighttpd.html" )
    
    static-file.exclude-extensions = ( ".php", ".pl", ".fcgi" )
    
    $SERVER["socket"] == ":443" {
            ssl.engine = "enable"
            ssl.pemfile = "/etc/lighttpd/example.com.pem"
            ssl.ca-file = "/etc/lighttpd/ca.pem"
    }
    
    include       "/etc/lighttpd/mime.conf"
    include_shell "cat /etc/lighttpd/conf.d/*.conf"
    
  10. /etc/lighttpd/conf.d/10-redirect.conf:
    server.modules += ( "mod_redirect" )
    
    $HTTP["scheme"] == "http" {
            $HTTP["host"] =~ ".*" {
                    url.redirect = (".*" => "https://%0$0")
            }
    }
    
  11. /etc/lighttpd/conf.d/20-auth.conf (replace network-cidr, example.com, EXAMPLE.COM, protected-path and application-name; network-cidr represents the network containing the hosts which have access to Kerberos authentication):
    server.modules += ( "mod_auth" )
    
    $HTTP["remoteip"] == "network-cidr" {
    	auth.backend                  = "gssapi"
    	auth.backend.gssapi.keytab    = "/etc/lighttpd/keytab"
    	auth.backend.gssapi.principal = "HTTP/www.example.com@EXAMPLE.COM"
    
            auth.require = (
    		"/protected-path" => (
    			"method"  => "gssapi",
    			"realm"   => "EXAMPLE.COM",
    			"require" => "valid-user"
    		),
    	)
    } else {
    	auth.backend                   = "htpasswd"
    	auth.backend.htpasswd.userfile = "/etc/lighttpd/htpasswd"
    
    	auth.require = (
    		"/protected-path" => (
    			"method"  => "basic",
    			"realm"   => "application-name",
    			"require" => "valid-user"
    		),
    	)
    }
    
  12. /etc/lighttpd/conf.d/30-accesslog.conf:
    server.modules += ( "mod_accesslog" )
      
    accesslog.use-syslog   = "enable"
    accesslog.syslog-level = 6
    
  13. /etc/lighttpd/conf.d/30-fastcgi.conf:
    server.modules += ( "mod_fastcgi" )
    
    fastcgi.server = (
            ".php" => ((
                    "bin-path" => "/usr/bin/php-cgi",
                    "socket" => "/tmp/php-fastcgi.socket"
            ))
    )
    
  14. Set the ownership of lighttpd's sensitive files using chown root:www-data /etc/lighttpd/*.pem /etc/lighttpd/htpasswd, and set the permissions on these files with chmod 640 /etc/lighttpd/*.pem /etc/lighttpd/htpasswd.
  15. /etc/php.ini: Set doc_root = "/mnt/sda1/var/www/example.com", error_log = syslog, and date.timezone = America/New_York.

Configuring the Prosody chat server

  1. /etc/prosody/certs/example.com.cert: Concatenate your certificate, the immediate certificate, and the root certificate to produce etc/prosody/certs/example.com.cert.
  2. /etc/prosody/certs/example.com.key: Place your private key in etc/prosody/certs/example.com.key.
  3. /etc/prosody/prosody.cfg.lua (replace example.com):
    admins = { }
    
    modules_enabled = {
    	"roster";
    	"saslauth";
    	"tls";
    	"dialback";
    	"disco";
    	"private";
    	"vcard";
    	"legacyauth";
    	"version";
    	"uptime";
    	"time";
    	"ping";
    	"pep";
    	"register";
    	"posix";
    };
    
    allow_registration = false;
    
    pidfile = "/var/run/prosody/prosody.pid";
    	
    ssl = { 
    	key = "/etc/prosody/certs/example.com.key";
    	certificate = "/etc/prosody/certs/example.com.cert";
    	c2s_require_encryption = true;
    	s2s_require_encryption = true;
    }
    
    log = {
    	{ levels = { "error" }; to = "syslog";  };
    	{ levels = { "error" }; to = "file"; 
    		filename = "/var/log/prosody/prosody.err";  };
    	{ levels = { min = "info" }; to = "file"; 
    		filename = "/var/log/prosody/prosody.log";  };
    }
    
    VirtualHost "example.com"
    	enabled = true
    
  4. Set the ownership of Prosody's sensitive files using chown prosody /etc/prosody/certs/*, and set the permissions on these files with chmod 600 /etc/prosody/certs/*.
  5. For each prosody user, prosodyctl register USERNAME example.com PASSWORD, replacing USERNAME, PASSWORD, and example.com. (Use LDAP?)

Configuring the Postfix SMTP server and Bogofilter spam filter

  1. /etc/postfix/master.cf (replace host.example.com):
    # Do not filter mail from localhost (e.g., from Roundcube or a ssh tunnel).
    127.0.0.1:smtp      inet  n       -       n       -       -       smtpd
    
    # Filter all other mail through bogofilter (see below).
    host.example.com:smtp      inet  n       -       n       -       -       smtpd
       -o content_filter=filter
    
    pickup    unix  n       -       n       60      1       pickup
    cleanup   unix  n       -       n       -       0       cleanup
    qmgr      unix  n       -       n       300     1       qmgr
    tlsmgr    unix  -       -       n       1000?   1       tlsmgr
    rewrite   unix  -       -       n       -       -       trivial-rewrite
    bounce    unix  -       -       n       -       0       bounce
    defer     unix  -       -       n       -       0       bounce
    trace     unix  -       -       n       -       0       bounce
    verify    unix  -       -       n       -       1       verify
    flush     unix  n       -       n       1000?   0       flush
    proxymap  unix  -       -       n       -       -       proxymap
    proxywrite unix -       -       n       -       1       proxymap
    smtp      unix  -       -       n       -       -       smtp
    relay     unix  -       -       n       -       -       smtp
    showq     unix  n       -       n       -       -       showq
    error     unix  -       -       n       -       -       error
    retry     unix  -       -       n       -       -       error
    discard   unix  -       -       n       -       -       discard
    local     unix  -       n       n       -       -       local
    virtual   unix  -       n       n       -       -       virtual
    lmtp      unix  -       -       n       -       -       lmtp
    anvil     unix  -       -       n       -       1       anvil
    scache    unix  -       -       n       -       1       scache
    filter	  unix	-	n	n	-	-	pipe
       flags=Rq user=bogofilter argv=/usr/sbin/postfix-bogofilter -f ${sender} -- ${recipient}
    
  2. /etc/postfix/main.cf (replace host.example.com, example.com, and mailrelay.example.com):
    mail_owner = postfix
    setgid_group = postdrop
    
    myhostname = host.example.com
    myorigin = example.com
    mynetworks = 127.0.0.0/8 192.168.1.0/24
    
    queue_directory = /mnt/sda1/var/spool/postfix
    data_directory = /mnt/sda1/var/lib/postfix
    mail_spool_directory = /mnt/sda1/var/spool/mail
    virtual_mailbox_base = /mnt/sda1/var/spool/mail
    
    relay_domains = $mydestination
    
    smtpd_delay_reject = yes
    smtpd_helo_required = yes
    smtpd_helo_restrictions =
            permit_mynetworks,
            reject_non_fqdn_helo_hostname,
            reject_invalid_helo_hostname,
            permit
    
    strict_rfc821_envelopes = yes
    disable_vrfy_command = yes
    
    smtpd_relay_restrictions =
    	permit_mynetworks,
    	permit_sasl_authenticated,
    	reject_unauth_destination
    
    # Note, we leave reject_rbl_client-like checks for later processing.
    smtpd_recipient_restrictions =
            permit_mynetworks,              
    	reject_unauth_pipelining,              
    	reject_invalid_hostname,
            reject_non_fqdn_sender,                
            reject_non_fqdn_recipient,             
            reject_unknown_sender_domain,          
            reject_unknown_recipient_domain,
            reject_unauth_destination,      
            reject_rbl_client zen.spamhaus.org,
            reject_rbl_client bl.spamcop.net,  
            check_recipient_access cdb:/etc/postfix/recipient_access,
            check_sender_access cdb:/etc/postfix/sender_access,      
            reject
    
    relayhost = [mailrelay.example.com]:587
    smtp_tls_security_level = encrypt
    smtp_sasl_auth_enable = yes
    smtp_sasl_password_maps = cdb:/etc/postfix/saslpasswd
    smtp_sasl_security_options = noplaintext, noanonymous
    smtp_sasl_tls_security_options = noanonymous
    
    virtual_mailbox_domains = $mydomain
    virtual_mailbox_maps = ldap:/etc/postfix/ldap-users.cf
    virtual_mailbox_lock = fcntl
    virtual_uid_maps = ldap:/etc/postfix/ldap-uids.cf
    virtual_gid_maps = static:8
    virtual_alias_maps = cdb:/etc/postfix/virtual
    virtual_transport = lmtp:unix:private/dovecot-lmtp
    
    message_size_limit = 0
    mailbox_size_limit = 0
    virtual_mailbox_limit = 0
    
    unknown_local_recipient_reject_code = 550
    unknown_address_reject_code  = 554                               
    unknown_hostname_reject_code = 554                               
    unknown_client_reject_code   = 554
    
    biff = no
    html_directory = no
    manpage_directory = no
    readme_directory = no
    inet_protocols = ipv4
    
    mailbox_transport = lmtp:unix:private/dovecot-lmtp
    
  3. /etc/postfix/virtual (replace recipient@example.com):
    root recipient@example.com
    
  4. /etc/postfix/saslpasswd (replace mailrelay.example.com, user@example.com, and password):
    [mailrelay.example.com]:587 user@example.com:password
    
  5. /etc/postfix/recipient_access (replace unrestricted_recipient@example.com):
    unrestricted_recipient@example.com OK
    
  6. /etc/postfix/sender_access (replace permitted_sender@example.com):
    permitted_sender@example.com OK
    
  7. Use postmap to compile each of virtual, saslpasswd, recipient_access, and sender_access.
  8. /etc/postfix/aliases: Set the recipient of root's mail and run postalias /etc/postfix/aliases.
  9. /etc/postfix/ldap-users.cf (replace ldap-server and dc=example,dc=com):
    server_host = ldaps://ldap-server
    server_port = 636
    search_base = ou=people,dc=example,dc=com
    version = 3
    query_filter = uid=%u
    result_attribute = uid
    
  10. /etc/postfix/ldap-uids.cf (replace ldap-server and dc=example,dc=com):
    server_host = ldaps://ldap-server
    server_port = 636
    search_base = ou=people,dc=example,dc=com
    version = 3
    query_filter = uid=%u
    result_attribute = uidNumber
    
  11. Check the ownership of Postfix's spool directories.
  12. Create a bogofilter user in /etc/passwd, /etc/group, and /etc/shadow. Set the user's shell to /bin/false and his home directory to /mnt/sda1/var/spool/bogofilter.
  13. Create the directory /mnt/sda1/var/spool/bogofilter. Restrict the permissions of this directory so that only the bogofilter user may access it. Configure bogofilter to make use of the directory by modifying /etc/bogofilter.cf:
    bogofilter_dir=/mnt/sda1/var/spool/bogofilter
    
  14. Set the ownership of /etc/bogofilter.cf with chown bogofilter /etc/bogofilter.cf.

Configuring the Dovecot POP3/IMAP server

  1. Set up Dovecot's Kerberos principal by running kadmin.local on the network's Kerberos server, and executing the following commands (replace example.com and EXAMPLE.COM):
    1. add_principal -randkey imap/www.example.com@EXAMPLE.COM
    2. (if needed) purgekeys -all imap/www.example.com@EXAMPLE.COM
    3. ktadd -k keytab imap/www.example.com@EXAMPLE.COM
  2. Copy keytab from the network's Kerberos server to /etc/dovecot on Herald. Set the ownership and permissions of the file with chown dovecot keytab and chmod 600 keytab, respectively.
  3. /etc/dovecot/dovecot.conf (replace example.com):
    protocols = imap pop3 lmtp
    
    auth_gssapi_hostname = "$ALL"
    auth_mechanisms = plain login gssapi
    auth_krb5_keytab = /etc/dovecot/keytab
    
    userdb {
    	driver = ldap
    	args = /etc/dovecot/dovecot-ldap.conf
    }
    
    passdb {
    	driver = passwd-file
    	args = /etc/dovecot/passwd
    }
    
    service imap-login {
            inet_listener imaps {
                    port = 0
            }
    }
    
    service pop3-login {
            inet_listener pop3s {
                    port = 0
            }
    }
    
    service lmtp {
    	unix_listener /mnt/sda1/var/spool/postfix/private/dovecot-lmtp {
    		user = postfix
    		group = postfix
    		mode = 0600
    	}
    }
    
    protocol lmtp {
    	postmaster_address = postmaster@flyn.org
    	hostname = flyn.org
    	mail_plugins = $mail_plugins sieve
    }
    
    plugin {                                                                             
            fts_autoindex=yes                                                            
    	sieve_default = /etc/dovecot/sieve/default.sieve
    }
    
    ssl = required
    ssl_cert = </etc/dovecot/example.com.cert
    ssl_key = </etc/dovecot/example.com.key
    ssl_dh = </etc/dovecot/dh-param.pem
    mail_location = \
    	mbox:/mnt/sda1/var/spool/mail/%n-folders:INBOX=/mnt/sda1/var/spool/mail/%n
    mail_access_groups = mail
    default_login_user = nobody
    disable_plaintext_auth = yes
    auth_username_format = %Ln
    mbox_write_locks = fcntl
    
  4. /etc/dovecot/dovecot-ldap.conf (replace dc=example,dc=com):
    hosts = localhost
    base = ou=people,dc=example,dc=com
    user_attrs = homeDirectory=home,uidNumber=uid,gidNumber=gid
    user_filter = (&(objectClass=posixAccount)(uid=%n))
    
  5. /etc/dovecot/passwd (replace placeholder fields; hope to eventually use Kerberos):
    user:{PLAIN}password:uid:gid:Full Name:/var/run/dovecot:/bin/false
    
  6. /etc/dovecot/sieve/default.sieve:
    require "fileinto";
    
    if header :matches "X-Bogosity" "Spam,*" {
    	fileinto "Junk";
    }
    
  7. /etc/dovecot/dh-param.pem: Generate Diffie-Hellman parameters using openssl dhparam -out dh-param.pem -2 2048.
  8. Run sievec /etc/dovecot/sieve.
  9. Copy certificate and private key to example.com.cert example.com.key, respectively.
  10. Set the ownership of Dovecot's files using chown -R dovecot /etc/dovecot, and set the permissions on the most sensitive files with chmod 600 /etc/dovecot/example.com.key /etc/dovecot/passwd.

Configure Git

  1. Add /usr/bin/git-shell to /etc/shells.
  2. Create a git user in /etc/passwd, /etc/group, and /etc/shadow. Set the user's shell to /usr/bin/git-shell and his home directory to /mnt/sda1/var/git.
  3. Install the authorized users' public SSH keys at /mnt/sda1/var/git/.ssh/authorized_keys.

Configure the host firewall

  1. /etc/config/firewall:
    config defaults
    	option drop_invalid 1
    	option input ACCEPT
    	option output ACCEPT
    	option forward ACCEPT
    
    config zone
    	option name lan
    	option network lan
    	option input DROP
    	option output ACCEPT
    	option forward DROP
    
    # Allow SSH connections from LAN.
    config rule
    	option target ACCEPT
    	option src lan
    	option proto tcp
    	option dest_port 22
    
    # Allow SMTP connections from LAN.
    config rule                 
    	option target ACCEPT
    	option src lan
    	option proto tcp
    	option dest_port 25
    
    # Allow IMAP connections from LAN.
    config rule                 
    	option target ACCEPT
    	option src lan
    	option proto tcp
    	option dest_port 143
    
    # Allow HTTP connections from LAN.
    config rule
    	option target ACCEPT
    	option src lan
    	option proto tcp
    	option dest_port 80
    
    # Allow HTTPS connections from LAN.
    config rule
    	option target ACCEPT
    	option src lan
    	option proto tcp
    	option dest_port 443
    
    # Allow Jabber client-to-server connections from LAN.
    config rule
    	option target ACCEPT
    	option src lan
    	option proto tcp
    	option dest_port 5222
    
    # Allow Jabber server-to-server connections from LAN.
    config rule
    	option target ACCEPT
    	option src lan
    	option proto tcp
    	option dest_port 5269
    

Configure basic system settings

  1. /etc/config/fstab:
    config mount
            option device   /dev/sda1
            option target   /mnt/sda1
    	option fstype   ext4
            option options  rw
            option enabled  1
            option enabled_fsck 0
    
  2. /etc/config/system:
    config system
    	option hostname	herald.flyn.org
    	option timezone	EST5EDT,M3.2.0,M11.1.0
    
    config timeserver ntp
            list server     0.openwrt.pool.ntp.org
            list server     1.openwrt.pool.ntp.org
            list server     2.openwrt.pool.ntp.org
            list server     3.openwrt.pool.ntp.org
            option enabled 1
            option enable_server 0
    
  3. /etc/config/freifunk-watchdog:
    config process
    	option process dropbear	
    	option initscript /etc/init.d/dropbear
    
    config process
            option process crond
            option initscript '/etc/init.d/cron'
    	
    config process
    	option process lighttpd
    	option initscript /etc/init.d/lighttpd
    
    config process
    	option process ntpd
    	option initscript /etc/init.d/ntpd
    
  4. /etc/config/dropbear:
    config dropbear
    	option PasswordAuth 'off'
    	option RootPasswordAuth 'off'
    	option Port         '22'
    
Email: www@flyn.org — ✉ 6110 Campfire Court; Columbia, Maryland 21045; USA