====== Upgrade qpsmtpd to 0.96 ======

We currently run a very old qpsmtpd version (0.84 was released in 2010). A lot of improvements are available in newer release. Today, the last version is 0.96.

The goal is to update qpsmtpd to a newer release, then check if new plugins can be enabled to enhance security and spam filtering. All this is tracked on [[https://bugs.contribs.org/show_bug.cgi?id=8861|Bug #8861]]

===== Update qpsmtpd core package =====

The first step is to update the core qpsmtpd package to the latest version, adapt the spec file if needed, rebase needed patches.


===== Check qpsmtpd-plugins and smeserver-qpsmtpd for duplicated plugins =====

Some plugins might have been merged in the core qpsmtpd package, and should be removed from qpsmtpd-plugins/smeserver-qpsmtpd in that case

==== qpsmtpd-plugins ====

The list of plugins provided by qpsmtpd-plugins is the following:

  * autowhitelist_relayrcpt
  * bcc
  * check_goodrcptto
  * denysoft_multi_rcpt
  * exe_filter
  * handler
  * per_user_config
  * whitelist_soft

None of them are provided by qpsmtpd. Two are probably a bit similar (per_user_config vs user_config and whitelist_soft vs dns_whitelist_soft), but as they do not clash, I see no reason to remove them. They are not used in the stock configuration anyway

==== smeserver-qpsmtpd ====

The list of plugins bundled in smeserver-qpsmtpd is the following:

  * check_smtp_forward
  * disclaimer
  * dkim_sign
  * logging/logterse
  * peers
  * tnef2mime
  * virus/pattern_filter

None of those plugins are provided by qpsmtpd. But the functionnalities provided by dkim_sign are now merged in the dkim plugin. We can keep the dkim_sign plugin here, so those who have cponfigured it manually won't break their installation on upgrade, but we'll have to adjust the wiki documentation to use the new dkim plugin. See https://wiki.contribs.org/Email#DKIM_Setup

===== Adapt smeserver-qpsmtpd =====

Plugins have been renamed, some might need new/different arguments. We need to adapt each of those in smeserver-qpsmtpd

This is the list of plugins we use, in order

  * logterse: no change
  * tls: no change
  * auth_cvm_unix_local: no change
  * check_earlytalker: **renamed earlytalker**
  * count_unrecognized_commands: no change
  * bcc: no change
  * check_relay: **renamed relay**
  * check_norelay: **merged into the relay plugin**
  * require_resolvable_fromhost: **renamed resolvable_fromhost**
  * check_basicheaders: **renamed headers**
  * rhsbl: no change
  * dnsbl: no change
  * check_badmailfrom: **renamed badmailfrom**
  * check_badrcptto_patterns: **doesn't exist anymore, merged with badrcptto**
  * check_badrcptto: **renamed badrcptto**
  * check_spamhelo: **renamed helo**
  * check_smtp_forward: no change
  * check_goodrcptto: no change
  * rcpt_ok: no change
  * pattern_filter: no change
  * tnef2mime: no change
  * spamassassin: no change
  * clamav: no change
  * qmail-queue: no change

===== Improve our config =====

The last step is to see if we can make use of new features/plugins to improve security and spam filtering on SME. Here's a potential list of improvements

  * Create a random dhparam on each install and use it in the tls plugin
  * Check if we can make use of the naughty plugin
  * The headers plugin (replacing check_basicheaders) can check for several missing headers (From,Date,Subject,Message-ID,Received), the default being only From (even Date is not enabled anymore by default as it blocks some legit emails).
  * The headers plugin now accepts different values for future and past dates (offset after/before which the email is considered invalid and is rejected). The previous only had a single value. This is controlled by the MaximumDateOffset prop (smtpd). We should allow different offsets (1 for future and 5 for past for example)
  * We might consider changing the clamav plugin (which uses clamdscan) to the clamdscan plugin. This should be a bit more efficient as we don't spawn the clamdscan binary, but directly talk to the clamd daemon
  * The new **bogus_bounces** plugin can be enabled
  * Check what can be done on a default setup with **dkim** and **dmarc** plugins
    * We could enable dkim verification for inbound emails
    * We could generate rsa keys for each domain declared from the server-manager
    * Add a helper (or a panel in the server-manager) to get the TXT entry to add in the DNS
    * See what we can do with dmarc for inbound
    * See what we can do with dmarc for outbound
  * See if we could combine dspam with spamassassin, and if it can provide better spam detection
  * Test and see if the **karma** plugin is efficient
  * Test and enable the **loadcheck** plugin
  * Test and enable the **uribl** plugin
  * See how we could use per user settings. This is ambitious, but would allow to have per user bayes database. We could start by only using it for single recipient email. Multi recipients are harder to manage (we need to add a header during the SMTP transaction, the the LDA must detect iof the header is present, and re-submit the email to spamassassin to have it re-scanned with the per user setting)

===== To add in the release notes =====

  * The badrcptto_patterns doesn't exist anymore (it wasn't used by default). It's content must be merged in the badrcptto config file
  * The karma plugin is now ready to be used
  * The helo plugin can now check more than just the helo hostname
  * DKIM, SPF and DMARC are used
  * The loadcheck plugin can defer inbound emails when your server load is too high
  * The uribl plugin is ready to be used
  * RBLList, SBLList and the new UBLList must now be comma separated. Previous configuration will be migrated automatically. For RBLList, you can use a semicolon to separate the service address and a reject message. This is useful for lists which doesn't have a TXT entry to get a reject message from, but only provides A entries.

===== Documentation =====
==== Karma ====

The karma plugin tracks sender history. For each inbound email, various plugins can raise, or lower the "naughtiness" of the connection (eg, if SPF check passes, if the message is spammy etc...). For each host sending us email, the total number of connections, and the number of good and bad connections is recorded in a database. If a host as more bad than good connections in its history, emails will be rejected for 1 day. 3 settings are available for this plugin:

  * Karma (enabled|disabled): Default value is disabled. Change to enabled to use the plugin
  * KarmaNegative (integer): Default value is 2. It's the delta between good and bad connection to consider the host naughty enough to block it for 1 day. Eg, with a default value of two, a host can be considered naughty if it sent you 8 good emails and 10 bad ones
  * KarmaStrikes (integer): Default value is 3. This is the threshold for a single email to be considered good or bad. Eg, with the default value of 3, an email needs at least 3 bad karmas (reaches -3) for the connection to be considered bad. On the other side, 3 good karmas are needed for the connection to be considered good. Between the two, the connection is considered neutral and won't be used in the history count

Example:
<code bash>
db configuration setprop qpsmtpd Karma enabled KarmaNegative 3
signal-event email-update
</code>

==== URIBL =====

The URIBL plugin works a bit like RHSBL, except that it checks domain names found in the body of the email. For each URI identified, the corresponding domain name can be submitted to a BL list (through DNS queries). Two settings are available:

  * URIBL (enabled|disabled): Default is disabled. Set this to enabled to use the plugin
  * UBLList: (Comma separated list addresses): Default value is **multi.surbl.org:8-16-64-128,black.uribl.com,rhsbl.sorbs.net**. This can be the same as RBLList. You can also set bitmask to use for combined lists (in the default value, the bitmask is 8-16-64-128)

Example:
<code bash>
db configuration setprop qpsmtpd URIBL enabled UBLList multi.surbl.org,black.uribl.com
signal-event email-update
</code>

==== Helo ====

Previously, the helo plugin was just checking for some known bad helo hostnames used by spammers (aol.com and yahoo.com). Now, it can check much more than that. This plugin is always enabled and has a single setting:

  * HeloPolicy: (lenient|rfc|strict). The default value is **lenient**. See https://github.com/smtpd/qpsmtpd/blob/master/plugins/helo for a description of the various tests done at each level

Example:

<code bash>
db configuration setprop qpsmtpd HeloPolicy rfc
signal-event email-update
</code>

==== Inbound DKIM / SPF / DMARC ====

DMARC is a policy on top of DKIM and SPF. By default, SPF and DKIM are now checked on every inbound emails, but no reject is attempted. The dmarc plugin can decide to reject the email (depending on the sender policy). dkim and spf plugins are always enabled. dmarc has two settings:

  * DMARCReject (enabled|disabled): Default value is disabled. If set to enabled, the dmarc plugin can decide to reject an email (if the policy of the sender is to reject on alignment failure)
  * DMARCReporting (enabled|disabled): Default value is enabled. If set to enabled, enable reporting (which is the **r** in dma**r**c). Reporting is a very important part of the DMARC standard. When enabled, you'll record information about email you receive from domains which have published a DMARC policy in a local SQLite database (/var/lib/qpsmtpd/dmarc/reports.sqlite). Then, once a day, you send the aggregate reports to the domain owner so they have feedback. You can set this to disabled if you want to disable this feature
  * SPFRejectPolicy (0|1|2|3|4): Default value is 0. Set the policy to apply in case of SPF failure when the sender hasn't published a DMARC policy. Note: this is only used when no DMARC policy is published by the sender. If there's a DMARC policy, even a "p=none" one (meaning no reject), then the email won't be rejected, even on failed SPF tests.
    * 0: do not reject anything
    * 1: reject when SPF says fail
    * 2: reject when SPF says softfail
    * 3: reject when SPF says neutral
    * 4: reject when an error occurred (like a syntax error in SPF entry) or if no SPF entry is published
  * Inbound DKIM checks are only used by DMARC. No reject solely based on DKIM is supported

Example:
<code bash>
db configuration setprop qpsmtpd DMARCReject disabled SPFRejectPolicy 2
signal-event email-update
</code>
==== Outbound DKIM signing / SPF / DMARC policy ====

Everything is now ready for you to sign your outbound emails, and publish your public key, as well as your SPF and DMARC policy. A default DKIM key is created in /home/e-smith/dkim_keys/default. To enable DKIM signing for all the domain you manage:

<code bash>
db configuration setprop qpsmtpd DKIMSigning enabled
signal-event email-update
</code>

If you want to disable dkim signing for a domain, you can use:
<code bash>
db domains setprop domain.com DKIMSigning disabled
signal-event email-update
</code>

The default behavior is to use the same key pair for all your domains. But you can create other key pairs for specific domain if you want. For example, if you want to use a specific key pair for the domain.net domain:

<code bash>
cd /home/e-smith/dkim_keys
mkdir domain.net
cd domain.net
echo default > selector
openssl genrsa -out private 2048
openssl rsa -in private -out public -pubout
chown qpsmtpd:qpsmtpd private
chmod 400 private
signal-event email-update
</code>

Now, the emails using a domain.net sender address will be signed by this new key instead of the default one.

=== Publishing your DNS entries ===

Signing your outbound emails is just part of the process. You now need to publish some DNS entries so everyone can check if the email they receive matches your policy. This part is not to be done on your SME Server, but on your public DNS provider. A script helps you by creating some sample DNS entries already formatted for a bind-like zone file. To use it:

<code bash>
qpsmtpd-print-dns <domain name>
</code>
If omitted, the primary domain name is assumed.

Example output:
<code>
Here are sample DNS entries you should add in your public DNS
The DKIM entry can be copied as is, but others will probably need to be adjusted
to your need. For example, you should either change the reporting email adress
for DMARC (or create the needed pseudonym)


default._domainkey IN TXT "v=DKIM1;p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAs/Qq3Ntpx2QNdRxGKMeKc2r9ULvyYW633IbLivHznN9JvjJIbS54PGIEk3sSxvZSdpTRAvYlxn/nRi329VmcDK0vJYb2ut2rnZ3VO3r5srm+XEvTNPxij5eU4gqw+5ayySDjqzAMEMc5V7lUMpZ/YiqnscA075XiMF7iEq8Quv1y0LokmgwtxzOXEZap34WXlKyhYzH+D""fabF6SUllmA0ovODNvudzvEOanPlViQ7q7d+Mc3b7X/fzgJfh5P9f5U+iSmzgyGctSb6GX8sqsDMNVEsRZpSE3jd2Z33RDWyW21PGOKB/ZrLiliKfdJbd3Wo7AN7bWsZpQsei2Hsv1niQIDAQAB"
@ IN SPF "v=spf1 mx a -all"
@ IN TXT "v=spf1 mx a -all"
_dmarc IN TXT "v=DMARC1; p=none; adkim=s; aspf=r; rua=mailto:dmarc-feedback@domain.net; pct=100"
</code>

All you have to do now is publish those records

==== Load ====
The loadcheck plugin can temporarily deny inbound emails if your server is overloaded. This plugin is always enabled and has a single setting:

  * MaxLoad (int number): Default is 7. If your load is above this value, emails from the outside will be deferred.