Allow hosts to signal that Hardenize is allowed to test more frequently

[ivanr]

At the moment, although we allow reports to be refreshed, the results of SMTP assessments continue to be cached. This is because many SMTP servers are quite sensitive to too many connections, too fast. We wish to enable those who wish to be retested to signal so. This could be done with a special DNS TXT RR.

[ivanr]

Here's the plan, roughly:

1. We will support hosts publishing their policy in a DNS TXT RR record;
2. In the first version, the policy will support only two features, to allow or to forbid our assessments. Example policies: "v=H1; allow" and "v=H1; disallow". Crucially for this ticket, if assessments are allowed, we will not cache SMTP results.
3. We will look for the policy first under the "_hardenize" subdomain of the host being tested (e.g., _hardenize.blog.example.com); if not found we will go up the hierarchy (e.g., next to _hardenize.example.com), stopping when the first public suffix is encountered (e.g., _hardenize.com won't be checked).

[flummer]

So if fancywebsite.com has an MX record pointing to smtp.fancyhosting.com the TXT RR record should be at _hardenize.smtp.fancyhosting.com to control the mailserver caching, and the TXT RR record at _hardenize.fancywebsite.com will control the caching for everything else, right?

[ivanr]

@flummer that's correct, although a record at _hardenize.fancyhosting.com would work equally well.

[ivanr]

By the way, before we do any work on this ticket, we've decided to try to reduce caching. As of a few minutes ago (in v1.0.927), we force caching only for very widely used SMTP servers, extracted from Alexa's top 1m list. As a result, small operators will see fresh results every time.

[AliceWonderMiscreations]

I'll likely never be an Alexa Top 1000 so I probably don't need this DNS record, but what exactly do I need to do to make sure my SMTP servers (Postfix 2.11.11 soon to be 3.3.x) allows your rapid probe connections?

[ivanr]

@AliceWonderMiscreations That will depend on the error message. I am not familiar with Postfix, but IIRC there is an option that limits the number of connections in a period of time. Increasing that value should help. Or you can simply whitelist outbound.hardenize.com.