Enforce Grammar for default_off Messages

[cschanaj]

Reason

Unlike the platform attribute, there is not any grammatical requirement on the default_offattribute, This make the process of performing automated check to re-activate disable ruleset difficult.

To illustrate the situation, we have mismatch, mismatches, mismatched and their capitalization variants for the mismatched error. These variants make it hard for contributors to perform audit. See #9582, #9842

A particularly bad example could be BufferedIO.xml. It should be a platform=cacert ruleset, but the message is given in default_off. (It is a mismatch now)

Goal

The goal is to standardize the default_off attributes and to enforce the grammar with relaxng.xml such that automated audit can be done easier.

Steps

1. Modify default_off attributes in the existing ruleset to an agreed set of keyword, add the ruleset to ruleset-coverage-whitelist.txt if necessary. See Update default_off="mismatched" rulesets #9884.

2. Update CONTRIBUTING.md to explain the keyword we use.

3. Enforce the grammar with relaxng.xml.

Resource

Patterns

Each keyword should be separated by a comma and a space

(keyword1|keyword2)(, (keyword1|keyword2))*

Note

Some rule use ' (single quote) instead of " (double quote), it will be great if you can do a replacement as wel; :)

Keywords

List of keywords (feel free to suggest new ones)

regional
refused
timeout
cert-algo
cert-chain // curl: (60) SSL certificate problem: unable to get local issuer certificate
expired
self-signed
mismatched
ssl-error // reset, or any other ssl error not stated above 
status-unexpected // 200 Error pages
status-others // other than 200, i.e. 4xx/5xx
loops // secure connection redirects to plaintext
content-different // visual/ contextual difference, also for different status code
breaks-site // in a board sense.
breaks-third-parties
cors  // CORS issues 
ruleset-test-failed // failed ruleset test (avoid space within keywords)
request-owner
request-user
others // details should be stated in the ruleset comment

Existing Variants (200+ Variants)

Updated 2017.06.28 Please refer to default-off-attributes.txt

[Bisaloo]

Aren't cert-chain and cert-local-issuer the same thing?

[Bisaloo]

• [York.edu] Add subdomains and reactivate ruleset #9911 fixes Certificates are invalid
• [Pastebin.ca] Add subdomains and reactivate ruleset #9913 fixes No HTTPS support anymore
• [RecycleNow.com] Update, rename and reactivate ruleset #9918 fixes Site does not support HTTPS currently

[cschanaj]

@Bisaloo You're right, I will modify the list according.

[Bisaloo]

• What is the different between mismatched and status-mismatched?
• What do you propose when the https implementation is broken (SSL_ERROR_NO_CYPHER_OVERLAP or SSL_ERROR_RX_RECORD_TOO_LONG like for urlquery.net)?

[cschanaj]

@Bisaloo

• mismatched refers to a certificate mismatched error.

• status-mismatched refers to the situation where HTTP and HTTPS response clients with different HTTP status code, like HTTP (200) and HTTPS (404 with valid certificates). The problem is that it breaks Travis build and requires some test url. Now I think it should be merged with ruleset-test-failed, do you have any preference or suggestion?

• I would like a ssl-error keyword to be used in the situation you mentioned (and any other complex SSL errors not stated above) because these errors make little difference to the ruleset auditing process (please correct me if I am wrong).

Remark

• I would like to remove audit-required from the list of keyword because it is too ambiguous.

[Bisaloo]

status-mismatched refers to the situation where HTTP and HTTPS response clients with different HTTP status code, like HTTP (200) and HTTPS (404 with valid certificates). The problem is that it breaks Travis build and requires some test url. Now I think it should be merged with ruleset-test-failed, do you have any preference or suggestion?

I see. I would put that as content-different.

I would like a ssl-error keyword to be used in the situation you mentioned (and any other complex SSL errors not stated above) because these errors make little difference to the ruleset auditing process (please correct me if I am wrong).

No, I agree. I think we can also merge reset with ssl-error.

[cschanaj]

Alright. Let's put Different HTTP status code as content-different and reset as ssl-error.

Updated I think that we will need add a section for the default_off attributes in CONTRIBUTING.md when we are closed to resolving this issues.

[J0WI]

I'm not really happy with default_off="regional". Maybe a user wants to enable/disable only the rules for his own region, but not for other regions.

EDIT: Sorry, accidentally closed.

[jeremyn]

What exactly is the "ruleset auditing process" mentioned in #9906 (comment)? The specific process should make a difference in what keywords we want. (And if it doesn't make a difference, then what's the point of this issue anyway?)

I'm not sure what regional is for. I guess it's for the "only available in China" problems? Chinese sites are a special case due to the Great Firewall. We should get @gloomy-ghost 's and @ivysrono 's input on what they would like to see for that, though I know they are both busy with other things right now.

We should have only a small number of categories. Each extra category is one more thing to get confused about and one more potential argument to have with contributors.

For a while now in reviewing rulesets I've preferred invalid certificate to a more specific categorization in ruleset top comments. Here I would combine expired, self-signed, mismatched, cert-chain, and cert-algo into invalid-certificate.

I think other is slightly better wording than others.

Small, infrequent problems l should be moved into other. For example if there are only a few CORS rulesets, then cors should be included under other.

I don't know what request-owner and request-user mean.

We might want to allow extra attributes, such as date-expired="2017-05-14" for invalid-certificate.

Without knowing more about what an audit looks like, I would prefer just these keywords: regional (to cover "only available in/always breaks in X region"), refused, timeout, invalid-certificate, other, and possibly cors if there are a lot of default_off for those.

[jeremyn]

I just want to add that I think it's great people are working on automating the review process 😃

[cschanaj]

@jeremyn thank you for joining the discussion!

What exactly is the "ruleset auditing process" mentioned in #9906 (comment)?

To me, the audit process means to re-activate (or delete) the default_off rulesets if possible (either automatically/ manually). We have around 4k default_off ruleset and quite a number of them have not been re-activate for years. When a cleanup is necessary, the huge number of variants of the default_off make the process troublesome.

In particular, a contributor might want to focus on a specific type of default_off, e.g. an expired is much more likely to be fixed by web-master than a refused. And a cert-chain might require contributor to use a new Firefox profile. Without a consistent default_off message, a large portion of time will be spent on searching for ruleset.

I'm not sure what regional is for. I guess it's for the "only available in China" problems?

Yes. There is only 5 files will be applying the regional keyword AFAIK. See #9929. @gloomy-ghost, @ivysrono please let me know what do you prefer.

We should have only a small number of categories.

I agree that we should have fewer categories. Suppose that we have an automated process since they are basically the same to a robot (or a script), but I am not sure if this will affect manual audit or not. (I use a bot to re-generate a ruleset for a default_off host almost every time I would like to perform an audit and I do create default_off ruleset myself). :)

I am OK with expired, self-signed, mismatched, cert-chain and cert-algo merged into the category cert-invalid if there is no problem with that.

I don't know what request-owner and request-user mean.

request-owner mean web-master/ site owner want to have the ruleset disabled, like #9528

request-user https-everywhere user reported that the ruleset is causing problems, like #9371

We might want to allow extra attributes, such as date-expired="2017-05-14" for invalid-certificate.

If adding extra attributes is an option, I would really love to see an date-expired attribute. Beside, we might want to have something like phase-out="2022" (YYYY+5) to schedule the removal of default_off ruleset.

I would prefer just these keywords: regional (to cover "only available in/always breaks in X region"), refused, timeout, invalid-certificate, other, and possibly cors if there are a lot of default_off for those.

Which category should failed ruleset test go into in this case? We have 1.7k such ruleset (nearly half of the the default_off

[jeremyn]

@cschanaj I get the gist of what the auditing process is -- someone does a manual or semi-automated search over the rulesets looking for categories of default_off -- but the specific process/program/script used and its goals matter. For example if we want a detailed analysis then we should use fine-grained categories, but if we are trying to quickly identify "junk" rulesets then maybe broad categories are better.

Perhaps more compelling is that I don't like the idea of the hassle of constraining default_off values to support an auditing program that we can't see and have no input into. We don't want to have to justify these categories to contributors on the sole basis of "@cschanaj says this helps them with their auditing script" (no offense intended to you). Even worse is imagine in five years, suppose none of us here are involved with the project any more, and the future maintainers see the categories in the XML grammar but have no idea why they are there and what may break if they change them or if they add new categories. It introduces a dependency that we can't publicly see.

For regional, maybe we should mention China (gfw-of-china-issues or something) specifically since it is so far the only region that has special issues.

request-owner could instead be at-website-admin-request, so that default_off="at-website-admin-request" reads in a nice way.

request-user is not a valid category, we should only disable a ruleset if there is a verifiable reason or the site owner requests it. We wouldn't say default_off="someone-reported-a-problem".

I disagree with a phase-out attribute, we don't need to include a specific policy like a five-year grace period in the ruleset. Also doing even that simple math by hand would be tedious and error-prone. People doing the later analysis can decide if a ruleset should be disabled based on policy or common sense.

failed ruleset test is also not a valid category, it basically means broken. Since you say there are 1.7k of these rulesets then unfortunately unless a robust automated review program appears, no one is going to systematically review them all, so I suggest we mark them as unknown-legacy or something as a one-off. Future maintainers can delete them as a group if they become too burdensome to keep for some reason.

[jeremyn]

@Hainish This issue is getting slightly complicated and has future maintenance implications, so you may want to review this and provide feedback.

[pipboy96]

@zoracon Do you think this should be enforced by ruleset checker at some point?

[zoracon]

@zoracon Do you think this should be enforced by ruleset checker at some point?

To be honest, not sure. I want to go over the ruleset checker at some point after I finally get the fetch test issues resolved. But closing this for now since it seems like this came to a decent conclusion.