Return a simple mapping of trivial rulesets for given update channel

[Hainish]

Type: feature request

Reference: https://trac.torproject.org/projects/tor/ticket/28005

Exciting news - Tor Project is using our update channels in their alpha release to assist with the discovery of next-generation onion services!

To move this forward, Tor Browser developers have asked us to provide a hook in HTTPS Everywhere to return an object which contains a mapping of { host, to } for simple rulesets in a given update channel. This should work for rulesets in the format specified in the example ruleset channel Tor has provided us.

[Hainish]

I'll begin work on it this week.

[pipboy96]

@Hainish Wouldn't it make more sense to separate this from HTTP to HTTPS redirects, and how would it work if we deprecated all to="http://*" rules?

[Hainish]

@pipboy96 I think in order to support this, making deprecation of all to="http://*" URLs an option in the codepath that we can default to true rather than ripping out all the code for this is preferable.

[pipboy96]

@Hainish You mean Tor Browser's HTTPS Everywhere version is now going to be different from the regular one?

[Hainish]

@pipboy96 I think we could add a hook for HTTPSE to flip that option programmatically. Having a different extension in Tor Browser seems to invite a long-term maintenance burden

[pipboy96]

@Hainish I get it, but how is that flag going to be activated? WebExts can no longer read user.js? Will we just turn it on based on user agent?

[cschanaj]

The JSON in default.rulesets.1579180748.gz looks like this when prettified.

{
    "timestamp": 1579180748,
    "reference": "4437a4168878143ce3fdc63bbd355d537b1cea57",
    "rulesets":
    [
        {
            "name": "Nytimes.securedrop.tor.onion",
            "target":
            [
                "nytimes.securedrop.tor.onion"
            ],
            "rule":
            [
                {
                    "from": "^http[s]?://nytimes.securedrop.tor.onion/",
                    "to": "https://nyttips4bmquxfzw.onion/"
                }
            ]
        }
    ]
}

So HTTPS Everywhere is going to keep a list of *.tor.onion address specifically for the Tor Browser?

[Hainish]

@pipboy96 at the browser level, they can do this through the extension message passing interface such as they do in acatarineu/tor-browser@28005#diff-f2877e5f680c088366ab9ef15860e3e1R43

[Hainish]

@cschanaj not specifically for *.tor.onion, I'd like to be more generic than that. I'm not exactly sure the implementation details just yet, but I think we could request it for a specific update channel or perhaps the request for mapping could include a regex filter.

[cschanaj]

I tried the test build in the https://trac.torproject.org/projects/tor/ticket/28005, this is a fabulous enhancement to the onion browsing experience where a human-readable address can be used instead of the random-looking one. HTTPSE should definitely help to move forward this.

[pipboy96]

@cschanaj They will keep these as a separate ruleset bundle which would be updated using update channels feature.

[pipboy96]

@Hainish Should enabling this flag also add the update channel, or would it be added in some other way?

[pipboy96]

@cschanaj There is a problem with this screenshot. The certificate doesn't match the domain (it should only match the "real" onion address), but the browser shows that it does. Was it automatically whitelisted by browser or is it unintentional?

[jayvdb]

to="http://

IIUC, there is nothing to deprecated in the rules in this repo -- the three which use this are all disabled, and I have made it an assertion in my library as I was told that it was already firm policy not to downgrade at #18867 (comment) .

git grep -E '(to="http://|downgrade)'
Canon.xml:      <!-- Domains without TLS support/with downgrade/Broken certificate:
Cato-Institute.xml:             to="http://www.cato.org/$1" downgrade="1" /> -->
Tesco.xml:              to="http://www.tesco.com/" downgrade="1" /-->
Yandex.xml:             to="http://static-maps.yandex.ru/" downgrade="1" /-->

[pipboy96]

@jayvdb The problem is, onion services usually use HTTP, and we don't allow rewrites that result in a http:// URL.

[Hainish]

@pipboy96

@jayvdb The problem is, onion services usually use HTTP, and we don't allow rewrites that result in a http:// URL.

We enforce this through policy, not in the code. And that policy only effects the default ruleset update channel, not any others.