Safari Support (new safari API)

[jellyfishcoder]

The new safari API seems like it may support https-everywhere. https://developer.apple.com/library/prerelease/content/documentation/NetworkingInternetWeb/Conceptual/SafariAppExtension_PG/index.html#//apple_ref/doc/uid/TP40017319 is the documentation. It is currently for the new safari 10 in macOS Sierra. I believe it may allow for an extension to rewrite HTTP requests as HTTPS, but Im not completely sure.

[J0WI]

Is this also for iOS Safari?

[jellyfishcoder]

@J0WI, not yet, still only content blockers for ios, but it looks like extensions may be a future possibility.

[heubergen]

As far as I understand, the functionality could be done now under iOS with the make-https action as noticed here.
Can we have any comment from the devs about that?

[dato]

In particular, the above linked What’s New in Safair 10.0 says:

The make-https action is now available for iOS content blockers.

Just above that, it also says:

You can now create macOS-native Safari app extensions to sell and distribute in the App Store. Content Blockers for iOS can be easily ported to macOS. [Emphasis mine.]

Thanks in advance for investigating!

[Saklad5]

Could we have an answer as to the feasibility of HTTPS Everywhere using this new API?

[Hainish]

Related, this from email to our [email protected]:

I’m contacting you about your HTTPS Everywhere extension. You wrote in
your FAQ : « if you happen to know a way to perform secure request
rewriting in [Safari], feel free to let us know »

Well, there is now a way. In May 2016, Safari Content Blocking API has
been improved to be able to force HTTPS when available. It was then
tested in Safari Developer Preview. Eventually, the feature became
available to everyone in Safari 10 (iOS and macOS).

Here is the webkit thread :
https://bugs.webkit.org/show_bug.cgi?id=148966

Apple mentions the « make-https » action in Safari 10 at the bottom of
this page :
https://developer.apple.com/library/content/releasenotes/General/WhatsNewInSafari/Articles/Safari_10_0.html

Have you tried this ? Are you planning on a Safari version of HTTPS
everywhere ?

So it seems the answer is that it seems possible for at least the trivial rulesets. I'd have to investigate if it is possible for more complicated rulesets.

[Saklad5]

Even just the trivial rulesets would be better than nothing, right?

[CaioBianchi]

Any follow-up on this?

[edlebert]

I tried to get a safari extension built with the new content-blocker action, but I have never tried to build a safari extension before and I could not get it to work. :(

[javabean]

Should you want to play with a Safari Content Blocker Extension, you may want to start with the simple, no-frill https://github.com/krishkumar/BlockParty
A more advanced one (and probably overkill for research): https://github.com/AdguardTeam/AdguardForiOS

[edlebert]

I have it working locally. There are some other limitations: 1) New safari extensions need to be packaged with a native app, and I don't have that working yet. 2) This will only work with simple rules. It will not work for rewriting to different hosts, which is pretty common in HTTPS Everywhere. 3) It does nothing for secure cookies. 4) I don't have a developer account so my local extension cannot be distributed. But once I get a download-and-install version working I'll put up a github project for everyone.

[Saklad5]

Thanks for the attempt, despite the caveats.

I really wish Apple would just implement such functionality themselves if they aren’t willing to open up the API enough. This sort of feature is helpful for literally anyone using Safari, and Apple could easily fork HTTPS Everywhere and do this themselves.

[javabean]

@edlebert @Saklad5 you should probably create 2 issues (one for the redirection rules, one for the secure cookies) at https://bugreport.apple.com to let Apple know of the required functionality. It doesn't mean they will abide by your request, but at least they will know there is a functional need. ;-)

[CaioBianchi]

Webkit shipped Safari TP21 today: https://webkit.org/blog/7265/release-notes-for-safari-technology-preview-21/

Bugfix (?):
https://trac.webkit.org/changeset/209821

[infused-kim]

@edlebert do you have an update of this? Perhaps you could post whatever you have now on github?

[freediverx]

Any updates on this? Any comments from Apple?

[ecliptik]

With the passage of SJ Res 34 today, having Safari support for HTTPS Everywhere is most welcome.

[Saklad5]

@ecliptik Um, that was a few days ago, and isn’t binding until the House of Representatives passes it. However, since both chambers are apparently mostly comprised of people that are dangerously and inexcusably ignorant when it comes to technological security and privacy, your point is valid.

HTTPS won’t stop ISPs from monitoring what webpages you visit, though. For that, a VPN or proxy like Tor is necessary as well. macOS is very good at handling system-wide SOCKS proxies, so this isn’t too difficult. Homebrew makes it even easier to go that route.

[ecliptik]

@Saklad5 it was voted on in the house today, all that's left is for the president to sign it into law,

https//www.eff.org/deeplinks/2017/03/congress-sides-cable-and-telephone-industry

The EFF warns about broken encryption due with ISPs using Explicit Trust Proxies for MiTM attacks with it, we could use all the help we could get right now,

https//www.eff.org/deeplinks/2017/03/five-ways-cybersecurity-will-suffer-if-congress-repeals-fcc-privacy-rules

[fuzzy76]

AFAIK, deep packet inspection is way too much work for ISPs just for tracking users (the volume of packets is too high). So they probably stick to tracking DNS lookups, which HTTPS does not prevent. Changing DNS-server probably helps though. Just don't change to another provider that already sells your data, like the worlds largest advertising company or something.

[heubergen]

May we can leave the political disusscion on twitter and reddit and focus us here more on the technical aspect?

[afarrow]

I worked on this for a few hours last night and was able to make some progress. I put my work in a repo here: https://github.com/afarrow/HTTPS-Everywhere-Safari-Test

The readme should explain what I learned and provide a good starting point for anyone wanting to work more on this.

[jessesquires]

Thanks @afarrow ! 🙌

So it sounds like Safari support is blocked due to its inability to handle large JSON files? Is this a known issue or do you know of any workarounds? From this WWDC talk it sounds like it's possible to block dynamically via the Javascript API, too.

[afarrow]

@jessesquires

Yes, right now it seems the limitation is on the size of the JSON files. I also don't know of any way to do more complex rewrites (ex: rewrite example.com to secure.example.com), set the secure cookie flag or have exclusions. But I think there's value in the extension even if we can't do any of those things. I think the most promising area to explore is probably building the extension with XCode where you can use Swift or JavaScript. I haven't explored any workarounds to the JSON size issue yet but I'm going to spend some time later this week working on it.

[javabean]

As far as I know, there is a hard limit of 50k rules per extension (json), not on the binary size of the json.

[afarrow]

@javabean Yes, I ran into the 50k limit last night, thanks for confirming my suspicions. I don't know why the JSON files I was generating earlier were failing with a lot fewer rules than that. Perhaps an issue with the script I wrote. Do you know if that limit is per JSON file? For example, could you have 2 JSON files with 50k rules each to get around the limitation? Or is that 50k limit in place regardless of how many JSON files the rules are spread over?

Edit:
Oh, and just an update on the XCode idea. I tried to paste a ~20k line JSON file into XCode and it crashed. So I'm back to using the Safari Extension Builder but I'm now experimenting with using the JavaScript API that extensions have access to: https://developer.apple.com/library/content/documentation/Tools/Conceptual/SafariExtensionGuide/Introduction/Introduction.html

[javabean]

I'm not sure a Safari Extension would enable us to make-https: extensions can manipulate the DOM, but since it is already loaded it is too late to change the load type.
An extension can use content-blocking rules, but we're back to square 1 where we hit the 50k limit.

As far as I know, 1 extension can have only 1 Safari Content Blocker (and hence 1 JSON file, and 50k rules max).
You can still opt to build more than 1 extension to cover the whole range of rules we have, even it it will be weird for the user (she will have to install 2+ applications / extensions)!

How many rules do we currently have in the https-everywhere database (the ones that can be expressed as Safari Content Blocker rules)?

[afarrow]

I'm not sure a Safari Extension would enable us to make-https: extensions can manipulate the DOM, but since it is already loaded it is too late to change the load type.

The JavaScript API actually includes the class SafariBeforeNavigateEvent. This event fires "when the user navigates to a new page, before the page is loaded". So for example I experimented the other night with adding a Global HTML Page which is basically an HTML file that you use to add JavaScript to your extension. My global html file looks like this:

<!DOCTYPE html>
<html>
  <script src="./test.js"></script>
</html>

Then in the test.js file I was able to add:

safari.application.addEventListener('beforeNavigate', handleEvent, true);

function handleEvent(event) {...}

and the handleEvent function fired before the page was ever loaded. And as Apple states on the API page, "you can use the preventDefault method to prevent the new page from loading".

So my two ideas that stemmed from this were either:

1. Switching out the JSON files on the fly based on what site was requested (don't think this would work well if at all because Safari has to transform the JSON to bytecode)
2. Manually rewriting the URL in the handleEvent function. So far I haven't found a way to do that though

How many rules do we currently have in the https-everywhere database (the ones that can be expressed as Safari Content Blocker rules)?

A rough count is 82678 rules. I got this by running a script that ignored all files that had more than one rewrite rule, exclusions or had the default_off attribute. That's out of about 125k rules without any of those conditions.

Edit:
I added the HTML and JS files I mentioned above to my repo: https://github.com/afarrow/HTTPS-Everywhere-Safari-Test

[Saklad5]

Having 2 or 3 HTTPS Everywhere extensions would be far better than what we have now, which is nothing.

[Saklad5]

And as I understand it, the Safari Extension API is consistent across all of Apple’s ecosystem. By making a Safari version, you are also making an iOS version.

[heubergen]

@Saklad5 That's correct. A Safari Extension can easily be ported to an iOS App if we use the official API.

[chris1000]

Out of curiosity has support been picked up for Safari yet?

[Hainish]

@chris1000 not yet, and we don't have rolling out support for safari scheduled. This would involve ensuring that the necessary WebExtensions APIs are supported, in addition to the legal review of their privacy policy.

[bouk]

Hi everyone

I've been working on an iOS app version of HTTPS everywhere. For doing so I've created this script which processes the rulesets to extract a list of hosts that just require a 'simple' redirect: https://gist.github.com/bouk/bc4b1f406c3d4e83d28ade3f1b18a4e0

This list can then be inserted into a rule like

[
  {
    "action": {
      "type": "make-https"
    },
    "trigger": {
      "url-filter": ".*",
      "if-domain": [
"*3cdn.net",
"*3scale.net",
etc.
}
]

Which also supports more than the 50k rule limit, it seems there is no limit on the number in if-domain.

It would be good to hunt down overly complicated rulesets, essentially everything generated by make-trivial-rule is supported while others can't be guaranteed

[Saklad5]

@bouk That’s great! Safari’s content blockers are platform-agnostic as I understand it, so that will work on the Mac too!

[Revod]

Any news/updates about this?

[Hainish]

@Revod unfortunately, after legal review we found the Apple Developer License Agreement contains provisos that EFF is unable to comply with. This is for reasons similar to what we stated in this 2014 blog post.

We don't want to encourage others to sign the agreement. Third party implementations of HTTPS Everywhere using our rulesets certainly don't run afoul of our licensing of the project, but there's a caveat.

I'm not a lawyer, but my understanding is that Apple enforces restrictions on redistribution via its Licensing Agreement. So posting the project on the Safari store seems to be a violation of the GPL. So it seems that any contributor who had code merged under the GPL could appeal it's listing in the Safari store. This is an unfortunate outcome of being licensed restrictively, and this has been done in the past.

So I'm not a licensing expert, but I think if anyone chose to list HTTPS Everywhere in the Safari store it would be on shaky grounds.

[bouk]

@Hainish thank you for the reply.

Since the only part of the of the iOS app that would require GPL licensing is the ruleset, which is essentially just a .txt file with a list of hosts, it might be possible to make that file accessible and redistributable from the app, and license the rest of the app under Mozilla Public License. It seems that this would mean that people can redistribute the GPL-licensed part of the app as much as they want. The question is of course whether that's legally sound, or whether having that list of hosts be GPL-licensed means that the whole app needs to be GPL, which we would need legal advice on.

I think I will proceed with that plan, and we'll just see what happens.

[Hainish]

@bouk Something that may be helpful here: the way I understand it, the stipulations of the GPL are such that the restrictiveness only applies to code which is included in the same program. This differs from, say, the AGPL, which extends to network interaction. So here's what may interest you:

We plan on delivering the rulesets out-of-band of the extension itself, making it available on the web (along with a signature), so that the mainline HTTPS Everywhere extension as well as any third-party can download and apply these rules. If an iOS extension were to be developed, it could pull in the rulesets (which have been pre-compiled and provided on a webserver) at runtime, and not run afoul of the GPL.

[Hainish]

@chris1000 unfortunately it won't be in the same format as iOS content blockers.

[bouk]

I have a working app now, the source code can be found here: https://github.com/bouk/HTTPS4All
This is the tool I use to convert the rules into a hosts list: https://github.com/bouk/https-everywhere-host-list. This is the uploaded to https://hosts.https4all.org/hosts.txt, which the app downloads every time it's opened.

I have a build up on testflight, please email my at the email on my GitHub profile if you would like to give it a shot.

[bouk]

Hey everyone, just an update. I’ve released the iOS version, and you can now download it here: https://itunes.apple.com/gb/app/https4all/id1305430042?mt=8

[wryanthomas]

Thanks 'bouk'. That's awesome.

Anyone know of (similar) progress on a Safari for Mac extension?

[pipboy96]

I guess this is ok to close since EFF already said they can't enter a required contract with Apple, and there will be no official version of HTTPS Everywhere for Apple.

[fuzzy76]

How about the new webextension support in Safari 14? Source: https://9to5mac.com/2020/06/24/apple-adds-webp-hdr-support-and-more-to-safari-with-ios-14-and-macos-big-sur/