Batch trivialize unnecessary cloudfront.net rewrites #15998

cschanaj · 2018-07-02T03:32:43Z

Attached the source code used to perform the batch rewrites

https://gist.github.com/cschanaj/31abc5c1f08416683aeb54e74d3846ff

Bisaloo · 2018-07-02T08:29:51Z

src/chrome/content/rules/CKSource.com.xml

@@ -55,7 +55,7 @@


 	<rule from="^http://c\.cksource\.com/"
-		to="https://d2xc7e2akbvihw.cloudfront.net/" />
+		to="https://c.cksource.com/" />


This rule can be removed (covered by the trivial rewrite)

Bisaloo · 2018-07-02T08:30:35Z

src/chrome/content/rules/ComiXology.com.xml

@@ -87,7 +87,7 @@
 		to="https://www.comixology.com/" />

 	<rule from="^http://support\.comixology\.com/"
-		to="https://d3jyn100am7dxp.cloudfront.net/" />
+		to="https://support.comixology.com/" />


This rule can be removed (covered by the trivial rewrite)

Bisaloo · 2018-07-02T08:31:16Z

src/chrome/content/rules/DealerRater.com.xml

@@ -28,9 +28,9 @@
 		to="https://$1dealerrater.com/" />

 	<rule from="^http://cdn-static\.dealerrater\.com/"


This subdomain is not listed in the targets

Bisaloo · 2018-07-02T08:31:23Z

src/chrome/content/rules/DealerRater.com.xml

@@ -28,9 +28,9 @@
 		to="https://$1dealerrater.com/" />

 	<rule from="^http://cdn-static\.dealerrater\.com/"
-		to="https://d3go5n7g2hd1g0.cloudfront.net/" />
+		to="https://cdn-static.dealerrater.com/" />

 	<rule from="^http://cdn-user\.dealerrater\.com/"


This subdomain is not listed in the targets

Bisaloo · 2018-07-02T08:32:51Z

src/chrome/content/rules/Filemobile.xml

@@ -44,7 +44,7 @@
 		to="https://s3.amazonaws.com/$1.filemobile.com/" />

 	<rule from="^http://cimg\.filemobile\.com/"
-		to="https://d2ksawk1637r8x.cloudfront.net/" />
+		to="https://cimg.filemobile.com/" />


This rule can be removed (covered by the trivial rewrite)

Bisaloo · 2018-07-02T08:33:11Z

src/chrome/content/rules/GCaptain.com.xml

@@ -70,7 +70,7 @@ Non-2xx HTTP code: http://cfs1.gcaptain.com/ (200) => https://d2m8pnbf2v4ae2.clo
 		to="https://d38ecmhxsvwui3.cloudfront.net/" />

 	<rule from="^http://cfs1\.gcaptain\.com/"
-		to="https://d2m8pnbf2v4ae2.cloudfront.net/" />
+		to="https://cfs1.gcaptain.com/" />


This rule can be removed (covered by the trivial rewrite)

Bisaloo · 2018-07-02T08:34:20Z

src/chrome/content/rules/MESO-Rx.xml

@@ -47,7 +47,7 @@ Non-2xx HTTP code: http://www.thinksteroids.com/ (200) => https://www.thinkstero


 	<rule from="^http://cdn\.thinksteroids\.com/"
-		to="https://d1uxx6tvzy89fj.cloudfront.net/" />
+		to="https://cdn.thinksteroids.com/" />


This rule can be removed (covered by the trivial rewrite)

Bisaloo · 2018-07-02T08:35:44Z

src/chrome/content/rules/Rovio.com.xml

@@ -65,7 +65,7 @@
 		<test url="http://assets.rovio.com/rovio/favicon.ico" />

 	<rule from="^http://assets-production\.rovio\.com/"
-		to="https://ductl1gjw76cl.cloudfront.net/" />
+		to="https://assets-production.rovio.com/" />


This rule can be removed (covered by the trivial rewrite)

Bisaloo · 2018-07-02T08:35:56Z

src/chrome/content/rules/Sailthru.xml

@@ -112,7 +112,7 @@ Fetch error: http://me.sailthru.com/ => https://www.sailthru.com/: Too many redi
 		to="https://na-sj08.marketo.com/" />

 	<rule from="^http://media\.sailthru\.com/"
-		to="https://d1t6td7a2qcurl.cloudfront.net/" />
+		to="https://media.sailthru.com/" />


This rule can be removed (covered by the trivial rewrite)

Bisaloo · 2018-07-02T08:37:44Z

src/chrome/content/rules/scmagazine.com.xml

@@ -60,7 +60,7 @@
 		to="https://www.scmagazine.com/" />

 	<rule from="^http://media\.scmagazine\.com/"
-		to="https://dw5yb2c18zulu.cloudfront.net/" />
+		to="https://media.scmagazine.com/" />


This rule can be removed (covered by the trivial rewrite)

Bisaloo · 2018-07-02T08:39:09Z

I suggested simplifications. It's okay if you think the manual removal of redundant rules is outside the scope of this PR.

On the other hand, #15998 (review) and #15998 (review) should be addressed.

cschanaj · 2018-07-02T09:19:03Z

@Bisaloo I have removed the redundant rules and addressed the missing targets. It seems that DealerRater.com.xml require a bit more modifications. Travis is passing now. Thank you very much!!

Bisaloo · 2018-07-02T09:24:49Z

I'll wait a bit before merging in case @gloomy-ghost @Hainish @jeremyn or @J0WI wants to double check.

J0WI · 2018-07-03T22:54:24Z

I'm not so familiar with nodejs, but I think this is a valuable enhancement.
How did you find the hosts in list.js?

Hainish · 2018-07-04T00:14:17Z

In general ruleset parsing scripts should be checked in, along with the methodology to generate any data sets that the parsing script relies on (such as the hosts in list.js).

Some of the host clean-up scripts that @RReverser has run haven't been checked in, but I think this is a good practice in general, and we may want to start a util/ just for this.

RReverser · 2018-07-04T01:01:25Z

Some of the host clean-up scripts that @RReverser has run haven't been checked in

I think I have checked in all of them, which one you have in mind?

RReverser · 2018-07-04T01:04:03Z

Looking at the diff in this PR, many of these should be possible to actually trivialise further by running node utils/trivialize-rules on top of these changes. @cschanaj would you want to try that?

Bisaloo · 2018-07-05T11:11:31Z

@RReverser, it looks like this was done already in 61cec8e. Do you think the subsequent commits allow for further simplification by the script?

RReverser · 2018-07-05T11:17:00Z

@Bisaloo Ah, I've just seen Adk2.xml which can be trivialised further, but it can't by automatic script as there is securecookie tag (which might be used for cookie on other subdomains).

cschanaj · 2018-07-05T22:46:27Z

@J0WI @Hainish the script I used to find the host is rather dirty and include a bit of manual steps.

./make.sh
node index.js > input.txt
cat input.txt | while read host; do curl -IsS https://$host/ -o /dev/null && echo $host; done > success.txt
manual changes to success.txt to list.json

https://gist.github.com/cschanaj/2ee89cb791eeda0e764b167c5c3f4ede

@RReverser I agree that some rulesets can be futher trivialized but the securecookie tags forbidden that. I tried to make some changes (still work in progress) to the script to identify static cookies and it seem to work. I can submit a PR later if I can verify the modified script is really working. https://github.com/cschanaj/https-everywhere/blob/trivialize-rules.js/utils/trivialize-rules/trivialize-rules.js

Hainish · 2018-07-05T23:42:14Z

@cschanaj it's okay if the scripts are sloppy, I think the important part is that the process is documented and can be reproduced in the future. If someone comes along and cleans it up (or you have time to down the line), all the better.

Bisaloo · 2018-07-14T06:46:16Z

@cschanaj, it would be great if you could add your script to the repo as suggested by @Hainish.

Otherwise, this is ready to merge.

cschanaj · 2018-07-15T08:33:47Z

@Bisaloo @Hainish I have pushed a final version of the script in 87a9087. Sorry for force pushing a few time. This PR is ready to merge if you agree.

Bisaloo · 2018-07-16T16:03:24Z

Can I edit the commit message to remove intermediate commits and just keep Batch trivialize unnecessary cloudfront.net rewrites when I squash and merge or do you wish to keep all intermediate steps?

cschanaj · 2018-07-16T22:34:31Z

@Bisaloo please feel free to edit the commit message. thanks!!

Hainish · 2018-07-17T01:29:21Z

Thanks for this PR @cschanaj and for the review @Bisaloo!

cschanaj closed this Jul 2, 2018

cschanaj reopened this Jul 2, 2018

Pasu Chan Chak Shing added 7 commits July 2, 2018 12:22

Batch trivialize unnecessary cloudfront.net rewrites

aa3f52b

Remove cdni.condenast.co.uk from Conde_Nast.co.uk.xml, gone from DNS

c5553f1

Remove jawbone.com from Jawbone.com.xml, timeout

3550f0f

node utils/trivialize-rules/trivialize-rules.js

61cec8e

Remove forums and www.forums from RightScale.com.xml, self-signed

02caaa9

Whitelist failing rulesets from rules-test

0d6ac63

Remove thumbnails.inkfrog.com from InkFrog.com.xml, expired

bcf3fac

cschanaj closed this Jul 2, 2018

cschanaj reopened this Jul 2, 2018

Fix a typo in IndiaMART.xml

64dbd3a

Bisaloo reviewed Jul 2, 2018

View reviewed changes

Bisaloo self-assigned this Jul 2, 2018

Pasu Chan Chak Shing added 6 commits July 2, 2018 16:47

Remove complex rules for c.cksource.com

b6c7757

Remove complex rules for support.comixology.com

a7e6b2d

Add cdn-static and cdn-user to DealerRater.com.xml

2db7289

Remove complex rules for cimg.filemobile.com

891c109

Remove complex rules for cfs1.gcaptain.com

67eda8c

Remove complex rules for cdn.thinksteroids.com

0ea31c5

Pasu Chan Chak Shing added 5 commits July 2, 2018 16:52

Remove complex rules for assets-production.rovio.com

41a8961

Remove complex rules for media.sailthru.com

0ca9ffb

Remove complex rules for media.scmagazine.com

ba54f8a

Update DealerRater.com.xml, no longer redirects to http

a57dafc

Remove cdn-static.dealerrater.com, fix fetch-test

2979aae

Hainish added the top-1k label Jul 9, 2018

cschanaj mentioned this pull request Jul 12, 2018

Detect static securecookies to trivialize more rules #16029

Merged

Add scripts to trivialize complex rewrites to CDNs

87a9087

Bisaloo merged commit 812d5bb into EFForg:master Jul 17, 2018

Bisaloo removed their assignment Jul 17, 2018

cschanaj deleted the cf-rewrite branch July 17, 2018 12:59

cschanaj mentioned this pull request Jul 26, 2018

Trivialize complex rewrites to Cloudfront.net #15970

Closed

88 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Batch trivialize unnecessary cloudfront.net rewrites #15998

Batch trivialize unnecessary cloudfront.net rewrites #15998

cschanaj commented Jul 2, 2018 •

edited

Loading

Bisaloo Jul 2, 2018

Bisaloo Jul 2, 2018

Bisaloo Jul 2, 2018

Bisaloo Jul 2, 2018

Bisaloo Jul 2, 2018

Bisaloo Jul 2, 2018

Bisaloo Jul 2, 2018

Bisaloo Jul 2, 2018

Bisaloo Jul 2, 2018

Bisaloo Jul 2, 2018

Bisaloo commented Jul 2, 2018

cschanaj commented Jul 2, 2018 •

edited

Loading

Bisaloo commented Jul 2, 2018

J0WI commented Jul 3, 2018

Hainish commented Jul 4, 2018

RReverser commented Jul 4, 2018

RReverser commented Jul 4, 2018

Bisaloo commented Jul 5, 2018

RReverser commented Jul 5, 2018

cschanaj commented Jul 5, 2018

Hainish commented Jul 5, 2018

Bisaloo commented Jul 14, 2018

cschanaj commented Jul 15, 2018

Bisaloo commented Jul 16, 2018

cschanaj commented Jul 16, 2018

Hainish commented Jul 17, 2018

		@@ -28,9 +28,9 @@
		to="https://$1dealerrater.com/" />

		<rule from="^http://cdn-static\.dealerrater\.com/"

Batch trivialize unnecessary cloudfront.net rewrites #15998

Batch trivialize unnecessary cloudfront.net rewrites #15998

Conversation

cschanaj commented Jul 2, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Bisaloo commented Jul 2, 2018

cschanaj commented Jul 2, 2018 • edited Loading

Bisaloo commented Jul 2, 2018

J0WI commented Jul 3, 2018

Hainish commented Jul 4, 2018

RReverser commented Jul 4, 2018

RReverser commented Jul 4, 2018

Bisaloo commented Jul 5, 2018

RReverser commented Jul 5, 2018

cschanaj commented Jul 5, 2018

Hainish commented Jul 5, 2018

Bisaloo commented Jul 14, 2018

cschanaj commented Jul 15, 2018

Bisaloo commented Jul 16, 2018

cschanaj commented Jul 16, 2018

Hainish commented Jul 17, 2018

cschanaj commented Jul 2, 2018 •

edited

Loading

cschanaj commented Jul 2, 2018 •

edited

Loading