Privacy 2020 queries

[max-ostapenko]

Progress on #913

Online tracking

• top100_trackers_by_websites.sql - using EasyList; by device/category
• percent_of_tracked_websites_by_country.sql
• percent_of_websites_with_fingerprint.sql
• Persistent identifiers (browser storage)

Cookies

• top100_cookies_set_from_header.sql
• percent_of_websites_with_iab_tcf_banner.sql
• percent_of_websites_with_atsdd_schema.sql
• percent_of_websites_with_cmp.sql
• percent_of_websites_by_cmp.sql

Privacy policies

• percent_of_websites_with_privacy_links.sql

[rviscomi]

Be sure to update this from "Draft" to "Ready for review" so we can get more eyes on it

[rviscomi]

Hi @max-ostapenko can you give us an update on the status of this chapter's analysis? Are there only 2 queries?

[rviscomi]

@max-ostapenko how's this one coming along? Do you need any help?

[max-ostapenko]

@rviscomi I lost my billing account last week by accident so have wasted a couple of days to get a new free trial credit.
Now I'm finishing 3rd parties chapter.
Looking forward to your review after pushing the new queries.

[foxdavidj]

@max-ostapenko How are the queries coming along? Is this something you think you can finish this week?

[max-ostapenko]

@OBTo @ydimova I'm finishing with 3rd parties chapter today, and expect to have all queries for Privacy chapter drafted and data available in Sheets within next couple days.

[rviscomi]

@max-ostapenko how is this analysis coming along? This is the last week to get the analysis in.

[rviscomi]

Hi @max-ostapenko. Checking in again on this PR. Do you think you can get this finished by the end of the day today?

@ydimova you've also expressed an interest in contributing to the analysis, are you able to help complete the remaining queries?

[max-ostapenko]

@rviscomi visualised half of the queries. And finishing CMPs and trackers stats.
Maybe you could have a look at the committed ones already?

[rviscomi]

Hi @max-ostapenko, I've made some changes to the chart in the first sheet so that desktop and mobile are shown as separate series/colors:

Note that I hid the cell with the column name of the percentage to avoid it showing up in the chart with its raw SQL name.

Could you copy/paste that chart to the other sheets to retain the same formatting, and update them with the new sheets' data?

When do you expect to have the remaining queries implemented and ready for review? I'm very concerned about this chapter meeting the December 9 deadline given that it's been blocked on analysis for a while.

[max-ostapenko]

@rviscomi still need some time to identify how to make a query on top of EasyList.

[rviscomi]

Is this the list? https://easylist.to/easylist/easylist.txt

The way I'd do it would be to parse the list in JS and generate something that can be processed by SQL, like an array of objects. The prefixes also look significant but I'm not sure what they mean. They can be fields of the object, for example: [{prefix: '@@', params: '&foo&bar'}]. Then in the SQL you can return that from a UDF to get it strictly typed. Unnest the list in SQL, join it with the request URLs, and you're on your way. Hope that helps.

[ydimova]

@max-ostapenko here you can find an overview of the different rules and what they mean https://adblockplus.org/forum/viewtopic.php?t=7702&start=0. We could make a list of regex expressions for instance if we can parse the rules correctly.
Another option is to use WhoTracksMe. They have the eTLD+1 of the trackers.
Let me know if you want me to give it a try for easlylist or WhoTracksMe.

[rviscomi]

There are also ~3k known ad domains in the httparchive.almanac.third_parties table:

SELECT
  domain AS host
FROM
  `httparchive.almanac.third_parties`
WHERE
  date = '2020-08-01' AND
  category = 'ad'

You could join those with NET.HOST(request_url) to determine if it's an ad. cc @patrickhulce

[tunetheweb]

Can you merge main into this branch to pick up #1535 to avoid check errors that will happen next time you change this branch.

[rviscomi]

@max-ostapenko @ydimova can you give a status update on this analysis? Friendly reminder that we're launching in ~2 weeks ☺️

[rviscomi]

I'm going to merge these queries as-is and any changes can be applied in a follow-up PR.