Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve basic on-device data analysis #87

Closed
bassosimone opened this issue Oct 7, 2019 · 1 comment
Closed

Improve basic on-device data analysis #87

bassosimone opened this issue Oct 7, 2019 · 1 comment
Assignees
@bassosimone
Labels
enhancement New feature or request epic A collection of issues

Comments

@bassosimone
Copy link
Contributor

No description provided.

@bassosimone bassosimone added enhancement New feature or request P0 epic A collection of issues labels Oct 7, 2019
@bassosimone bassosimone self-assigned this Oct 7, 2019
@bassosimone bassosimone changed the title Basic, on-device data analysis 2.1.3 Improve basic on-device data analysis Oct 7, 2019
@hellais hellais removed OTFy2 labels Oct 7, 2019
bassosimone added a commit to ooni/netx that referenced this issue Oct 28, 2019
@bassosimone
Make events actionable for OONI's nervous system
f15f7e3 
After trial and error, I have determined that, in general, to decide
whether we need follow-up measurements, we need just the result of
four events in the HTTP round trip:

- RESOLVE_DONE
- CONNECT
- HANDSHAKE_DONE
- ROUND_TRIP_DONE

In some cases it may be possible to run operations inline while in
others it's probably best to run concurrently or later.

Regardless of the implementation strategy, this diff makes sure that
by combining this four events only, we have enough information to
conclude whether to trigger more measurements.

This conclusion is based on my analysis using ooni/jafar, it may be
of course that later we'll need to tweak this a bit.

I've been doing refactoring for quite some time to prepare this.

Master issue: ooni/probe-engine#87
@bassosimone bassosimone mentioned this issue Oct 28, 2019
Merged
bassosimone added a commit to ooni/netx that referenced this issue Oct 28, 2019
@bassosimone
Make events actionable for OONI's nervous system (#78)
518daf9 
After trial and error, I have determined that, in general, to decide
whether we need follow-up measurements, we need just the result of
four events in the HTTP round trip:

- RESOLVE_DONE
- CONNECT
- HANDSHAKE_DONE
- ROUND_TRIP_DONE

In some cases it may be possible to run operations inline while in
others it's probably best to run concurrently or later.

Regardless of the implementation strategy, this diff makes sure that
by combining this four events only, we have enough information to
conclude whether to trigger more measurements.

This conclusion is based on my analysis using ooni/jafar, it may be
of course that later we'll need to tweak this a bit.

I've been doing refactoring for quite some time to prepare this.

Master issue: ooni/probe-engine#87
bassosimone added a commit to ooni/netx that referenced this issue Oct 28, 2019
@bassosimone
dnsdialer: allow to divert lookuphost
80a70fe 
A MeasurementRoot can contain an alternative LookupHost function
which may be used to bypass the usual chain.

This is also functional to the OONI reactive system.

See ooni/probe-engine#87
@bassosimone bassosimone mentioned this issue Oct 28, 2019
Merged
bassosimone added a commit to ooni/netx that referenced this issue Oct 28, 2019
@bassosimone
dnsdialer: allow to divert lookuphost (#79)
944c6ea 
A MeasurementRoot can contain an alternative LookupHost function
which may be used to bypass the usual chain.

This is also functional to the OONI reactive system.

See ooni/probe-engine#87
@bassosimone bassosimone mentioned this issue Oct 28, 2019
Merged
bassosimone added a commit to ooni/netx that referenced this issue Oct 28, 2019
@bassosimone
Returns follow-up experiment types for TLS
991be9c 
Work related to ooni/probe-engine#87
@bassosimone bassosimone mentioned this issue Oct 28, 2019
Merged
bassosimone added a commit to ooni/netx that referenced this issue Oct 28, 2019
@bassosimone
Returns follow-up experiment types for TLS
b765c3c 
Work related to ooni/probe-engine#87
bassosimone added a commit to ooni/netx that referenced this issue Oct 28, 2019
@bassosimone
Returns follow-up experiment types for TLS
22017dc 
Work related to ooni/probe-engine#87
bassosimone added a commit to ooni/netx that referenced this issue Oct 28, 2019
@bassosimone
Returns follow-up experiment types for TLS
cf27fec 
Work related to ooni/probe-engine#87
bassosimone added a commit to ooni/netx that referenced this issue Oct 29, 2019
@bassosimone
Make nervous system data more immediately actionable
6e04eb4 
Implement a scoreboard and write there what we see. The follow up
action may or may not be required.

This is also functional to the OONI reactive system.

See ooni/probe-engine#87
@bassosimone bassosimone mentioned this issue Oct 29, 2019
Merged
bassosimone added a commit to ooni/netx that referenced this issue Oct 29, 2019
@bassosimone
Make nervous system data more immediately actionable (#84)
48cfb0e 
Implement a scoreboard and write there what we see. The follow up
action may or may not be required.

This is also functional to the OONI reactive system.

See ooni/probe-engine#87
bassosimone added a commit to ooni/netx that referenced this issue Oct 29, 2019
@bassosimone
Extend scoreboard to also track TLS handshake RST
1470a23 
The scoreboard package will probably benefit from some more
precise unit testing with respect to returned keys.

Work related to ooni/probe-engine#87
bassosimone added a commit to ooni/netx that referenced this issue Oct 29, 2019
@bassosimone
Extend scoreboard to also track TLS handshake RST (#82)
5972a62 
The scoreboard package will probably benefit from some more
precise unit testing with respect to returned keys.

Work related to ooni/probe-engine#87
@hellais hellais changed the title 2.1.3 Improve basic on-device data analysis Improve basic on-device data analysis Oct 31, 2019
bassosimone added a commit to ooni/netx that referenced this issue Oct 31, 2019
@bassosimone
Expose the scoreboard
c4c3676 
Functional to enable follow-up measurements.

Xref: ooni/probe-engine#87
@bassosimone bassosimone mentioned this issue Oct 31, 2019
Merged
bassosimone added a commit to ooni/netx that referenced this issue Oct 31, 2019
@bassosimone
Expose the scoreboard (#107)
5197c35 
Functional to enable follow-up measurements.

Xref: ooni/probe-engine#87
bassosimone added a commit to ooni/netx that referenced this issue Oct 31, 2019
@bassosimone
porcelain: Implement automatic DNS fallback
099ad99 
Xref:
- ooni/probe-engine#87
- ooni/probe-engine#88
@bassosimone bassosimone mentioned this issue Oct 31, 2019
Merged
bassosimone added a commit to ooni/netx that referenced this issue Oct 31, 2019
@bassosimone
porcelain: Implement automatic DNS fallback (#108)
894dd76 
Xref:
- ooni/probe-engine#87
- ooni/probe-engine#88
This was referenced Nov 1, 2019
Merged
Merged
@bassosimone
Copy link
Contributor Author

bassosimone commented Nov 1, 2019 •
edited
Loading

I am going to mark this issue as complete. The https://github.com/ooni/netx library is the place where I have implemented the following functionality:

  1. extraction of low-level events that are "rooted" to a specific HTTP request, dial for cleartext or TLS connection, system/UDP/TCP/DoH/DoT domain name resolution

  2. the above has been implemented by mocking several Go standard library interfaces and standard HTTP-observing callbacks to obtain as much events as possible without adding artificial restrictions (e.g. not running concurrent measurements)

  3. easy binding of such events when post-processing data because there are a bunch of consistent IDs that allow to join data collected at DNS level, with data collected a network level, with data collected at HTTP level

  4. easy attribution of errors to major operations that failed (DNS resolution, TCP connect, TLS handshake, HTTP protocol interaction) by returning an implementation of the error interface that contains this information (this is less easy that it seems; when a DoH resolve fails because of a TLS handshake error, you want the failed operation to be the TLS handshake, not the parent DNS lookup)

  5. functionality called "scoreboard" to record odd stuff that we've noticed and we want to investigate later

  6. implementation of automatic SNI blocking follow-up mini-experiment based on the "scoreboard" that has been discussed with @fortuna as part of a broader scope design document

  7. automatic detection of "bogon" replies (e.g. 10.4.4.17) and fallback to preconfigured DoT or DoH resolvers in such case, thus also performing an opportunistic DoT/DoH mini experiment

To implement all of this, I needed to deeply refactor the library. I have tagged this issue in a bunch of refactoring PRs that are mostly relevant to it, however, I'd say that all commits since ooni/netx@556048a until ooni/netx@aa7ac63 have been guided by the objective of implementing all the above.

On this note, I also wrote a tool https://github.com/ooni/jafar to simulate several kind of censorship and of unreliable networks (for #88) such that I could test whether the code I was writing was correctly measuring specific scenarios and was able to correctly bypass SNI blocking and react to returned bogon address.

This was referenced Nov 1, 2019
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bassosimone bassosimone

Labels
enhancement New feature or request epic A collection of issues
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bassosimone @hellais