From e8ffc2e14b5d51390be0c5b3a12fcd27f2943dee Mon Sep 17 00:00:00 2001 From: jburg <38106050+JaxGames5225@users.noreply.github.com> Date: Fri, 23 Dec 2022 23:40:37 -0500 Subject: [PATCH 01/14] Created ts-039-shadowsocks.md file --- nettests/ts-039-shadowsocks.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 nettests/ts-039-shadowsocks.md diff --git a/nettests/ts-039-shadowsocks.md b/nettests/ts-039-shadowsocks.md new file mode 100644 index 00000000..8b137891 --- /dev/null +++ b/nettests/ts-039-shadowsocks.md @@ -0,0 +1 @@ + From 3f1e20dc1d6c6ec7ec9b8abf4f4cb264a9998cdd Mon Sep 17 00:00:00 2001 From: jburg <38106050+JaxGames5225@users.noreply.github.com> Date: Sat, 24 Dec 2022 00:09:48 -0500 Subject: [PATCH 02/14] Updated Shadowsocks Spec up Expected output --- nettests/ts-039-shadowsocks.md | 76 ++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) diff --git a/nettests/ts-039-shadowsocks.md b/nettests/ts-039-shadowsocks.md index 8b137891..8c8fadae 100644 --- a/nettests/ts-039-shadowsocks.md +++ b/nettests/ts-039-shadowsocks.md @@ -1 +1,77 @@ +# Specification version number +2022-12-23-000 + +# Specification name + +Shadowsocks + +# Test preconditions + +An internet connection + +# Expected impact + +Ability to detect the censorship of fully-encrypted protocols, specifically Shadowsocks + +# Expected inputs + +None + +# Test description + +The main goal of the test is to inform the user (and the community) whether or not they are experiencing censorship on connections that send fully encrypted packets that appear random, as well as to record information about censored packets in order to better understand the censorship algorithm. The test seeks to accomplish these goals by doing the following: + +1. If no IP is given by the user, select an IP from the list of IP addressed in the affected range +2. Complete a TCP handshake with the IP address and send a stream of zero bytes as a control test. If this control test succeeds then proceed with the test, otherwise end the test and return the error +3. Complete a TCP handshake with the IP address and send a stream of random bytes. If this connection times out, we attempt to connect once more to check for residual censorship. If the residual censorship test results in a timeout, we end the test, record information about the blocked packet, and inform the user they are experiencing censorship. Otherwise we continue with the test +4. Step 3 is repeated 19 more times to account for the blocking rate +5. If no errors occurred and the test was completed, all connections are then closed and the test informs the user they are not experiencing censorship. + +# Expected output + +## Parent data format + +This is the base data format(s) that this test will adhere to (it is +implicit that it will follow df-000-base). + +## Required output data + +This is data that should be part of the base dataformat without which the +test cannot properly be interpreted. + +## Data specification version number + +Question: Isn't this implicit in the test specification number, is there a reason +why we should have two versions one for the data format and one for the +test specification? Would changing the dataformat not imply changing the +test version number? + +## Semantics + +List the extra keys that will be part of the report that are not part of +the parent data format. Be sure not to have keys that clash with the +parent data format. + +## Possible conclusions + +Based on the ouput data what conclusions can you draw? + +## Example output sample + +## Expected Post-processing efforts + +Question: What exactly is meant by this? Is this meaning the possible +difficulties that a person doing post-processing may encouter? + +# Privacy considerations + +There are a few! + +# Packet capture considerations + +We capture all packets on the interface foo for bar units of time. + +# Other notes + +Bikesh{r}ed! From 7815821b47f13ff29f09bf6cc7e711ad85200741 Mon Sep 17 00:00:00 2001 From: jburg <38106050+JaxGames5225@users.noreply.github.com> Date: Tue, 3 Jan 2023 02:06:03 -0500 Subject: [PATCH 03/14] Completed Shadowsocks spec --- nettests/ts-039-shadowsocks.md | 91 +++++++++++++++++++++------------- 1 file changed, 57 insertions(+), 34 deletions(-) diff --git a/nettests/ts-039-shadowsocks.md b/nettests/ts-039-shadowsocks.md index 8c8fadae..f8c13b6e 100644 --- a/nettests/ts-039-shadowsocks.md +++ b/nettests/ts-039-shadowsocks.md @@ -30,48 +30,71 @@ The main goal of the test is to inform the user (and the community) whether or n # Expected output -## Parent data format - -This is the base data format(s) that this test will adhere to (it is -implicit that it will follow df-000-base). - ## Required output data -This is data that should be part of the base dataformat without which the -test cannot properly be interpreted. - -## Data specification version number - -Question: Isn't this implicit in the test specification number, is there a reason -why we should have two versions one for the data format and one for the -test specification? Would changing the dataformat not imply changing the -test version number? +* The result of the test, 'success' or failure type +* Whether or not the censorship was detected ## Semantics -List the extra keys that will be part of the report that are not part of -the parent data format. Be sure not to have keys that clash with the -parent data format. +* Success: True if all 20 connections and control test succeeded +* ConnectionCount: Number of successful connections +* FinalPopcount: The popcount of the triggering packet +* FirstSix: True if first six bytes of the final payload are printable +* TwentyContig: True if there exist twenty contiguous bytes of printable ASCII in the final payload +* HalfPrintable: True if at least half of the final payload is made up of printable ASCII +* PopcountRange: True if final popcount is less than 3.4 or greater than 4.6 +* MatchesHTTP: True if fingerprinted as HTTP +* MatchesTLS: True if fingerprinted as TLS +* Payload: Payload of final packet +* Censorship: True if all 20 connections succeeded +* Error: String of error ## Possible conclusions -Based on the ouput data what conclusions can you draw? +Ability to determine if the user is in a location where they are experiencing censorship on fully encrypted traffic and what packet triggered the censorship. ## Example output sample -## Expected Post-processing efforts - -Question: What exactly is meant by this? Is this meaning the possible -difficulties that a person doing post-processing may encouter? - -# Privacy considerations - -There are a few! - -# Packet capture considerations - -We capture all packets on the interface foo for bar units of time. - -# Other notes - -Bikesh{r}ed! +```JSON +{ + "annotations":{ + "architecture":"amd64", + "engine_name":"ooniprobe-engine", + "engine_version":"3.16.0-alpha", + "platform":"macos" + }, + "data_format_version":"0.2.0", + "input":null, + "measurement_start_time":"2023-01-03 06:53:40", + "probe_asn":"AS6128", + "probe_cc":"US", + "probe_ip":"127.0.0.1", + "probe_network_name":"Cablevision Systems Corp.", + "report_id":"", + "resolver_asn":"AS6128", + "resolver_ip":"167.206.251.142", + "resolver_network_name":"Cablevision Systems Corp.", + "software_name":"miniooni", + "software_version":"3.16.0-alpha", + "test_keys":{ + "success":true, + "connection_count":19, + "final_popcount":4.074525745257453, + "first_six":false, + "twenty_contig":false, + "half_printable":false, + "popcount_range":false, + "matches_http":false, + "matches_tls":false, + "payload":"KLpodhNrDfHPs6cEYBe096yVZdxqZ3udlhcs/ziiC11KHXcs2LUfa/CpiiLyo2NfguJ99k+k23XWE59+lw723HpsGJUKJnHop2BLXUCVUJDektT6Hm9rYTeBtAvqPZP+LVQ+WmqpoU7OFpeM3m7mVTut2AfSaH8TPhaDG377uYXz2tvZy+Oa7d/AsLzl4DKc707x+tITtFj4V/Gg2RfaHZe4C9tH9Wujw/62PiM6IgT3IK9fXT2QB0O9ZinY9+KxwVs7AYbXhoYdMoF9+s1wIL1f1NNx/Khgx6eYovROsj4768niLIPy6ketR0jZAA1CLidDAaWOvEDc/Tgv5vHcenUR0VawQFhGTfu+J6z4GEoQoi6e+N1HqvRoLXCd/OWdgybHVBlpPc8Wr7K8xrvdMwGIGKN+rpClGiFwxLJQkptr5kr9oZmM3T9cBy2ViZjdRM7HW3c8YmrGmw0jyVDszHcl4kBHeANgOEGtAudqvoxKPbLZYxvke64wu5RGr3CUEpwAnJW4GgPvl1KSWt9n5HSC0+Lhtbrcd7iUtlufoRjHrw3IGDt+n+S4F1tvV+4cslBRcv+wlJx4zFL+We+gJSg2CUFVLqOdRgpB73lBTe1Sb2vBB1RSZ3Cn0WTwhpbFVASpDS8nnJsD+CSVmXVpOy0PxvrYLA/UY6mE0kFBfqH9oVC8A+TN0IA3/vkzwZ/P9Xs8HRP5xm6shPvpy19MD9YWSK0Co3EXUpQrt4TW4kPeMbt/Dgpxe72zcuh6N9pjp3oR1fz1ioMOIp+1yalhB3XqgYAALUzpYI1Ya2A4if9qQq9nvVdLqDKFTehxKW1+mgJ+3/I7EG+6yprd7UGuQSpc49Yg/LhBchiXhIqTcgpNNNNClnjh31UTQwYT2NjYWuWK0ijGQfDjwP9bgYOPGaUOyzjkZTnWL1ejAaa5saA3q9TzKdZoY5Pw3BbO0WXP6SH2H1hhS/dB8XQPPLnq9jHj", + "censorship":false, + "error":null + }, + "test_name":"shadowsocks", + "test_runtime":6.178643611, + "test_start_time":"2023-01-03 06:53:34", + "test_version":"0.1.0" +} + +``` From c4097049523ad6bb15ad49e82f5b54a480a77c71 Mon Sep 17 00:00:00 2001 From: jburg <38106050+JaxGames5225@users.noreply.github.com> Date: Thu, 12 Jan 2023 18:10:17 -0500 Subject: [PATCH 04/14] Update and rename ts-039-shadowsocks.md to ts-039-randomtraffic.md --- ...-shadowsocks.md => ts-039-randomtraffic.md} | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) rename nettests/{ts-039-shadowsocks.md => ts-039-randomtraffic.md} (82%) diff --git a/nettests/ts-039-shadowsocks.md b/nettests/ts-039-randomtraffic.md similarity index 82% rename from nettests/ts-039-shadowsocks.md rename to nettests/ts-039-randomtraffic.md index f8c13b6e..30bd5d96 100644 --- a/nettests/ts-039-shadowsocks.md +++ b/nettests/ts-039-randomtraffic.md @@ -4,7 +4,7 @@ # Specification name -Shadowsocks +Random Traffic # Test preconditions @@ -12,7 +12,11 @@ An internet connection # Expected impact -Ability to detect the censorship of fully-encrypted protocols, specifically Shadowsocks +Ability to detect the censorship of fully-encrypted protocols which encrypt every byte of traffic in an attempt to appear completely random. + +``` +Note: This does not include TLS as TLS has a standard handshake to begin with. +``` # Expected inputs @@ -22,8 +26,8 @@ None The main goal of the test is to inform the user (and the community) whether or not they are experiencing censorship on connections that send fully encrypted packets that appear random, as well as to record information about censored packets in order to better understand the censorship algorithm. The test seeks to accomplish these goals by doing the following: -1. If no IP is given by the user, select an IP from the list of IP addressed in the affected range -2. Complete a TCP handshake with the IP address and send a stream of zero bytes as a control test. If this control test succeeds then proceed with the test, otherwise end the test and return the error +1. If no IP address is given by the user, select an IP address from the list of IP addresses in the affected range +2. Complete a TCP handshake with the IP address and send a stream of null bytes as a control test. If this control test succeeds then proceed with the experiment, otherwise attempt the control test with a new IP address two more times or until the control test is successful. If no control test succeeds end the test and return the error. 3. Complete a TCP handshake with the IP address and send a stream of random bytes. If this connection times out, we attempt to connect once more to check for residual censorship. If the residual censorship test results in a timeout, we end the test, record information about the blocked packet, and inform the user they are experiencing censorship. Otherwise we continue with the test 4. Step 3 is repeated 19 more times to account for the blocking rate 5. If no errors occurred and the test was completed, all connections are then closed and the test informs the user they are not experiencing censorship. @@ -37,7 +41,7 @@ The main goal of the test is to inform the user (and the community) whether or n ## Semantics -* Success: True if all 20 connections and control test succeeded +* Success: True if no errors occurred * ConnectionCount: Number of successful connections * FinalPopcount: The popcount of the triggering packet * FirstSix: True if first six bytes of the final payload are printable @@ -47,12 +51,12 @@ The main goal of the test is to inform the user (and the community) whether or n * MatchesHTTP: True if fingerprinted as HTTP * MatchesTLS: True if fingerprinted as TLS * Payload: Payload of final packet -* Censorship: True if all 20 connections succeeded +* Censorship: False if all 20 connections succeeded * Error: String of error ## Possible conclusions -Ability to determine if the user is in a location where they are experiencing censorship on fully encrypted traffic and what packet triggered the censorship. +Ability to determine if the user is experiencing censorship on fully-encrypted traffic and what packet triggered the censorship. ## Example output sample From 39201f225697b2c534ef4cdac97736be7e11709b Mon Sep 17 00:00:00 2001 From: jburg <38106050+JaxGames5225@users.noreply.github.com> Date: Fri, 13 Jan 2023 10:42:06 -0500 Subject: [PATCH 05/14] Update ts-039-randomtraffic.md --- nettests/ts-039-randomtraffic.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nettests/ts-039-randomtraffic.md b/nettests/ts-039-randomtraffic.md index 30bd5d96..71603c9c 100644 --- a/nettests/ts-039-randomtraffic.md +++ b/nettests/ts-039-randomtraffic.md @@ -24,7 +24,7 @@ None # Test description -The main goal of the test is to inform the user (and the community) whether or not they are experiencing censorship on connections that send fully encrypted packets that appear random, as well as to record information about censored packets in order to better understand the censorship algorithm. The test seeks to accomplish these goals by doing the following: +The main goal of the test is to inform the user whether or not they are experiencing censorship on connections that send fully encrypted packets that appear random, as well as to record information about censored packets in order to better understand the censorship algorithm. The test seeks to accomplish these goals by doing the following: 1. If no IP address is given by the user, select an IP address from the list of IP addresses in the affected range 2. Complete a TCP handshake with the IP address and send a stream of null bytes as a control test. If this control test succeeds then proceed with the experiment, otherwise attempt the control test with a new IP address two more times or until the control test is successful. If no control test succeeds end the test and return the error. From 7ebd29e709948db05de8ec4287e1b3282b944e49 Mon Sep 17 00:00:00 2001 From: jburg <38106050+JaxGames5225@users.noreply.github.com> Date: Fri, 13 Jan 2023 10:43:37 -0500 Subject: [PATCH 06/14] Update ts-039-randomtraffic.md --- nettests/ts-039-randomtraffic.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nettests/ts-039-randomtraffic.md b/nettests/ts-039-randomtraffic.md index 71603c9c..9101c0bc 100644 --- a/nettests/ts-039-randomtraffic.md +++ b/nettests/ts-039-randomtraffic.md @@ -1,6 +1,6 @@ # Specification version number -2022-12-23-000 +2023-01-13-000 # Specification name From 0ebaad23b6320007dee60a3ea6c66c33e83d0642 Mon Sep 17 00:00:00 2001 From: jburg <38106050+JaxGames5225@users.noreply.github.com> Date: Wed, 12 Apr 2023 16:28:47 -0400 Subject: [PATCH 07/14] Rename and wrap changes --- ...ndomtraffic.md => ts-040-randomtraffic.md} | 37 ++++++++++++++----- 1 file changed, 27 insertions(+), 10 deletions(-) rename nettests/{ts-039-randomtraffic.md => ts-040-randomtraffic.md} (71%) diff --git a/nettests/ts-039-randomtraffic.md b/nettests/ts-040-randomtraffic.md similarity index 71% rename from nettests/ts-039-randomtraffic.md rename to nettests/ts-040-randomtraffic.md index 9101c0bc..4b4cee96 100644 --- a/nettests/ts-039-randomtraffic.md +++ b/nettests/ts-040-randomtraffic.md @@ -12,7 +12,8 @@ An internet connection # Expected impact -Ability to detect the censorship of fully-encrypted protocols which encrypt every byte of traffic in an attempt to appear completely random. +Ability to detect the censorship of fully-encrypted protocols which encrypt every +byte of traffic in an attempt to appear completely random. ``` Note: This does not include TLS as TLS has a standard handshake to begin with. @@ -24,13 +25,27 @@ None # Test description -The main goal of the test is to inform the user whether or not they are experiencing censorship on connections that send fully encrypted packets that appear random, as well as to record information about censored packets in order to better understand the censorship algorithm. The test seeks to accomplish these goals by doing the following: - -1. If no IP address is given by the user, select an IP address from the list of IP addresses in the affected range -2. Complete a TCP handshake with the IP address and send a stream of null bytes as a control test. If this control test succeeds then proceed with the experiment, otherwise attempt the control test with a new IP address two more times or until the control test is successful. If no control test succeeds end the test and return the error. -3. Complete a TCP handshake with the IP address and send a stream of random bytes. If this connection times out, we attempt to connect once more to check for residual censorship. If the residual censorship test results in a timeout, we end the test, record information about the blocked packet, and inform the user they are experiencing censorship. Otherwise we continue with the test -4. Step 3 is repeated 19 more times to account for the blocking rate -5. If no errors occurred and the test was completed, all connections are then closed and the test informs the user they are not experiencing censorship. +The main goal of the test is to inform the user whether or not they are experiencing +censorship on connections that send fully encrypted packets that appear random, as +well as to record information about censored packets in order to better understand +the censorship algorithm. The test seeks to accomplish these goals by doing the +following: + +1. If no IP address is given by the user, select an IP address from the list of IP + addresses in the affected range +3. Complete a TCP handshake with the IP address and send a stream of null bytes as + a control test. If this control test succeeds then proceed with the experiment, + otherwise attempt the control test with a new IP address two more times or until + the control test is successful. If no control test succeeds end the test and + return the error. +5. Complete a TCP handshake with the IP address and send a stream of random bytes. + If this connection times out, we attempt to connect once more to check for residual + censorship. If the residual censorship test results in a timeout, we end the test, + record information about the blocked packet, and inform the user they are experiencing + censorship. Otherwise we continue with the test +7. Step 3 is repeated 19 more times to account for the blocking rate +8. If no errors occurred and the test was completed, all connections are then closed + and the test informs the user they are not experiencing censorship. # Expected output @@ -45,7 +60,8 @@ The main goal of the test is to inform the user whether or not they are experien * ConnectionCount: Number of successful connections * FinalPopcount: The popcount of the triggering packet * FirstSix: True if first six bytes of the final payload are printable -* TwentyContig: True if there exist twenty contiguous bytes of printable ASCII in the final payload +* TwentyContig: True if there exist twenty contiguous bytes of printable ASCII in + the final payload * HalfPrintable: True if at least half of the final payload is made up of printable ASCII * PopcountRange: True if final popcount is less than 3.4 or greater than 4.6 * MatchesHTTP: True if fingerprinted as HTTP @@ -56,7 +72,8 @@ The main goal of the test is to inform the user whether or not they are experien ## Possible conclusions -Ability to determine if the user is experiencing censorship on fully-encrypted traffic and what packet triggered the censorship. +Ability to determine if the user is experiencing censorship on fully-encrypted +traffic and what packet triggered the censorship. ## Example output sample From 1c6b91e5505833fdd3a77916a98f8f655885191e Mon Sep 17 00:00:00 2001 From: jburg <38106050+JaxGames5225@users.noreply.github.com> Date: Wed, 12 Apr 2023 16:33:07 -0400 Subject: [PATCH 08/14] Corrected semantics section --- nettests/ts-040-randomtraffic.md | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/nettests/ts-040-randomtraffic.md b/nettests/ts-040-randomtraffic.md index 4b4cee96..460af61e 100644 --- a/nettests/ts-040-randomtraffic.md +++ b/nettests/ts-040-randomtraffic.md @@ -56,19 +56,21 @@ following: ## Semantics -* Success: True if no errors occurred -* ConnectionCount: Number of successful connections -* FinalPopcount: The popcount of the triggering packet -* FirstSix: True if first six bytes of the final payload are printable -* TwentyContig: True if there exist twenty contiguous bytes of printable ASCII in +This experiment generates a "test keys" result object containing the following keys: + +* success: True if no errors occurred +* connection_count: Number of successful connections +* final_popcount: The popcount of the triggering packet +* first_six: True if first six bytes of the final payload are printable +* twenty_contig: True if there exist twenty contiguous bytes of printable ASCII in the final payload -* HalfPrintable: True if at least half of the final payload is made up of printable ASCII -* PopcountRange: True if final popcount is less than 3.4 or greater than 4.6 -* MatchesHTTP: True if fingerprinted as HTTP -* MatchesTLS: True if fingerprinted as TLS -* Payload: Payload of final packet -* Censorship: False if all 20 connections succeeded -* Error: String of error +* half_printable: True if at least half of the final payload is made up of printable ASCII +* popcount_range: True if final popcount is less than 3.4 or greater than 4.6 +* matches_http: True if fingerprinted as HTTP +* matches_tls: True if fingerprinted as TLS +* payload: Payload of final packet +* censorship: False if all 20 connections succeeded +* error: String of error ## Possible conclusions From 625f5a79cb862e4e5ff6c216e1d8759f12b696f3 Mon Sep 17 00:00:00 2001 From: jburg <38106050+JaxGames5225@users.noreply.github.com> Date: Wed, 12 Apr 2023 17:42:47 -0400 Subject: [PATCH 09/14] Updated error explanation and cleaned up test description --- nettests/ts-040-randomtraffic.md | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/nettests/ts-040-randomtraffic.md b/nettests/ts-040-randomtraffic.md index 460af61e..e36c7def 100644 --- a/nettests/ts-040-randomtraffic.md +++ b/nettests/ts-040-randomtraffic.md @@ -32,19 +32,21 @@ the censorship algorithm. The test seeks to accomplish these goals by doing the following: 1. If no IP address is given by the user, select an IP address from the list of IP - addresses in the affected range -3. Complete a TCP handshake with the IP address and send a stream of null bytes as + addresses in the affected range. +2. Complete a TCP handshake with the IP address and send a stream of null bytes as a control test. If this control test succeeds then proceed with the experiment, otherwise attempt the control test with a new IP address two more times or until the control test is successful. If no control test succeeds end the test and return the error. -5. Complete a TCP handshake with the IP address and send a stream of random bytes. - If this connection times out, we attempt to connect once more to check for residual - censorship. If the residual censorship test results in a timeout, we end the test, - record information about the blocked packet, and inform the user they are experiencing - censorship. Otherwise we continue with the test -7. Step 3 is repeated 19 more times to account for the blocking rate -8. If no errors occurred and the test was completed, all connections are then closed +3. Repeat 20 times + 1. Complete a TCP handshake with the IP address and send a stream of random bytes. + If this connection times out, we attempt to connect once more to check for residual + censorship. + 2. If the residual censorship test results in a timeout, we end the test, + record information about the blocked packet, and inform the user they are experiencing + censorship. Otherwise we continue with the test. If any error other than a timeout + occor the test terminates. +5. If no errors occurred and the test was completed, all connections are then closed and the test informs the user they are not experiencing censorship. # Expected output From d6a810e3e5d35da6448bafa26955c9864f9d0e8e Mon Sep 17 00:00:00 2001 From: jburg <38106050+JaxGames5225@users.noreply.github.com> Date: Thu, 13 Apr 2023 23:24:34 -0400 Subject: [PATCH 10/14] Expanded payload definition --- nettests/ts-040-randomtraffic.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/nettests/ts-040-randomtraffic.md b/nettests/ts-040-randomtraffic.md index e36c7def..f8aeb5e6 100644 --- a/nettests/ts-040-randomtraffic.md +++ b/nettests/ts-040-randomtraffic.md @@ -65,12 +65,13 @@ This experiment generates a "test keys" result object containing the following k * final_popcount: The popcount of the triggering packet * first_six: True if first six bytes of the final payload are printable * twenty_contig: True if there exist twenty contiguous bytes of printable ASCII in - the final payload + the final payload * half_printable: True if at least half of the final payload is made up of printable ASCII * popcount_range: True if final popcount is less than 3.4 or greater than 4.6 * matches_http: True if fingerprinted as HTTP * matches_tls: True if fingerprinted as TLS -* payload: Payload of final packet +* payload: Payload of the triggering packet or final successful packet if the censorship + was not triggered * censorship: False if all 20 connections succeeded * error: String of error From 52648aa25eb5687c0eacbba82ab4e15a05ecab32 Mon Sep 17 00:00:00 2001 From: jburg <38106050+JaxGames5225@users.noreply.github.com> Date: Thu, 20 Apr 2023 15:34:23 -0400 Subject: [PATCH 11/14] Updated expected impact section --- nettests/ts-040-randomtraffic.md | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/nettests/ts-040-randomtraffic.md b/nettests/ts-040-randomtraffic.md index f8aeb5e6..6047e571 100644 --- a/nettests/ts-040-randomtraffic.md +++ b/nettests/ts-040-randomtraffic.md @@ -13,11 +13,23 @@ An internet connection # Expected impact Ability to detect the censorship of fully-encrypted protocols which encrypt every -byte of traffic in an attempt to appear completely random. +byte of traffic in an attempt to appear completely random. These protocols include +but are not limited to Shadowsocks, VMess, and Obfs4. This does not include TLS as +TLS has a standard handshake to begin with. + +It is important to note that this experiment is based off of the paper "How the +Great Firewall of China Detects and Blocks Fully Encrypted Traffic" written by +Mingshi Wu. The paper investigated and characterized the rules used by the GFW +to passively block fully encrypted traffic. The nettest produces traffic which +will be blocked by the GFW. Blocking in this context means that, once the offending +payload has been observed, the GFW installs rules that null-route traffic for +the server endpoint for a given amount of time. Blocking is also nondeterministic +meaning it can take multiple connections to the same destination endpoint with +offending payload to trigger this form of blocking. The nettest then records the +characteristics of the generated traffic along with whether it was blocked and +what are the characteristics of the payload that eventually triggered blocking. + -``` -Note: This does not include TLS as TLS has a standard handshake to begin with. -``` # Expected inputs From d1ef89f92e6bf02f41666384c7ab3ed04eec4ffc Mon Sep 17 00:00:00 2001 From: jburg <38106050+JaxGames5225@users.noreply.github.com> Date: Thu, 20 Apr 2023 15:50:24 -0400 Subject: [PATCH 12/14] Updated test description --- nettests/ts-040-randomtraffic.md | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/nettests/ts-040-randomtraffic.md b/nettests/ts-040-randomtraffic.md index 6047e571..4fd6897c 100644 --- a/nettests/ts-040-randomtraffic.md +++ b/nettests/ts-040-randomtraffic.md @@ -29,8 +29,6 @@ offending payload to trigger this form of blocking. The nettest then records the characteristics of the generated traffic along with whether it was blocked and what are the characteristics of the payload that eventually triggered blocking. - - # Expected inputs None @@ -40,18 +38,18 @@ None The main goal of the test is to inform the user whether or not they are experiencing censorship on connections that send fully encrypted packets that appear random, as well as to record information about censored packets in order to better understand -the censorship algorithm. The test seeks to accomplish these goals by doing the -following: +the censorship algorithm. It should be noted that this experiment contains a set of +TCP endpoints known to possibly host circumvention servers. The test seeks to accomplish +these goals by doing the following: -1. If no IP address is given by the user, select an IP address from the list of IP - addresses in the affected range. -2. Complete a TCP handshake with the IP address and send a stream of null bytes as +1. Select a TCP endpoint from the list of TCP endpoints in the affected range at random. +2. Complete a TCP handshake with the TCP endpoint and send a stream of null bytes as a control test. If this control test succeeds then proceed with the experiment, - otherwise attempt the control test with a new IP address two more times or until + otherwise attempt the control test with a new TCP endpoint two more times or until the control test is successful. If no control test succeeds end the test and return the error. 3. Repeat 20 times - 1. Complete a TCP handshake with the IP address and send a stream of random bytes. + 1. Complete a TCP handshake with the TCP endpoint and send a stream of random bytes. If this connection times out, we attempt to connect once more to check for residual censorship. 2. If the residual censorship test results in a timeout, we end the test, From 2ba4b25f42345863c852e8da1377485f466fafff Mon Sep 17 00:00:00 2001 From: jburg <38106050+JaxGames5225@users.noreply.github.com> Date: Thu, 20 Apr 2023 16:04:08 -0400 Subject: [PATCH 13/14] Updated test description --- nettests/ts-040-randomtraffic.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/nettests/ts-040-randomtraffic.md b/nettests/ts-040-randomtraffic.md index 4fd6897c..3f4f0450 100644 --- a/nettests/ts-040-randomtraffic.md +++ b/nettests/ts-040-randomtraffic.md @@ -43,11 +43,10 @@ TCP endpoints known to possibly host circumvention servers. The test seeks to ac these goals by doing the following: 1. Select a TCP endpoint from the list of TCP endpoints in the affected range at random. -2. Complete a TCP handshake with the TCP endpoint and send a stream of null bytes as - a control test. If this control test succeeds then proceed with the experiment, - otherwise attempt the control test with a new TCP endpoint two more times or until - the control test is successful. If no control test succeeds end the test and - return the error. +2. Complete a TCP handshake with the TCP endpoint as a control test. If this control test + succeeds then proceed with the experiment, otherwise attempt the control test two more + times with a new TCP endpoint or until the control test is successful. If none of the + three control tests succeed end the test and return the error. 3. Repeat 20 times 1. Complete a TCP handshake with the TCP endpoint and send a stream of random bytes. If this connection times out, we attempt to connect once more to check for residual From 1d275ccf149e321d32865d180e67f0fdcdc8aae0 Mon Sep 17 00:00:00 2001 From: jburg <38106050+JaxGames5225@users.noreply.github.com> Date: Thu, 20 Apr 2023 16:19:15 -0400 Subject: [PATCH 14/14] Changed key name from error to failure --- nettests/ts-040-randomtraffic.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/nettests/ts-040-randomtraffic.md b/nettests/ts-040-randomtraffic.md index 3f4f0450..f9f0c130 100644 --- a/nettests/ts-040-randomtraffic.md +++ b/nettests/ts-040-randomtraffic.md @@ -82,7 +82,7 @@ This experiment generates a "test keys" result object containing the following k * payload: Payload of the triggering packet or final successful packet if the censorship was not triggered * censorship: False if all 20 connections succeeded -* error: String of error +* failure: String of error ## Possible conclusions @@ -124,7 +124,7 @@ traffic and what packet triggered the censorship. "matches_tls":false, "payload":"KLpodhNrDfHPs6cEYBe096yVZdxqZ3udlhcs/ziiC11KHXcs2LUfa/CpiiLyo2NfguJ99k+k23XWE59+lw723HpsGJUKJnHop2BLXUCVUJDektT6Hm9rYTeBtAvqPZP+LVQ+WmqpoU7OFpeM3m7mVTut2AfSaH8TPhaDG377uYXz2tvZy+Oa7d/AsLzl4DKc707x+tITtFj4V/Gg2RfaHZe4C9tH9Wujw/62PiM6IgT3IK9fXT2QB0O9ZinY9+KxwVs7AYbXhoYdMoF9+s1wIL1f1NNx/Khgx6eYovROsj4768niLIPy6ketR0jZAA1CLidDAaWOvEDc/Tgv5vHcenUR0VawQFhGTfu+J6z4GEoQoi6e+N1HqvRoLXCd/OWdgybHVBlpPc8Wr7K8xrvdMwGIGKN+rpClGiFwxLJQkptr5kr9oZmM3T9cBy2ViZjdRM7HW3c8YmrGmw0jyVDszHcl4kBHeANgOEGtAudqvoxKPbLZYxvke64wu5RGr3CUEpwAnJW4GgPvl1KSWt9n5HSC0+Lhtbrcd7iUtlufoRjHrw3IGDt+n+S4F1tvV+4cslBRcv+wlJx4zFL+We+gJSg2CUFVLqOdRgpB73lBTe1Sb2vBB1RSZ3Cn0WTwhpbFVASpDS8nnJsD+CSVmXVpOy0PxvrYLA/UY6mE0kFBfqH9oVC8A+TN0IA3/vkzwZ/P9Xs8HRP5xm6shPvpy19MD9YWSK0Co3EXUpQrt4TW4kPeMbt/Dgpxe72zcuh6N9pjp3oR1fz1ioMOIp+1yalhB3XqgYAALUzpYI1Ya2A4if9qQq9nvVdLqDKFTehxKW1+mgJ+3/I7EG+6yprd7UGuQSpc49Yg/LhBchiXhIqTcgpNNNNClnjh31UTQwYT2NjYWuWK0ijGQfDjwP9bgYOPGaUOyzjkZTnWL1ejAaa5saA3q9TzKdZoY5Pw3BbO0WXP6SH2H1hhS/dB8XQPPLnq9jHj", "censorship":false, - "error":null + "failure":null }, "test_name":"shadowsocks", "test_runtime":6.178643611,