RiseupVPN measurements frequently say "likely blocked" when server is down

[agrabeli]

During the OONI Partner Training 2021 (https://ooni.org/post/ooni-partner-training-2021/), participants reported that the RiseupVPN test frequently reports false positives, saying that RiseupVPN is "likely blocked" when it's not (instead, their servers are down).

Perhaps we can add control measurements to the RiseupVPN test? Or perhaps we can somehow detect and annotate when the test fails due to server issues?

[kalikaneko]

Status of riseup infra can be checked at https://riseupstatus.net/

There's a json of the current status https://riseupstatus.net/index.json

I don't seem to find a machine-parseable version of the history, but that's perhaps a feature request to https://github.com/cstate/cstate

[bassosimone]

I have started adapting riseupvpn to make it more robust. Since ooni/probe-cli#1363 and ooni/probe-cli#1360, the probe does not ever flag riseupvpn as failed, rather we only consider it an experimental test for collecting data useful to riseup (as discussed in ooni/probe-cli#1125 (review)).

Additionally, since ooni/probe-cli#1355, riseupvpn can be conditionally enabled and disabled using the check-in API and the default status is disabled. Which means that we have a mechanism in the probe to make sure we're able to prevent this test from running in case of data quality issues.

The final step to transform riseupvpn into a data-collection only nettest is to modify the scoring logic of the pipeline to be more lax, as I documented in ooni/backend#745.

So, as far as the probe is concerned, this issue is solved. Further improvements will include using richer input to serve input to riseupvpn, which will make it more robust and effective (see #2559).

As such, I feel safe with closing this issue because I did enough work in the probe to mitigate and the rest of the work that is needed is either documented follow up work or backend work.