Consider an upper bound on trusting entity statuses when they go missing

[CDR-API-Stream]

Description

Issue #433 set expectations on how a data holder handles entities going missing from Register APIs. As part of feedback, a request was made to set an upper bound on how long trust is maintained.
#433 (comment)

This issue has been raised to continue this conversation.

Area Affected

Participant Statuses
The statement to consider updating:

If Data Holders do not receive a status for a Data Recipient or Data Recipient Software Product, or receives a status that is not recognised, Data Holders SHOULD ignore the value and use the previous status value retrieved from the CDR Register

Change Proposed

Consider an upper bound for this period of trust. 24 hours is currently being proposed.

[CDR-API-Stream]

This issue is being discussed in Maintenance Iteration #10.
 
There is a line to be drawn as to where the technical standards should specify participant behaviour in an edge case and where operational mechanisms apply instead.
 
As there are multiple considerations to be made when deciding on an upper bound, we have described some options to help drive the discussion:
 
Option 1: Provide a 24 hour upper bound.
Specifying an upper bound adds an additional state to the model, where data holder behaviour needs to be defined.
In this scenario, we would need to differentiate the behaviour when the register APIs are unavailable  <= 24 hours and > 24 hours.
Operational/security considerations would also need to be made on whether 24 hours is sufficient a period of time prior to the ecosystem becoming "untrusted"
 
Option 2: Provide a larger upper bound of 7 days
Specifying an upper bound with a longer timeframe gives greater flexibility to the operation of the CDR before the ecosystem becomes "untrusted". However, the larger the upper bound becomes, the less meaningful it is to have it.
 
Option 3: Specify no upper-bound
Specifying no upper-bound means that no additional state needs to be defined in the standards however, operational requirements would need to be considered for the scenario. When the CDR Register APIs are unable to convey the trust in the ecosystem,  alternative mechanisms would need to be available to inform data holders.

[CDR-API-Stream]

This piece of work is related to data holder expectations when there is a register outage.
Register repo issue 99: ConsumerDataStandardsAustralia/register#99

Data Holders (DH) are to trust the Local Cache in the event of an outage to the Register indefinitely until advised by the ACCC

The ACCC will be relying on out-of-band communications to advise participants on the status of the platform and therefore what data sharing and consent obligations they have.

This position could be treated as precedence and aligning the behaviour of data holders for "trusting entity statuses when they go missing" should be considered

[CDR-API-Stream]

Proposal:

Register repo issue 99 resulted in no upper-bound on data holder caching during a Register outage.

This pattern can be adopted for the scenario when trusted entity statuses go missing.

Therefore, the proposal is to adopt Option 3: Specify no upper-bound

Out-of-band communications will be relied upon if the CDR Register publishes missing statuses.

[perlboy]

Issue 99 covers a complete outage of a trusted source (ie. Register) not a specification violation of the trusted source and is therefore, while a related example, not a like for like in implementation impact.

As per feedback on #433 this behaviour is "standard" behaviour in CTS between tests and as such implementations have been built around this context, notably missing participants are being silently deleted (ie. without revocations), placed in frozen state (human intervention) or causing no updates at all (ie. the entire response is considered untrusted and all updates are ignored).

Consequently, please treat this as a Future Dated Obligation so that alignment can occur within release schedules.

[CDR-API-Stream]

Discussion through maintenance iteration 10 has highlighted the Future Dated Obligation (FDO) mechanism is used to define logical dates functionality takes effect. It is not used to project manage CTS or CDR Register delivery
 
The testing and operational aspects of the CDR can define their own release schedules without relying on an FDO to drive this. Therefore, an FDO does not need to be defined for the ACCC to work with industry to communicate impacts to CTS.
 
The DSB proposes that the standards do not specify an upper limit (Option 3: Specify no upper-bound) and do not set an FDO.

[perlboy]

CTS is an operational requirement to be approved by the Registrar (despite what individuals may say if a complete fail of the CTS was submitted the Holder would be rejected). Additionally the CTS Register API would be unable to convey trust in its current operation because it would have missing entries without sufficient state change (ie. the entries are corrupt).

ACCC requires that CTS is tested in a "production like" environment and requests a coupling of this to an attestation from the Holder. On this basis it would appear that consideration to achieve the operational requirement would trigger the need for the CTS to establish an alternate mechanism outlined or should Holders ignore the Standards on the basis the Regulator has given apparently conflicting direction?

[CDR-API-Stream]

This recommendation Option 3: Specify no upper-bound without specifying an FDO will be recommended to the chair.

@perlboy Impacts to CTS and operational processes are driven by the ACCC. This issue will be left open to give the ACCC an opportunity to respond. This will maintain traceability on the operational impacts you have raised

[ACCC-CDR]

@perlboy - thanks for your feedback. It should be noted that CTS’ behaviour with respect to this issue and caching is different to that of RAAP and the Register.

CTS has been designed to be “stateless” so that participants can run a numbered set of tests in any order and run each test scenario as many times as required to complete the certification process. A consequence of this stateless behaviour is that participants who are engaged in the CTS testing process cannot reliably assume that the results of one test will be carried forward to a subsequent test. For example, a test designed for a Data Holder (DH) participant may require that DH to interact with a simulated Accredited Data Recipient (ADR), including steps to determine the status of the ADR or the ADR’s Software Product. However, the DH shouldn’t assume that the simulated ADR will still exist when a subsequent test is run and nor should the DH rely on the cached value/s for that ADR.

Due to the stateless nature of CTS, participants will encounter an influx of participant metadata records when performing the necessary polling of the CTS Register during testing.  Therefore, the participant may choose to implement a process to regularly purge test data (e.g. after each test / at the end of each testing day / at the completion of CTS testing) to return their solution to a known, baseline state.  Participant metadata should not be removed whilst executing a CTS test (i.e. between test steps) as it may result in errors.

If you are experiencing issues related to the “stateless” manner in which CTS operates, please contact the ACCC via the CDR Service Management Portal or via the CDR Technical Operations Mailbox for further advice.

[perlboy]

@perlboy - thanks for your feedback. It should be noted that CTS’ behaviour with respect to this issue and caching is different to that of RAAP and the Register.

Then the CTS has failed to meet one of its core objectives specified in the Guidance Material:

The CTS tests if participants can conform with the Consumer Data Standards (CDS) and CDR Register (the Register) design.

The fact they are different is meaningful especially in state transitions.

CTS has been designed to be “stateless” so that participants can run a numbered set of tests in any order and run each test scenario as many times as required to complete the certification process. A consequence of this stateless behaviour is that participants who are engaged in the CTS testing process cannot reliably assume that the results of one test will be carried forward to a subsequent test. For example, a test designed for a Data Holder (DH) participant may require that DH to interact with a simulated Accredited Data Recipient (ADR), including steps to determine the status of the ADR or the ADR’s Software Product.

It's unclear what this is meant to be clarifying other than specifically stating that the CTS is designed to be the opposite of reality - that is, that the Register is entirely dedicated to managing ecosystem state and the CTS behaviour is to rewrite history every test.

However, the DH shouldn’t assume that the simulated ADR will still exist when a subsequent test is run and nor should the DH rely on the cached value/s for that ADR.

And yet this Decision Proposal specifically seeks to explicitly clarify that if an ADR disappears the previous state is retained for an infinite period (Option 3). It is worrying that the ACCC still doesn't seem to understand the problem of what the response is. "Disappearing" Software Products is not conformant with the CDR Register as per the Standards:

Dormant entries are defined as those records that have reached and maintained an end state in the CDR Register for the past 2 years

Additionally the Standards specify the end state table which the CTS does not appropriately follow:

The CTS doesn't transition Software Products through states but instead they simply go from Active to non-existent in Software Product Status. This is a direct violation of the clearly documented Standards.

Due to the stateless nature of CTS, participants will encounter an influx of participant metadata records when performing the necessary polling of the CTS Register during testing.  Therefore, the participant may choose to implement a process to regularly purge test data (e.g. after each test / at the end of each testing day / at the completion of CTS testing) to return their solution to a known, baseline state. 

This is confusing the issue. New software products, appropriately advertised in the CTS are fine.

Participant metadata should not be removed whilst executing a CTS test (i.e. between test steps) as it may result in errors.

The behaviour of what happens when metadata goes missing is, at this point in time, undefined within the Standards and the verification of Data Recipient statuses is usually an atomic action for legal reasons.

If you are experiencing issues related to the “stateless” manner in which CTS operates, please contact the ACCC via the CDR Service Management Portal or via the CDR Technical Operations Mailbox for further advice.

I don't need advice, I need the ACCC CTS to behave the way the ACCC is expecting participants to behave in the real ecosystem. All the explanation above seems to do is double-talk it's way to a justification of why it's ok that the Conformance Testing Suite doesn't behave like reality. The explanation provided seems more like a bureaucratic excuse making response rather than a well considered engineering one.

The action is simple, the CTS should behave like the Register.

[CDR-API-Stream]

This recommendation Option 3: Specify no upper-bound without specifying an FDO was accepted by the chair documented in DP 237 and the requirement published in V1.17.0 of the Consumer Data Standards.

Operational considerations on how participants are informed of missing or invalid statuses are left for the ACCC to drive and are not part of the standards maintenance process.

For feedback or issues relating to the operation of the Consumer Data Right or conformance testing please contact the ACCC via the CDR Service Management Portal or via the CDR Technical Operations Mailbox.