Decision Proposal 145 - Strategy for Reporting & Metrics

[CDR-API-Stream]

This decision proposal outlines a roadmap and strategy for consulting on the reporting of metrics in the CDR ecosystem.

The proposal for consultation is attached below:
Decision Proposal 145 - Strategy for Reporting & Metrics.pdf

Feedback is now open for this proposal for an extended period to account for the end of year break. Feedback is planned to be closed on Monday 25th January 2021, now extended to Friday 29th of January 2021.

[TT-Frollo]

Bringing forward the consultation on metrics can provide greater insight into the performance of the ecosystem.
The sequence of the consultation into 2 quarters is beneficial so that the first 3 can achieve an earlier implementation.

Frollo suggests that the breaking down of the existing aggregated endpoint metrics by endpoint should include the acknowledgement that data holders can use a different provider for Product Reference Data and therefore a different endpoint would be required as a priority change to the standards. Frollo already provides a PRD solution to several data holders that is separate to their authenticated banking API’s. This current single aggregated API can limit competition in PRD provision as more work to integrate metrics to a single end point is required.

Frollo also suggests that the consent metrics are included in this first quarter as it can be associated to the CX experience work related to pending metrics and how many consents are withdrawn due to issues when providing consent.

We do not see the current manual regulatory reporting as being onerous and therefore not a priority for change.

[anzbankau]

Thanks for the opportunity to comment on the upcoming proposals.

We are broadly supportive of the proposal to extend the scope of metric reporting. Owing to the substantial volume of change already tabled for 2021, we suggest prioritizing these decision proposals. We believe that Item 6 “Refusals and Error Handling” is worthy of earlier consideration as it is the most likely to have an impact on current implementations.

We also suggest CX metrics be deferred and considered at the same time as consent metrics. Whilst we haven’t had the opportunity to review the detail of the proposal, we would like to raise for consideration the privacy aspect to gathering statistics on consumer behaviour. We suggest if there is a need to gain insights into consumer behaviour this may be better served through qualitative research with permission of consumers. We would also like to question whether test outcomes need an alternative channel to what is already established, as this would be extra complexity added to the solution.

[NationalAustraliaBank]

NAB welcomes the opportunity to provide feedback into the metrics and reporting items as per the consultation plan outlined in the DP.

[WestpacOpenBanking]

Westpac welcomes the opportunity to provide input into the strategy for reporting metrics. The implementation burden placed on data recipients and data holders to provide statistics across the listed topics may be considerable. We recommend that the DSB take a collaborative value prioritisation based approach to the expansion of reporting metrics across identified and not-yet identified metric categories. Value should be considered in terms of use cases, access patterns and the implementation costs placed on data recipients and data holders. A set of workshops may be a productive way to consider priorities and use cases holistically. Near real time data, very high accuracy (which can also negatively affect API performance if audit logging calls are required to be synchronous as is currently the case) and statistics that require new audit functionality are all likely to be more costly to implement. As a specific example, the computation of customerCount or recipientCount in near real time rather than over longer timescales is of limited value when the data is only consumed daily. We recommend that the strategy should allow for a range of data frequencies (such as weekly, monthly, 6-monthly, etc) based on factors including the benefit and value and required validation activities that may be expected to data to ensure accuracy.

In relation to the identified topics we have the following specific feedback.

1. Endpoint Metrics

• The proposed changes to de-aggregate summary statistics so that they are computed by endpoint may make it difficult for the ACCC to achieve an accurate view as to if a participant is meeting the non-functional requirements. As an example, the peak TPS values for each endpoint may occur at different times of day and hence could not be used to compute the peak TPS. It may be desirable to retain aggregated statistics to avoid this issue.

2. Brand Aware Metrics

• A data holder may have a very large number of brands, especially if they are engaged in white-labelling.
• As per our comment on topic 1, de-aggregation of summary statistics by brand may make it difficult to understand if data holders are meeting non-functional requirement expectations.
• Common base URIs and services are currently permitted by the registry in some cases. In some instances it will not be possible to split statistics by brand. For example, a call to Get Products may have a response containing many brands.

3. CX Metrics for Data Holders

• Many of the proposed statistics would require the implementation very fine-grained audit logging which would require considerable effort from participants.
• Inclusion of usage metrics for specific functionalities (e.g. joint account management) may help to understand adoption and preferences.

4. Public Reporting and Dashboards

• If Get Metrics is change to be a public API rather than an authenticated API then ‘at the time of the call’ statistics should be excluded. This is because Public API architectures are best suited to relatively static data in order to allow the use of CDNs to protect against DDOS attacks.
• Some types of information, for example conformance monitoring in relation to security standards may be best kept authenticated for security reasons.
• It would be important to consider any privacy implications as a result of public audit data and review this if additional new data is added over time – can the data reveal information about individuals in e.g. the case where only a single customer has used a service?

5. Conformance Monitoring (ACCC / CDR Register)

• Conformance monitoring statistics have dependencies on conformance monitoring tools.
• Current tools have limited test coverage and are not amenable to automated reporting.

6. Refusals and Error Handling

• There are likely to be a very large number of error scenarios and refusal scenarios across data holders. Coming up with an exhaustive list of scenarios may require considerable effort and it would be undesirable to limit scenarios beyond the reasons allowed in the rules.
• There is a dependency on current error handling consultations.

7. Consent Metrics

• See our comments on item 3.

8. Testing and Assurance

• See our comments on item 5.

9. Data Recipient Metrics

• See all comments above.

[commbankoss]

The proposed consultations are important for the development of the CDR regime. Commonwealth Bank appreciates the DSB’s consideration of the following recommendations:

1. The consultation period to address these topics is extended. The DSB is aiming to address these nine consultations by the end of Q2, which would allow only about 2 weeks per consultation on average. This is a heavily compressed timeframe in light of the breadth of scope covered, especially alongside existing and future consultations outside of reporting and metrics. This is particularly a concern given the recent Rules 2.0 amendment which will likely result in urgent technical consultations of significant scope.

2. The consultations are sequenced to align to the urgency and impact each will have on the ecosystem. Several of the proposed consultations will have a greater impact and are potentially more ’urgent’ than others. For example, ‘Refusals and Error Handling’ is valuable, however there are arguably more urgent issues pending, and given the current rate of consumer uptake this consultation should be postponed until we are sufficiently informed of the problems to be solved. Commonwealth Bank’s preferred phased consultation order (in order of priority) follows:

Phase 1

• Brand Aware Metrics
• Conformance Monitoring (ACCC/CDR Register)
• Testing and Assurance

Phase 2

• Consent Metrics
• Endpoint Metrics

Phase 3

• CX Metrics for Data Holders
• Public Reporting and Dashboards (ACCC/CDR Register)
• Data Recipient Metrics
• Refusals and Error Handling

[CDR-API-Stream]

As per community request the deadline for feedback has been extended to Friday 29th of January 2021.

[PratibhaOrigin]

Feedback from Origin –

This is a high level strategy paper – it is still unclear from a policy perspective whether these are the appropriate metrics. We cannot provide detailed comments at this time – we will have more comments once the detailed metrics are developed and consulted on.

Few queries from the high-level look –
• Will it be mandatory to implement such monitoring and analytics endpoints? Or data holder can opt out?
• If all reported data is travelling via gateway , can a gateway be responsible for collecting and providing metrics data?
• PI protection must be heavily considered.
• Will it be required to build certain set of APIs/endpoints for Conformance Monitoring?
• Testing and Assurance will affect our delivery pipeline so it must be discussed prior implementation phase.

[CDR-API-Stream]

This consultation is being closed and we will publish an approach based on the feedback shortly.

[CDR-API-Stream]

Thank you all for the feedback provided. This will be fed into the consultations for individual topics, some of which will be released in the next few weeks.

The schedule decided upon, after incorporating feedback and balancing this against the needs of the regime as a whole is as follows:

Decision Proposal Topic	Consultation Period
Endpoint Metrics	Q1 2021
Brand Aware Metrics	Q1 2021
CX Metrics for Data Holders	Q1 2021
Refusals and Error Handling	Q2 2021
Public Reporting and Dashboards	Q2 2021
Data Recipient Metrics	Q3 2021
Conformance monitoring	Q3 2021
Testing and Assurance	Q3 2021

Note that this is the consultation schedule and should not be confused with the implementation schedule. Implementation timeframes will be part of the content for these consultations.