Decision Proposal 192 - AEMO Exposed End Points

[CDR-API-Stream]

The attached decision proposal outlines a recommendation for the API contracts that will be exposed by AEMO to Retailers.

The decision proposal is attached below:
Decision Proposal 192 - AEMO Exposed End Points.pdf

Feedback is welcome on this proposal. Consultation on this proposal will close on 9th August 2021

[SelenaLiuEA]

Hi All,

This Decision Proposal mentions "AEMO is unable to determine how long the current customer has had control of any specific NMI but they are able to determine how long the retailer (or more specifically, participant ID) has had control of each NMI".

EnergyAustralia has explored this issue further in relation to the provision of Historical Metering Data under Issue #195 and welcomes any feedback against that issue stream.

Decision Proposal 195 - Candidate Usage End Points · Issue #195 · ConsumerDataStandardsAustralia/standards (github.com)

Thank you

[PratibhaOrigin]

Thanks for the opportunity to provide feedback on this topic.

Based on the points highlighted in this paper, we wanted to raise few concerns –

1. Paper states –

“The retailers will, in effect, be able to operate as a proxy of the data provided by AEMO.”

• Based on this, assuming the retailer (data holder) is not supposed to store the data obtained from AEMO in any form and just pass it to the ADR. Please confirm.
• However, the NMI to Service point IT transformation is required before passing the data.
• If the data holder is not supposed to store the data, in case of reporting/breach, can the rules team make this clear where the responsibility sits with - AEMO OR retailer? As data holders won’t have any data stored, they will not be able to reproduce it in case of any complaints related to data shared for AEMO data sets.

2. Paper states –

“All occurrences of the servicePointId field, whether in a request, a response, or as a query parameter should be populated with the NMI instead of a service point ID using the ID permanence rules. AEMO will not be able to translate between a NMI and servicePointId as they will not be aware of the recipient or the subject associated with the consent. As a result, the retailer will be expected to translate the NMI provided in the servicePointId field into an ID conformant with the ID permanence rules.”

• This transformation from servicePointID to NMI or NMI to servicepointID will have added complexity and time lag while fulfilling a data sharing request.
• We request the DSB to take this into account while working on the non-functional requirements for energy – the timing requirements for retailer (data holder) should not consider the delay or time take by AEMO to fulfill the data request.
• However, it should take into consideration the added time for transformation on NMI to servicepointID and vice-versa before passing on the data.

3. Paper states –

“ AEMO is unable to determine how long the current customer has had control of any specific NMI but they are able to determine how long the retailer (or more specifically, participant ID) has had control of each NMI. If a request for usage data spans a time period when the retailer was not in control of the NMI then AEMO will not respond with an error but will not share data outside the period of control of the retailer. It remains an obligation of the retailer to ensure that the data requested and then shared with the ADR is not outside the bounds of control of the specific customer.1”

• Based on the above comment , is it correct to say that for a specific usage data request , historic meter/usage data wont be shared. The usage data for the period customer is with current retailer (data-holder) will only be in scope.

4. This multiple hand-off points and added logic to transform critical data before requesting and sharing data without actually storing the data has an added risk of errors and delays. As suggested previously, in light of comments raised above, we wanted to raise the query to rules team again – can a retailer(data holder) share the meter data from its own record instead of requesting its from AEMO and adding complexities of multiple handoffs , delays etc.

[EnergyAustraliaBA]

Thank you for the opportunity to provide feedback.

After reviewing the Decision proposal documentation, we have few questions to clarify namely,

• Is the intent that the AEMO endpoints are defined separately on GitHub, next to the equivalent endpoints exposed by retailers? Potentially renaming the EndPoints to stop confusion with the endpoints to be exposed by retailers?
• Can DSB define correctly the AEMO endpoints in the Draft Energy rules instead of describing the differences?
• As stated on page 3, Additional Notes (second bullet point) “End points that require knowledge of the NMIs that belong to the consumer have been excluded from the AEMO end point set. This includes Get Bulk Usage and Get Bulk DER. When a retailer is required to respond to these end points they should call the equivalent end point for specific service points and provide the specific list of NMIs to AEMO.“ This might not be as simple as stated especially if there is a large amount of NMIs and a large period requested. How do we intend to mitigate if this is the case?
• Will it be possible to provide a sample swaggers for the AEMO end points that describes the attributes as per the paper published?

[biza-io]

Please find attached the Biza.io Response to Decision Proposal 192: AEMO Exposed End Points:
DP192 - AEMO Exposed End Points.pdf

While the reasons for recommendations are outlined within the document we restate them here as follows:

1. Redraft the proposal to explicitly describe the AEMO endpoints being delivered. There is a precedent for the disclosure of separated but related endpoints within how the existing Standards describe the Admin APIs, and doing so would aid in client library production for Holders.

2. Alter the names, operationIds and paths of the AEMO endpoints to be deliberately separated from the ADR facing endpoints. As AEMO is, in effect, a Data Holder, albeit within a closed loop, there is already a standard specified for this within Extensibility, eg. /cds-au/v1/aem.

3. Mandate that the x-fapi-interaction-id provided to AEMO aligns with the generated x-fapi-interaction-id the Holder would respond with if it was absent from an ADR request.

4. Remove the introduced x-cds-arrangement header as it does not appear to serve a valid function. Should x-cds-arrangement remain, clarity should be sought as to whether there is a scenario where an Arrangement Identifier is not established (ie. during consent flow) but endpoint data is required.

5. Remove the following APIs for AEMO delivery:
   a. Get DER For Service Point
   b. Get Usage for Service Point

   This would represent a 33% reduction in build requirements associated with this proposal.

6. Consider the introduction of a Get Specific Service Points into the DP194 proposal and instead align this proposal to it. Alternatively, as per Recommendation 1, introduce such an endpoint explicitly in this specification rather than ambiguously redefining the existing Get Service Points as outlined in the Summary of alterations for AEMO endpoints column of the API breakdown table.

7. Remove commentary related to links as it is ambiguous. Traditional standards documents would, at most, place this under Implementation Considerations.

[CDR-API-Stream]

In response to @SelenaLiuEA

EnergyAustralia has explored this issue further in relation to the provision of Historical Metering Data under Issue #195 and welcomes any feedback against that issue stream.

Thank you for the work posted on DP 195. It is recommended that participants take a look at that work as it seeks to address some fundamental issues in the energy sector for CDR.

The DSB will respond to that feedback in the #195.

[CDR-API-Stream]

In response to @PratibhaOrigin

“The retailers will, in effect, be able to operate as a proxy of the data provided by AEMO.”

• Based on this, assuming the retailer (data holder) is not supposed to store the data obtained from AEMO in any form and just pass it to the ADR. Please confirm.

Yes, that is correct

• However, the NMI to Service point IT transformation is required before passing the data.

Yes, that is correct

• If the data holder is not supposed to store the data, in case of reporting/breach, can the rules team make this clear where the responsibility sits with - AEMO OR retailer? As data holders won’t have any data stored, they will not be able to reproduce it in case of any complaints related to data shared for AEMO data sets.

This is beyond the scope of the DSB consultation. We have passed this feedback on to the rules team.

• This transformation from servicePointID to NMI or NMI to servicepointID will have added complexity and time lag while fulfilling a data sharing request.
• We request the DSB to take this into account while working on the non-functional requirements for energy – the timing requirements for retailer (data holder) should not consider the delay or time take by AEMO to fulfill the data request.
• However, it should take into consideration the added time for transformation on NMI to servicepointID and vice-versa before passing on the data.

We have taken this into account in #193 and appreciate the repost of this feedback in that thread.

“ AEMO is unable to determine how long the current customer has had control of any specific NMI but they are able to determine how long the retailer (or more specifically, participant ID) has had control of each NMI. If a request for usage data spans a time period when the retailer was not in control of the NMI then AEMO will not respond with an error but will not share data outside the period of control of the retailer. It remains an obligation of the retailer to ensure that the data requested and then shared with the ADR is not outside the bounds of control of the specific customer.1”

• Based on the above comment , is it correct to say that for a specific usage data request , historic meter/usage data wont be shared. The usage data for the period customer is with current retailer (data-holder) will only be in scope.

Not quite, there is nuance to the response here. Usage history is in scope for CDR (even beyond the current period of relationship with the current retailer) but there are technical barriers to being able to deliver this data safely under CDR. EA has posted some excellent work on summarising this problem space under #195.

1. This multiple hand-off points and added logic to transform critical data before requesting and sharing data without actually storing the data has an added risk of errors and delays. As suggested previously, in light of comments raised above, we wanted to raise the query to rules team again – can a retailer(data holder) share the meter data from its own record instead of requesting its from AEMO and adding complexities of multiple handoffs , delays etc.

We will pass this feedback to the rules team, however, the DSB understands that they are aware of this feedback already.

[CDR-API-Stream]

In response to feedback from @energyaustraliaDD

• Is the intent that the AEMO endpoints are defined separately on GitHub, next to the equivalent endpoints exposed by retailers? Potentially renaming the EndPoints to stop confusion with the endpoints to be exposed by retailers?
• Can DSB define correctly the AEMO endpoints in the Draft Energy rules instead of describing the differences?

Yes, that is the intent. This proposal did not contain a detailed specification as the goal was to get consensus on the general approach. To turn this into actual standards we will need to expand on the proposal with explicit, detailed specs.

• As stated on page 3, Additional Notes (second bullet point) “End points that require knowledge of the NMIs that belong to the consumer have been excluded from the AEMO end point set. This includes Get Bulk Usage and Get Bulk DER. When a retailer is required to respond to these end points they should call the equivalent end point for specific service points and provide the specific list of NMIs to AEMO.“ This might not be as simple as stated especially if there is a large amount of NMIs and a large period requested. How do we intend to mitigate if this is the case?

It is not expected that this would be difficult as only the request would be impacted. The retailer would also need to implement this logic for the account end points to identify the NMIs related to held accounts. If specific technical issues are identified, however, a standards response would be developed in response using standard processes.

• Will it be possible to provide a sample swaggers for the AEMO end points that describes the attributes as per the paper published?

Yes, that will be the next step.

[CDR-API-Stream]

In response to the feedback from @biza-io

Please find attached the Biza.io Response to Decision Proposal 192: AEMO Exposed End Points:
DP192 - AEMO Exposed End Points.pdf

Thank you for the detailed feedback. We will incorporate this in the formulation of detailed specifications for the proposed endpoints.

While the reasons for recommendations are outlined within the document we restate them here as follows:

1. Redraft the proposal to explicitly describe the AEMO endpoints being delivered. There is a precedent for the disclosure of separated but related endpoints within how the existing Standards describe the Admin APIs, and doing so would aid in client library production for Holders.

Yes, that is the intent. This proposal was intended to consult on the general approach. As there has been no feedback against the recommendation the next step will be to move to a more detailed definition of the end points.

2. Alter the names, operationIds and paths of the AEMO endpoints to be deliberately separated from the ADR facing endpoints. As AEMO is, in effect, a Data Holder, albeit within a closed loop, there is already a standard specified for this within Extensibility, eg. /cds-au/v1/aem.

Making a clear separation from the retailer endpoints is good feedback and will be adopted. Leveraging the extenisibility model is one approach. Another could be to formally incorporate a treatment of secondary data holder endpoints as this may be a recurring pattern in future sectors.

3. Mandate that the x-fapi-interaction-id provided to AEMO aligns with the generated x-fapi-interaction-id the Holder would respond with if it was absent from an ADR request.

This was implied but not clear. It would be a good clarification. The interaction ID should be a mandatory request header for the AEMO specific end points.

4. Remove the introduced x-cds-arrangement header as it does not appear to serve a valid function. Should x-cds-arrangement remain, clarity should be sought as to whether there is a scenario where an Arrangement Identifier is not established (ie. during consent flow) but endpoint data is required.

The purpose of the header was to provide audit level traceability to ensure that retailers are not arbitrarily calling the AEMO end points and are doing so in response to an authorised consent.

5. Remove the following APIs for AEMO delivery:
   a. Get DER For Service Point
   b. Get Usage for Service Point
   This would represent a 33% reduction in build requirements associated with this proposal.

Why should these end points be removed? This would transfer a one time build cost from AEMO to the equivalent build cost for every retailer since the cost for AEMO to make the GET X For Service Point into a proxy for GET X For Specific Service Points can be done server side as well as client side. The overall implementation cost delta would not appear to stack up.

6. Consider the introduction of a Get Specific Service Points into the DP194 proposal and instead align this proposal to it. Alternatively, as per Recommendation 1, introduce such an endpoint explicitly in this specification rather than ambiguously redefining the existing Get Service Points as outlined in the Summary of alterations for AEMO endpoints column of the API breakdown table.

There is a need for the ADR to have an endpoint to actually get a list of service points for a customer. There does not appear to be a need for a Get Specific Service Points endpoint (based on previous consultation). It is not clear how this would improve the overall standards. The DSB will discuss this feedback internally, however, and we welcome further justification in the consultation on NMI Standard Data endpoints.

7. Remove commentary related to links as it is ambiguous. Traditional standards documents would, at most, place this under Implementation Considerations.

The DSB does not concur with this feedback.

[CDR-API-Stream]

After reviewing the feedback provided it would appear that there is strong support for the recommendations in the decision proposal.

It is acknowledged that this is a high level proposal only and specific endpoint definitions need to be developed. As a result a decision will not be taken to the DSB Standards Chair at this stage until a draft specification has been published.

[biza-io]

Remove the introduced x-cds-arrangement header as it does not appear to serve a valid function. Should x-cds arrangement remain, clarity should be sought as to whether there is a scenario where an Arrangement Identifier is not established (ie. during consent flow) but endpoint data is required.
The purpose of the header was to provide audit level traceability to ensure that retailers are not arbitrarily calling the AEMO end points and are doing so in response to an authorised consent.

The presence of this header provides absolutely no guarantee of any of this and AEMO has no method of verifying it because the arrangement information is not disclosed and the identifier easily forged. Traceability is a problem solved more explicitly by x-fapi-interaction-id already and, if necessary, can be used as a reference point to retrieve arrangement identifiers through regulatory spot checking although it is unclear what control is intended here.

Additionally and as noted in our response it remains unclear how this header could be populated if the information being requested is required for the establishment of consent in the first place.

   Remove the following APIs for AEMO delivery:
 a. Get DER For Service Point
    b. Get Usage for Service Point
    This would represent a 33% reduction in build requirements associated with this proposal. 

Why should these end points be removed? This would transfer a one time build cost from AEMO to the equivalent build cost for every retailer since the cost for AEMO to make the GET X For Service Point into a proxy for GET X For Specific Service Points can be done server side as well as client side. The overall implementation cost delta would not appear to stack up.

We believe we have outlined the reasoning in our response and recommend the DSB reread it to acquire an understanding of the assessment that has been done. We paste a screenshot of the relevant section for the convenience of the DSB:

We are wary of the term proxy being used in the above response as its context appears to be different to its use in the original proposal and is misleading relative to the implementation reality. By definition a data transform is required to pass calls between ADR->Holder and Holder->AEMO because doing so incorporates alternate authentication methods, in-line header filtering, in-line header additions (including some with apparent business logic) and rewriting of meta data to name a few.

In summary, there is no transfer of build cost as the requirement for the retailer calling AEMO does not dictate a need to distinctly separate between 1 OR >1 service point details. If a request is received on the CDR facing side for a single service point it can be packaged as a downstream request of a single servicePointId and then unwrapped to a single response. Likewise if a request is received on the CDR facing side for multiple service points the same downstream methods could be called but the input parameters would have multiple servicePointId and the whole array returned.

[CDR-API-Stream]

Please find attached the final decision of the Chair:
Decision 192 - AEMO Exposed End Points.pdf