Multiple missing values from mandatory fields served by AEMO APIs

[perlboy]

Description

The AEMO APIs are serving payloads which are non-compliant with respect to the Data Standards. This results in validation errors as the response is passed from AEMO side to the Recipient side resulting in a holder either:
a) Responding to the Recipient with an Unexpected Error with isSecondaryDataHolder set to true
b) The Holder presenting a schema invalid payload because the SDH provided an incorrect one

When requesting a solution from AEMO the response has been to request the Retailer request the upstream provider (i.e. the metering provider) to update the requested data. This is neither scalable nor sustainable and ultimately is placing unreasonable pressure on the entire CDR workflow because a mandated holder is failing to deliver schema validated payloads.

Area Affected

We have identified at least the following fields which have this issue:

• All fields in EnergyServicePointDetail.distributionLossFactor
• EnergyElectricityACConnectionV1.commissioningDate

Change Proposed

A few solutions:

1. Fix the data. That's not really a Standards problem and I'm not so sure it's realistically viable.
2. AEMO modifies it's API outputs with dummy but schema valid responses
3. Update the Standards to mark identified fields as Optional

[JohnLaird-SA]

This issue is something we have also ran into over the last few weeks which has led to some design choices to relax data validation for specific attributes, which is not ideal.

It would be great to have better guidance on the standards for Open Energy and how they have been implemented by the different agents.

I would only suggest that the 2nd proposal of dummy output also contains some other identifying attribute to ignore the invalid values.

[johnAEMO]

Thank you for advising us of the missing distributionLossFactor fields.
These have been missing since 17:00 AEST on 27/6/23 and now reinstated and have been available since 03:00 AEST on 1/7/2023.
DistributionLossFactors are reset annually by industry at the end of June and this change was not as seamless as expected.

Regarding the missing DER commissioningDate values, our analysis has indicated that these are only missing where there is no DER present (which is in approximately 80% of NMIs/SPs) and therefore no commissioning. AEMO has been operating under #526 which addressed the handling of these instances, since November 2022.
We are happy to reopen the #526 discussion should this be required.

[CDR-API-Stream]

The issue regarding EnergyDerRecord.acConnections.commissioningDate was discussed with some participants in a meeting facilitated by AEMO on 6th July 2023.

• The DSB clarified that providing an empty array, in this case acConnections, is compliant with the standards
• However, If the acConnections array was not empty, the commissioningDate field being mandatory would need to be provided

Based on the information we have, there does not seem to be any changes required to the standards. But if more information is provided, the issue can be revisited.

[perlboy]

As recently as today we continue to observe invalid payloads from AEMO, notably a set of acConnections without a commissioning date. We are currently observing between 2-5 of these per day and rejecting them as invalid with a response of GeneralError/Unexpected and isSecondaryDataHolderError set to true.

To be clear the AEMO payload is neither complying with the Standards (commissioningDate is missing) nor to the discussion in 526 as availablePhaseCount and installablePhaseCount are being returned as 1 not the proposed value of 0. acConnections is not being returned as an empty array but an array with an invalid schema.

Again, we request a change to make this field optional as it appears to impact a vast majority of records (80%+) and is causing ongoing operational issues for which AEMO does not appear to be providing relief. We are caught between receiving invalid data from the Market Operator yet having an obligation to pass compliant responses to Data Recipients.

[Justin-AEMO]

Thank you for the notification. An issue has been identified with the DER Register API. The team are currently working on remediation at the moment. We will update this thread once the fix has been applied.

[ShaneDoolanFZ]

Hi @Justin-AEMO. Just checking in on whether there is a target date for remediation? Thanks.

[Justin-AEMO]

Planned fix for DER register API to be applied 20 August 2023.

[ShaneDoolanFZ]

Great. Thanks for your prompt response @Justin-AEMO.

[johnAEMO]

Issue has been closed