Decision Proposal 194 - Candidate NMI Standing Data End Points

[CDR-API-Stream]

This decision proposal contains a recommendation for the candidate URIs and end points for NMI Standing Data. This proposal has been developed with the aid of AEMO.

The decision proposal is embedded below:
Decision Proposal 194 - Candidate NMI Standing Data Payloads.pdf

This consultation will be open for feedback until the 17th September 2021.

[EnergyAustraliaBA]

Thank you for the opportunity to review and allowing us to provide feedback on the proposal paper published on NMI standing data. Following are list of questions and clarifications we have.

1. Is it correct to assume that the “Candidate.NMI.Standing.Data.Payloads.” only shows the current state of the NMI related data?
2. What’s the rationale for “servicePointClassification” in the CDR as per summary of changes proposed? - For CDR I would query some values in servicePointClassification in CDR. I understand MSATS has values like Interconnector, Cross Boundary etc, but not sure if makes sense for CDR.
3. validFromDate what is the definition of “…first became valid” as per page 8. Is it NMI created in MSATS or First meter installed on NMI?
4. lastUpdateDateTime: the date and time that the information for this service point was modified. Should it include “last” modified?
5. relatedParticipants (Page 13) role As the LNSP is fixed for the NMI, but the FRMP can change in time. Does this only show the Current FRMP? (as no effective dates provided)
6. meters (page 13) does the array show all meters that have ever been located at the NMI? As status has Removed as well? There are no effective dates in relation to these meters, what is the value of Removed Meters?
7. readType (page 15)- currently this is set to be 3 characters, but will it be extended to 4 characters to include intervallength.
8. lastReadDate/Quality (page 4)- flagged as mandatory, will AEMO NMI/meters have this value to make this mandatory?
9. What’s the rationale for removing the following:

• streamStatus
• profileName
• peakDemand1
• peakDemand2

10. Whats the reasoning behind adding "registerConsumption" – what values are to be expected? How is this different to “registerConsumptionType” introduced to replace “dataStreamType”.

[CDR-API-Stream]

In response to @EnergyAustraliaBA:

1. Is it correct to assume that the “Candidate.NMI.Standing.Data.Payloads.” only shows the current state of the NMI related data?

Yes

2. What’s the rationale for “servicePointClassification” in the CDR as per summary of changes proposed? - For CDR I would query some values in servicePointClassification in CDR. I understand MSATS has values like Interconnector, Cross Boundary etc, but not sure if makes sense for CDR.

This is a value held by AEMO in MSATS and it is therefore designated and valid for sharing. Up until now we have not received feedback that the inclusion of this field would be a problem. In general we would do not assess the likely value of a field as this can be highly contextual to a specific ADR, service and consumer.

3. validFromDate what is the definition of “…first became valid” as per page 8. Is it NMI created in MSATS or First meter installed on NMI?

We defer to the AEMO's definition of this field under MSATS as this field has a direct corollary in the MSATS data.

4. lastUpdateDateTime: the date and time that the information for this service point was modified. Should it include “last” modified?

This field is effectively last the modified field. This is the naming convention CDR uses for such fields.

5. relatedParticipants (Page 13) role As the LNSP is fixed for the NMI, but the FRMP can change in time. Does this only show the Current FRMP? (as no effective dates provided)

Yes. We believe that to be the case

6. meters (page 13) does the array show all meters that have ever been located at the NMI? As status has Removed as well? There are no effective dates in relation to these meters, what is the value of Removed Meters?

The expectation is that the meter data held by AEMO will be returned. If this includes removed and disconnected records then they should also be returned.

7. readType (page 15)- currently this is set to be 3 characters, but will it be extended to 4 characters to include intervallength.

This field is structured according to the current structure defined by MSATS. There is no intention to change the meaning of this field for the CDR.

8. lastReadDate/Quality (page 4)- flagged as mandatory, will AEMO NMI/meters have this value to make this mandatory?

We have flagged this mandatory in consultation with AEMO that this classification would be acceptable.

9. What’s the rationale for removing the following:

• streamStatus
• profileName
• peakDemand1
• peakDemand2

We received feedback from the industry in previous rounds of consultations that these should be removed and there was no counter feedback to suggest they should be retained.

1. Whats the reasoning behind adding "registerConsumption" – what values are to be expected? How is this different to “registerConsumptionType” introduced to replace “dataStreamType”.

Thank you for picking this up. This is a typo in the proposal document. The added field (as seen in the payload body) is "registerConsumptionType"

[PratibhaOrigin]

Thanks for the opportunity to provide feedback on this topic.

Few queries /comments below –

1. Query on get service points end point -

GET /energy/electricity/servicepoints - Obtain a list of service points owned by the customer that has authorised the current session

As AEMO is the data holder and supporting this endpoint, how is AEMO supposed to know which sites/NMIs/service points are related to specific customer as the don’t have the customer data.
This information will in fact sit with retailer (primary data holder). Is the retailer (primary data holder) supposed to pass the list of NMIs or some parameters while requesting this data from AEMO regarding other fields mentioned in this payload.
Further clarification on this will helpful.

2. Paper suggests –

The data covered by this proposal is expected to be provided to the primary data holder directly by AEMO as the secondary data holder. This means that the existing data structures defined by AEMO for data transfer within the electricity market is the domain of data that can be included for sharing.

Confirmation required on the above that it refers to the data structures contained in existing data transfer methods and not the actual existing data transfer methods for NMI standing data. i.e, we are not referring to existing B2B method for sharing this data . The requested CDR data will be shared between AEMO and retailer via AEMO exposed end points as referred in DP – 192.

3. GET /energy/electricity/servicepoints/{servicePointId}

Will {servicePointId} be a NMI when the retailer calls this API ?
If so, then can we add an additional request parameter - servicepoint mapping which can be added into the response field of the ' servicePointId ' . This will avoid extra step of transformation at the retailer's end before sending the response to the ADR.

4. Response payloads for both APIs -

GET /energy/electricity/servicepoints and GET/energy/electricity/servicepoints/{servicePointId}

• servicePointId - Tokenised ID of the service point to be used for referring to the service point in the CDR API suite. To be created in accordance with CDR ID permanence requirements

• nationalMeteringId - The independent ID of the service point, known in the industry as the NMI

AEMO will be sending NMI ID in both servicePointId and nationalMeteringId fields of the payload. Is this correct ?

5. servicePointStatus -
   OFF_MARKET (Applies when the service point is no longer settled in the NEM)

Query related to the above Off-market scenario.
We have 3 scenarios for embedded networks –
A. Currently On-market and has a market NMI ID allocated
B. Currently Off-market. , but previously On-market and hence still has a market NMI allocated.
C. Currently Off-market and always been off-market. Doesn’t have a market NMI allocated.

Is it fair to say that scenario A and B are in scope of CDR but not scenario C.
We may need some input from rules and CXteam here too. If the scenario C is not in scope – are we supposed to make this available for CDR account sharing on account selection screen of the Data holder ?

6. Paper states –

readType –
Code to denote the method and frequency of Meter Reading.
The value is formatted as follows:

• First Character = Remote (R) or Manual (M);
• Second Character = Mode: T = telephone W = wireless P = powerline I = infra-red G = galvanic V = visual
• Third Character = Frequency of Scheduled Meter Readings: 1 = Twelve times per year 2 = Six times per year 3 = Four times per year D = Daily or weekly .For example, MV3 = Manual, Visual, Quarterly.

Just a clarification – after the MSDR changes , readtype is a 4 character field and not 3 character as mentioned in the paper. This needs to be updated here too in accordance to market changes.

7. nextScheduledReadDate and lastReadDate - should not be applicable if remotely read.

8. multiplier – As part of MSDR , it has been updated to an enumerated list. Will the same be updated here ?

9. We also had query around the use-cases for some of the data values on how they will be useful for CDR. Are there any use-cases mentioned by ADR where these values will be used. Or the intention is to do a NMI standing data dump in sync with the data values available in MSATs and upto the ADRs how they want to utilise the values.
   For example –
   • servicePointClassification - INTERCONNECTOR
   • distributionLossFactor

[ghost]

On behalf of AGL Australia CDR working group:

AGL welcomes the opportunity to provide feedback on the proposed paper. The following is the list of our comments:

• Unlike cross sectorial data clusters such as billing, NMI standing data forms part of the inner workings of the energy industry that requires an intimate knowledge to accurate represent and use. Beyond the technical description of each attribute, we suggest complimenting the technical standards with real world examples, or at least awareness of the accuracy and/or correct interpretation of the information provided.
  • For example, just because the nextScheduledReadDate returns DD/MM/YYYY does not necessarily mean the consumer should expect to see someone turn up at their home on that date.
  • Likewise, the difference between ACTUAL vs SUBSTITUTE vs FINAL_SUBSTITUTE of a the last read quality which can influence the quality of the usage and subsequently billing information if it was cross validated.
• Over the course of the customer relationship with the retailer, for various reason the underlying NMI standing data can change. The risk of assuming only the “current state” available to ADRs might be limiting and may cause interpretation problems. We suggest enhancing this API to allow for historical NMI standing data. This will ensure consistency across historical usage.
  • For example, a change in meter type can influence historical usage trends and other insights.
• We noticed some inconsistency in representing dates. For example, nextScheduledReadDate is described as “DateString” while validFromDate is described as “Object”.
• We want to confirm our understanding that, although there are differences in formats like date and date-time between MSATS and CDR; AEMO will be returning all fields in CDR formats documented in “common field types” in standards.

Thank you

[johnAEMO]

Thank you for the opportunity to respond to the above Decision Proposal.

Our response is in the attached word document.
Decision Proposal 194 - AEMO Response v1.0.docx

[CDR-API-Stream]

Thanks everyone for the feedback. This consultation will now be closed and we will respond to the final items of feedback in short order.

[CDR-API-Stream]

In response to the feedback from @PratibhaOrigin:

1. Query on get service points end point -

GET /energy/electricity/servicepoints - Obtain a list of service points owned by the customer that has authorised the current session

As AEMO is the data holder and supporting this endpoint, how is AEMO supposed to know which sites/NMIs/service points are related to specific customer as the don’t have the customer data. This information will in fact sit with retailer (primary data holder). Is the retailer (primary data holder) supposed to pass the list of NMIs or some parameters while requesting this data from AEMO regarding other fields mentioned in this payload. Further clarification on this will helpful.

This is correct. The retailer will past a list of NMIs in the request payload to AEMO. As highlighted in Decision Proposal 192, this endpoint will change from GET to POST for retailer requests to AEMO

2. Paper suggests –

The data covered by this proposal is expected to be provided to the primary data holder directly by AEMO as the secondary data holder. This means that the existing data structures defined by AEMO for data transfer within the electricity market is the domain of data that can be included for sharing.

Confirmation required on the above that it refers to the data structures contained in existing data transfer methods and not the actual existing data transfer methods for NMI standing data. i.e, we are not referring to existing B2B method for sharing this data . The requested CDR data will be shared between AEMO and retailer via AEMO exposed end points as referred in DP – 192.

Yes, CDR data will be shared via AEMO exposed endpoints

3. GET /energy/electricity/servicepoints/{servicePointId}

Will {servicePointId} be a NMI when the retailer calls this API ? If so, then can we add an additional request parameter - servicepoint mapping which can be added into the response field of the ' servicePointId ' . This will avoid extra step of transformation at the retailer's end before sending the response to the ADR.

The mapping of NMI to servicePointId must be maintained by the Retailer under the ID permanence requirements. Orchestrating this mapping via the AEMO endpoints would not be a pattern that the DSB would recommend without good reason.

4. Response payloads for both APIs -

GET /energy/electricity/servicepoints and GET/energy/electricity/servicepoints/{servicePointId}

• servicePointId - Tokenised ID of the service point to be used for referring to the service point in the CDR API suite. To be created in accordance with CDR ID permanence requirements
• nationalMeteringId - The independent ID of the service point, known in the industry as the NMI

AEMO will be sending NMI ID in both servicePointId and nationalMeteringId fields of the payload. Is this correct ?

Yes

5. servicePointStatus - OFF_MARKET (Applies when the service point is no longer settled in the NEM)

Query related to the above Off-market scenario. We have 3 scenarios for embedded networks – A. Currently On-market and has a market NMI ID allocated B. Currently Off-market. , but previously On-market and hence still has a market NMI allocated. C. Currently Off-market and always been off-market. Doesn’t have a market NMI allocated.

Is it fair to say that scenario A and B are in scope of CDR but not scenario C. We may need some input from rules and CXteam here too. If the scenario C is not in scope – are we supposed to make this available for CDR account sharing on account selection screen of the Data holder ?

As this is a scope question it is likely that it can only be clarified by the Rules team. We will pass this feedback on to them for awareness

6. Paper states –

readType – Code to denote the method and frequency of Meter Reading. The value is formatted as follows:

• First Character = Remote (R) or Manual (M);
• Second Character = Mode: T = telephone W = wireless P = powerline I = infra-red G = galvanic V = visual
• Third Character = Frequency of Scheduled Meter Readings: 1 = Twelve times per year 2 = Six times per year 3 = Four times per year D = Daily or weekly .For example, MV3 = Manual, Visual, Quarterly.

Just a clarification – after the MSDR changes , readtype is a 4 character field and not 3 character as mentioned in the paper. This needs to be updated here too in accordance to market changes.

Thank you, this is useful feedback. AEMO has also provided feedback along these lines and we will incorporate changes arising from MSATS procedures once they are firm. Note that this is likely to be an ongoing process even after implementation is complete.

7. nextScheduledReadDate and lastReadDate - should not be applicable if remotely read.

The attributes are aligned with MSATS and are expected to be provided accordingly. We will re-confirm and update descriptions accordingly

8. multiplier – As part of MSDR , it has been updated to an enumerated list. Will the same be updated here?

As mentioned above, the standards will align to current MSATS procedures. They can be updated as necessary closer to the new MSATS procedures being finalised.

9. We also had query around the use-cases for some of the data values on how they will be useful for CDR. Are there any use-cases mentioned by ADR where these values will be used. Or the intention is to do a NMI standing data dump in sync with the data values available in MSATs and up to the ADRs how they want to utilise the values. For example

• servicePointClassification - INTERCONNECTOR
• distributionLossFactor

The latter scenario is correct. The standards are developed according to designation and not nominated use cases.

[CDR-API-Stream]

In response to feedback from @AElharouny:

AGL welcomes the opportunity to provide feedback on the proposed paper. The following is the list of our comments:

• Unlike cross sectorial data clusters such as billing, NMI standing data forms part of the inner workings of the energy industry that requires an intimate knowledge to accurate represent and use. Beyond the technical description of each attribute, we suggest complimenting the technical standards with real world examples, or at least awareness of the accuracy and/or correct interpretation of the information provided.
  • For example, just because the nextScheduledReadDate returns DD/MM/YYYY does not necessarily mean the consumer should expect to see someone turn up at their home on that date.
  • Likewise, the difference between ACTUAL vs SUBSTITUTE vs FINAL_SUBSTITUTE of a the last read quality which can influence the quality of the usage and subsequently billing information if it was cross validated.

Thank you. Our CX team are developing a data language to help understand some of the technical and industry specific concepts. They usually provide additional description that elaborates on the data as the cluster/permission language is difficult to comprehend on their own. It would be at the DH’s discretion to clarify specifics like the examples above.

Over the course of the customer relationship with the retailer, for various reason the underlying NMI standing data can change. The risk of assuming only the “current state” available to ADRs might be limiting and may cause interpretation problems. We suggest enhancing this API to allow for historical NMI standing data. This will ensure consistency across historical usage.

• For example, a change in meter type can influence historical usage trends and other insights.

The standing data payload is a reflection of MSATS standing data. A data recipient can build up a “change over time” view of the data with multiple requests to the standing data payload combined with usage data information.

With regard to the ability for recipients to be able to interpret data, this is best managed via feedback helping to enhance the payloads to ensure they can be interpreted properly (particularly in reference to the example provided). It needs to be noted that customer may approve access to usage data without approving access to standing data. If the standing data is required to interpret usage data then the usage payloads should be considered inadequate.

Would you recommend any changes to the usage payloads to ensure they can be interpreted in isolation?

• We noticed some inconsistency in representing dates. For example, nextScheduledReadDate is described as “DateString” while validFromDate is described as “Object”.
• We want to confirm our understanding that, although there are differences in formats like date and date-time between MSATS and CDR; AEMO will be returning all fields in CDR formats documented in “common field types” in standards.

Thank you, we will correct the inconsistencies. Yes, AEMO (and all CDR Data Holders) need to provide data in CDR compliant format.

[CDR-API-Stream]

It has been brought to our attention by the CX team (who are doing data language standards) that the scope for standing data detail should be:
energy:electricity.servicepoints.detail:read
to match the scope for the equivalent list endpoint which is
energy:electricity.servicepoints.basic:read

This change will be included in the decision.

[CDR-API-Stream]

Further feedback to @PratibhaOrigin after referring the question to the rules team:

1. servicePointStatus -
   OFF_MARKET (Applies when the service point is no longer settled in the NEM)

Query related to the above Off-market scenario.
We have 3 scenarios for embedded networks –
A. Currently On-market and has a market NMI ID allocated
B. Currently Off-market. , but previously On-market and hence still has a market NMI allocated.
C. Currently Off-market and always been off-market. Doesn’t have a market NMI allocated.

Is it fair to say that scenario A and B are in scope of CDR but not scenario C.
We may need some input from rules and CXteam here too. If the scenario C is not in scope – are we supposed to make this available for CDR account sharing on account selection screen of the Data holder ?

The response from the rules team is as follows:

Treasury’s rules team has considered these scenarios and advised that only the consumer in scenario A is intended to be in scope for purposes of the CDR. To be eligible under the CDR rules, the consumer needs to not only have a NMI, but for their NMI to be settled in the wholesale market, with a financially responsible market participant (FRMP) allocated to it, and their retail arrangement must be with a retailer (and not an exempt seller). The rules team’s understanding is that in scenario B, the consumer will have a NMI but there won’t be a FRMP allocated to the NMI, and therefore the consumer in scenario would not be eligible to share their data via the CDR. Nor would the consumer in scenario C. For scenarios B and C, the rules team does not expect that data holders would make the account information available for off-market accounts on their account selection screen, as this may cause confusion.

[CDR-API-Stream]

Responses to the feedback with AEMO (from the PDF submission):

• At the top of page 3 the first dot point, we would recommend changing to “The underlying quality of the data held for industry participants by AEMO is low…”
• Typo under Summary of Changes table page 5, should read registerConsumptionType
• Typo under Summary of Changes table page 5, installationType should read NCONUML and is unmetered load (not metered) and was introduced because of Global Settlement (not 5MS)

Thank you for the specific feedback, we will update to fix the highlighted errors.

• page 6, presumably ServicePointId will be populated by the retailer (as it will be unknown by AEMO)

This is correct (this was determined in #192)

• page 7, ServicePointClassification we would recommend removal of SAMPLE, BULK, INTERCONNECTOR, CROSS_BOUNDARY as these do not have end use customers –

The values will be removed as recommended

• page 8, isGenerator we would recommend that this is removed as it is predominantly used for settlement and can be misleading if relied upon to flag local generation (refer rather to the DER API)

The attribute will be left in the payload, the description will be updated to clarify that the value of this attribute does not indicate consumer generation

• page 8, vaildFromDate we would recommend changing the type to DateString and the definition as per the NOTE on Effective Dates

We will update the type to DateString

• page 8, lastUpdateDateTime we would recommend removing this field or if it must be included define it as per the NOTE on Effective Dates

The attribute will be left in the payload as there is not enough justification to remove it and it is a useful field for recipients

• page 12, distributionLossFactor description we would recommend changing the definition to remove the words “and value” as the value is described in the next field

Note the standing data description includes both code and value. As discussed offline, we will leave the description as is

• page 13, relatedParticipants role we would recommend the inclusion of a third role with the description
  • DRSP - (wholesale Demand Response and/or market ancillary Service Provider and note that where it is not relevant for a NMI it will not be included)

The value will be added as recommended

• page 14, meters status we would recommend the removal of the REMOVED code (as the meter is no longer active)
• page 14, meters installationType we would recommend the removal of the PROF and SAMPLE codes as these are for internal purposes only
• Typo page 14, meters installationType should read NCONUML and is unmetered load (not metered) and was introduced because of Global Settlement (not 5MS)

The above changes will be made as requested

• page 15, meters readType we would recommend changing the description to include the potential for:
  “Optional Fourth Character = to identify what interval length the meter is capable of reading. This includes five, 15 and 30 minute granularity as the following:

  • A – 5 minute
  • B – 15 minute
  • C – 30 minute
  • D – Cannot convert to 5 minute (i.e. due to metering installation de-energised)
  • M - Manually Read Accumulation Meter

  For example, MV3M = Manual, Visual, Quarterly, Manually Read Accumulation Meter; RWDC = Remote, Wireless, Daily, 30 minutes interval.”

The change will be made as requested. We note that this aligns with feedback from Origin

• page 15, meters nextScheduledReadDate we would recommend changing the description as it is not required for Interval meters, only Basic meters

As discussed offline, the description will be updated to align with MSATS

• page 15, meters lastReadDate and LastReadQuality: These fields are not standing data, AEMO calculates this information as part of the NMI discovery service APIs for Retailers. The fields should be removed as they are not NMI Standing data and hence do not fall under the CDR designation.

The attributes will removed from standing data based on the above feedback. Note that they are available/can be derived from usage data payloads

• page 15, meters registers we would recommend the insertion of a new field “suffix” after registerId. This would be a string, Mandatory and is the Unique identifier of the suffix within this service point. It is not globally unique

The attribute will be added and called “registerSuffix” in alignment with usage data.

• page 15, meters registers averagedDailyLoad we would recommend including in the description that this is calculated annually

Ok. We will incorporate this change

• page 16, meters registers registerConsumptionType we would recommend modifying the last four codes removing “NON_MARKET_”

Ok. NOTE: The change is being requested as registerConsumptionType is no longer non_market due to 5MS changes

• page 16, meters registers networkTariffType we would recommend be changed to networkTariffCode and made Optional

Ok. NOTE: The change is being requested as the attribute was historically optional and can be blank

• page 16, meters registers timeOfDay we would recommend be made an enum with the possible values:
  • ALLDAY
  • INTERVAL
  • PEAK
  • BUSINESS
  • SHOULDER
  • EVENING
  • OFFPEAK
  • CONTROLLED
  • DEMAND

Ok. We will incorporate this change

• page 16, meters registers controlledLoad we would recommend modifying the second paragraph of the description modifying “No” to read ““No” or similar…”
• page 16, meters registers controlledLoad we would recommend modifying the last paragraph of the description to read “If the register relates to a Controlled Load, it may contain a description of that Controlled Load or a “Yes” or similar. Note that this is a free text field and is currently being restructured in consultation with industry.”

As discussed offline, the “controlledLoad” attribute will be made into a boolean with “true” or “false” as possible values

• Typo page 16, meters registers consumptionType, remove “899”

Thank you for picking up the typo, it will be removed

[CDR-API-Stream]

The Chair has approved the attached decision for this consultation:
Decision 194 - Candidate NMI Standing Data End Points.pdf