Noting Paper 248 - Energy PRD

[CDR-API-Stream]

The attached noting paper outlines the exposure patterns and obligations regarding product reference data (ie. Generic Tariffs) in the energy sector.

---

edit 11th May

An updated version of the noting paper is provided below:
Noting Paper 248 - Energy PRD.pdf

This version includes:

• statements indicating which patterns do, or do not, imply self-hosting by the energy retailer
• an additional statement regarding certificates is also provided for pattern 2 noting paper has been updated
• minor fixes of typos

The original noting paper is attached below for reference:
Noting Paper 248 - Energy PRD 04-14.pdf

---

This paper is not a proposal and therefore this thread is not a formal consultation. Feedback or requests for clarification on the paper are, however, welcome.

[perlboy]

Firstly, thank you for preparing this Noting Paper on short notice to clarify the position with regards to the presentation of Generic Plan data, aka Product Reference Data (PRD) for Energy. Reading this through there seems to be some missing considerations that are alluded to but not specifically addressed.

In response it is important to define a number of factual requirements relevant for this noting paper:

1. PublicBaseUri is used for PRD information and the Status/Outages endpoints. This is noted briefly in Pattern 2 but is, in fact, a global requirement regardless of technical solution. There is potential for more flexibility on this requirement if the Register was to bifurcate PublicBaseURI into two separate base uris (PRD and Status/Outages)
2. The current Metrics API requires the statistical collection of Unauthenticated endpoint activity vis-à-vis the activity attributed to the PublicBaseUri
3. Metrics API is required to be available to the Regulator for Retailers delivering Consumer Data Request Service (CDRS)
4. ACCC currently requires additional 6-monthly reporting of PRD Requests via a manual form submission process

With these requirements now stated it is now possible to assess the suitability of the described patterns in more detail.

Pattern 1: Direct to AER

• The PublicBaseUri for Retailers adopting this pattern would be set to the AER
• Metrics collection for these endpoints would, by definition, be the responsibility of AER particularly since Admin Metrics endpoints for Retailers would presumably not be available until they had exposed their CDRS
• It is unclear how AER would differentiate between retailers.
  • In the "traditional" PRD context this would be using a brand filter but metrics would be skewed for unfiltered requests.
  • Alternatively a base path could be specified for each retailer on the AER base domain although this could potentially increase implementation complexity for AER
• As noted this would no longer be viable when the Retailer was required to delivery Status and Outages APIs
• 6-monthly ACCC reporting would be the responsibility of AER unless AER was to provide each Retailer with a report that the Retailer could submit via the existing channels

Suitability: Non-CDRS Retailer ✅ CDRS Retailer ⛔

Pattern 2: DNS Redirection

• The proposal suggests the use of a DNS CNAME and the diagram indicates that Consumer traffic would "flow" through the Retailer DNS to AER.
  • This appears to be technically inaccurate, a DNS CNAME provides a resolver outcome (ie. a name to an IP). This is then used to initiate HTTPS communication on which to perform API calls. Those API calls would be made using the Retailers Host header directly to AER and AER services would be expected to respond appropriately to these retailer specific hostnames.
  • AER would need to maintain or be given TLS certificates that match this Host header presumably via SNI for each of the Retailers selected domains which has the potential to run afoul of corporate policies with regards to the assignment and provision of TLS certificates to hostnames within the corporate namespace
  • AER TLS certificate maintenance would be an exponential requirement as Retailers came online. Certificate rotation would need to be heavily automated to ensure sustainability which, while possible (through the likes of letsencrypt + http challenge proofs) may run afoul of retailer corporate policies
• Metrics would need to continue to be collected by AER and would need to be the responsibility of AER because the Retailer is not hosting the API services themselves.
• While a back channel reporting mechanism may be possible the Metrics endpoint has a relatively high collection frequency (at least once every 24hrs) which, when considered in the context of processing requirements for AER to produce such a report for 100s of Retailers may be impossible to achieve.
• As noted, once a CDRS service was operated by a Retailer the Status & Outage APIs would make this pattern non-viable
• There is a reference to the concept that when a Retailer goes live they can continue to use the same domain. I assume this is a reference to the fact the CNAME can be changed to a service hosted by the Retailer as a DNS CNAME provides no other avenue for consistency enablement. It's not involved in the actual API comms only the name resolution to the API itself.

Suitability: Non-CDRS Retailer ✅ CDRS Retailer ⛔

Pattern 3: Proxy

• This proposal implements a back-to-back API between the Retailer and AER
• While it is simpler to reference this as a Proxy this isn't technically accurate due to the implication that Metrics must be collected for reporting on Retailer Admin APIs. That is to say using a traditional "reverse proxy" pattern with COTS software would be unlikely to meet the requirements when considering the metrics collection requirements
• This pattern introduces the potential challenge that the latency between the Retailer Gateway and AER contributes to the overall time budget with respect to NFRs
• This pattern introduces the potential challenge that issues with the AER endpoint cascades implicitly into the Retailer. This is of particular concern because metrics may be incorrectly skewed based on performance outcomes outside of the Retailers control. Given the NFRs define a 95th percentile obligation the distribution of such calls could give rise to the Regulator raising incidents (as it is already very quick to do) for NFR failures for which the Retailer will be unable to ameliorate.
• This pattern is, in essence, a similar pattern to the Secondary Data Holder architecture (ie. AEMO) but without such a designation the Metrics API will not expose potential discrepancies between how long it takes AER to provide a response and how long it takes the Retailer to distribute that response to the Consumer

Suitability: Non-CDRS Retailer ✅ CDRS Retailer (provided the above concerns are handled) ✅

Pattern 4: Self-Hosted

• This is essentially an alignment to the Banking sector PRD delivery
• As noted, this would allow consumer access to be monitored by the retailer and therefore Metrics to be collated in much the same way as is already occurring in Banking

Suitability: Non-CDRS Retailer ✅ CDRS Retailer ✅

Pattern 5: Hybrid

• This pattern proposes to implement a similar pattern to Pattern 3: Proxy however with an added business rule transformation for supply of non AER products (such as C&I from internal Retailer systems)
• This pattern would require the dynamic merging of external and internal data for the purposes of Get Generic Plans. Given this endpoint has various filters and pagination behaviours this is likely to be more technically complex than perhaps meets the initial evaluation
• The issues associated with Metric collection and potential latency from AER provided endpoints outlined in Pattern 3 remain

Suitability: Non-CDRS Retailer ✅ CDRS Retailer (provided the above concerns and those in Pattern 3 are handled) ✅

Final note: No alternative solutions have been proposed because this isn't a proposal.

[CDR-API-Stream]

This is a very good summary of the position outlined in the paper and the suitability assessment presented is accurate.

The callout that AER would need to be provided certificates for DNS Redirection to work is a good one and should be added to the paper.

There is one item to note arising from this commentary, and this is something we may need to add to the paper, or clarify via other guidance.

An energy retailer that is not providing their own Product Data Request Service is not required to report on the PRD end points and is not accountable for the non-functional requirements related to that service. This means that the retailer can exclude these endpoints from the unauthenticated metrics in the Get Metrics API and in periodic reporting to the ACCC.

This is why it is highlighted as a disadvantage for the Self-Hosted and Hybrid patterns. These patterns result in the retailer actually presenting a Product Data Request Service themselves and the rules state that they would then become accountable for all of the obligations of that service (including metrics collection).

We did discuss separating PublicBaseURI into two different URIs but decided against pursuing this route due to the complexity of change required for the Register and existing participants for a situation that is probably unique to the energy sector.

Apologies if the diagram for the DNS redirection pattern looks incorrect. I was trying to make the pictures simple and representative for visual readers and differentiated pass through traffic from redirected traffic by the use of the API endpoint notation (line + circle). It is correct to highlight that the DNS redirection pattern does not result in the retailer actually receiving any traffic as this is one of the advantages of that approach.

[perlboy]

This is a very good summary of the position outlined in the paper and the suitability assessment presented is accurate.

Thanks!

The callout that AER would need to be provided certificates for DNS Redirection to work is a good one and should be added to the paper.

I know this is a noting paper only but administratively that's going to be quite painful so maybe AER should be providing comment (@joe-aer, I'll drag you in here 😄 ). It may be worth adding a note that to reduce party to party (ie. AER to Retailer) dependencies the Retailer might be requested to setup a TXT record in addition to the CNAME to support Email to DNS TXT contact DCV... I'm really not sure though, even putting aside security policies etc, pretty dubious on the sustainability of this (with 100s of retailers AER is going to need a dedicated cert renewal person....)

There is one item to note arising from this commentary, and this is something we may need to add to the paper, or clarify via other guidance.
An energy retailer that is not providing their own Product Data Request Service is not required to report on the PRD end points and is not accountable for the non-functional requirements related to that service. This means that the retailer can exclude these endpoints from the unauthenticated metrics in the Get Metrics API and in periodic reporting to the ACCC.

Hmm, this is an important clarification and is appreciated.

This is why it is highlighted as a disadvantage for the Self-Hosted and Hybrid patterns. These patterns result in the retailer actually presenting a Product Data Request Service themselves and the rules state that they would then become accountable for all of the obligations of that service (including metrics collection).

More for clarification than anything, I believe this is a reference to this part of the rules?

9.4 Reporting requirements
Reports that must be prepared—data holder
(1) A data holder must prepare a report for each reporting period that:
[snip]
(c) sets out the number (if any) of:
(i) product data requests; and
[snip]
received by the data holder during the reporting period; and

This covers the 6 monthly reporting requirement ✅ but I don't believe the Rules speak to the Get Metrics endpoint directly? It would be good to add a clarifying statement on Get Metrics to explicitly call out it is only metrics received directly by the Data Holder?

We did discuss separating PublicBaseURI into two different URIs but decided against pursuing this route due to the complexity of change required for the Register and existing participants for a situation that is probably unique to the energy sector.

I'm not confident such assumptions can be made and while I'm not seeking to push it here there's some direct benefits even in banking for doing so. I'm dubious about making decisions because a single participant (the Register) classifies it as a complex change - by that definition very little will get done.

Apologies if the diagram for the DNS redirection pattern looks incorrect. I was trying to make the pictures simple and representative for visual readers [snip]

Hmm, I guess it's a style choice. A dotted line to Retailer DNS then a solid line to AER would be clearer imo.

[perlboy]

During the Maintenance Iteration call there was some discussion in other business related to this delivery item. During the call there was an implication that, because the Status/Outages endpoint is part of the Standards to deliver on the above objective it would be necessary for a Retailer to provide a mechanism to deliver the PRD information, at least with, Pattern 3.

The overriding feedback from the DSB has always been that the Rules override the Standards and on that basis it seems important to specifically highlight components from the Rules which appear to be incongruous with the clarifications apparently intended in this Noting Paper but further stated in the Iteration call.

The Rules

CDR Rules: https://www.legislation.gov.au/Details/F2022C00187
Energy Sector Designation Instrument: https://www.legislation.gov.au/Details/F2020L00833

There's a number of relevant components in the rules here.

Definition of required product data

Noteworthy is that Section 9 & 10 of the designation instrument specifically relate to retail arrangements for electricity and gas. And part 12 of the designation instrument specifically states AER & "Victorian agency" as the designated source for Section 9:

Definition of product specific data
Schedule 4, 1.3 Meaning of terms for types of data in the context of Energy

Product Data Request Service

Product Data Request Service (Energy)
Schedule 4, Part 4, 4.2 Product data request service:

Further during the call there seemed to be some conflating of the difference between a Product Data Request Service and a CDR Data Request Service. Despite what the Standards may seek to imply a CDR Data Request Service implicitly includes many more obligations that, should unauthenticated endpoints be considered within such a scope, would be impossible to achieve because they must be conducted by eligible CDR consumers which are impossible to identify - and indeed - probably aren't eligible because they are looking at plans of alternative retailers:

Reporting Requirements

Division 9.3, Part 9.4 Reporting Requirements specifies reporting requirements to the ACCC for Product Data Requests:

Rules Observations

1. Product Data is clearly defined within the sectors and it is clearly tied to a Product Data Request Service (PDRS). There has been some suggestion that the unauthenticated endpoints other than PRD are not a PDRS and therefore must be a CDRS. This is not only technically impossible but in fact the other unauthenticated endpoints don't appear to have any classification at all (ie. the Data Standards alone mandate them, the Rules provide no coverage for them)
2. A Product Data Request Service is an online service, not a specific PRD API but rather an online service. Irrespective of the method (ie. the above scenarios, proxy or not) the online service is not an obligation on Retailers and in fact Retailers are explicitly excluded from needing to deliver this service.
3. Not withstanding (2), the designation instrument explicitly states the designated data is from AER & Victorian agency except where a plan is tailored. It's quite possible a Retailer presenting the data - unless it was sourced directly back from the AER - wouldn't actually be complying with the designation instrument.
4. Regardless of Scenario 3 -> 5, a Retailer would have "received" the request and, assuming they are considered a data holder at that point - and it's very murky as to whether they would be - would need to provide reporting to the Regulator

Standards Observations

The Standards, assuming a Retailer is required to deliver PRD (a big if):

1. Do not exclude metric collection for these endpoints. It may be implied but is never stated.
2. Define a non-functional requirement for "All unauthenticated endpoints" which means the above concerns related to Scenario 3 remain
3. Do not specify an acceptable latency for PRD Data in Energy
4. Do not specify the handling of error behaviours should AER be unavailable and a request is presented to the Data Holder

Action

Despite this being a Noting Paper what its production has highlighted is that the Standards don't appear to be fit for purpose for the apparent intended implementation and certainly don't appear to be aligned with the obligations outlined in the Rules. The Rules not only do not require a Retailer to provide a PDRS, they appear to explicitly exclude such a case - on this basis the Standards implying that it would need to be provided seems in conflict with the Rules and indeed is quite possibly not enforceable.

The stated reason for not splitting PublicBaseUri (which looks like it would resolve this issue fairly cleanly) was "complexity of change required for the Register". The net result of this is a, potentially very large, uplift in Retailers delivery requirements for something which the Rules don't appear to require.

On this basis alone it would appear that the splitting of PublicBaseUri should and indeed must be revisited especially since it doesn't appear to have actually been consulted on.

[CDR-API-Stream]

Hi @perlboy, in response to part of your previous feedback...

This is why it is highlighted as a disadvantage for the Self-Hosted and Hybrid patterns. These patterns result in the retailer actually presenting a Product Data Request Service themselves and the rules state that they would then become accountable for all of the obligations of that service (including metrics collection).

More for clarification than anything, I believe this is a reference to this part of the rules?

9.4 Reporting requirements
Reports that must be prepared—data holder
(1) A data holder must prepare a report for each reporting period that:
[snip]
(c) sets out the number (if any) of:
(i) product data requests; and
[snip]
received by the data holder during the reporting period; and

This covers the 6 monthly reporting requirement ✅ but I don't believe the Rules speak to the Get Metrics endpoint directly? It would be good to add a clarifying statement on Get Metrics to explicitly call out it is only metrics received directly by the Data Holder?

Actually, the section of the rules being referred to is 4.2 (2) in Schedule 4, Part 4. Specifically:

4.2  Product data request service

             (1)  Despite rule 1.12, a data holder of energy sector data, other than the AER and the Victorian agency, is not required to provide a product data request service.

             (2)  However, if such a data holder chooses to provide an online service that can be used to make product data requests, the service must comply with rule 1.12.

Basically, a retailer doesn't have to provide a Product Data Service but if they choose to do so they have to do it according to the obligations under the rules.

[CDR-API-Stream]

In response to the feedback from @perlboy, with a call to action to split the PublicBaseUri value on the register, the appropriate place to raise this is as a CR in standards maintenance. It may be the easiest solution for many retailers although the implementation constraint noted is also material in nature.

This would not, however, resolve the branding issues related to the presentation of the PRD endpoints for energy which may be important for some retailers.

Regarding your analysis of the rules, the noting paper was written to specifically clarify some of the ambiguities you raise and our understanding of the rules aligns with the content of the noting paper.

[perlboy]

Basically, a retailer doesn't have to provide a Product Data Service but if they choose to do so they have to do it according to the obligations under the rules.

This is a contradiction to the guidance given in the Noting Paper. The DSB appears to imply that "proxying" (which it technically isn't) is different to providing a Product Data Request Service which, for the purposes of the Rules, it isn't.

The retailer can voluntarily provide this data but, by the very nature of it being voluntary, requests to these endpoints will return a 404 at the PublicBaseUri related url for these endpoints if they choose not to. It's baffling why the DSB appears so intent on confusing this but clearly it is now the prerogative of the Holders and potentially Vendors delivering on Holder obligations to take a position in the absence of the DSB providing unambiguous guidance.

[biza-io]

Biza.io cannot speak directly for its customers however, we have requested our customers make their feedback in this matter known via the appropriate channels.

When providing technical guidance to our customers we have noted there are a significant number of unresolved operational concerns with the models outlined in the Noting Paper. Further we note that the DSB itself has acknowledged issues with the presented solution and has taken 2 months to suggest the creation of a CR to rectify a clear technical design failure.

A prudent action would be for the DSB to reopen consultation on these endpoints and reevaluate prior feedback in this new context if it intends to continue to suggest these endpoints must be functional at the Retailers published PublicBaseUri. Should the DSB choose to do this Biza.io will proceed to produce the change requests it believes are necessary to support this operating topology.

As this thread now potentially forms a key piece of evidence in any legal proceedings arising from the Regulator launching enforcement action with regards to Data Holders obligations associated with these endpoints we wish to place in the same record the historical occurrences where @CDR-API-Stream or members of the DSB have provided statements which appear to contradict the clarification this Noting Paper was apparently providing.

Decision Proposal 190, 11 October 2021, James Bligh

This data cluster is designated to be provided by the Australian Energy Regulator (AER) and the Victorian Department of Environment, Land, Water and Planning (DELWP) on behalf of Retailers.

Energy Draft Feedback Cycle 3, 30 May 2021, @CDR-API-Stream

It was raised that it is not clear how plans for Corporate and Institutional customers will be handled. It has been clarified that EME and VEC do not currently hold these plans so, if they are to be made available, that would be a change at the discretion of AER and DELWP. The CDR designation is to make available the data held by EME and VEC and therefore C&I plans will only come into scope if EME or VEC data sets are expanded.

Design Paper: a peer-to-peer data access model in the energy sector, 27 May 2021, @CDR-API-Stream

The design paper does not address this as this is not an area of the designation that is impacted by the P2P model change. Generic product data will be delivered to ADRs by EME and VEC using data provided to them by retailers as per the current designation instrument for the energy sector.

Energy Draft Feedback Cycle 1, 8 Mar 2021, @CDR-API-Stream

The designated data holders for the generic tariff structures is EME and VEC. Currently retailers are not expected to implement the generic tariff endpoints.

Decision Proposal 111, 4 August 2020, James Bligh pdf:

As stipulated in the designation instrument the data covered by this proposal is expected to be provided by the Australian Energy Regulator and the Victorian Government as the specified data holders.

[PratibhaOrigin]

Origin supports the policy views and solution issues raised by @perlboy . That is, we do not believe that there is any legislative requirement in relation to retailers providing product data held by Energy Made Easy or Victorian Energy Compare.

In the energy sector, plan information can be split into two data categories:

1. General product data – this is generally available, public offer data (data held by Energy Made East or Victorian Energy Compare); and
2. Tailored customer product data – this is data that is specific to the customers (ie data held only by retailers)

The intention of the Energy Designation for CDR and the CDR Rules was that 1) general product data be provided by Energy Made Easy and Victorian Energy Compare) while 2) tailored customer product data be provided by energy retailers. The distinguishing difference being general product data was related to the availability of market offers from any retailer operating in the market and tailored product data was related to the current product the customer was being supplied on.

Further to the analysis of the Energy CDR Rules by @perlboy , this distinction is clear in the Energy CDR Designation Instrument. Section 9 of the Energy Designation Instrument is in relation to “information about retail arrangements”. This is information relating to an arrangement such as tariffs, fees, benefits or rebates. Section 12 of the Energy Designation then defines who is the Data Holder for each of the relevant data sets. The Table in section 12 of the Designation Instrument (extract below) sets that the AER and Victorian Government is designated Data Holder for information related to retail arrangements except to the extent the information relates to a tailored arrangement (item 4 below). Retailers are then designated in item 5 below as a Data Holder for tailored retail information.

It is our view, that it is clear that the AER and Victorian Government have been designated as Data Holders for general retail arrangement information. It was not the intention of the Rules nor the Energy Designation Instrument for retailers to have an involvement in providing general product data. Origin does not believe that we should be required to be involved in the process nor take on the additional responsibilities and requirements that come with being assigned this role. This proposal by the DSB goes beyond the intention of the Rules.

[CDR-API-Stream]

In response to the feedback from @biza-io, @PratibhaOrigin and @perlboy:

The requirement to supply the base URI for public endpoints to the Registrar and to expose status and outages endpoints arise from the obligation to support a Consumer Data Request Service under the rules. This obligation applies to the three major energy retailers from the November date and not the October date.

As stated above and in the noting paper, a simple proxy of the PRD end points to allow them to be exposed from the base public URI (as required for the above obligation) does not constitute voluntary hosting and also does not mean that the retailer becomes responsible for the obligations inherent in providing a Product Data Request Service. The retailer is technically facilitating the traffic but is not supplying the data and is not taking on the role of data holder under the rules. This means, for instance, that there is no obligation to provide NFRs under get metrics for the PRD end points (although the obligation to report on the public admin end points remains).

The statement that proxying the PRD end points is equivalent to voluntary hosting is not the case. The DSB confirmed this understanding with stakeholders in the ACCC and Treasury prior to publishing the noting paper.

Statements have also been made indicating that it is technically difficult and costly to implement a simple pass through proxy for PRD end points. When the concept of AER and DELWP being designated for the PRD end points, as a way to reduce retailer implementation costs, was being consulted on the DSB asked whether a proxy model would be technically difficult and the feedback from retailers was that this would not be the case. This feedback aligned with the experience in the banking sector where similar patterns are used to support OSPs and also aligns with a priori expectations. If a retailer's specific technical context makes a proxy model difficult, however, we would appreciate understanding the specifics that makes this the case. This would help us formulate guidance to reduce impact as the current position of the standards is based on the feedback previously received.

It should also be noted that the patterns presented in the noting paper are intended to be helpful but not exhaustive or comprehensive. Any solution that meets the obligation that the base public URI exposes both the admin end points and the PRD end points would be sufficient. If we have missed a viable pattern that meets these outcomes we would happy to add it to the noting paper.

[perlboy]

Wow, just wow.

The requirement to supply the base URI for public endpoints to the Registrar and to expose status and outages endpoints arise from the obligation to support a Consumer Data Request Service under the rules. This obligation applies to the three major energy retailers from the November date and not the October date.

This appears to be conflating the definition of a Consumer Data Request Service and, if taken literally, would mean that the status & outages endpoint would comply with the rules associated with CDR eligibility (1.13(1)(a)(i)) and accredited persons (1.13(1)(b)(i)). On this basis these endpoints would be MTLS and authorisation bound as there is no other way of assessing eligibility and accreditation.

The outages & status endpoint have no data set classification and ultimately appear only captured by Division 8.4, 8.11(1)(h) involving "provision of administrative or ancillary services by CDR participants to facilitate the management and receipt of communications between CDR participants".

As stated above and in the noting paper, a simple proxy of the PRD end points to allow them to be exposed from the base public URI (as required for the above obligation) does not constitute voluntary hosting and also does not mean that the retailer becomes responsible for the obligations inherent in providing a Product Data Request Service.

I keep saying it but it's like trying to explain first year network engineering to someone who simply isn't interested in learning. This isn't a simple proxy but a back to back API which comes with it problems.

As a really simple demonstration (don't know why the DSB can't see this) if Retailer X implemented a reverse ("simple") proxy the net result would be that their endpoint would expose all retailers plans from the AER endpoint. If the retailer implemented a reverse proxy with the brand=Retailer X set in the query string they would be serving a non compliant endpoint if the requesting client chose to include their own brand field. Further if Retailer X sells things under multiple brands the Generic Tariffs specification does not support a list of brands.

I could go further but the constant trivialisation of this suggestion to a "simple proxy" is either the DSB being willing and demonstratively blind to the problems present or simple not having sufficient competence to grasp the technical concepts being put forward. Please stop making assertions that don't appear to be factually correct.

The retailer is technically facilitating the traffic but is not supplying the data and is not taking on the role of data holder under the rules. This means, for instance, that there is no obligation to provide NFRs under get metrics for the PRD end points (although the obligation to report on the public admin end points remains).

The rules define both product data and consumer data request services as an "online service". The DSB can twist things however it likes but it's quite obvious a judge during a determination will assess an "online service" as something on the internet.

The statement that proxying the PRD end points is equivalent to voluntary hosting is not the case. The DSB confirmed this understanding with stakeholders in the ACCC and Treasury prior to publishing the noting paper.

Then the DSB, the ACCC & Treasury are redefining the legal definition of an "online service" and this appears to be legally precarious in nature if applied to any other carriage service doing any other online activity.

Statements have also been made indicating that it is technically difficult and costly to implement a simple pass through proxy for PRD end points. When the concept of AER and DELWP being designated for the PRD end points, as a way to reduce retailer implementation costs, was being consulted on the DSB asked whether a proxy model would be technically difficult and the feedback from retailers was that this would not be the case.

Where was this stated by any retailer in any thread anywhere? Or is the DSB now going to refer to faceless "people" as the evidence to support this statement? If anything retailers appear to have raised issues with the specification on numerous occasions and were shutdown on the basis they were only intended to be delivered by AER & DELWP.

A search for the word proxy on the Standards repository mentions proxy zero times since 2019 except where the DSB also referred to the AEMO secondary data holder as a "proxy" (which it later admitted it wasn't). Whatever narrative the DSB wishes to believe seems to be one of fiction:

This feedback aligned with the experience in the banking sector where similar patterns are used to support OSPs and also aligns with a priori expectations.

Once again, another conflation. The banking sector doesn't do a proxy model yet this is used as a factoid for justification. For clarity an OSP (like Biza.io) doesn't use a similar pattern because doing so would be akin to a DDoS protection service for which no service alteration is expected. It would appear the DSB fundamentally doesn't understand how OSPs are delivering services in CDR (or maybe it simply doesn't care).

If a retailer's specific technical context makes a proxy model difficult, however, we would appreciate understanding the specifics that makes this the case. This would help us formulate guidance to reduce impact as the current position of the standards is based on the feedback previously received.

This isn't a decision proposal so is not a consultation, all I'll do is repaste what Biza.io already stated: "A prudent action would be for the DSB to reopen consultation on these endpoints and reevaluate prior feedback in this new context if it intends to continue to suggest these endpoints must be functional at the Retailers published PublicBaseUri. Should the DSB choose to do this Biza.io will proceed to produce the change requests it believes are necessary to support this operating topology."

It seems peculiar that the DSB now expects the Regulated party to solve for a clear technical deficiency in its design.

It should also be noted that the patterns presented in the noting paper are intended to be helpful but not exhaustive or comprehensive. Any solution that meets the obligation that the base public URI exposes both the admin end points and the PRD end points would be sufficient. If we have missed a viable pattern that meets these outcomes we would happy to add it to the noting paper.

To be clear, the onus is on the DSB and the Chair to provide a functional data standard to comply with. In this matter it has failed to do so and as it continues to persist with it's apparent commitment to a forced implementation without rectifying the concerns raised this conversation is now bordering on the Chair failing to have consulted effectively on this topic.

[agl-cdrprogram]

The PRD noting paper 248 is in contradiction with rules & energy designation instrument. It has significantly expanded the scope of what is required to be done by data holder under the CDR Rules by including the facilitation of a product data request service. AGL’s position is that there is no legislative requirement in relation for data holders to provide a product data request service. AGL also does not consider the ‘Noting paper’ to be the appropriate vehicle to effectively make changes to requirements under the Rules.

Alternative Option Proposed:
Noting Paper 248 is a technical workaround addressing a capability gap that could be addressed through an enhancement within Registrar. It would more effective and economical for the registrar platform to support multiple base-URIs instead of asking multiple retailers to do short term and long-term options.

Please find attached more supporting information for reference.

PRD Changes & Impacts.pptx

[CDR-API-Stream]

The Treasury have confirmed the following regarding this thread:

• As previously stated the facilitation of PRD endpoints is not equivalent to self hosting or the implementation of a Product Data Request Service
• The requirement under the standards that a single PublicBaseUri be used to access both PRD and Admin endpoints is not clearly enforceable under the rules for the initial retailers on the 15th November and is therefore not mandatory
• Voluntary implementation by Retailers is possible and would be welcome
• The Treasury rules team will seek to amend the rules in coming rules development cycles to ensure clarity of enforceability for this issue

[biza-io]

Biza.io thanks the DSB for clarifying this is not required as part of the current Rules setting. We look forward to collaborating with Treasury on how to technically and reasonably enable this service in the future.

[Sbendat]

Hi everyone,

Excuse me for not reading every word of what is an extremely lengthy thread regarding Noting Paper 248, but I think I have already run into issues regarding this noting paper and trying to access custom PublicBaseUris for unauthenticated endpoints. Both the Origin and AGL PublicBaseUris no longer follow the same format prior to Nov 8th, and consequentially, the GET requests made using with these new PublicBaseUris are sending a 404 error; previously using the old format, they worked without a problem.

Does it seem like this is all related?

It would be great to have a standardised PublicBaseUri path for these endpoints like almost every retailer currently has. It makes data retrieval exponentially easier for data recipients and will minimise maintenance costs as more retailers join the Consumer Data Request Service.

From a data recipient standpoint having a standardisation to the PublicBaseUri is most preferable. More standardisation and no changes to essential endpoint paths. Trying to work with endpoints that keep changing the PublicBaseUri is pretty frustrating.

[perlboy]

Hi @RubberDucky73,

The Origin and AGL PublicBaseUri points at their own endpoints for the purposes of disclosing Status & Outages. This will be something which will repeat as more retailers come online as it is impossible to not implement Generic Plans and provide these mandated endpoints. I highlighted this in my April response:

PublicBaseUri is used for PRD information and the Status/Outages endpoints. This is noted briefly in Pattern 2 but is, in fact, a global requirement regardless of technical solution. There is potential for more flexibility on this requirement if the Register was to bifurcate PublicBaseURI into two separate base uris (PRD and Status/Outages)

The new uris don't return 404 for the endpoints mandated on Holders (Status & Outages).

As a standalone question, most likely separated from this thread, there is an ambiguity that is present as to whether the Generic Plans paths are actually in scope for the Standards because they aren't mandated. The implication here is that this would imply that other completely unrelated endpoints (like Banking endpoints for Energy providers, or Telco for Energy & Banking) should now also provide a Standards defined response. If this was the case it would result in a global CDR wide changes being required whenever any endpoint is added which seems counterintuitive.

Once again, repeating the suggestion from April, the cleanest solution here if the government is so inclined is to bifurcate (ie. separate) the endpoints advertised for PRD/Plans from Status/Outages.

[Sbendat]

HI @perlboy, thanks for the quick reply. Yeah, I can't talk to the Status and Outages endpoints as I haven't used those yet. My issues are primarily with the PRD/plans endpoints. Though from my perspective of working with multiple endpoints, the more that the Uri paths are standardised and follow the same formatting, the easier my work has been up until this point. Apologies for the detour!

[jxeeno]

Hi all,

I've created a CR in standards maintenance to discuss separation of PRD and status/outage endpoints in the Register API.

ConsumerDataStandardsAustralia/standards-maintenance#561