Real-Time-Carbon-Footprint-Proposal

[seanmcilroy29]

From Adrian Cockcroft:
I have been working on something I'd like to share and discuss at the meeting this week. I've got a proposal for a Real-Time energy and carbon API standard that would hopefully provide the information needed to do SCI and other work in the cloud. I'm presenting at QCon London on March 29th and will share it publicly there. I'm hoping to get the cloud providers to at least consider it as a future direction to work towards, and it would appear as something like CloudWatch metrics from AWS or as a Prometheus Exporter. Summary of the data that is currently available, and the data we'd like to be available from the three leading cloud providers, is in the attached doc. I've shared it with MSFT, AWS and GCP folk in the last few days. This is in no way intended to be a finished standard proposal, and there is no buy-in from anyone yet, it's an initial SWAG at what I think is needed, and which could be an end point to work towards over the next few years.

Real-Time-Carbon-Footprint-Proposal - https://drive.google.com/file/d/1vGiZM6R8ZkXxR6M7ELFqI0cAJM9JKP_f/view?usp=share_link

[adrianco]

Thanks for the discussion today. I need to do a longer writeup of the idea than is on the slides, but I think my starting point would be a Prometheus Exporter configured to generate a proposed schema, with simulated data sources to start with, and a configurable model that included whatever data is available. Someone mentioned a possibly similar energy exporter might already exist.

[Sealjay]

For the data format, I strongly think we should extend existing projects like the lf energy specification; otherwise again we will end up with competing specifications. That’s where the convention work landed.
There are many tools monitoring carbon already, see this issue again:
Green-Software-Foundation/opensource-wg#60

We could ask teams from those tools to work with you on this.
@Henry-WattTime @jawache @adrianco

[Sealjay]

Bumping this again, prior to the SC call. I don't think we need the Green-Software-Foundation/opensource-wg#60 project if we also have a real time carbon footprint project.

[KnowledgeRatio]

Reviewed the material and agree that pursuing both would be a near complete overlap of work and standards. I'm happy to defer to @adrianco's initiative given the analysis that is already present. That said, I would stress the need to look at other open specifications that offer this, and particularly LFEnergy:

https://github.com/carbon-data-specification
http://carbondataspec.org/

[Sealjay]

It may be one of the shortest GSF projects in our (also short) history- but have tabled a discussion in today's OSS WG to close the convention project:
Green-Software-Foundation/opensource-wg#83

@KnowledgeRatio if you can't make it, could you add a comment please?

[KnowledgeRatio]

I would support closing the CDIF project because I feel that there is more progress and also a larger outcome (real-time) that is to be achieved by the other project. I offer to support/contribute to the RTCF project when that is off the ground. What I would like to see from RTCF is to go beyond the CSP services and take a bigger picture view of carbon data/impact.

I will also attach the requirements I was able to draft and prioritise which could be useful for posterity and the superseding project.

[plnoel]

Thank you for formalizing and structuring this proposal. Totally supporting this initiative, we are definitely suffering from the lack of insight on the energy use of the various services we use with the cloud providers to optimize our footprint.

[jawache]

I think one interesting characteristic of this standard is that it's a standard for monitoring carbon emissions, i think that should be something that clearly surfaces from this proposal, the SCI can be used for monitoring but not necessary, this is primarily focused on monitoring.

So reducing the distance between someone making a change to a software system and then seeing the impact of that change, it's could have the same impact on the sustainability industry as moving from say a yearly release waterfall model to daily releases (CI/CD) did to the software industry. Instead of a figure every year and then trying to figure out what changes drove that number, if the figure comes every minute you'll be able to correlate changes to the system to changes in carbon emissions, which is a much stronger driver of behavioral change than a yearly regulatory report.

[Henry-WattTime]

@Oleg-Zhymolokhov @srini1978 @navveenb please share this with the Carbon QL team for their review and potential collaboration.

[adrianco]

Update: I took off on some vacation time so haven't made much progress. My plan is to contact the cloud providers and get their reactions. Also Kubecon was last week, and now that's over I will start a discussion with the Kepler project and other CNCF working groups that are relevant. I had a meeting with AWS scheduled yesterday but it was cancelled as they were dealing with the layoffs that dropped (my contact is still there).

[adrianco]

There's some relevant work on the Kepler discussion thread I started. Vincent Caldera from Red Hat has built a real time carbon monitoring dashboard for an OS-Climate Kubernetes cluster (running the OpenShift based OSC Data Commons), and shows the results here: https://www.youtube.com/watch?v=dd-6rqp_qlA

[adrianco]

I've made some contacts with AWS and Microsoft people so far, and am not in a hurry, it's going to take time to get everyone aligned around something useful. My next step is to flesh out my proposal in a bit more detail and to convene a discussion at GSF about the right place/process to take this forward. I'm also thinking about writing an Amazon PRFAQ form of the proposal - which would iterate as we get more feedback, but help with engaging the relevant AWS people.

[adrianco]

I've written this up as a PR-FAQ - have passed it by some people at AWS, Azure and GCP, but not had substantive feedback yet. Positive feedback from Harness.io and CloudZero. Green-Software-Foundation/pr-faqs#10