WattTime API integration and other data sources discussion

[Willmish]

The WattTime API can be used as one of the sources for determining carbon index of different regions. We will also need to later add support for other data-sources - e.g. Electricity map.

To use it with the CarbonAware SDK, we need to:

• Add a C# interface, interacting with the API, which can be used by the SDK
• Add a basic mapping of Azure region names to WattTime region names. (some Azure regions may have datacenters in multiple WattTime regions, need to decide how we will handle that)
• Add a more detailed mapping of: Azure, AWS, IBM and other cloud provider regions.
• Integrate the WattTime C# interface, along with the mappings into the CarbonAware SDK.
• Make sure it can be accessible via the WebAPI interface of the CarbonAware SDK.
• Add tests for the interface.

C# WattTime API development workflow:

• Use a secure way of storing the generated API key, e.g. using dotEnv files (consider using/allowing other options later (e.g. Azure KeyVault integration)
• Create a basic CLI interface for requesting carbon index for specified regions and at specified times
• Rewrite the interface to make it easily expandable/usable from the main SDK (add responses to the current default requests - get emission data by location (by time), get emission data by locations (by time), get best emission data by locations (by time).
• Add demo mappings for Azure region names to WattTime names (e.g. focus just on westus, eastus, southuk/uk/ukwest)
• Link the interface to be used with the WebAPI of the SDK (and "forward" the data from WattTime as responses)

[yuu1127]

@Willmish
I'm interested in your proposal.
For mapping the region between WattTime and cloud providers, I think it is better to use identifiers with the layer structure such as namespace in C++ (e.g. azure::us::es , aws::us::ws). This could help users to call queries easier.

We are also thinking it is necessary to determine the interface first -
we've also posted #55 and #57 to discuss extending the project scope of the SDK.
I think we can work together in this area because ideally the SDK should support collecting I of SCI in various environments.
(We aim to use the SDK on-premise)

[Willmish]

Hi @yuu1127,
Great to see some interest!
I saw your message regarding the new discussion on slack, haven't gotten around reading the proposal yet.

I will not be available this week, but I am planning to set up a meeting revolving around WattTime API and this issue in particular next week on Tuesday 17.05 at 10:30 BST. This will be take place on slack (more info there, on the #proj-carbon-aware-sdk channel)

Regarding the identifiers, it does seem like a reasonable mapping. Do you suggest this replaces the default names cloud providers use for their regions? (so azure:uk:south or azure:uk:s instead of southuk)

If so, only thing I am worried about is that this introduces another format to represent those regions, apart from the ones defined by WattTime, and the cloud providers. (It does unify them in a sense, but does create new abbreviation format developers need to follow).

Maybe instead use another parameter to indicate cloud provider and use the abbreviations for those regions provided by that cloud platform?

E.g.:

• Azure region southuk could be represented as a tuple: (azure, southuk)
• AWS region eu-west-2 (which is located in the south of UK), could be represented as a tuple (aws, eu-west-2)

Or merge the two ideas and have a layered structure (like you suggested), which uses the original names of the regions, given by the cloud providers?

The reason why (I think) sticking to the names for the regions given by the cloud providers is crucial, is because the developers who work with those tool (E.g. AWS devs), will most likely know the regions they are interested in by heart, and won't have to go and look through another lookup to see which region has what mapping on the SDK. Hope that makes sense.

[yuu1127]

Hi @Willmish,

yes for identifier it's better to use the name with extensibility for future usage.
I understand developers may want to use the regional name for each cloud that they use.
So we need to develop plugins for this.

Another concern is whether we need to make mapping for the cloud between Watttime.
As far as I know, cloud vendor has their own power supply system.
( For GCP they say 100 % renewable energy is used https://cloud.google.com/sustainability)
Watttime seems to take carbon intensity value (I score for SCI) from Balancing Authority(BA).
I feel it does not so make sense to apply this value to the cloud since cloud provider might have their own I value.
What is your opinion?

but I am planning to set up a meeting revolving around WattTime API and this issue in particular next week on Tuesday 17.05 at 10:30 BST

great !! I will attend this if I can make it

[luwangshell]

Hi, I have done some work with WattTime api. def love to contribute here. :)

[danuw]

Sorry to be late the party but exactly what we were discussing with @Willmish about watttime seeming too generic and seeking more accurate values (from cloud provider at least or other source?) so the I is more accurate.
Question was also around what else can be used to make this even more accurate?

Anyway please count me in too :)

[yuu1127]

I agree, it seems there are not many choices for sources of "I"(carbon intensity) as far as I know.
So it's a good starting point to develop Watttime API first (and electricity map maybe later on ?).

the I is more accurate.

Yes if it's possible to get "I" sources from Datacenter (also cloud provider),
we can develop an adapter to connect
the device and SDK to store data related to "I".

[Willmish]

Converted to discussion, as there is some existing WattTIme API support, but there are some interesting ideas here worth keeping.

[Willmish]

We should also consider other local APIs as data sources -> e.g. for UK's grid: https://carbonintensity.org.uk/
(Similarly to Electricitymap integration added by @yuu1127)

Maybe create a collection here or in separate discussion of APIs/data sources worth looking into?