Telemetry Based and Benchmark Based SCI Calculations

[jawache]

In the SCI the conversation currently revolves a lot around the idea that you can measure the total carbon emissions for an application. For a lot of applications, esp. open source applications, this is not possible and we should be more inclusive and start talking about an alternative method of calculating the SCI.

For example if you are the maintainer of an open source library you have very little idea of where you library is used, the hardware, energy consumption, the carbon intensity. It would be impossible for you to be able to calculate a total carbon emissions. For those projects their only option for calculating the SCI is via a benchmark, a program which puts the application through a consistent set of use cases, a consistent load and then calcultes teh result. There are plenty of examples of performance benchmarks out there e.g. https://github.com/python/pyperformance or https://github.com/tensorflow/benchmarks/tree/master/perfzero and they measure the E, M and some thought needs to be put into I for these use cases.

Reducing the problem space a little bit.

There are two broad categories for applications, those that run in the cloud (>1 machine) those that run on one device (=1 machine).

There are also two methods of calculation, via telemetry (so you can calculate the total) or via benchmark.

+---------+--------------------+---------------------+
|         |     Telemetry      |      Benchmark      |
+---------+--------------------+---------------------+
| Cloud   |  Total / Baseline  |  Total of Benchmark |
| Device  |  Total / Baseline  |  Total of Benchmark |
+---------+--------------------+---------------------+

For example, if you are tensorflow you may choose perfzero as your benchmark.

• You run perfzero and calculate SCI as (E * I) + M using the numbers you have directly measured from running that benchmark.
• The SCI might then be reported as X per perfzero, with the name of the benchmark as the baseline.

If for example you are an AI Cloud SaaS then you would calculating using telemetry, like so:

• T = (E * I) + M
• Then you decide your baseline, if it's number of users then you would calculate it as
• SCI = T / number of users which gives you something like SCI = X per user.

[atg-abhishek]

This is a good point @jawache and I imagine that there might be some of our use cases that can demonstrate this application a bit more concretely. Which section in the SCI spec do you see this being included/mentioned?

[atg-abhishek]

This is a good point @jawache and I imagine that there might be some of our use cases that can demonstrate this application a bit more concretely. Which section in the SCI spec do you see this being included/mentioned?

[atg-abhishek]

@jawache I've added this discussion on the agenda for this week. If there are any other details that you'd like to add in here before the meeting this week, please feel free to do so :)

[srini1978]

Love the idea of coming up with benchmarks. . If we are going the benchmark route, as @atg-abhishek has called out, we need to host a few internal applications and benchmark them for E,I and M values and report it . I am not very sure if it would be a specification thingy as this is more of an example of the application of the standard than the standard itself.

[atg-abhishek]

Thanks @srini1978, we actually have an ongoing project that I'm leading called the GCC which might be a good avenue for bringing some of these ideas. A notice about that was sent out in the newsletter by @jawache

[jawache]

Thanks @atg-abhishek and @srini1978. I think the choice of benchmark should be left to each product. There is so much variability and context there. We can perhaps define what the requirements of a benchmark are, e.g. "the benchmark MUST be representative of real world production use of the application".

In terms of where in the spec, I would say this is a description of the "methodology" of the specification, so in that section just above

"These are used to calculate total carbon emissions (C) and carbon intensity (CI):"

Since then we will be describing two methods, one to calculate an intensity by first calculating a total through telemetry and another to calculate an intensity directly by using a benchmark.