SCI feedback from greeNsort

[greeNsort]

Please find here a detailed suggestion for pragmatic enhancement of the SCI. The core point I make is that the SCI - although formally correct - requires information that is not available to software developers (which hardware is used for how long etc.) and often is not relevant for the 'greenness' of software (local energy/CO2 ratio). I show pragmatic alternatives that work for developers and that have been proven useful in the greeNsort project. If you like, I happily explain my view face-to-face in a video-session. Looking forward to talking about this and integrating it into the GSF SCI.
Best
Jens

[atg-abhishek]

Thank you for this very detailed submission @greeNsort - we will review it in the upcoming meeting and perhaps @seanmcilroy29 once we have done that we could invite @greeNsort to share their findings as an invited contributor if they are not already a part of the GSF?

[Henry-WattTime]

@greeNsort It would be great to have you join one of our meetings. We'll reach out to coordinate.

[ElmarBorgmeier]

Jens, your input is well thought and I think it's very useful.
You focus on an specific case: analyzing sorting algorithms. This is what the current projects of the standards group are about: Applying the SCI concept to different cases and see how it turns out in those cases. Feedback from your case is one among others, such a an online shop and an IoT scenario.

There are some aspects about your case that make it special. I think that

• some of your conclusions stem from this specific point of view.
• some of your observations relate to your specific goal for the SCI, what it should express and what it can be used for.

Your viewpoints are absolutely valid in my opinion and we should take them very seriously. At the same time, we want to include other scenarios, so we need to be aware that some aspects of your conclusions might not be suitable from a more general perspective.

Here comes more specific feedback from me. (This is my viewpoint only, just an input for further discussion.)

Measuring Units: I do think that we haven't reached a final understanding of the usage-units R. Some aspects here:

• Embodied emissions are related to time, so including a time factor into R seems reasonable, possibly eben necessary.
• On the other hand, embodied emissions relate to a specific set of hardware. So it seems reasonaly to use a unit that describes what this set of hardware can handle and when we need to scale out. The typical scenario: A server that handles R users well, but when the number of users increases beyond R, a second server is needed. In this case, R = "number of users" seems best, because this defines the range to which M is valid. (The same argument might also lead to definig R as "number of concurrent sessions" oder "concurrent transactions", depending on the software system.)
• Also, "per user" will probably be the most useful unit in many cases. As a user, it gives me an indication what it means if (or when) I use this service. It helps me to make decisions.

Measuring Energy
There are practical issues measuring energy in realistic scenarios, I agree. I guess it will be an ongoing discussion in the community how to do this best. But I don't think this as an issue for the SCI definition.
You're testing a single algorithm. That kinda leads to a view like a physics scientist studying single particles. In other cases I will want to measure entire software systems or services instead. This is more like measuring gas behaviour scientifically: You can ignore the behaviour of individual particles because you analyze the lot of them. Both are valid scientific approaches. If I, as a user, want to get a feeling for: "What is the climate impact of a typical google search?", I am not interested in a isolated value for a search algirothm. Instead, I refer to the entire environment. This leads to different types of measurements. Simpler ones, in a way, because there are no isolation issues.

Location-Based carbon emissions
This is not in your text, but I loved your argument in the disussion: "We should calculate the CO2/kwh value using the marginal CO2-input into the grid. (=the worst input)" Because optimization is probably the only thing that actually reduces energy usage. It will not reduce the amount of wind/solar/water power available. Instead, it will reduce the amount of extra energy needed, which comes from burning fossile fuels. So the actual climate impact is not limited to local average values for CO2/kwh. It ist beyond average.

I agree that we should discuss your questions. (Discuss them again, it's not as if we haven't been done that a few times.)

I would like to add something I brought up in a different discussion: There a two points of view here, depending on what you want to achieve with the SCI:

• Software does not have emissions all by itself. As the SCI spec says, it induces them via the hardware. So, if I am solely interested in describing the software, the SCI schould take the hardware out of the equation as much as possible. Which can be achieved by defining constants for I and for the different pieces of hardware included in M. This "generic" version of SCI is useful for me when I decide which software to buy and run on my own hardware.
• Software systems do have emissions. (Systems meaning: hardware included.) There are cases when I am more interested in the impact of the system. Like: When I want to decide between to cloud based services. In this case, the hardware is known and it seems ridiculous not to take it into account. (The SCI currently refers to a software system. Which seems reasonable since it sort of expresses a carbon value which cannot be calculated without a set of hardware.))

Sorry, I need to attend a meeting now. Hopefully, I will have time later or tomorrow to add more feedback.

[atg-abhishek]

Thank you for the very detailed and well thought our response @ElmarBorgmeier - let's discuss this further in the call today.

[greeNsort]

Thank you Elmar,
here are my preliminary thoughts:

Measuring Units

My most fundamental approach of relating cost to /R would be relating cost to value. Operationalizing a value-based /R is hard. But even if we operationalize something that has value shaved-off, we should still remember the ultimate goal: creating maximum value at minimum cost. Now, let's take an example: cost per transaction. Not all transactions are alike. Let's start with a clearly valuable one: an emergency-call or its digital-transactional equivalent, when for example an autonomous car reports an accident. For such a transaction being reliable we should we willing to accept a higher CO2 price than for a trivial transaction (say a centralized bank-transfer) or even a harmful transaction (say a criminal transaction in a anonymous Ponzi scheme). Yes, I am talking about bitcoin. What I say is, that the same gCO2 which might be acceptable for a normal banking transaction should not be acceptable for a bitcoin transaction. Now the situation with bitcoin is even worse: I made the calculation and compared the cost of a bitcoin transaction to the cost of a reasonably redundant Oracle Exadata transaction (a system I know very well). To my own surprise, the bitcoin transaction is by a factor in the magnitude of 1e11 (eleven zeros) more expensive than a traditional DB-transaction. This is an easy case: either way, via value or cost, we come to the same conclusion.

Aside of value, defining /R should respect certain rules. One is the structure of energy. 'Energy' is defined as power over time. Note that 'Work' has a very related structure: it is defined as power over distance. I suggest that /R represents a certain piece of work. Doing maximum work at minimum cost is a reasonable criterion. Some 'pieces' of work have clear boundaries, such as a transaction or writing a 2-page letter. Other work might not have clear boundaries in a technical definition. Say streaming a video. For streaming a video we need 'time' in /R. The cost of streaming an hour of video. However, remembering the value: better the cost of streaming a film than of streaming an hour. We should be careful in not extending the use-case of putting time in /R too much. What about an hour of Twitter? Just technically minimizing the cost of an hour of Twitter is neglecting the task that society rightfully expect from us: minimizing the cost of something meaningful. Hence: Twitter for which purpose? Take one possible purpose: getting up-to-date on the latest news. Our aspiration as responsible software producers should be to deliver maximum (unbiased) latest news at minimum cost. Maybe the answer is something completely new and different from Twitter. May be the answer is even something technically boring: information filtering and aggregation by traditional press. Expect future politicians to make meaningful comparisons: what is the purpose and CO2-cost of some relavant SaaS-service, what is the the CO2 cost of a technically completely different alternative serving the same purpose, and then expect regulation. If we define /R purely technically (hour of twitter) we fail to do something useful, and as an industry consortium we then have to reckon with the accusation of preventing relevant comparisons (aka greenwashing).

Measuring Energy

I like your example of measuring gas. It relates to my idea of defining R in a meaningful way, not just technically. I agree that the difficulty of measuring energy should not make us drop a proper definition of SCI. But I stress again, that a definition of SCI that is hard to use in practice simply would not be used. Therefore we should either provide tools that make measurement easy, or provide proxy alternatives. Inspired by our discussion in the Meeting, I tried again to come up with a better measuring process than the one using perf in a separate process. My second attempt with PAPI, a couple of years after the first failed also: PAPI has a low-level API that I find too complicated, and it has a high-level API that looked promising: enclose a critical code section with PAPI_hl_region_begin and PAPI_hl_region_end, compile, and then according to need instruct the API via environment variables what to measure. Result is one or many JSON files. That's the theory. Here what I experienced. First installing PAPI packages under Ubuntu and trying to use it failed: outdated version. OK, downloading, configuring and making a new version. Still no Energy measurements. OK, PAPI is a facade hiding very different techniques of Energy measurement that need to be compiled in, most suitable RAPL and POWERCAP. It turns out, RAPL counters overflow so quickly (< 1 sec) that the high-level API is useless. With RAPL, one would need to interrupt the running algorithm each fraction of a second using the low-level API, doable with a loop, undesirable in a recursive implementation. OK, then powercap. The counters do not overflow as quickly, but there is a bug in PAPI giving nonsense values (on my CPU), although the powercap counters look good. Frustrating. I ended up with writing my own interface for reading from the linux powercap kernel counters directly. Unfortunately this is platform specific, specific for one-socket machines, even CPU-specific and fails for my second machine with a different CPU. OK, now I extended my interface that it handles two different Intel CPUs. Anyhow, I am happy now to have a measurement method that is much better for my specific purpose. I even have a calibration method allowing adjustment for background processes. This was time-consuming and did not lead to a generic solution. It probably fails as soon as a want to measure an AMD. Now don't forget that I was extremely motivated to get this done. But we want people to optimize their code who have little time and maybe also less motivation.

Location-Based carbon emissions

Thanks for enjoying the 'marginal' thought. I basically agree on the two point of views:

• selecting the better software for the purpose: I look forward to publication of GSF standard factors for converting Energy to CO2 and will happiliy integrate those in my testbed. I still believe that the Footprint-measures have value in themselves. Just measuring Energy and RAM is more abstract and closer to software as such than measuring CO2. What if we want to measure and optimize for methane? Or for rare-earth elements? Or simply for cost in dollars? Just optimizing algorithms for runtime performance and hardware parsimony is more generic.
• selecting the better SaaS: of course I want to compare end-to-end money, methane, and yes particularly CO2.

Kind regards
Jens

[greeNsort]

One more remark: I agree that the greeNsort case is special in focusing on algorithms for standard libraries, not software. Hence I should have better used the adjective 'sustainable' than 'generic'. Developing algorithms is laying foundations for decades, we hardly can foresee which on which hardware they will run. Hence sustainability implies focusing on basic stuff like fewer operations, better locality, less RAM. Optimizing for specific CPUs - say exploiting specific instructions - still makes sense in software implementations but creates artifacts that are not resilient to changes in hardware, hence are not sustainable over long time horizons. Reusable software-components and libraries are some middle-ground between algorithms and software, regarding half-life we have: software < components < algorithms. I think the end-to-end-CO2 approach is most suitable to software and least to (fundamental) algorithms.

[greeNsort]

Was my pleasure. Unfortunately I was today busy in calls of my regular job. Please let me know in advance if you want me to participate again. Kind regards     Gesendet: Donnerstag, 05. Mai 2022 um 14:18 Uhr Von: "Abhishek Gupta" ***@***.***> An: "Green-Software-Foundation/software_carbon_intensity" ***@***.***> Cc: "greeNsort" ***@***.***>, "Mention" ***@***.***> Betreff: Re: [Green-Software-Foundation/software_carbon_intensity] SCI feedback from greeNsort (Discussion #248)   Thank you for the very detailed and well thought our response @ElmarBorgmeier[https://github.com/ElmarBorgmeier] - let's discuss this further in the call today. — Reply to this email directly, view it on GitHub[#248 (reply in thread)], or unsubscribe[https://github.com/notifications/unsubscribe-auth/ASKZNGXZUVJ5AOWF47QBVW3VIO4CLANCNFSM5TIKI42A]. You are receiving this because you were mentioned.Message ID: ***@***.***>

[atg-abhishek]

@seanmcilroy29 will send out a recording of the WG call with this discussion and a link to the presentation for the broader SWG to review and then we will have a detailed discussion on the points that were raised.

[atg-abhishek]

Hey @seanmcilroy29 just checking if we had done this so that we can include on the agenda for the meeting this week? No worries if not, we can include it for next week's meeting. Thanks!

[seanmcilroy29]

@atg-abhishek - I've checked and it looks like the recording was deleted as the 7 days had elapsed.

[seanmcilroy29]

@atg-abhishek - I've retrieved the recording - link available via Standards WG GSF G-Drive doc

[atg-abhishek]

Thanks @seanmcilroy29 !

[Henry-WattTime]

WG: This conversation is more focused on algorithms. The SCI is more focused on measuring the impact of comprehensive software.
If you are sure you don't have control over some of the other non-algorithm specific hardware, use consistent assumptions/values across those variables. This is especially true when comparing different algorithms.

Challenge: many developers don't know things about their software (for example, what hardware the future software will run on). Does the WG have a position on what should be done in that situation.
-WG could provide what those default assumptions are
-Provide a range of potential options (onsite, hyperscale data center, local desktop/laptop)
-allow to make reasonable assumption about energy consumption and embodied carbon
-could be load tested hardware or a reasonable global assumption, or based on targeted client. In all cases, must remain the same between situations.
-Request: some guardrails that begin to answer these questions. Can we request from Open Data team?