Research project on Sustainable Computing

[seanmcilroy29]

From Chris Xie (Huawei)
We are intending to initiate a research project on Sustainable Computing topic, in particular, how programming languages (Rust, C, Java, Python etc.) affect the carbon efficiencies of the full computing stack, and translate the efficiencies into economic impacts in dollar terms, using the SCI methodology as a tool for such studies. Effectively this project could be a case study or an application of SCI and intends to apply the SCI methodology to the Social Cost of Carbon (SCC) study of programming languages and software systems. Apply meaning to SCI scores is not limited to dollar terms, but it could also be carbon efficiency rankings of comparable computing systems, ICT solutions, 5G base stations, etc. Please see the attached document for details. Please feel free to ping me for further discussions, ideas, collaborations. - SoftwareSCC_GSF.pdf

[Henry-WattTime]

Henry: good luck identifying a social cost of carbon
Tammy: efficiency of software isn't the primary driver, it is how the underlying hardware interprets the software
w/ Srini studied different languages doing similar tasks. Is is software? Hardware? How to quantify?
Ben: If interpreter is the same, shouldn't the software be the same? What is the scale of the difference? Marginal gains or significant? Lots of existing research around this.
New research would be as it relates to SCI
GSF position? Does it matter or not matter? It's a tool in the toolbelt, so it could have an effect?
To switch, need a reason/ROI. Value/social cost could be a way to tie it back to value.
Gadhu: everything else remaining the same, how much does changing the language have an effect, what's the scale? Is this worth investing in?
Cost of the work vs overall gain in total emissions reduction
value could be combined cost and overall value.
(henry example, lots of little apps vs one large app
Srini: Performance efficiency discussion tend to be theoretical. Study comparison of languages.
Motivation: Choices related to performance requirements and key requirements. Don't optimize unless required to. Emissions may be a new incentive to care about performance.
Good for greenfield development, emissions could be a factor
Inherent efficiency of language, and quality of the actual code.

Question for Asim? Python is ~60% less efficient than java, implies switching, but could fix Python
Marcos: can we work with the open source community to increase the efficiency of software? Open area to explore
-when in the thick of development, can forget efficiency, efficiency in the language side helps

Navveen: wrote application in nodejs and rust, keep stuff constant, rust was very efficient (~5-10%).
hardware/users/etc the same
tradeoffs, limited libraries and APIs, new language
consider tradeoffs
case study still internal, calculated SCI score

RESOURCES:
Rust comparison: https://aws.amazon.com/blogs/opensource/sustainability-with-rust/
Energy efficiency of Languages: https://greenlab.di.uminho.pt/wp-content/uploads/2017/09/paperSLE.pdf
Ziliang at Texas State has done some research into this
Social Cost of Carbon: https://www.rff.org/publications/journal-articles/comprehensive-evidence-implies-a-higher-social-cost-of-co2/

Core question:
Same application with SCI score, ported to different language, what is impact on score (operational/energy emissions only, no change in hardware and embodied emissions)?

[Henry-WattTime]

WG:
Navveen: We had discussed creating a reference application earlier for SCI and comparing languages, with other factors being the same. Trade-offs on computing, skills, and expertise can provide cost angle and simulation. Benchmarks would be useful in this exercise
Ciril: +1 with Navveen. There seem to be two steps: 1. Comparing Languages 2. Assessing Pricing of the associated Carbon

Gadhu: What will be really useful is just some detailed benchmarking of various aspects of the stack - language, pattern etc for achieving a particular function so that you can compare the carbon cost of using these aspects to design your software system.
Help enable a decision about what language to use. When making a carbon decision, low level aspects. Create benchmarks for different architectures. SCI will depend on many factors, so in this case keep everything but the language the same between comparison (same hardware, location, time). Then you can know which language is more carbon efficient. Then you can optimize other parameters (hardware, switches, etc). This would allow a baseline comparison. Unlikely that this research would be done in a corporate setting, good for academic study. See Ziliang' study that compares ML models does this for model comparisons.

Pindy: comparing languages - could we do a longitudinal study compare the same solutions using different languages - can the professor create simulations

[Henry-WattTime]

update in August

[chrisxie-fw]

During today's discussion (5/25/2023), I shared an idea and an ultimate vision that someday, major github open source projects will have an SCI score or SSC (Social Cost of Carbon dollars sign) along side the number of forks and likes, or some type of carbon efficiency rating of the software, in absolute terms or relative terms, or some sort of carbon efficiency certification. To achieve that, one of the approaches is to define a standard/specification of a controlled environment (hardware, software, versions of SCI spec, locale etc) where such rating/evaluation of SCI/SCC is produced. This way, anyone can take such an environmental specification and produce an SCI/SSC dollars of their own software systems. The produced SCI/SCC scores will be compliant and certified by GSF. Of course, it will make sense for GSF as an organization to build an initial reference implementation of the controlled environment, and ideally the compliant environment built by 3rd parties should produce similar SCI/SCC scores results. This could also be an extension of the sustainable computing research project. This idea is a bit like UL Labs that measures product safety. In this cause, software product or systems are measured for its carbon footprint, in CO2's or in SSC dollars. The point of this idea is to translate SCI scores to real life meaning, real life value, in terms of dollar amount, or understandable certification or meaning of carbon efficiency of software systems. Again, the initial scope of the standard is to define the parameters of the controlled testing environment in order to create the SCI/SCC scores, so that anyone can take the standard specification to build their own testing lab environment to test their software to create their own SCI/SCC score, and GSF could issue the certificate that their score meets or complies with GSF's standard of certification. GSF still maintains the authority of the certificate and the evolution of the SCI/SCC spec, and still is responsible for the definition of the SCI, SCC spec, the testing environment spec, and the future iterations of these specifications. If this vision is achieved, SCI will have a really meaningful impact to the whole software industry. The value of SCI standard will be truly realized. Look forward to continued brainstorming of ideas of translating SCI to real life meanings and applications.

[Henry-WattTime]

WG: recommends merging with SCER

[ArneTR]

Hey @chrisxie-fw ,

we have just submitted an article that describes an automated open source approach to measure the SCI. Is that something that fits what you are suggesting?

Green-Software-Foundation/sci-guide#52

@Henry-WattTime
We also saw the project from the GSF to have the SCER for many applications. We have done something similar in the past by using for instance TPC Benchmarks for databases as standardized usage scenario and then running automated SCI benchmarks against it.

See an example for a TPC Select benchmark here for:

• PostgreSQL: https://metrics.green-coding.berlin/stats.html?id=69958765-14bb-44c0-90cd-8ef084c0ecdd
• MariaDB: https://metrics.green-coding.berlin/stats.html?id=c818a440-15b8-4da9-9614-96c3296fbb8a
• Comparison-View: https://metrics.green-coding.berlin/compare.html?ids=c818a440-15b8-4da9-9614-96c3296fbb8a,69958765-14bb-44c0-90cd-8ef084c0ecdd

The repo to match it is here: https://github.com/green-coding-berlin/DBMS-bench

Is that something that could be interesting for the SCER project?

[Henry-WattTime]

Hello @ArneTR, the TPC benchmarks are very interesting and very applicable to the SCER conversation. @mvaltas, is this something to look at in more detail for the SCER?

[ghost]

Yes, definitelly it is something to look at, the result of mgCO2E per TPC-C select looks very well within the bounds of what want to grade software in.

[ArneTR]

Just for cross-referencing: I also made a post in the SCER project about other possible benchmarks on this topic like for browsers etc. : Green-Software-Foundation/scer#1

[chrisxie-fw]

Hi @ArneTR , you presented a great tool that could potentially be helpful in SCER project, thank you!

I was thinking if some kind of labelling be possible for different software projects, similar to what France has done to food nutritional score labelling: Nutri-Score: https://en.wikipedia.org/wiki/Nutri-Score .

• Referencing nutri-score's algorithm, what would be the key factors that determine the carbon efficiency of a certain piece of software, noting that SCI score is an absolute value, and label rating is a relative value.
• In the case of PostgreSQL and MariaDB mentioned above, in that test scenario, for example, if PostgreSQL got a SCI score of 1, MariaDB got a 2, can we then label PostgresSQL an SCER label of "A", MariaDB an SCER label of "B"? Again, these are relative labelling, in its particular test cases. Would the labelling be useful? to who? in what circumstances?
• Another example: does it make sense to measure SCI scores of idle laptops, one with windows OS, one with Linux OS, one with macOS, of similar hardware configurations? This may relate to the application boundaries of SCI, as in: what areas of applications that SCI score is most meaningful and makes the most sense. SCER may be an extension of SCI that presents SCI in relative terms, like the Nutri-Score.

[chrisxie-fw]

I really like the "Cleaner than 90% of pages tested", because it gives a relative rating of the page/website in comparison to the others tested. The relative rating scheme is what the rating of "A to D/F" is meant to convey to the end user.

[chrisxie-fw]

One other thing I would like to propose is regarding "0.1g of CO2/view". 0.1g of CO2 may not ring a bell for an average visitor of the website, but if it is converted to dollars in Social Cost of Carbon, that may start to get more attentions to the carbon rating of the site. The same logic may go with the software carbon efficiency rating SCER.

[chrisxie-fw]

One last thing is with the scope of the SCER standard. Our thinking is that the Standard may be more about generic framework of getting to a relative carbon efficiency rating, rather than the application specific rating, because there are just too many software applications to compute the rating. So our thinking is to extract the common process out of the calculating the rating, and make that a standard, so that each application specific domain can use this standard framework to create the carbon rating of their own application domain, such as database software, operating system software, ML software, etc. So we may want to identify the common process of getting a CE (Carbon Efficiency) rating, which may involve the definition of a benchmark (e.g. the most commonly used operations/functions of the software, in the case of database, it may be read/write operations), and the relative CE rating calculation formula or algorithm (in the case of database, the SCI scores of several databases from different vendors are submitted, the algorithm may be to calculate the average rating of each database, and assign a label of A to D based on their relative position in the submitted SCI score range). The key principles here are (1) the standard is generic, common, non-application-specific, (2) relativity in the labelling/rating.

[ArneTR]

I really like the Social Cost of Carbon! Interestingly seems to me that it varies by country ... I would have expected it to be a world-wide number, since this is a world-wide problem ... maybe too easy thinking :)

I will bring this up with our team internally and we will probalby bring it into the next release of our tool next week.

What we currently have is this:

This was the typical form the EPA calculators and such displayed the CO2e data in a more tangible format. The SCC is way better IMHO.

2. I also do agree on the pre-defined standard benchmarks that software has to adhere rather the other way round.
   Especially for the very different world of E-Mail programs one can define 2-3 very basic SCER benchmarks where a software has to hit certain "checkpoints / endpoints" and then you get a comparable rating.
   Obviously the software has much more but this is then the "luxury" part and up to the user to wage if he wants that for the maybe increased runtime cost.

A place where this has worked well I think is for instance with cars, where a "standard" test is run how much liter per kilometer a car consumes.
Or in graphic-cards and drivers. Stuff like CineBench and 3DMark have just proposed what they thought is reasonable (to my knowledge at least) and created an industry standard benchmark around this.

I think what would make this tangible for the user is to create an examplary case on an application that has so far no standard tests around. DBs have TPC-C already around .... So I guess doing this for an E-Mail application or a calender or a word processor would be very interesting to illustrate the concept and get the community working on drafting and deciding for standard tests.

What do you think? Is that maybe even something the WG is already planning?

[chrisxie-fw]

The "Trees, Gasoline..." tangible depictions look great! If SCC is incorporated, that would be fantastic!

I envision that SCER standard may help create a standard framework, or a "meta" standard (standard of standards), that helps create more application specific standards like "CineBench", but in Carbon terms, or SCC terms.