Goals/criteria for a case study to validate the SCI standard

[dtoakley]

There is an idea to validate the SCI standard using some existing software, to help us learn whether the standard is/isn't providing the intended value. If it isn't, the case study would hopefully help us identify gaps in the standard. Personally, I think we're still some way off from being able to do this effectively, however as we think about selecting a piece of software, I wanted to open a discussion on the goals of the case study. I currently see three potential potential goals:

1. Validate whether the SCI standard can be used to quantify the carbon emissions of a piece of software. E.g I have a software application running in the cloud and I want to quantify it's carbon emissions over time. Can I use the SCI standard to do this?
2. Validate whether the SCI can be used to reduce the carbon emissions from a piece of software. E.g I want to make changes to the same software application and quantify the change in emissions. Can I use the SCI standard to do this?
3. Validate whether a piece of software can be built as an implementation of the SCI standard. E.g. I want to offer a product/service/tool that allows people to quantify their carbon emissions from their software. Can I use and meet the SCI standard in doing so?

Keen to read/hear what others think, but my sense is that 2 can't be achieved without 1, and 2 brings us the most value for learning if/how the standard can help create an immediate real-world impact on the software we build. If this is true, here are some suggested criteria for deciding on a piece of software:

1. It's simple enough to run the software in commonly used environments (locally, the cloud, on-premise, etc)
2. Software is complex enough that there are possible changes (code, infrastructure, dependancies, etc) that could be made to potentially reduce carbon emissions. But also simple enough so we can understand how it works.
3. It's open source - so the WG can best understand the case study and the underlying software under test
4. It's written in one of the top X most common languages

There could be others, but thought I'd share those to start the discussion on both the goals for the case study and how we select a piece of software for it.

[atg-abhishek]

This is great @dtoakley - I think you capture requirements for the case study from 1 and 2 quite well. I think once we have a better idea from running some case studies (based on the requirements that you outline here), we can start to think about 3. It should be something that we have in the back of our mind for sure but I feel we would need to have the SCI take a bit more of a concrete form before we are able to think a bit more about 3.

[atg-abhishek]

Will: This reminds us of the ML Reproducibility checklist (@atg-abhishek to add in a link for this)

[atg-abhishek]

Here is the link to the checklist: https://www.cs.mcgill.ca/~jpineau/ReproducibilityChecklist.pdf

[jawache]

For (2) I think it all boils down to the 3 characteristics of the SCI. It should be sensitive to energy efficiency, carbon awareness and hardware efficiency. As long as we ensure that remains true then (2) will be true since everything a software engineer can do falls into one of those three buckets. If an engineer makes their app more carbon aware, makes it do more when there are more renewables on the grid, and that doesn't reduce the SCI, then why would an engineer bother to make their application carbon aware.

For (1) at some level, it is a number that takes as input Energy Used * Carbon Intensity + Embodied Carbon. The rules for how we calculate energy, carbon intensity and embodied carbon are where things get sticky since it really changes a lot depending on the type of app, I would suggest a selection of use cases to make sure we are representing a broad swathe of software teams.

• You are the open-source maintainer of TensorFlow
• You sell a SaaS ML product that uses TensorFlow
• You are a mobile app developer
• You are a web developer.
• You are the Windows team at Microsoft
• You are the Android team at Google