Extending Software Carbon Intensity for AI

[navveenb]

I have put together some high-level points to drive discussions on how we can extend SCI to AI. Please note, these are initial ideas that will be discussed, iterated, and agreed upon by the standards working group.

.........
Goal of this specification
The purpose of this proposed specification is to assist AI practitioners—developers, data scientists, engineers, and decision-makers—in understanding and reducing the carbon footprint of AI systems. By making informed choices about model design, computational efficiency, and deployment strategies, practitioners can minimize emissions while maintaining performance.

Reducing the SCI score of an AI system is achievable by:

• Improving Energy Efficiency: Optimizing algorithms and code to use less energy for the same computational tasks by simplifying models (such as reducing the number of layers in a neural network), using pre-trained models to reduce computational requirements (such as leveraging transfer learning from existing models), and employing prompt engineering and efficient fine-tuning techniques to minimize additional training (such as using few-shot learning or optimizing prompts to achieve desired outputs with fewer computational resources).
• Enhancing Hardware Efficiency: Utilizing hardware that delivers higher performance per watt, such as specialized accelerators like GPUs, TPUs, or AI-specific chips.
• Adopting Carbon-Aware Practices: Timing computational tasks to coincide with periods of lower carbon intensity in the energy grid, such as running training jobs when renewable energy availability is high.

This specification will address the unique challenges of measuring and reducing the SCI of AI systems, including scenarios where practitioners have limited access to underlying utilization data, such as when using Gen AI services via APIs.

Scope
We will develop a methodology for calculating the carbon emissions rate (SCI score) of AI software systems, including both classical AI and generative AI applications. This specification aims to provide a reliable, consistent, and comparable measure that practitioners can use to set targets and track progress in reducing carbon emissions throughout the AI lifecycle—from development and training to deployment and inference.

Rationale for Development
As AI systems continue to grow in complexity and ubiquity, their energy consumption and associated carbon emissions are becoming increasingly significant. Current SCI specifications do not fully address the unique characteristics and challenges of AI applications. By extending the SCI to AI systems, we aim to:

• Provide Clear Guidelines: Offer AI practitioners a tailored framework for measuring and reducing carbon emissions.
• Promote Sustainability: Encourage the development of energy-efficient AI models and practices.
• Enhance Transparency: Foster openness in reporting the environmental impact of AI systems.
• Facilitate Standardization: Establish standard methodologies for calculating SCI in AI, enabling comparability across different systems and organizations.

Development Goals
In developing this extended specification, we will focus on:

• Addressing Measurement Challenges: Provide guidance to estimate energy consumption and emissions, even when direct utilization data is unavailable.
• Defining Software Boundaries: Clarifying which components should be included in the SCI calculation for AI systems.
• Adapting Functional Units: Identifying appropriate functional units that reflect how AI systems scale.
• Incorporating AI-Specific Actions: Including strategies and actions unique to AI that can reduce carbon emissions.
• Updating Reporting Practices: Establishing guidelines for transparent reporting, including assumptions and uncertainties.

Measurement Challenges in AI Systems
We recognize that measuring the SCI of AI systems presents unique challenges, which we will address in the specification:

• Limited Access to Utilization Data

• When using AI services provided via APIs (e.g., Azure OpenAI Service, Google Vertex AI), practitioners may not have direct access to the underlying hardware utilization or energy consumption data. We will provide methodologies to estimate energy consumption using proxy metrics.

• Variability in Workloads

• AI workloads can vary significantly in computational intensity based on factors such as model complexity, input data size, and requested output length, especially in generative AI applications. Our specification will account for this variability in the SCI calculation.

• Estimating Energy Consumption

• Without direct measurements, estimating energy consumption requires reliance on proxy metrics and assumptions, which can introduce uncertainties. We will provide guidelines on how to make informed estimates and document assumptions transparently.

• Defining Software Boundaries

• Determining which components to include in the SCI calculation can be complex, especially when third-party services and client-side computations are involved. We will define clear boundaries to ensure consistency and completeness.

Procedure
The steps required to calculate and report an SCI score for an AI system will be:

Bound: Define the software boundary, including all components contributing to the AI system's operation.
Scale: Choose a functional unit (R) that best represents how the AI system scales, such as per inference, per token processed, or per training run.
Define: Decide on the quantification method for each component—using real-world measurements, proxy metrics, or modeled estimates.
Quantify: Calculate the SCI value for each component and sum them to get the total SCI score for the AI system.
Report: Disclose the SCI score, the system boundary, the calculation methodology, and any assumptions or uncertainties.

Methodology Summary
Same as SCI, with the following specific to AI

• Energy (E)

• Energy consumed by the AI system includes:

• • Training Energy: Energy used during model training and fine-tuning.
• • Inference Energy: Energy used during the deployment phase for making predictions or generating outputs.
• • Data Processing Energy: Energy consumed in data preprocessing, augmentation, and loading.
• • Client-Side Computation: Energy used in computations performed on end-user devices.
• • Units: kilowatt-hours (kWh).

• Estimating Energy Consumption Without Direct Data

• We will provide guidance for estimating energy consumption when direct measurement is not possible:

• • Proxy Metrics: Use token counts, latency, and model size as indicators of computational effort.
• • Published Data: Refer to academic research or industry reports providing energy consumption estimates for similar models or services.
• • Assumed Hardware Profiles: Make informed assumptions about the underlying hardware based on publicly available information.

• Region-Specific Carbon Intensity (I)

• • The carbon intensity of the electricity grid where the computation takes place will be considered.

• Estimating Embodied Emissions Without Direct Data

• We will guide practitioners on estimating embodied emissions:

• • Industry Averages: Use average embodied emissions values for typical hardware (e.g., GPUs, servers).
• • Usage Allocation: Estimate your share of the hardware's embodied emissions based on usage metrics like API call counts.

• Software Boundary

• We will define the software boundary to encompass:

• • Compute Resources: Servers, GPUs, TPUs used during training and inference.
• • Storage Systems: Databases and data storage services used for datasets.
• • Networking Equipment: Data transfer between systems and to end-users.
• • Third-Party Services: AI services accessed via APIs, considering their computational impact.
• • Client-Side Computations: Especially important in generative AI where some processing may occur on user devices.
• • Supporting Infrastructure: Monitoring, logging, and backup systems specific to the AI application.
• • Development Infrastructure: Compute resources used during model development, testing, and fine-tuning.

• Defining the Boundary with Limited Data Access

• When using third-party AI services, we will recommend:

• • Including Client-Side Operations: Account for all computations performed on your end.
• • Estimating Third-Party Impact: Use proxy metrics to estimate the impact of API calls.
• • Including Network Emissions: Account for emissions from data transfer between your application and the AI service.

• Functional Unit (R)

• We will help practitioners select a functional unit that accurately reflects how the AI system scales. Possible choices include:

• • Per Inference: Common for deployed AI services responding to requests.
• • Per 1,000 Tokens Processed: Useful in generative AI where token count correlates with computational effort.
• • Per Training Run: For evaluating the impact of training new models or fine-tuning existing ones.
• • Per User Session: When interactions vary but can be averaged per user.
• • Per API Call: If requests are relatively uniform in complexity.

• Consistency in the functional unit will be essential for accurate SCI calculation and comparison.

Quantification Method

• Measurement
• We will encourage the use of real-world data where possible:
• • Energy Monitoring Tools: Software utilities or hardware meters that track energy consumption during AI tasks.
• • Cloud Provider Data: Energy usage reports provided by cloud service vendors.
• • Hardware Sensors: Direct measurements from devices like GPUs and CPUs.
• Estimation Using Proxy Metrics
• When direct measurement is not feasible, we will provide guidance on:
• • Token Count: Estimating energy consumption based on the number of tokens processed.
• • Latency: Using response time as an indirect indicator of computational effort.
• • Model Complexity: Considering the size and type of the model being used.
• • Request Frequency: Calculating total impact based on the number of API calls or inferences.
• Modeling
• We will recommend creating models to estimate energy consumption:
• • Algorithmic Complexity: Using computational complexity to estimate energy usage.
• • Hardware Specifications: Basing estimates on the power ratings and efficiency of typical hardware used for AI workloads.
• • Simulation Tools: Employing software that simulates energy consumption based on workload characteristics.

Core Characteristics
The SCI score will reflect improvements resulting from AI-specific optimizations:

• Model Compression: Techniques like pruning and quantization that reduce model size and computational requirements will lower the SCI score.
• Efficient Architectures: Using models designed for efficiency, such as transformers with fewer parameters, will reduce emissions.
• Prompt Optimization: In generative AI, crafting prompts that achieve desired results with fewer tokens will decrease the SCI score.

Encouraging Granular Data Use

• High-Resolution Data: We will recommend using the most detailed data available for energy consumption, carbon intensity, and embodied emissions.
• Provider Collaboration: We will encourage engagement with service providers to obtain more precise data.
• Proxy Data: When granular data is unavailable, we will suggest using the best available estimates and documenting the sources.

[navveenb]

Pls have a look - @Henry-WattTime @jawache @seanmcilroy29

[jawache]

Hey @navveenb thanks for this, looks great, so glad you're taking a lead here.

I agree with most of what your proposing at the high level, but in terms of deliverables I believe we should consider splitting this into two (or more) distinct deliverables.

1. The SCI for AI Spec which covers just these three points:

• Defining Software Boundaries: Clarifying which components should be included in the SCI calculation for AI systems.
• Adapting Functional Units: Identifying appropriate functional units that reflect how AI systems scale.
• Updating Reporting Practices: Establishing guidelines for transparent reporting, including assumptions and uncertainties.

2. A separate Guide/WhitePaper/Course for how to compute it which includes these points

• Addressing Measurement Challenges: Provide guidance to estimate energy consumption and emissions, even when direct utilization data is unavailable.
• Incorporating AI-Specific Actions: Including strategies and actions unique to AI that can reduce carbon emissions.

Rationale for separating these out is that the field of software measurement is moving very very fast but standards move slow. If we put advice on how to measure a standard in the standard then it will quickly become out of date. But we can update a guide very quickly.

I would also propose that perhaps the guide should be a course with a certificate of completion like our green software practitioners course, that way people have a clear action they can take after we publish the SCI for AI.

3. Supporting Deliverables
   We should think through other things we can create in parallel which would support the SCI for AI.

• An Impact Framework template, ready for people to add their own data and compute their SCI for AI score.
• Publish case studies, so people have access to examples when they view the spec.
• A repository to publish SCI for AI scores with Impact Framework Manifest files as evidence.

Process

In terms of process I'd propose that if the SWG is agreed then we launch a workshop designed to quickly get an initial baseline for the two points above.

Rough idea for now is for us to design a workshop with our consultant. Then the SWG/GAIC identify the kinds of people we want in this workshop, and then put a call out for applications. The GAIC and SWG should both be involved in the selection of those initial workshop participants.

[Henry-WattTime]

Addition from WG: should be doing actual calculations at the same time
Just challenging to compute parts of the data.
See example: https://www.rockpool.tech/posts/is-training-in-scope-for-ai-sci

[navveenb]

Yes, @jawache , thanks for the inputs. Agreed, we need to divide it into three parts - Specs, Guide/Awareness, and Tools. In terms of the process, A workshop is the best way forward to create an initial baseline for these elements.

[Henry-WattTime]

Notes from Standards WG call:

Green AI working group:
Lots of discussions on the topics, includes lots of AI topics
-What are the boundaries
-Phases of AI development
What should we not be doing: ethical AI, some other things that are not the focus
Green AI has lots of expertise in AI, but not expertise in the SCI necessarily

There are other terms being used in AI - narrow AI, general AI, superintelligent AI
locking down the definitions will help the boundary topic

Proposed: joint workshop between standards and Green AI experts
have specific set of questions to answer AI
How to nominate people to that
Invite external experts?
Standards WG designs workshop
Output of workshop?
-step one of the draft standard, a markdown file

Step two:
-standards WG actually writes standard
-testing it with real examples

Step three;
-have a packet of spec/course/guide/tooling to actually implement spec

Timeline for SCI for AI?
Workshop early next year, aiming for spec to follow that shortly

AI has already changed quickly, currently subscription approach
Hard to measure for these 'hidden' APIs
Focus on full knowledge systems, where fully lifecycle is known
Push AI orgs to share the information
This is the classic 'managed services' problem
-ideally we are defining the standard that the services are using, they publish an SCI
-that SCI can be used by different organizations
-(This is the dream, the entire supply chain is publishing scores)
example: Redhat expert had info on steps on how to build an AI model from the ground up
-can be used to build lifecycle model/diagram
How to relate information to people consuming services
-most people are not building the models themselves, but consuming and API
-some research papers exploring this (Navveen to share?)

Is there a reference architecture for AI that we can measure?
-model the managed services to compute

WG consensus: Design the workshop on October 24, need Russ

[Henry-WattTime]

WHO gets into the workshop:
Green AI committee
Standards WG
Create an application process
Selection criteria are a shared responsibility between two groups
Standards experts & AI experts
Level of expertise, how do we vet them, how do we manage application process

Engineer the workshop so that the outcome
foundational information to write the standard SCI for AI
-What is in scope/what is the boundary
-What is the functional unit
-How do we cover different types of AI

Baseline/starting point is the existing SCI, we're extending it for AI
-How do we apply this to AI?

Output from Green AI workshops may be useful for definitions

Collecting reading material for the workshop, task for committees

#391