2024.02.21

[seanmcilroy29]

---

2024.02.21 Agenda/Minutes

---

Time: Bi-weekly @ 1600 (GMT) - See the time in your timezone

• Co-Chair - Marco Valtas (Thoughtsworks)
• Co-Chair - Chris Xie- (Futurewei)
• Convener – Sean Mcilroy (Linux Foundation)

Antitrust Policy

Joint Development Foundation meetings may involve participation by industry competitors, and the Joint Development Foundation intends to conduct all of its activities in accordance with applicable antitrust and competition laws. It is, therefore, extremely important that attendees adhere to meeting agendas and be aware of and not participate in any activities that are prohibited under applicable US state, federal or foreign antitrust and competition laws.

If you have questions about these matters, please contact your company counsel or counsel to the Joint Development Foundation, DLA Piper.

Recordings

WG agreed to record all Meetings. This meeting recording will be available until the next scheduled meeting.
Meeting recording link

Roll Call

Please add 'Attended' to this issue during the meeting to denote attendance.

Any untracked attendees will be added by the GSF team below:

• Full Name, Affiliation, (optional) GitHub username

Agenda

• Approve agenda
• Approve previous Meeting Minutes

Introductions

• Any new members?

PR submissions

• Review 1st draft for SCER for LLMs #28

Review IMDA Email

• Review Email from IMDA - information not for public consumption

Submitted Issues

• Research for current ways of measuring CO2 emissions of LLM's #25
• Common Components of a Software Categorization (2.1.1) should also include "Size of the Software" as one of the other aspects.  #26
• Rating Example needs some correction #27
• Logo for Software Carbon Efficiency Rating Project #29
• 2024.02.21 #30
• The scope of SCER specification: would SCER address the specifications of data collection, and the resulted rating publication? #14
• Should SaaS be part of the software categorization?  #10
• Should these aspects of categorizing software be OR or AND relationship?  #11
• Benchmarking #12
• Examples for SCER Reference Applications and Technical Integration & Visualisation #1

Draft Timeline

• Formation and Kick-off: Within one month of the working group's establishment on GitHub (Starting Date: Oct 18, 2023).
• Carbon Efficiency Best Practices Guide: Develop and finalize the guide within six months **(Target Completion Date: [Insert Date]).**
• Software Carbon Efficiency Rating (SCER) Standard: Develop and publish the SCER standard within eight months (Target Completion Date: [Insert Date]).
• Badge Program Implementation: Establish the software carbon efficiency badge program within eight months (Target Completion Date: [Insert Date]).
• Documentation and Guidelines: Create and release comprehensive documentation and guidelines within six months (Target Completion Date: [Insert Date]).
• Community Engagement Initiatives: Plan and execute engagement initiatives continuously, with an initial series of webinars starting within three months (Starting Date: [Insert Date]).

Spec Review

• Proposal to discuss the Spec updates at the next WG meeting?

Next Steps

• PR/blog/email notification to GSF members
• Info SWG
• Circulate Draft Standard

AOB

• Member submissions

Future meeting Agenda submissions

• If you have any Agenda submissions, please let Sean know via email [email protected] or open an Issue

Next Meeting

• 07th Mar @ 9 am P/T

Adjourn

• Motion to adjourn

---

Meeting Action Items / Standing Agenda / Future Agenda submissions

• Add here

[ghost]

Attended

[chrisxie-fw]

attended.

[chrisxie-fw]

The team agreed on the following meeting minutes and next steps:

• @chrisxie-fw to submit the updated research and work plan to Dev branch, so that other contributors can comment, revise, update.
• The SCER standard includes 1. Software Categorization, 2. Benchmark definition, 3. Rating definition(algorithm etc), 4. Visualization and Labelling. For LLMs, it appears that Hugging Face has done 1 & 2 already. The team believes that SCER for LLM could add value on 3&4.
• Next steps: To confirm that value proposition, team agrees to further study Hugging Face's carbon efficiency evaluation process for LLMs, and to identify potential gaps and opportunities where SCER for LLM can provide value. The next meeting agenda is to share the findings, preferably in the form of a document as record to be shared in the Dev branch. @seanmcilroy29 @chrisxie-fw @mvaltas @mrchrisadams @veverkap

[seanmcilroy29]

MoM

Energy consumption and carbon footprint of large language models.
Chris has proposed a framework for evaluating and comparing community agencies using large language models. Additionally, he has raised concerns about the energy consumption and carbon footprint of high-end Nvidia chips, which consume the same energy as a small nation and have a larger carbon footprint by 2024. To address the issue of differences in profiles, life cycles, and efficiency tuning, Chris suggests separating large language models into production and interference phases. He also emphasizes the importance of adopting more efficient and sustainable practices in AI model training to reduce energy consumption and emissions.

Carbon footprint of AI training.
Marco suggests that we should categorize the efficiency of training based on the amount of carbon emissions it produces, taking into account both the training and deployment phases. Sean O and Marco discuss the carbon emissions associated with AI models during the discussion, and Chris shares his insights on how Hugging Face accounts for these emissions. Chris also talks about a project that successfully reduced carbon emissions by 20 metric tonnes, using a French supercomputer powered by nuclear energy. The team also talks about Carbon AI, a tool that aims to estimate the carbon footprint of AI models, including national language models, by tracking the consumption of computer resources and energy mix. Chris discusses the most popular hugging rates and their downloads, likes, and environmental impact. Marco raises a question about why commonly visited regions are not included in the evaluation parameter and suggests some potential alternatives for providing value.

AI model rating and training costs.
Chris: Algorithms become a software category, with training and operating distinct.

Standardizing AI model training and evaluation.
Sean O and Chris discussed incorporating a new AI model into an existing specification. The group consider various factors, such as training data and hardware embodied carbon. Chris emphasizes the importance of standardizing AI models and their applications. He highlights the need to identify different phases of the model lifecycle.

AI model training costs and efficiency.
Chris suggested that "inbody" should be defined as the cost of training a model. This could help justify the cost of training and spread it across millions of users. On the other hand, Marco proposed that the training cost should be separate from the user perspective. He suggested focusing on hardware requirements and tokens per kilowatt for efficient deployment. The group discussed the cost of training large language models (LLMs) and how it should be factored into the product itself. Marco also mentioned open-source projects for national model safety and workload testing and highlighted how Google and OpenAI advocate for a training phase to lower the carbon cost of software development. Chris expressed interest in reviewing and learning from the framework for setting up standards or practices. He aims to achieve a similar attitude towards lowering carbon costs in software development.

AI model benchmarking and rating.
Marco leads a discussion about the Open AI leaderboard, which is a platform for benchmarking AI models. This includes metrics related to energy and safety. The group discusses how to obtain and rate data to benchmark these models. Marco suggests collaborating with a benchmarking agency to promote reading and standardize a linking algorithm. Sean O suggests preparing something to present to potential partners, such as visualizations of their data. Marco suggests creating a website that scrapes their data and provides value in return. Chris suggests investigating if potential partners' definition of integrity aligns with their own. Marco agrees and suggests drilling down a subset of their data to create a rating for a specific category.

Action Items
[ ] Review the draft SCER spec for RMS submitted by Chris
[ ] Review the IMDA email sent by Sean M.
[ ] Review the issues added by Kitty
[ ] Review Chris' updated AI use case study
[ ] Review the Hugging Face benchmarks and process
[ ] Gather data from Hugging Face and calculate ratings
[ ] Build a demo site with ratings using Hugging Face data
[ ] Align integrity definitions between Hugging Face and Green Software Foundation
[ ] Identify a subset of LLM categories to focus on