2024.03.06

[seanmcilroy29]

---

2024.03.06 Agenda/Minutes

---

Time: Bi-weekly @ 1600 (GMT) - See the time in your timezone

• Co-Chair - Marco Valtas (Thoughtsworks)
• Co-Chair - Chris Xie- (Futurewei)
• Convener – Sean Mcilroy (Linux Foundation)

Antitrust Policy

Joint Development Foundation meetings may involve participation by industry competitors, and the Joint Development Foundation intends to conduct all of its activities in accordance with applicable antitrust and competition laws. It is, therefore, extremely important that attendees adhere to meeting agendas and be aware of and not participate in any activities that are prohibited under applicable US state, federal or foreign antitrust and competition laws.

If you have questions about these matters, please contact your company counsel or counsel to the Joint Development Foundation, DLA Piper.

Recordings

WG agreed to record all Meetings. This meeting recording will be available until the next scheduled meeting.
Meeting recording link

Roll Call

Please add 'Attended' to this issue during the meeting to denote attendance.

Any untracked attendees will be added by the GSF team below:

• Full Name, Affiliation, (optional) GitHub username

Agenda

• Approve agenda
• Approve previous Meeting Minutes

Introductions

• Any new members?

PR Review

• Update landscape research and analysis #31

Issues Dashboard

• Hugging Face uses CodeCarbon to benchmark LLMs energy consumption in kWh #32
• Establish an instant-messaging group for real time brainstorming for research #33
• Add a meeting agenda: presenting SCER for LLMs in AI_Dev conference Responsible AI track for Sustainable AI life cycle #34
• Research for current ways of measuring CO2 emissions of LLM's #25
• Common Components of a Software Categorization (2.1.1) should also include "Size of the Software" as one of the other aspects.  #26
• Rating Example needs some correction #27
• The scope of SCER specification: would SCER address the specifications of data collection, and the resulted rating publication? #14
• Should these aspects of categorizing software be OR or AND relationship?  #11
• Benchmarking #12
• Examples for SCER Reference Applications and Technical Integration & Visualisation #1

Draft Timeline

• Formation and Kick-off: Within one month of the working group's establishment on GitHub (Starting Date: Oct 18, 2023).
• Carbon Efficiency Best Practices Guide: Develop and finalize the guide within six months **(Target Completion Date: [Insert Date]).**
• Software Carbon Efficiency Rating (SCER) Standard: Develop and publish the SCER standard within eight months (Target Completion Date: [Insert Date]).
• Badge Program Implementation: Establish the software carbon efficiency badge program within eight months (Target Completion Date: [Insert Date]).
• Documentation and Guidelines: Create and release comprehensive documentation and guidelines within six months (Target Completion Date: [Insert Date]).
• Community Engagement Initiatives: Plan and execute engagement initiatives continuously, with an initial series of webinars starting within three months (Starting Date: [Insert Date]).

Spec Review

• Proposal to discuss the Spec updates at the next WG meeting?

Next Steps

• PR/blog/email notification to GSF members

AOB

• Member submissions

Future meeting Agenda submissions

• If you have any Agenda submissions, please let Sean know via email [email protected] or open an Issue

Next Meeting

• 20th Mar @ 9 am P/T

Adjourn

• Motion to adjourn

---

Meeting Action Items / Standing Agenda / Future Agenda submissions

• Add here

[sk16-dev]

Attended

[chrisxie-fw]

attended

[chrisxie-fw]

Meeting minutes:

• team approved to merge the pull request: Update landscape research and analysis #31
• team resolved a number of issues, the remaining issues are to be resolved in future meetings
• team will read through the current research documents and URLs and have a collective understanding of the current landscape in LLM's carbon footprint and existing solutions
• team will come back and share what the current market gaps are, and where we think SCER WG can contribute value to
• one potential opportunity that the team have discussed is the need for transparency of LLM's carbon efficiency benchmarking workflow/process, so that the numbers are reproducible by any entity following a publicly available standard process (open source standard). This appears to be aligned with the mission of SCER for LLMs. So one idea is to discuss with major LLM providers to see if they are willing to open source their internal LLM benchmarking workflow. If they are not willing, then SCER for LLMs will consider to devise an open source version of such a workflow and publish the standard, workflow, and the resulting carbon efficiency ratings and numbers as open source projects under GSF. This topic will need to be further discussed and developed into concrete action items.

[KeertiHCL]

attended