Use-Case: Measuring the total carbon emissions for a given ML workload (training/inference)

[will-iamalpine]

AzureML is a great use case to explore, due to the high cost (including carbon) of ML workloads. They are discrete 'jobs' and can showcase the carbon intensity specification work.

[atg-abhishek]

@buchananwp can you also add in

1. the external presentation here for reference and
2. perhaps brief answers to Goals/criteria for a case study to validate the SCI standard #23

Thanks!

[atg-abhishek]

Will: Using a lifecycle methodology thinking about training, inference, etc. will help

Vaughn: where does the inference happen will also help - embodied cost in the device that is on the edge vs. having to go to the cloud

Abhishek: you can cache frequently used results rather than doing fresh inferences every time

Will: Triton better packs inferences on the GPU which helps to get more in terms of the utilization of the hardware

[atg-abhishek]

@buchananwp what was the next step that we needed on this?

[navveenb]

@atg-abhishek @buchananwp , I can help with this specification. We can discuss it in our weekly meeting.

[navveenb]

I am sharing some thoughts -

Some factors that affect carbon emissions for a given ML workload

1. Choice of server infrastructure - machine types, CPU, GPU, TPU etc
2. Machine learning model - Type of network
3. Data storage and transfer - Data stored and accumulated for ML model
4. Pre-processing of Data - Is building a data pipeline and computation included as part of ML workload
5. Training cycles for creating the final model
6. Training cycles for updating the model - Strategy like use of transfer learning.
7. Final model size and versions of ML model - Storage of current model and previous ML models (for audit, compliance etc)
8. Deployment type - Cloud or Edge (as network transfer, hardware optimisation on edge, specialised hardware edge devices might come into play)
9. Fine tuning model infrastructure like caching v/s runtime serving of model.

In terms of measurement, in my view we should look at the above factors holistically to come up with the total carbon emission model.