Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add warnings and errors to ImpactsOutput #99

Merged
merged 13 commits into from
Dec 13, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 11 additions & 6 deletions docs/tutorial/impacts.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,14 @@
# Environmental Impacts

Environmental impacts are reported for each request in the [`Impacts`][impacts.modeling.Impacts] pydantic model and features multiple [criteria](#criteria) such as the [energy](#energy) and [global warming potential](#global-warming-potential-gwp) per phase ([usage](#usage) or [embodied](#embodied)) as well as the total impacts.
Environmental impacts are reported for each request in the [`ImpactsOutput`][tracers.utils.ImpactsOutput] pydantic model and features multiple [criteria](#criteria) such as the [energy](#energy) and [global warming potential](#global-warming-potential-gwp) per phase ([usage](#usage) or [embodied](#embodied)) as well as the total impacts.

To learn more on how we estimate the environmental impacts and what are our hypotheses go to the [methodology](../methodology/index.md) section.

```python title="Structure of Impacts model"
from ecologits.impacts.modeling import *
from ecologits.tracers.utils import ImpactsOutput
from ecologits.impacts.modeling import ADPe, Embodied, Energy, GWP, PE, Usage

Impacts(
ImpactsOutput(
energy=Energy(), # (1)!
gwp=GWP(),
adpe=ADPe(),
Expand All @@ -22,13 +23,17 @@ Impacts(
gwp=GWP(),
adpe=ADPe(),
pe=PE(),
)
),
warnings=None, # (4)!
errors=None
)
```

1. Total impacts for all phases.
2. Usage impacts for the electricity consumption impacts. Note that the energy is equal to the "total" energy impact.
3. Embodied impacts for resource extract, manufacturing and transportation of hardware components allocated to the request.
4. List of [`WarningMessage`][status_messages.WarningMessage] and [`ErrorMessage`][status_messages.ErrorMessage].


You can extract an impact with:

Expand All @@ -43,10 +48,10 @@ Or you could get **value range** impact instead:

```python
>>> response.impacts.usage.gwp.value
Range(min=0.16, max=0.48) # Expressed in kgCO2eq (1)
RangeValue(min=0.16, max=0.48) # Expressed in kgCO2eq (1)
```

1. [`Range`][impacts.modeling.Range] are used to define intervals.
1. [`RangeValue`][utils.range_value.RangeValue] are used to define intervals.

## Criteria

Expand Down
2 changes: 1 addition & 1 deletion docs/tutorial/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

The :seedling: **EcoLogits** library tracks the energy consumption and environmental impacts of generative AI models accessed through APIs and their official client libraries.

It achieves this by **patching the Python client libraries**, ensuring that each API request is wrapped with an impact calculation function. This function computes the **environmental impact based on several request features**, such as the **chosen model**, the **number of tokens generated**, and the **request's latency**. The resulting data is then encapsulated in an `Impacts` object, which is added to the response, containing the environmental impacts for a specific request.
It achieves this by **patching the Python client libraries**, ensuring that each API request is wrapped with an impact calculation function. This function computes the **environmental impact based on several request features**, such as the **chosen model**, the **number of tokens generated**, and the **request's latency**. The resulting data is then encapsulated in an [`ImpactsOutput`][tracers.utils.ImpactsOutput] object, which is added to the response, containing the environmental impacts for a specific request.


<div class="grid cards" markdown>
Expand Down
53 changes: 53 additions & 0 deletions docs/tutorial/warnings_and_errors.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# Warnings and Errors

EcoLogits may encounter situations where the calculation of environmental impacts has **high risk of inaccuracies or uncertainties** (reported as **warnings**), or where the calculation **fails** due to certain reasons like misconfiguration (reported as **errors**).

Warnings and errors are reported in the [`ImpactsOutput`][tracers.utils.ImpactsOutput] pydantic model within the `warnings` and `errors` fields respectively. Each warning or error contains a `code` (all listed below) and a `message` explaining the issue.

!!! note "Silent reporting of warnings and errors"

By default, warnings and errors are reported **silently**. This means you won't see any warning logged or exception raised. This approach ensures your program continues to execute and avoids spamming the log output, especially when executing many requests.

Code example on how to determine **if your request resulted in any warnings or errors** and how to retrieve them.

```python
from ecologits import EcoLogits

EcoLogits.init()

response = ... # Request code goes here

if response.impacts.has_warnings:
for w in response.impacts.warnings:
print(w)

if response.impacts.has_errors:
for e in response.impacts.errors:
print(e)
```


## Warnings

List of all the warnings that EcoLogits can report.

### `model-arch-not-released`

This warning is reported when the model architecture is not disclosed by the provider. Thus, the estimation of environmental impacts is based on a assumption of the model architecture (e.g. dense or mixture of experts, number of parameters).

### `model-arch-multimodal`

This warning is reported when the model is multimodal. EcoLogits uses energy benchmarking data from open source LLMs that can only generate text. Models that can generate (or use as input) data from other modalities such as image, audio or video are currently not fully supported.


## Errors

List of all the errors that EcoLogits can report.

### `model-not-registered`

This error is reported when the selected model is not registered. This can happen when the model has been released recently or if you are using a custom model (such as fine-tuned models). In the first case you can try updating EcoLogits to the latest version, if the error persists, you can [open up an issue :octicons-link-external-16:](https://github.com/genai-impact/ecologits/issues/new?assignees=&labels=bug&projects=&template=bug_report.yml).

### `zone-not-registered`

This error is reported when the selected geographical zone is not registered. This can happen if the configured zone does not exist or if the custom zone is not properly registered.
Loading
Loading