Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added DiskIO Reporter #825

Merged
merged 12 commits into from
Sep 26, 2024
Merged

Added DiskIO Reporter #825

merged 12 commits into from
Sep 26, 2024

Conversation

ArneTR
Copy link
Member

@ArneTR ArneTR commented Jun 30, 2024

This PR adds a Disk I/O reporter.

To make it container agnostic the data is pulled directly from the sysfs. This needs the IO delegator to be set as specified in the documentation.

I am currently under the assumption that this controller is always active for docker in root mode, but will double check soon.

The approach here is similar to the other cgroup reading metric providers. However while crafting this the idea came up to actually split Input and Output and not supply the value as a cumulated number.

I see here two options:

  • Making the reporter something like a base reporter that always reads everything and then only a part of the value gets extracted
  • Making a shallow reporter that works like the SDIA or XGBoost one and actually relies on another reporter to be present on the system.

Here I think @ribalba it makes sense to discuss how this was solved in the lm_sensors

@ArneTR ArneTR requested a review from ribalba June 30, 2024 13:25
Copy link

github-actions bot commented Jun 30, 2024

Old Energy Estimation

Eco-CI Output:

Label 🖥 avg. CPU utilization [%] 🔋 Total Energy [Joules] 🔌 avg. Power [Watts] Duration [Seconds]
Total Run (incl. overhead) 23.3636 1671.39 3.87 432
Measurement #1 23.2916 1671.39 3.88 431

🌳 CO2 Data:
City: Phoenix, Lat: 33.4475, Lon: -112.0866
IP: 20.169.14.176
CO₂ from energy is: 0.476346150 g
CO₂ from manufacturing (embodied carbon) is: 0.123255565 g
Carbon Intensity for this location: 285 gCO₂eq/kWh
SCI: 0.599602 gCO₂eq / pipeline run emitted

Copy link
Member

@ribalba ribalba left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My C is a little rusty. Not sure I found everything.

metric_providers/disk/io/cgroup/container/source.c Outdated Show resolved Hide resolved
metric_providers/disk/io/cgroup/container/source.c Outdated Show resolved Hide resolved
metric_providers/disk/io/cgroup/container/source.c Outdated Show resolved Hide resolved
metric_providers/disk/io/cgroup/container/source.c Outdated Show resolved Hide resolved
metric_providers/disk/io/cgroup/container/source.c Outdated Show resolved Hide resolved
metric_providers/disk/io/cgroup/container/source.c Outdated Show resolved Hide resolved
metric_providers/disk/io/cgroup/container/source.c Outdated Show resolved Hide resolved
Copy link

Eco-CI Output:

Label 🖥 avg. CPU utilization [%] 🔋 Total Energy [Joules] 🔌 avg. Power [Watts] Duration [Seconds]
Total Run (incl. overhead) 25.0178 1570.25 3.93 400
Measurement #1 24.9735 1570.25 3.95 398

🌳 CO2 Data:
City: Boydton, Lat: 36.6534, Lon: -78.375
IP: 20.161.78.144
CO₂ from energy is: 0.617108250 g
CO₂ from manufacturing (embodied carbon) is: 0.114125523 g
Carbon Intensity for this location: 393 gCO₂eq/kWh
SCI: 0.731234 gCO₂eq / pipeline run emitted

@ArneTR ArneTR requested a review from ribalba August 22, 2024 10:16
@ArneTR
Copy link
Member Author

ArneTR commented Aug 22, 2024

@ribalba All done. Quick Re-review please as I changed this also in all other C source files we have

Copy link
Member

@ribalba ribalba left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@ArneTR ArneTR merged commit 3caa354 into main Sep 26, 2024
4 checks passed
@ArneTR ArneTR deleted the diskio branch September 26, 2024 05:54
ArneTR added a commit that referenced this pull request Sep 27, 2024
* main:
  Authentication in GMT (#872)
  Phase name check (#893)
  Carbondb unique (#877)
  Added DiskIO Reporter (#825)
  Bump pylint from 3.3.0 to 3.3.1 (#913)
  Bump aiohttp from 3.10.5 to 3.10.6 (#912)
  Bump plotext from 5.3.1 to 5.3.2 (#914)
  Bump pandas from 2.2.2 to 2.2.3 (#910)
  Bump plotext from 5.2.8 to 5.3.1 (#909)
  Bump pylint from 3.2.7 to 3.3.0 (#911)
  Bump psycopg-pool from 3.2.2 to 3.2.3 (#901)
  Bump pydantic from 2.9.1 to 2.9.2 (#904)
  Bump fastapi[standard] from 0.114.2 to 0.115.0 (#905)
  Importer for Measurements (#908)
Copy link
Member

@ribalba ribalba left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants