Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reduce size of API production image #1008

Closed
krysal opened this issue Mar 23, 2023 · 1 comment · Fixed by #4285
Closed

Reduce size of API production image #1008

krysal opened this issue Mar 23, 2023 · 1 comment · Fixed by #4285
Assignees
@dhruvkb
Labels
💻 aspect: code Concerns the software code in the repository 🧰 goal: internal improvement Improvement that benefits maintainers, not users 🟨 priority: medium Not blocking but should be addressed soon 🧱 stack: api Related to the Django API 🧱 stack: ingestion server Related to the ingestion/data refresh server

Comments

@krysal
Copy link
Member

krysal commented Mar 23, 2023 •
edited by dhruvkb
Loading

Problem

Development python packages are included in the final production image for the API and the Ingestion Server. Those aren't needed there and represent a risk; we should be able to remove them easily following a strategy similar to the Catalog with a docker-compose.yml / docker-compose.override.yml files.

Description

Exclude dev packages from the openverse-api and openverse-ingestion_server images.
@krysal krysal added 🟨 priority: medium Not blocking but should be addressed soon ✨ goal: improvement Improvement to an existing user-facing feature 💻 aspect: code Concerns the software code in the repository 🧱 stack: api Related to the Django API 🧱 stack: ingestion server Related to the ingestion/data refresh server labels Mar 23, 2023
@krysal krysal self-assigned this Mar 23, 2023
@github-project-automation github-project-automation bot moved this to 📋 Backlog in Openverse Backlog Mar 23, 2023
@krysal krysal mentioned this issue Mar 23, 2023
Closed
7 tasks
@krysal krysal added 🧰 goal: internal improvement Improvement that benefits maintainers, not users and removed ✨ goal: improvement Improvement to an existing user-facing feature labels Mar 23, 2023
@obulat obulat moved this from 📋 Backlog to 🏗 In progress in Openverse Backlog Mar 28, 2023
dhruvkb pushed a commit that referenced this issue Apr 14, 2023
@AetherUnbound @krysal @stacimc
Wikimedia: re-attempt large batches with reduced parameter selection (#…
c476f92 
…1008)

* WIP

* Handle dates from CLI appropriately

* Debug testing to reproduce issue

* Use the correct key

* Added notes

* Basic outline for how to handle larger batches

* Preserve popularity data across query attempts by using a cache

* Remove logging basic config

* Debug

* Docs

* Cleanup + docs

* Undo debug

* Fix existing tests

* Bump give-up count to 100

* Add iiprop configuration to prop changes

* Use kwarg syntax instead

* Update tests to use data ingester as a fixture

* Rename param, add prop adjustment tests

* Add test for breaking on max iteration attempts

* WIP documentation

* Add test for cache usage

* Pin iterations to a specific amount

* Update docs!

* Change header levels, update DAGs.md

* Add a note about possible infinite loops

* Delete unused operators folder

* Generalize description of date argument

* Make some methods static, improve logging

* Improve documentation & logging

Co-authored-by: Staci Mullins <[email protected]>

* Remove vestigial check, reset iteration count

* Update DAG docs

* Handle cases where the continue token isn't exactly `gaicontinue||`

* Improve tests to handle case where continue token differs

* Undo debug change

* Remove debug change

Co-authored-by: Staci Mullins <[email protected]>

* Fix spacing

Co-authored-by: Krystle Salazar <[email protected]>

* Update DAGs.md

---------

Co-authored-by: Krystle Salazar <[email protected]>
Co-authored-by: Staci Mullins <[email protected]>
Co-authored-by: Krystle Salazar <[email protected]>
@dhruvkb dhruvkb mentioned this issue Apr 17, 2024
Merged
8 tasks
@dhruvkb dhruvkb mentioned this issue May 7, 2024
Merged
8 tasks
@openverse-bot openverse-bot moved this to 🏗 In Progress in Openverse Backlog May 7, 2024
@dhruvkb dhruvkb changed the title Reduce production images size (API & Ingestion Server) Reduce size of API production image May 9, 2024
@dhruvkb
Copy link
Member

dhruvkb commented May 9, 2024

I have reduced the scope of this issue to only cover the API and not the ingestion server because

  • The ingestion server does not use PDM so optimising it will be a separate endeavour that cannot share effort with the same endeavour for API.
  • The ingestion server is already planned to be phased out so there is not much to gain by optimising it in its current deprecated form.

@openverse-bot openverse-bot moved this from 🏗 In Progress to ✅ Done in Openverse Backlog May 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dhruvkb dhruvkb

Labels
💻 aspect: code Concerns the software code in the repository 🧰 goal: internal improvement Improvement that benefits maintainers, not users 🟨 priority: medium Not blocking but should be addressed soon 🧱 stack: api Related to the Django API 🧱 stack: ingestion server Related to the ingestion/data refresh server
Projects
Openverse Backlog
Archived in project
Milestone
No milestone
2 participants
@krysal @dhruvkb