Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate how to decouple AIP from Conda, and IF it brings improvements/flexibility #40

Closed
verovaleros opened this issue Oct 23, 2024 · 4 comments · Fixed by #42
Closed
Assignees
Labels
enhancement New feature or request
Milestone

Comments

@verovaleros
Copy link
Member

verovaleros commented Oct 23, 2024

The current AIP docker image is 1.92GB, which begs the question 'is all of this needed?'.

A big component on this docker size can be related to the use of Conda to manage the dependencies and requirements.

The task would be to investigate if this is the case, also if it's possible to not use conda and use virtual environments instead and how much would the size reduction would be.

All in all, the key question is: is this change worth it?

@verovaleros verovaleros added the enhancement New feature or request label Oct 23, 2024
@verovaleros verovaleros self-assigned this Oct 24, 2024
@verovaleros verovaleros moved this from 🆕 New to 🏗 In progress in Stratosphere AIP Oct 24, 2024
@verovaleros verovaleros added this to the v3.0.0 milestone Oct 24, 2024
@verovaleros
Copy link
Member Author

There are three key files that need modifying to decouple AIP from the use of Conda:

  • etc/docker/Dockerfile (same for M1)
  • etc/docker/entrypoint.sh
  • Additionally, the introduction of a .dockerignore file can help reduce even further the size of the container.
  • There may also be some packages in requirements.txt that may not be needed, such as ipython.

Ongoing tests show that with these are some other modifications the docker size can be reduced cca 70%.

@verovaleros
Copy link
Member Author

A complementary approach would be to use multi-stage builds. This would allow to have a base image that can be use for development and a reduced smaller size build with only the required packages for production.

@verovaleros
Copy link
Member Author

Ongoing tests show that a multi-stage build is what works best, using python-vevn instead of Conda, and removing unwanted requirements. Final size shrinks from 1.92GB to 371M.

@verovaleros
Copy link
Member Author

Proposed changes can be found currently in https://github.com/stratosphereips/AIP/tree/40-decouple-aip-from-conda

@verovaleros verovaleros moved this from 🏗 In progress to 👀 In review in Stratosphere AIP Oct 24, 2024
@verovaleros verovaleros mentioned this issue Oct 26, 2024
4 tasks
@verovaleros verovaleros linked a pull request Oct 26, 2024 that will close this issue
4 tasks
@verovaleros verovaleros moved this from 👀 In review to ✅ Done in Stratosphere AIP Oct 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

1 participant