Reduction of image size with multi-staging Dockerfile

[aulasau]

Hey everyone.
Currently, I'm trying to build a micromamba-based image for my Python app. And I have some problems with the image size.
There are plenty of libraries that I use in the app. For a simple Dockerfile with one stage, I've got an image with a 1.35 GB size.

So I decided to optimize this image. Instead of installing all libraries, I'm copying /opt/conda/lib/python3.11/site-packages(all installed lib files are here) to the second stage of the image. And that helps a little bit, right now, the image size is 920 MB.

Is there a better way to do this? And how can I reduce image even more? Personally I was shocked that libraries can take up so much space.
Maybe it's, in general, a bad idea to move site-packages to the next stage, what do you think?
I'm searching for some guidance.

Here is my Dockerfile

FROM mambaorg/micromamba:1.5.6 as base
WORKDIR /app

# pip requirements are already on this file, it's a cool feature by new conda versions
COPY _container_files/from_history_mamba_env.yaml  from_history_mamba_env.yaml

RUN --mount=type=cache,target=/var/cache/apt  \
    micromamba install -y -n base -f from_history_mamba_env.yaml  && \
    micromamba clean --all --force-pkgs-dirs --yes && \
    rm -f from_history_mamba_env.yaml

ARG MAMBA_DOCKERFILE_ACTIVATE=1
RUN python -m spacy download ru_core_news_sm && \
    python -m nltk.downloader stopwords -d '/opt/conda/nltk_data'


######### SECOND STAGE #########
FROM mambaorg/micromamba:1.5.6 as filter_bot_container

# micromamba has no python on board so we should install it
RUN --mount=type=cache,target=/var/cache/apt  \
    micromamba install -y -n base -c conda-forge python==3.11 && \
    micromamba clean --all --force-pkgs-dirs --yes

ARG MAMBA_DOCKERFILE_ACTIVATE=1
WORKDIR /app

COPY --from=base /opt/conda/lib/python3.11/site-packages /python_packages
COPY --from=base /opt/conda/nltk_data /opt/conda/nltk_data

USER root
# some necessary dependencies
RUN apt-get update && apt-get install --yes libatlas-base-dev libpq5
USER mambauser

ENV PYTHONPATH=/python_packages:$PYTHONPATH

COPY .. .

CMD python -m main

And here is the head of the content size list in python_packages (or site-packages on 1st stage) dir:

(base) root@9730e001ab04:/# du -h python_packages/ | sort -rh
627M	python_packages/
86M	python_packages/lingua/language-models
86M	python_packages/lingua
76M	python_packages/pandas
71M	python_packages/scipy
52M	python_packages/sklearn
45M	python_packages/ru_core_news_sm/ru_core_news_sm-3.7.0
45M	python_packages/ru_core_news_sm
39M	python_packages/numpy
38M	python_packages/pandas/tests
33M	python_packages/blis
32M	python_packages/ru_core_news_sm/ru_core_news_sm-3.7.0/vocab
31M	python_packages/spacy
25M	python_packages/pyrogram
22M	python_packages/pandas/_libs
20M	python_packages/pyrogram/raw
19M	python_packages/sqlalchemy
18M	python_packages/numpy/core
17M	python_packages/pip
16M	python_packages/pymorphy3_dicts_ru/data
16M	python_packages/pymorphy3_dicts_ru
14M	python_packages/pip/_vendor
14M	python_packages/nltk
12M	python_packages/pandas/core
11M	python_packages/scipy/stats
11M	python_packages/scipy/sparse
11M	python_packages/pyrogram/raw/types
9.1M	python_packages/cryptography
9.0M	python_packages/pygments
8.9M	python_packages/scipy/linalg
8.6M	python_packages/cryptography/hazmat
8.5M	python_packages/scipy/optimize
7.7M	python_packages/pygments/lexers
7.7M	python_packages/cryptography/hazmat/bindings
7.4M	python_packages/scipy/special
7.2M	python_packages/spacy/pipeline
6.8M	python_packages/numpy/core/tests
6.6M	python_packages/sklearn/utils
6.3M	python_packages/sklearn/metrics
...

As you can see, pandas -- 76M, scipy -- 71M, sklearn -- 52M.

And I read this blogpost that was mentioned in docs. The author suggests removing all these kinds of stuff like tests dir in libs manually.

Is there no way to do some magic trick for a better reduction mechanism?

[wholtz]

Hi @aulasau . It is unclear to me how a multi-stage build is helping you here. I guess you avoid having from_history_mamba_env.yaml in one of your layers, but this optimization seems like overkill for a file that small.

Have you tried using one of our alpine based images? That would be an easy way to save a few MBs. You are missing a && rm -rf /var/lib/apt/lists/* on the end of your RUN apt-get ... That will save you a little bit as well.

I imagine with a couple of find ... -delete commands you could trim this down by removing test/tests directories, but those type of optimizations cannot be universally applied.

[aulasau]

Yeah, I thought about alpine image. But on the documentation, there is a warning about additional testing. And I decided to keep it simpler with the main Docker branch.
Working right now on the deletion of tests and similar redundant files.

So I guess there is no miracle cure to make it vastly smaller. I had some hope for possible dirty hacks with package binaries or smth like that.
Python has never been good at optimizations of this kind :(

Anyway, thank you for your response!