Python dependencies differ between pre-built and built-from-source Docker images

[arman-ddl]

Description

For the same DHC version, e.g. 0.20.1, Python dependencies differ between pre-built Docker images via https://deephaven.io/core/docs/tutorials/quickstart/ and built-from-source images via https://deephaven.io/core/docs/how-to-guides/launch-build/.

If a DHC user starts off w/ a pre-built container, and then moves to building from source, the difference in Python dependencies can cause issues. For example, if one install pathway for one reason or another resolves numpy to 1.24.1 instead of 1.23.5, then that install will break and not the other (see #3422).

Additionally, systems that run DHC workers natively (like DnD) need to know which sets of Python dependencies are supported and tested against, so that they can install those and have confidence in stability.

In addition to stability, if a user migrated from pure DHC containers to a system that runs DHC workers natively, they (understandably) would expect a similar experience (and therefore dependencies). This cannot be guaranteed if there isn't a central source of supported Python dependencies to begin with.

Steps to reproduce

1. Install DHC 0.20.1 via https://deephaven.io/core/docs/tutorials/quickstart/
2. Install DHC 0.20.1 via https://deephaven.io/core/docs/how-to-guides/launch-build/
3. Run pip freeze via Docker terminal on both containers

Expected results

Equivalent Python dependencies (at least as much is feasibly possible).

Actual results

Different Python dependencies.

Pre-built image:

deephaven-core==0.21.0
deephaven-plugin==0.3.0
jedi==0.18.2
jpy==0.13.0
llvmlite==0.39.1
numba==0.56.4
numpy==1.23.5
pandas==1.5.3
parso==0.8.3
pyarrow==11.0.0
python-dateutil==2.8.2
pytz==2022.7.1
six==1.16.0

Built-from-source image:

deephaven-core @ file:///wheels/deephaven_core-0.20.1-py3-none-any.whl
deephaven-plugin==0.3.0
importlib-metadata==5.2.0
java-utilities==0.2.0
jedi==0.18.2
jpy==0.13.0
llvmlite==0.39.1
numba==0.56.4
numpy==1.21.6
pandas==1.3.5
parso==0.8.3
pkg_resources==0.0.0
pyarrow==9.0.0
python-dateutil==2.8.2
pytz==2022.7
six==1.16.0
typing_extensions==4.4.0
zipp==3.11.0

Additional details and attachments

Versions

• Deephaven: 0.20.1
• OS: macOS Monterey 12.6.2
• Browser: Chrome 109.0.5414.119
• Docker: 4.16.2 (95914)

[devinrsmith]

@margaretkennedy , the guide https://deephaven.io/core/docs/how-to-guides/launch-build/ is out-of-date in terms of relevance. It's really only meant for developers, and any images produced in this fashion should be for development purposes only. The production docker images are built https://github.com/deephaven/deephaven-server-docker. I think we may have tickets saying we need to update this.

[margaretkennedy]

@jjbrosnan I'll need your help on updating https://deephaven.io/core/docs/how-to-guides/launch-build/