Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: secure hermetic-build docker image #3196

Draft
wants to merge 42 commits into
base: main
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
5f5e0bd
chore: secure hermetic-build docker image
diegomarquezp Sep 12, 2024
15a19c5
fix python preparation
diegomarquezp Sep 12, 2024
ff0fc1a
apine image
diegomarquezp Sep 12, 2024
9a25dda
update reqs
diegomarquezp Sep 12, 2024
66d7f45
do not use BSD flags for `rm`
diegomarquezp Sep 12, 2024
d12877c
fixes to docker image
diegomarquezp Sep 16, 2024
4885459
fix reference to global site-packages
diegomarquezp Sep 16, 2024
b97c209
fix permissions
diegomarquezp Sep 17, 2024
3f94f0b
use SHAs directy
diegomarquezp Sep 17, 2024
08fe2cd
reduce image size
diegomarquezp Sep 17, 2024
539922a
use cloud build action
diegomarquezp Sep 17, 2024
1a34741
Update .cloudbuild/library_generation/library_generation.Dockerfile
diegomarquezp Sep 17, 2024
43f1ac0
update DEVELOPMENT.md
diegomarquezp Sep 17, 2024
1dc3629
use buildkit
diegomarquezp Sep 17, 2024
90dafe3
Merge remote-tracking branch 'origin/secure-hermetic-build-docker-ima…
diegomarquezp Sep 17, 2024
3f3deec
Merge branch 'main' into secure-hermetic-build-docker-image
diegomarquezp Sep 17, 2024
b730a4b
do not build image in integration test
diegomarquezp Sep 17, 2024
db2e8e7
remove wrong dependency
diegomarquezp Sep 17, 2024
fb98222
Merge remote-tracking branch 'origin/secure-hermetic-build-docker-ima…
diegomarquezp Sep 17, 2024
32fffb7
comment unwanted airlock repo
diegomarquezp Sep 17, 2024
51544a3
Update library_generation/DEVELOPMENT.md
diegomarquezp Sep 17, 2024
e9a5df4
remove redundant skipTests
diegomarquezp Sep 17, 2024
8f0ac9b
Merge remote-tracking branch 'origin/secure-hermetic-build-docker-ima…
diegomarquezp Sep 17, 2024
2c35db2
add links to confirm availablity of missing python packages
diegomarquezp Sep 17, 2024
d09124f
save point: owl-bot cli standalone and python repo using airlock
diegomarquezp Sep 17, 2024
34835a5
remove standalone executable for owlbot
diegomarquezp Sep 17, 2024
a3490e2
remove airlock registry for python
diegomarquezp Sep 17, 2024
efeff60
fix list of glibc shared objects
diegomarquezp Sep 17, 2024
ae0f349
add instructions for setting up airlock docker registry
diegomarquezp Sep 18, 2024
9177111
fix python setup
diegomarquezp Sep 18, 2024
8797e30
Revert "remove airlock registry for python"
diegomarquezp Sep 18, 2024
aa0fe85
fix integration test yaml
diegomarquezp Sep 18, 2024
bce332d
add hermetic-library-generation cloud build job
diegomarquezp Sep 18, 2024
d0a6da1
use CLOUD_LOGGING_ONLY to allow SA runner
diegomarquezp Sep 18, 2024
11a81d3
fix volumes
diegomarquezp Sep 18, 2024
565afda
retrigger build
diegomarquezp Sep 18, 2024
dd275ac
retrigger build
diegomarquezp Sep 18, 2024
2e5067d
retrigger build
diegomarquezp Sep 19, 2024
32ac053
retrigger build
diegomarquezp Sep 19, 2024
054bf42
retrigger build
diegomarquezp Sep 19, 2024
87a2e09
add python volumes
diegomarquezp Sep 19, 2024
42f331e
use cloud-sdk image for steps
diegomarquezp Sep 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
use SHAs directy
diegomarquezp committed Sep 17, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
commit 3f94f0b365627eaf626f7834afe991c2816c9f57
15 changes: 10 additions & 5 deletions .cloudbuild/library_generation/library_generation.Dockerfile
Original file line number Diff line number Diff line change
@@ -12,7 +12,9 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# Creates the owl-bot binary (no node runtime needed)
FROM us-docker.pkg.dev/artifact-foundry-prod/docker-3p-trusted/node:22.1-alpine as owlbot-cli-build

# node:22.1-alpine
FROM us-docker.pkg.dev/artifact-foundry-prod/docker-3p-trusted/node@sha256:487dc5d5122d578e13f2231aa4ac0f63068becd921099c4c677c850df93bede8 as owlbot-cli-build
JoeWang1127 marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do we plan to update these base images? It might be fine to not update this one, but for the Java and Python one, we may want to update the Maven/JDK/Python version regularly.

ARG OWLBOT_CLI_COMMITTISH=ac84fa5c423a0069bbce3d2d869c9730c8fdf550

# install tools
@@ -55,7 +57,8 @@ RUN npx postject owl-bot-bin NODE_SEA_BLOB sea-prep.blob \
RUN cp ./owl-bot-bin /owl-bot-bin

# Creates the generator jar
FROM us-docker.pkg.dev/artifact-foundry-prod/docker-3p-trusted/maven:3.8.6-openjdk-11-slim as ggj-build
# maven:3.8.6-openjdk-11-slim
FROM us-docker.pkg.dev/artifact-foundry-prod/docker-3p-trusted/maven@sha256:2cb7c73ba2fd0f7ae64cfabd99180030ec85841a1197b4ae821d21836cb0aa3b as ggj-build

WORKDIR /sdk-platform-java
COPY . .
@@ -72,7 +75,8 @@ RUN --mount=type=cache,target=/root/.m2 cp "/root/.m2/repository/com/google/api/
"/gapic-generator-java.jar"

# Builds the python scripts in library_generation
FROM us-docker.pkg.dev/artifact-foundry-prod/docker-3p-trusted/python:3.11-alpine as python-scripts-build
# python:3.11-alpine
FROM us-docker.pkg.dev/artifact-foundry-prod/docker-3p-trusted/python@sha256:0b5ed25d3cc27cd35c7b0352bac8ef2ebc8dd3da72a0c03caaf4eb15d9ec827a as python-scripts-build
JoeWang1127 marked this conversation as resolved.
Show resolved Hide resolved

# This will use GOOGLE_APPLICATION_CREDENTIALS if passed in docker build command.
# If not passed will leave it unset to support GCE Metadata in CI builds
@@ -102,7 +106,8 @@ RUN python -m pip install --target /usr/local/lib/python3.11 .
# Final image. Installs the rest of the dependencies and gets the binaries
# from the previous stages. We use the node base image for it to be compatible
# with the standalone binary owl-bot compiled in the previous stage
FROM us-docker.pkg.dev/artifact-foundry-prod/docker-3p-trusted/node:22.1-alpine as final
# node:22.1-alpine
FROM us-docker.pkg.dev/artifact-foundry-prod/docker-3p-trusted/node@sha256:487dc5d5122d578e13f2231aa4ac0f63068becd921099c4c677c850df93bede8 as final
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why can't we use a python base image?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The standalone owlbot seems to need a few runtime libraries (.so) in the linux environment that are available in the node image by default (besides the node runtime). The python image doesn't have them.
If we want to have a python image as base, I can try to use a python base and install node or whichever libraries the executable needs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From discussion with @blakeli0 and @JoeWang1127, let's use a vanilla Alpine based image.


ARG PROTOC_VERSION=25.4
ARG GRPC_VERSION=1.66.0
@@ -129,7 +134,7 @@ SHELL [ "/bin/bash", "-c" ]
# grpc plugin).
# Alpine, by default, only supports musl-based binaries, and there is no public
# downloadable distrubution of the grpc that is Alpine (musl) compatible.
# This is one of the recommended approaches
# This is one of the recommended approaches to ensure glibc-compatibility
# as per https://wiki.alpinelinux.org/wiki/Running_glibc_programs
WORKDIR /home
RUN git clone https://gitlab.com/manoel-linux1/GlibMus-HQ.git