Docker build with Ubuntu mirrors #16098

MiguelWeezardo · 2023-02-13T22:03:49Z

Description

In the last few weeks, we've seen failures in the job that builds the Docker image where it couldn't connect to ports.ubuntu.com or archive.ubuntu.com. Adding mirrors to apt should make it more resilient against such failures.

These two mirrors were chosen by getting a list of mirrors that include ports using this: https://askubuntu.com/questions/428698/are-there-alternative-repositories-to-ports-ubuntu-com-for-arm
and then checking which ones are the fastest using netselect, executed on an ec2 instance in us-east-2:

$ sudo netselect -v -s10 -t20 $(cat valid.txt )
Running netselect to choose 10 out of 14 addresses.      
...............................................................................................
    0 http://mirrors-ca.muzzy.us/ubuntu/
  173 http://mirrors.ocf.berkeley.edu/ubuntu-ports/
  417 http://mirror.kumi.systems/ubuntu-ports/
  621 http://jp.mirror.coganng.com/ubuntu-ports/
  716 http://mirror.deace.id/ubuntu/
 1134 http://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/
 1155 http://ubuntu-mirror.cloud.mu/ubuntu-ports/
 1281 http://mirror.misakamikoto.network/ubuntu-ports/
 1460 http://in.mirror.coganng.com/ubuntu-ports/
 1550 http://mirror.nishi.network/ubuntu-ports/

Out of these, the first one is invalid as it doesn't work over http, returning a 403.

Additional context and related issues

#15807 (comment)
https://github.com/trinodb/trino/actions/runs/4004028582/jobs/6872683430

This and #15840 both attempt to solve the same problem, but this solution is preferred as it doesn't rely on cache entries existing. #15840 can still be used to speed up Docker builds occasionally, but it isn't reliable enough to solve this problem.

Release notes

(x) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
( ) Release notes are required, with the following suggested text:

# Section
* Fix some things. ({issue}`issuenumber`)

MiguelWeezardo · 2023-02-13T22:05:13Z

Once I check all architectures build successfully, I will drop all commits except the "Add more mirrors" one

MiguelWeezardo · 2023-02-27T11:44:23Z

@hashhar Do you think this approach is worth considering?

MiguelWeezardo · 2023-03-08T15:22:24Z

@losipiuk Any chance of getting this merged?

nineinchnick · 2023-03-08T15:27:29Z

Test Docker image step ran in 15 minutes, and it's usually around 9 minutes. Can you rebase, so we can see if that was an outlier?

losipiuk · 2023-03-08T15:29:43Z

core/docker/build.sh

    docker build \
        "${WORK_DIR}" \
        --pull \
        --platform "linux/$arch" \
        -f Dockerfile \
        -t "${TAG_PREFIX}-$arch" \
        --build-arg "TRINO_VERSION=${TRINO_VERSION}"
+    rm -fr "${WORK_DIR}/default/apt/sources.list.d"


why not keep not put those mirror files in default/etc and have all of those for build of images for each architecture. Would it break sth? I think the non-matching mirros woudl not be used. Then you do not need to modify this script at all.

That was my initial approach, but if I remember correctly having all architectures defined at once made the apt update step longer since it tried to download package lists from all mirrors.

One performance improvement that might be worth checking out is having all mirrors defined while also using docker buildx to build multiple architectures in parallel. That way we'd have a single shared build context, and the parallel build gains might offset some of the cost of fetching all mirror package lists.

One performance improvement that might be worth checking out is having all mirrors defined while also using docker buildx to build multiple architectures in parallel. That way we'd have a single shared build context, and the parallel build gains might offset some of the cost of fetching all mirror package lists.

I have a WIP branch with this approach in branch michal/docker_ubuntu_mirrors_dockerx

MiguelWeezardo · 2023-03-08T17:26:52Z

Test Docker image step ran in 15 minutes, and it's usually around 9 minutes. Can you rebase, so we can see if that was an outlier?

It's down to 11m23s in the second execution, still a little more than the usual.

losipiuk · 2023-03-15T15:59:50Z

core/docker/default/apt/sources.list.d/mirrors-amd64.sources

+Enabled: yes
+Types: deb
+URIs: https://mirrors.ocf.berkeley.edu/ubuntu/ https://mirror.kumi.systems/ubuntu/
+Suites: jammy jammy-updates jammy-backports jammy-security


Will it always be jammy. Aren't they bumping debian version when there is new one in eclipse-temurin:17-jdk?

I think once eclipse-temurin:17-jdk updates their base image we should update the mirror descriptors manually.
Other than pinning ourselves to a specific hash of eclipse-temurin we have no control over the distro version they use.
They might even switch from Ubuntu to something which doesn't use APT.

The point is we will not notice that moment right?
Or will we get some error when they update?

The error might not directly mention a mismatch of Ubuntu versions, but I expect it might look like a huge update with APT trying to pull every installed package twice. I think it will either timeout the job or take long enough for us to notice.

I also wonder if using a tag instead of a hash for the base image is a good idea. I think it's better to use immutable hashes for build repeatability.

We can probably build these mirrorlists dynamically during image build by something like:

source /etc/os-release # the above exports a UBUNTU_CODENAME variable sed -i 's/UBUNTU_CODENAME/${UBUNTU_CODENAME}/g' /etc/apt/sources.list.d/*

And in the mirrorlist we commit to repo we replace occurences of actual release name with UBUNTU_CODENAME.

I also wonder if using a tag instead of a hash for the base image is a good idea

I think the point of not using specific version it to always get the most recent security patches in place

We can probably build these mirrorlists dynamically during image build by something like

This looks nice

I also wonder if using a tag instead of a hash for the base image is a good idea

I think the point of not using specific version it to always get the most recent security patches in place

I understand that was the intention, but it seems risky because a build of the same trino git hash might work today and fail tomorrow just because the tag was updated in between. I think a more repeatable approach would be to pin ourselves to a specific docker image hash and use dependabot to keep it up to date with the tag. In this case every docker image change would correspond with a commit in trino that we can roll back if it breaks something.

I do not think repeatablitity of build is a very important thing we want to address here. Having subsequent builds use safe docker as a base, without manual intervention to bump base image version is more important imo.

losipiuk

LGTM. Please squash the fixup

Future-proof against Ubuntu codename update This approach works as long the following assumptions hold: - the location of `/etc/os-release` file does NOT change - the name of the UBUNTU_CODENAME environment variable does NOT change - `eclipse-temurin` still uses Ubuntu as it's base

The recent merge of trinodb#16098 increased build robustness in the event of main Ubuntu repo failures, at the cost of 2-3 minutes for downloading Ubuntu mirror indexes. It turns out the 45 minute timeout was very close to the actual time needed to perform the entire job, and extending it is necessary to avoid cancelling the job.

cla-bot bot added the cla-signed label Feb 13, 2023

This was referenced Feb 13, 2023

Docker ubuntu mirrors MiguelWeezardo/trino#14

Closed

Use apt mirrors when building the Docker image #16091

Closed

MiguelWeezardo force-pushed the michal/docker_ubuntu_mirrors branch from a063086 to 9075bc8 Compare February 13, 2023 23:38

MiguelWeezardo requested review from nineinchnick, hashhar and electrum February 13, 2023 23:38

MiguelWeezardo marked this pull request as ready for review February 13, 2023 23:40

nineinchnick approved these changes Feb 14, 2023

View reviewed changes

MiguelWeezardo requested a review from findepi February 21, 2023 22:21

MiguelWeezardo added the dx Issues or pull requests related to developer experience label Feb 23, 2023

MiguelWeezardo requested review from losipiuk and martint and removed request for findepi and electrum March 8, 2023 15:20

losipiuk reviewed Mar 8, 2023

View reviewed changes

kokosing force-pushed the master branch from 3f05134 to 58d6356 Compare March 14, 2023 11:34

MiguelWeezardo force-pushed the michal/docker_ubuntu_mirrors branch from f093312 to 89f9d6e Compare March 14, 2023 13:10

losipiuk reviewed Mar 15, 2023

View reviewed changes

hashhar approved these changes Mar 22, 2023

View reviewed changes

losipiuk approved these changes Mar 22, 2023

View reviewed changes

MiguelWeezardo force-pushed the michal/docker_ubuntu_mirrors branch 2 times, most recently from 2cf3887 to 8e5c991 Compare March 22, 2023 14:21

MiguelWeezardo added 2 commits March 22, 2023 15:22

Bring Docker build.sh help message in-line with reality

a8a7912

MiguelWeezardo force-pushed the michal/docker_ubuntu_mirrors branch from 8e5c991 to a8a7912 Compare March 22, 2023 14:22

losipiuk merged commit 1d400c3 into trinodb:master Mar 23, 2023

github-actions bot added this to the 411 milestone Mar 23, 2023

MiguelWeezardo deleted the michal/docker_ubuntu_mirrors branch March 24, 2023 12:48

colebow mentioned this pull request Mar 29, 2023

Add Trino 411 release notes #16552

Merged

MiguelWeezardo mentioned this pull request Apr 3, 2023

Extend maven-checks job timeout by additional 5 minutes #16863

Closed

nineinchnick mentioned this pull request Apr 4, 2023

maven-checks job fails if Ubuntu repositories go down #15992

Closed

findepi mentioned this pull request Apr 12, 2023

Revert "Add more mirrors" #16973

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker build with Ubuntu mirrors #16098

Docker build with Ubuntu mirrors #16098

MiguelWeezardo commented Feb 13, 2023 •

edited

Loading

MiguelWeezardo commented Feb 13, 2023

MiguelWeezardo commented Feb 27, 2023

MiguelWeezardo commented Mar 8, 2023 •

edited

Loading

nineinchnick commented Mar 8, 2023

losipiuk Mar 8, 2023

MiguelWeezardo Mar 8, 2023

MiguelWeezardo Mar 8, 2023

MiguelWeezardo Mar 14, 2023

MiguelWeezardo commented Mar 8, 2023

losipiuk Mar 15, 2023

MiguelWeezardo Mar 17, 2023

losipiuk Mar 17, 2023

MiguelWeezardo Mar 19, 2023 •

edited

Loading

MiguelWeezardo Mar 19, 2023

hashhar Mar 19, 2023 •

edited

Loading

losipiuk Mar 19, 2023

MiguelWeezardo Mar 23, 2023

losipiuk Mar 23, 2023

losipiuk left a comment

Docker build with Ubuntu mirrors #16098

Docker build with Ubuntu mirrors #16098

Conversation

MiguelWeezardo commented Feb 13, 2023 • edited Loading

Description

Additional context and related issues

Release notes

MiguelWeezardo commented Feb 13, 2023

MiguelWeezardo commented Feb 27, 2023

MiguelWeezardo commented Mar 8, 2023 • edited Loading

nineinchnick commented Mar 8, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MiguelWeezardo commented Mar 8, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MiguelWeezardo Mar 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hashhar Mar 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

losipiuk left a comment

Choose a reason for hiding this comment

MiguelWeezardo commented Feb 13, 2023 •

edited

Loading

MiguelWeezardo commented Mar 8, 2023 •

edited

Loading

MiguelWeezardo Mar 19, 2023 •

edited

Loading

hashhar Mar 19, 2023 •

edited

Loading