Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for building AWS EC2 AMIs #132

Merged
merged 2 commits into from
Jun 30, 2022
Merged

Add support for building AWS EC2 AMIs #132

merged 2 commits into from
Jun 30, 2022

Conversation

cevich
Copy link
Member

@cevich cevich commented May 24, 2022 •
edited
Loading

Certain podman and aardvark-dns CI testing requires capabilities
not available in GCE. Respectively, executing on bare-metal and on the
arm64 architecture.

  • Add the necessary Makefile targets, packer definitions, script,
    and CI updates to produce AWS images.
  • Add the gvproxy package to Fedora images to support testing the
    podman-machine command.
  • Increase the build timeout since AWS is considerably slower and more
    failure-prone than GCE.
  • Update documentation to reflect how both GCE and EC2 images are built.

N/B: It was decided that AWS functionality in CI is urgently needed. While PR is possibly at the "good enough" stage for merging, there are some significant TODO's introduced:

  1. Cirrus-CI is currently freaking out and adding a crap ton of Dockerfile code annotations that are irreverent. You can turn them off in the github UI under the ... menu for each file.
  2. A unique AMI ID is produced for fedora-aws Cache Image and fedora-netavark-aws Cache Image. These need to be manually extracted from the output logs (or manifest.json artifact). The image ID posted by github-actions bot only applies to GCE images.
  3. Building AWS EC2 AMI's is vastly more complex than GCE. There are a TON of gotchas and non-obvious configurations. I have attempted to document & comment most of these but it's not 100% complete.
  4. The AWS builds sometimes timeout/fail/break for strange reasons, occasionally leaving running instances behind 😠 Some hand-holding of the build tasks is needed, including cleanup of leftover instances. In particular the m5zn.metal instances - they cost $4/hour to leave running.
  5. This PR is producing updated (GCE) Ubuntu 2204 images, but CI is currently broken for a number of tests. See Cirrus: Update VM images w/ updated bats buildah#4074 [DO NOT MERGE - WIP] Cirrus: use Ubuntu 22.04 LTS podman#14397 and Cirrus: Use pre-installed bats podman#14719

@github-actions
Copy link

Cirrus CI build successful. Image ID c6478779013922816 ready for use.

@github-actions
Copy link

github-actions bot commented Jun 7, 2022

Cirrus CI build successful. Image ID c5885465554321408 ready for use.

@cevich cevich changed the title [WIP] [CI:DOCS] Add support for building AWS EC2 AMIs [WIP] Add support for building AWS EC2 AMIs Jun 8, 2022
@cevich cevich force-pushed the aws_support branch 9 times, most recently from 9b81859 to 6c5f1c5 Compare June 10, 2022 16:17
@github-actions
Copy link

Cirrus CI build successful. Image ID c5852063325224960 ready for use.

@cevich cevich force-pushed the aws_support branch 2 times, most recently from d12bead to 5f1a9a3 Compare June 13, 2022 20:42
@github-actions
Copy link

Cirrus CI build successful. Image ID c5513834986209280 ready for use.

@cevich cevich force-pushed the aws_support branch 10 times, most recently from 025a9a9 to d1d0df1 Compare June 14, 2022 22:30
@github-actions
Copy link

Cirrus CI build successful. Image ID c5335014693928960 ready for use.

@cevich cevich force-pushed the aws_support branch 2 times, most recently from cec8f78 to 8403259 Compare June 16, 2022 18:37
@cevich cevich force-pushed the aws_support branch 6 times, most recently from 8ce9dfb to 755ffc6 Compare June 28, 2022 19:50
@github-actions
Copy link

Cirrus CI build successful. Image ID c4884187411906560 ready for use.

@cevich cevich changed the title [WIP] Add support for building AWS EC2 AMIs Add support for building AWS EC2 AMIs Jun 28, 2022
@cevich cevich force-pushed the aws_support branch 5 times, most recently from fda863c to d27ff2c Compare June 29, 2022 18:50
@github-actions
Copy link

Cirrus CI build successful. Image ID c4787187991445504 ready for use.

cevich added 2 commits June 30, 2022 11:41
@cevich
Improve image builder VM & Container
50b4698 
* Update the Containerfile comment for accuracy.
* Simplify package installation script.
* Distinguish certain tasks as container or VM-only during setup.

Signed-off-by: Chris Evich <[email protected]>
@cevich
Add support for building AWS EC2 images
25ab726 
Certain podman and aardvark-dns CI testing requires capabilities
not available in GCE.  Respectively, executing on bare-metal and on the
arm64 architecture.

* Add the necessary `Makefile` targets, packer definitions, script,
  and CI updates to produce AWS images.
* Add the `gvproxy` package to Fedora images to support testing the
  `podman-machine` command.
* Increase the build timeout since AWS is considerably slower and more
  failure-prone than GCE.
* Update documentation to reflect how both GCE and EC2 images are built.

Signed-off-by: Chris Evich <[email protected]>
@cevich cevich requested review from edsantiago, baude and lsm5 June 30, 2022 15:51
@cevich
Copy link
Member Author

cevich commented Jun 30, 2022

@edsantiago @baude @lsm5 PTAL (I know it's a lot). I'd like to get this in today or early tomorrow since I'll be on PTO next week. Hopefully I've not made too many egregious gaffs 😕

@cevich
Copy link
Member Author

cevich commented Jun 30, 2022

This is a flake (likely bug in the Ubuntu 2204 repositories):

�[0;32m    ubuntu: The following information may help to resolve the situation:�[0m
�[0;32m    ubuntu:�[0m
�[0;32m    ubuntu: The following packages have unmet dependencies:�[0m
�[0;32m    ubuntu:  libapparmor-dev : Depends: libapparmor1 (= 3.0.4-2ubuntu2) but 3.0.4-2ubuntu2.1 is to be installed�[0m
�[0;32m    ubuntu: E: Unable to correct problems, you have held broken packages.�[0m
�[0;32m    ubuntu:     exit(100)�[0m
�[0;32m    ubuntu: Retry attempts exhausted�[0m

I'll re-run the ubuntu Cache Image task, but last time this happened it took a day for them to fix it 😢

@cevich
Copy link
Member Author

cevich commented Jun 30, 2022

Re: failing ubuntu repos. I opened #143 in case it becomes a problem, it will provide a way to easily bypass that build.

@cevich
Copy link
Member Author

cevich commented Jun 30, 2022

Note to me: Latest podman image ami-06a41d8a81ab56afa.

@github-actions
Copy link

Cirrus CI build successful. Image ID c6457378097856512 ready for use.

@lsm5 lsm5 mentioned this pull request Jun 30, 2022
Merged
lsm5
lsm5 approved these changes Jun 30, 2022
View reviewed changes
Copy link
Member

@lsm5 lsm5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, in my limited experience 😐

@cevich
Copy link
Member Author

cevich commented Jun 30, 2022

Thanks for taking a look @lsm5 I know it's a lot 😞

I'm just going to merge this then. Any problems can be addressed in future PRs.

@cevich cevich merged commit d2cf123 into containers:main Jun 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lsm5 lsm5 lsm5 approved these changes

@edsantiago edsantiago Awaiting requested review from edsantiago

@baude baude Awaiting requested review from baude

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@cevich @lsm5