AWS boot testing for all supported image types #217

achilleas-k · 2023-10-20T15:16:58Z

Enables boot testing on AWS for all supported image types.

The following changes were made to image definitions and tests:

The edge-ami image uses the kernel options that are applied to all other EC2/AMI images.
The command line flags for the boot-aws command were modified.
The ec2-sap image for RHEL 8.4 is built with the lm_sensors.service disabled (in CI) because of a known issue.

supakeen · 2023-10-20T16:58:05Z

Awesome :)

achilleas-k · 2023-10-20T18:42:04Z

EC2 images are created compressed (xz) and should be decompressed before pushing to boot.

thozza

The changes look good, but the failing EDGE-ami boot test is IMHO caused by the fact that EDGE images don't have cloud-init installed by default... 🤔 Thus the ssh key is never configured as authorized in the VM...

achilleas-k · 2023-10-24T10:15:42Z

The changes look good, but the failing EDGE-ami boot test is IMHO caused by the fact that EDGE images don't have cloud-init installed by default... 🤔 Thus the ssh key is never configured as authorized in the VM...

I've been thinking about how to test these (while working on other stuff) and I guess the only way is to add a user to the image BP. I really wanted to have an empty baseline test for everything, but I guess there's no way to automate initial-setup (at least not the way we configure it?). We'll need to do the same with the installers as well anyway.

achilleas-k · 2023-10-25T00:16:07Z

It worked! Kinda https://gitlab.com/redhat/services/products/image-builder/ci/images/-/jobs/5366641949

test/configs/edge-ostree-pull-user.json

achilleas-k · 2023-10-25T21:40:38Z

Alright, we are booting and... failing on ARM for some reason:

● systemd-firstboot.service                                                                 loaded failed failed    First Boot Wizard

EDIT: this is good right? we're catching bugs 🐛

cgwalters · 2023-10-26T00:48:05Z

FWIW in coreos and derivatives https://github.com/coreos/fedora-coreos-config/blob/6b1b56baab105ffac5cb290d5083f75e23437d2e/manifests/ignition-and-ostree.yaml#L25
(systemd-firstboot also makes little sense in a system that uses cloud-init)

achilleas-k · 2023-10-26T10:26:14Z

FWIW in coreos and derivatives https://github.com/coreos/fedora-coreos-config/blob/6b1b56baab105ffac5cb290d5083f75e23437d2e/manifests/ignition-and-ostree.yaml#L25 (systemd-firstboot also makes little sense in a system that uses cloud-init)

We don't add cloud-init to the edge-ami. The image was added by the Edge team so I'm not sure if that was an oversight or deliberate. Maybe @7flying can shed some light?

It's also strange that it only fails on aarch64 (and consistently).
I haven't gotten around to checking the log for the failure yet.

7flying · 2023-10-26T14:22:36Z

FWIW in coreos and derivatives https://github.com/coreos/fedora-coreos-config/blob/6b1b56baab105ffac5cb290d5083f75e23437d2e/manifests/ignition-and-ostree.yaml#L25 (systemd-firstboot also makes little sense in a system that uses cloud-init)

We don't add cloud-init to the edge-ami. The image was added by the Edge team so I'm not sure if that was an oversight or deliberate. Maybe @7flying can shed some light?

It's also strange that it only fails on aarch64 (and consistently). I haven't gotten around to checking the log for the failure yet.

We deliberately did not add it since that would mean adding the RPM to every image using the same edge-commit and that did not make sense. Our docs recommend to add cloud-init in the blueprint of the edge-ami instead.

achilleas-k · 2023-10-26T18:31:35Z

Our docs recommend to add cloud-init in the blueprint of the edge-ami instead.

But we can't add packages just to the ami. The build just deploys the commit and creates the image and doesn't support layering packages on top. Is that something we should consider?

There's also the (IMO better) alternative that I want to do eventually but might take some time: Build the image directly without pulling in an ostree commit and export both the image and the commit tarball. That way we could build an edge-ami and get out both an edge-commit.tar that's specific to the AMI and the raw image for EC2.

achilleas-k · 2023-10-26T18:37:25Z

Also of course, adding it would require it being in the commit, I don't know why I thought it would be included in the ami. But for our test, we can build an ostree commit that has it to test against the edge-ami specifically.

@7flying do you know if adding cloud-init to an edge/iot-commit conflict will conflict with initial-setup?

7flying · 2023-10-27T07:11:14Z

Our docs recommend to add cloud-init in the blueprint of the edge-ami instead.

But we can't add packages just to the ami. The build just deploys the commit and creates the image and doesn't support layering packages on top. Is that something we should consider?

Good point, I meant to say, "to the commit that is just going to be used for the AMI". But I'll double check with docs to make that point clear.

Yes, ideally we would want a single edge commit to have all the packages that all the possible edge images would need, and then, depending on the image that you want to build some of them would be "disabled/removed", or we could use layers, but regardless, there shouldn't be more configuration burden for the user, if possible.

There's also the (IMO better) alternative that I want to do eventually but might take some time: Build the image directly without pulling in an ostree commit and export both the image and the commit tarball. That way we could build an edge-ami and get out both an edge-commit.tar that's specific to the AMI and the raw image for EC2.

Yes, having a single command-line spell that builds it all in one go would be the best solution.

7flying · 2023-10-27T07:15:50Z

Also of course, adding it would require it being in the commit, I don't know why I thought it would be included in the ami. But for our test, we can build an ostree commit that has it to test against the edge-ami specifically.

@7flying do you know if adding cloud-init to an edge/iot-commit conflict will conflict with initial-setup?

@achilleas-k no sorry, I did not test that

achilleas-k · 2023-10-27T13:05:26Z

Also of course, adding it would require it being in the commit, I don't know why I thought it would be included in the ami. But for our test, we can build an ostree commit that has it to test against the edge-ami specifically.
@7flying do you know if adding cloud-init to an edge/iot-commit conflict will conflict with initial-setup?

@achilleas-k no sorry, I did not test that

I think I'll add this as part of our test configs: Build an edge commit with cloud-init installed and use it as the base image for an edge-ami and then boot test it with cloud-init.

But I'll leave that for a follow-up PR.
For now, here, I'll see if I can figure out what's making the firstboot service fail to start (could be nothing, but it could be a side effect of some bug) and take it from there.

Thanks for the input!

achilleas-k · 2023-10-30T21:10:57Z

I modified the edge-ami image type to include all the kernel options that we add to other EC2/AMI image types.

achilleas-k · 2023-10-30T22:39:24Z

One failure left: https://gitlab.com/redhat/services/products/image-builder/ci/images/-/pipelines/1055416885 (ec2-sap on RHEL 8.4).

achilleas-k · 2023-10-30T23:21:23Z

One failure left: https://gitlab.com/redhat/services/products/image-builder/ci/images/-/pipelines/1055416885 (ec2-sap on RHEL 8.4).

Most likely it's an issue with just being slower to start:

$ sudo systemd-analyze
Startup finished in 2.601s (kernel) + 9.615s (initrd) + 1min 33.448s (userspace) = 1min 45.665s
multi-user.target reached after 38.479s in userspace

The keyscan, which we use to wait for the system, quits after 10 tries with 10 second intervals.

EDIT: Though it also failed to start lm_sensors.service, so the system is in any case degraded.

Enables boot testing on AWS for all supported image types.

If an image is an xz compressed file, unxz it before boot testing and delete the uncompressed image when done to avoid pushing it to the build cache.

jq . --sort-keys the whole file for cleaner edits.

The config isn't used by any image type currently. The private SSH key is set in the GitLab CI variables.

Add two new configs for image types that pull an ostree commit and add a user customization to the configuration. Same key that was added to the user-customizations config. The new configs are used for edge-ami so we can boot test with a predefined user and SSH key.

Doesn't make sense for our boot testing binary.

If the CI_PRIV_SSH_KEY env var exists, use that instead of generating a new key pair. This is used for boot testing images that have a key configured in the build config, for cases where user creation isn't possible at first boot.

Update the test README to mention that: - Images are boot tested when supported. - Images are uploaded to the S3 bucket as well as the manifest, not just the build info. A minor change is also made to both the test/README.md and the HACKING.md about building the `cmd/build` binary without sudo and putting it in `./bin/build` before running it with sudo.

Add instructions for how to use the boot-aws command.

In the boot test script, we don't need to use sudo to run systemctl is-system-running For the edge-ami boot test, we can't set passwordless sudo right now, so if it's not required, we can just run it as a regular user.

Environments don't yet support kernel arguments so doing this now would make the current patch set much bigger than we want it to be. Add a note to do it another time.

Add kernel options to the image type for edge images (raw, ami, and vsphere) and handle them in the edgeRawImage() function. Move the 'modprobe.blacklist=vc4' option, which was used to be set in the image function, to the image types instead.

achilleas-k · 2023-10-31T12:30:38Z

One failure left: https://gitlab.com/redhat/services/products/image-builder/ci/images/-/pipelines/1055416885 (ec2-sap on RHEL 8.4).

Most likely it's an issue with just being slower to start:
$ sudo systemd-analyze
Startup finished in 2.601s (kernel) + 9.615s (initrd) + 1min 33.448s (userspace) = 1min 45.665s
multi-user.target reached after 38.479s in userspace
The keyscan, which we use to wait for the system, quits after 10 tries with 10 second intervals.

EDIT: Though it also failed to start lm_sensors.service, so the system is in any case degraded.

So the timeout increase worked and the test script ran but like I saw in my local test, the lm_sensors service didn't start:
https://gitlab.com/redhat/services/products/image-builder/ci/images/-/jobs/5420205170

This is only an issue on RHEL 8.4 SAP.

EDIT: Known issue: https://issues.redhat.com/browse/RHEL-14096

- Create an empty config specifically for ec2-sap for all distros except RHEL 8.4. - Create a config for ec2-sap for RHEL 8.4 only that disables lm_sensors.service Related: RHEL-14096

thozza

LGTM

supakeen · 2023-10-31T16:51:22Z

Absolutely love it.

achilleas-k changed the title ~~test: AWS boot testing for all supported image types~~ AWS boot testing for all supported image types Oct 20, 2023

achilleas-k requested review from thozza and supakeen October 20, 2023 15:18

achilleas-k mentioned this pull request Oct 20, 2023

Boot test images in CI #173

Open

achilleas-k force-pushed the boot-test/ec2 branch from 530aa63 to 97ae07c Compare October 20, 2023 16:18

supakeen previously approved these changes Oct 20, 2023

View reviewed changes

supakeen enabled auto-merge October 20, 2023 16:57

achilleas-k dismissed supakeen’s stale review via 2aa2fbd October 20, 2023 20:58

achilleas-k force-pushed the boot-test/ec2 branch 2 times, most recently from 2aa2fbd to 9cc47a3 Compare October 20, 2023 21:36

thozza reviewed Oct 24, 2023

View reviewed changes

achilleas-k force-pushed the boot-test/ec2 branch 2 times, most recently from 031121c to 0717f77 Compare October 24, 2023 21:20

thozza reviewed Oct 25, 2023

View reviewed changes

test/configs/edge-ostree-pull-user.json Outdated Show resolved Hide resolved

achilleas-k force-pushed the boot-test/ec2 branch 2 times, most recently from 5f7f868 to 624113f Compare October 25, 2023 19:40

achilleas-k mentioned this pull request Oct 27, 2023

Test: Build edge-commit with cloud-init to use with edge-ami and edge-qcow2 #230

Open

achilleas-k added 17 commits October 31, 2023 00:27

test: AWS boot testing for all supported image types

b208300

Enables boot testing on AWS for all supported image types.

test: unxz images when necessary

fe2ed0a

If an image is an xz compressed file, unxz it before boot testing and delete the uncompressed image when done to avoid pushing it to the build cache.

test: enable boot testing for ostree image types

8cb5ce3

test/config-map: sort keys

97f31e7

jq . --sort-keys the whole file for cleaner edits.

test: update key in user-customizations config

1e3beb0

The config isn't used by any image type currently. The private SSH key is set in the GitLab CI variables.

cmd/boot-aws: add separate flag for private ssh key

0d9cace

cmd/boot-aws: remove share option

e3ce7f6

Doesn't make sense for our boot testing binary.

test: read priv key from CI in boot-image

cf864ec

If the CI_PRIV_SSH_KEY env var exists, use that instead of generating a new key pair. This is used for boot testing images that have a key configured in the build config, for cases where user creation isn't possible at first boot.

cmd/boot-aws: rename some flags for clarity

af3d9c9

HACKING: add instructions for using ./cmd/boot-aws

c28dd4b

Add instructions for how to use the boot-aws command.

test: don't use sudo for systemctl command in test script

e645d5c

In the boot test script, we don't need to use sudo to run systemctl is-system-running For the edge-ami boot test, we can't set passwordless sudo right now, so if it's not required, we can just run it as a regular user.

distro/rhel9: add the ami kernel options to edge-ami

4d12fa2

distro: add note to move amiKernelOptions to EC2 env

18d64d7

Environments don't yet support kernel arguments so doing this now would make the current patch set much bigger than we want it to be. Add a note to do it another time.

cmd/boot-aws: increase boot wait time to 5 mins

f08e0b5

achilleas-k force-pushed the boot-test/ec2 branch from b4c4053 to f08e0b5 Compare October 30, 2023 23:27

test: create configs for ec2-sap

4eae71f

- Create an empty config specifically for ec2-sap for all distros except RHEL 8.4. - Create a config for ec2-sap for RHEL 8.4 only that disables lm_sensors.service Related: RHEL-14096

achilleas-k force-pushed the boot-test/ec2 branch from 5aa8720 to 4eae71f Compare October 31, 2023 15:33

thozza approved these changes Oct 31, 2023

View reviewed changes

supakeen added this pull request to the merge queue Oct 31, 2023

Merged via the queue into osbuild:main with commit b6216aa Oct 31, 2023
7 checks passed

achilleas-k deleted the boot-test/ec2 branch October 31, 2023 19:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AWS boot testing for all supported image types #217

AWS boot testing for all supported image types #217

achilleas-k commented Oct 20, 2023 •

edited

Loading

supakeen commented Oct 20, 2023

achilleas-k commented Oct 20, 2023

thozza left a comment

achilleas-k commented Oct 24, 2023

achilleas-k commented Oct 25, 2023

achilleas-k commented Oct 25, 2023 •

edited

Loading

cgwalters commented Oct 26, 2023

achilleas-k commented Oct 26, 2023

7flying commented Oct 26, 2023 •

edited

Loading

achilleas-k commented Oct 26, 2023

achilleas-k commented Oct 26, 2023

7flying commented Oct 27, 2023

7flying commented Oct 27, 2023

achilleas-k commented Oct 27, 2023

achilleas-k commented Oct 30, 2023

achilleas-k commented Oct 30, 2023

achilleas-k commented Oct 30, 2023 •

edited

Loading

achilleas-k commented Oct 31, 2023 •

edited

Loading

thozza left a comment

supakeen commented Oct 31, 2023

AWS boot testing for all supported image types #217

AWS boot testing for all supported image types #217

Conversation

achilleas-k commented Oct 20, 2023 • edited Loading

supakeen commented Oct 20, 2023

achilleas-k commented Oct 20, 2023

thozza left a comment

Choose a reason for hiding this comment

achilleas-k commented Oct 24, 2023

achilleas-k commented Oct 25, 2023

achilleas-k commented Oct 25, 2023 • edited Loading

cgwalters commented Oct 26, 2023

achilleas-k commented Oct 26, 2023

7flying commented Oct 26, 2023 • edited Loading

achilleas-k commented Oct 26, 2023

achilleas-k commented Oct 26, 2023

7flying commented Oct 27, 2023

7flying commented Oct 27, 2023

achilleas-k commented Oct 27, 2023

achilleas-k commented Oct 30, 2023

achilleas-k commented Oct 30, 2023

achilleas-k commented Oct 30, 2023 • edited Loading

achilleas-k commented Oct 31, 2023 • edited Loading

thozza left a comment

Choose a reason for hiding this comment

supakeen commented Oct 31, 2023

achilleas-k commented Oct 20, 2023 •

edited

Loading

achilleas-k commented Oct 25, 2023 •

edited

Loading

7flying commented Oct 26, 2023 •

edited

Loading

achilleas-k commented Oct 30, 2023 •

edited

Loading

achilleas-k commented Oct 31, 2023 •

edited

Loading