Multiple pythons needed for podman-py testing #111

cevich · 2022-01-26T22:11:39Z

The one-size-fits-all approach is ever increasingly showing gaps and
inefficiencies WRT CI for the podman-py repository. Rather than
continue to increase the bloat of the golang-centric images, produce a
dedicated image for this repo. This allows pre-installing/caching all
the necessary python bits needed, avoiding the need to install them at
runtime. It will require updates to the podman-py setup in order to
make use of the new image effectively.

github-actions · 2022-01-26T23:17:34Z

Cirrus CI build successful. Image ID c6391889636098048 ready for use.

cevich · 2022-01-27T16:21:26Z

DO NOT USE c6391889636098048: The container images are getting "too big" due to the extra python versions. @jwhonce do any of your tests run in the container images or are VMs used exclusively?

If so...I can limit the extra python bits to only installing in VMs and not the containers.

jwhonce · 2022-01-27T18:47:58Z

@cevich I don't know, the cirrus.yaml has a container field. Does that mean the tests are only run in a container? (currently...)

cevich · 2022-01-27T20:29:20Z

Sorry, I thought maybe you'd know off-hand. Taking a look in podman-py, I see the gating task uses the container. Both test and latest use VMs, so we do have a mix 😢 Assuming make and make lint need to use those other python versions, my idea of asymmetrically installing won't work. Damn.

cevich · 2022-01-27T20:40:34Z

Let's try it this way: I moved most of the python bits into the VM-only list of installs. We'll need to confirm the images work with podman CI, but this will all but negate any container-image size concerns (unless it doesn't work)...

github-actions · 2022-01-27T21:38:59Z

Cirrus CI build successful. Image ID c6730261319647232 ready for use.

cevich · 2022-01-27T21:44:46Z

K, lessee if it works in podman. You're welcome to test c6730261319647232 in podman-py CI as well.

jwhonce · 2022-01-27T22:07:34Z

@cevich See https://github.com/containers/podman-py/pull/160/checks?check_run_id=4973220410 for errors

cevich · 2022-01-28T16:31:19Z

make
rm dist/* || :
rm: cannot remove 'dist/*': No such file or directory
/usr/bin/python3 -m pip install --user -r requirements.txt
/usr/bin/python3: No module named pip
make: *** [Makefile:16: podman] Error 1

Exit status: 2

Right, okay so we need some python stuffs in the containers then. Lemmie go into one manually where I can dnf install bit by bit until the tests pass...

cevich · 2022-01-28T16:48:15Z

...oof, okay, no won't work. make lint wants all the bunches of python versions too 😞 Probably the quickest way around this is to alter the gating CI job to use VMs and not containers. That will slow the job down a little, since the VM has to boot, but will not result in bloat or extra things to maintain on the image-side.

That said (and I remember we talked about this before), it seems like we need a custom image for the podman-py repo. CI. The python needs are unique enough that the one-size-fits-all approach shows weakness. This is more/less why I did custom images for netavark and aardvark.

This does take quite a bit of effort to pull off, and we're in the middle of the. podman 4.0 crunch. Is a working containers/podman-py#160 necessary/critical for the 4.0 efforts?

I can spend a little time on it today, since other folks are OOO - and I'm blocked on their help to move some items forward....

cevich · 2022-01-28T17:12:04Z

The fedora-podman-py Cache Image is the CI task to watch. Assuming it builds successfully, the hard-work can begin: getting it to work in the podman-py repo...

cevich · 2022-01-28T18:54:19Z

Repos timed out, retrying...

cevich · 2022-01-28T19:50:33Z

Timed out! ugh...it must be Friday. I think the repos. (or network) are just being slow today for whatever reason. I'll try again Monday.

cevich · 2022-01-31T20:25:46Z

Still timing out. I don't get it. The other VM images build in < 10 minutes. WTH going on with the new one it taking so long.

cevich · 2022-01-31T21:03:31Z

Found the problem...there was a dnf install ... command waiting for somebody to press y 😭

cevich · 2022-02-01T16:37:57Z

AAAAAND if it ain't one thing...it's something else...and something else...and...

Previously the VM images used for automation were generic/one-size-fits-all. However over time, various repos. have required numerous updates resulting in a great deal of bloat. To combat this, a dedicated VM image has been crafted for use just by this repo. Put this into use, and re-order tasks in `.cirrus.yml` to match more closely with the order of operations. Ref: containers/automation_images#111 Signed-off-by: Chris Evich <[email protected]>

github-actions · 2022-02-01T17:06:02Z

Cirrus CI build successful. Image ID c4728831859752960 ready for use.

The one-size-fits-all approach is ever increasingly showing gaps and inefficiencies WRT CI for the `podman-py` repository. Rather than continue to increase the bloat of the golang-centric images, produce a dedicated image for this repo. This allows pre-installing/caching all the necessary python bits needed, avoiding the need to install them at runtime. It will require updates to the `podman-py` setup in order to make use of the new image effectively. Signed-off-by: Chris Evich <[email protected]>

cevich · 2022-02-04T16:08:45Z

@jwhonce update: The new image seems to work with podman-py containers/podman-py#161 but I got a bit delete-happy yanking python stuff out of the other images. That seems to have broke the docker-py testing we do in podman. So I'm putting the general Fedora image back the way it was, and we'll give the new set of images another go 'round...

github-actions · 2022-02-04T17:05:52Z

Cirrus CI build successful. Image ID c5533341981081600 ready for use.

Mainly this is to confirm some changes needed for the podman-py CI setup don't disrupt operations here. Ref: containers/automation_images#111 Signed-off-by: Chris Evich <[email protected]>

This is necessary because both are pulled in by containers-common in the updates-testing repo. for F35. Since they will never be officially supported along with podman 4 in F35, ensure they're not present in any images. Signed-off-by: Chris Evich <[email protected]>

Mainly this is to confirm some changes needed for the podman-py CI setup don't disrupt operations here. Ref: containers/automation_images#111 Signed-off-by: Chris Evich <[email protected]>

Previously the VM images used for automation were generic/one-size-fits-all. However over time, various repos. have required numerous updates resulting in a great deal of bloat. To combat this, a dedicated VM image has been crafted for use just by this repo. Put this into use, and re-order tasks in `.cirrus.yml` to match more closely with the order of operations. Ref: containers/automation_images#111 Signed-off-by: Chris Evich <[email protected]>

Mainly this is to confirm some changes needed for the podman-py CI setup don't disrupt operations here. Ref: containers/automation_images#111 Signed-off-by: Chris Evich <[email protected]>

github-actions · 2022-02-08T19:32:22Z

Cirrus CI build successful. Image ID c4764556961513472 ready for use.

cevich · 2022-02-08T19:32:31Z

DO NOT USE: c5533341981081600 - has netavark/aardvark packages installed for F35 by default.

Mainly this is to confirm some changes needed for the podman-py CI setup don't disrupt operations here. Ref: containers/automation_images#111 Note: Glibc resolver configuration has changed from previous images. An additional setup command was added to remove systemd-resolved from the chain. Signed-off-by: Chris Evich <[email protected]>

Mainly this is to confirm some changes needed for the podman-py CI setup don't disrupt operations here. Ref: containers/automation_images#111 Also includes a minor steup fix WRT setting up for test-rpm build. Signed-off-by: Chris Evich <[email protected]>

cevich requested a review from jwhonce January 26, 2022 22:11

cevich force-pushed the mega_python branch from 5c8286a to 6554120 Compare January 26, 2022 22:15

cevich marked this pull request as draft January 27, 2022 16:19

cevich force-pushed the mega_python branch from 6554120 to 70a7954 Compare January 27, 2022 20:39

cevich mentioned this pull request Jan 27, 2022

[main] Cirrus: Update VM Images for 4.0 release containers/podman#13055

Merged

cevich force-pushed the mega_python branch from 70a7954 to dab4197 Compare January 28, 2022 17:10

cevich force-pushed the mega_python branch from dab4197 to cd147ea Compare January 31, 2022 20:53

cevich force-pushed the mega_python branch from cd147ea to 80c7d80 Compare February 1, 2022 16:06

cevich mentioned this pull request Feb 1, 2022

[CI:DOCS] Use CentOS-8 Stream for Image Builder VM #112

Merged

cevich mentioned this pull request Feb 1, 2022

Cirrus: Use dedicated VM image for automation containers/podman-py#161

Merged

cevich force-pushed the mega_python branch from 7d29941 to cf66ac4 Compare February 4, 2022 16:04

cevich marked this pull request as ready for review February 8, 2022 19:31

cevich merged commit 3fa3bdd into containers:main Feb 10, 2022

cevich deleted the mega_python branch July 19, 2022 14:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multiple pythons needed for podman-py testing #111

Multiple pythons needed for podman-py testing #111

cevich commented Jan 26, 2022 •

edited

Loading

github-actions bot commented Jan 26, 2022

cevich commented Jan 27, 2022

jwhonce commented Jan 27, 2022 •

edited

Loading

cevich commented Jan 27, 2022

cevich commented Jan 27, 2022

github-actions bot commented Jan 27, 2022

cevich commented Jan 27, 2022

jwhonce commented Jan 27, 2022

cevich commented Jan 28, 2022

cevich commented Jan 28, 2022

cevich commented Jan 28, 2022

cevich commented Jan 28, 2022

cevich commented Jan 28, 2022

cevich commented Jan 31, 2022

cevich commented Jan 31, 2022

cevich commented Feb 1, 2022

github-actions bot commented Feb 1, 2022

cevich commented Feb 4, 2022

github-actions bot commented Feb 4, 2022

github-actions bot commented Feb 8, 2022

cevich commented Feb 8, 2022

Multiple pythons needed for podman-py testing #111

Multiple pythons needed for podman-py testing #111

Conversation

cevich commented Jan 26, 2022 • edited Loading

github-actions bot commented Jan 26, 2022

cevich commented Jan 27, 2022

jwhonce commented Jan 27, 2022 • edited Loading

cevich commented Jan 27, 2022

cevich commented Jan 27, 2022

github-actions bot commented Jan 27, 2022

cevich commented Jan 27, 2022

jwhonce commented Jan 27, 2022

cevich commented Jan 28, 2022

cevich commented Jan 28, 2022

cevich commented Jan 28, 2022

cevich commented Jan 28, 2022

cevich commented Jan 28, 2022

cevich commented Jan 31, 2022

cevich commented Jan 31, 2022

cevich commented Feb 1, 2022

github-actions bot commented Feb 1, 2022

cevich commented Feb 4, 2022

github-actions bot commented Feb 4, 2022

github-actions bot commented Feb 8, 2022

cevich commented Feb 8, 2022

cevich commented Jan 26, 2022 •

edited

Loading

jwhonce commented Jan 27, 2022 •

edited

Loading