apiv2: test using docker-py integration tests #5386
Comments
openshift-ci-robot
added
the
kind/feature
|
take a closer look at the tests ... they use swarm and a bunch of stuff we will never do. |
|
they also dont support multiple registries and consider docker.io to be the one and only registries making their short name use difficult to deal with. |
Yep those aren't features I care about either. But it's a great way to demonstrate to potential adopters what works with apiv2 (in a very granular fashion). I'm not even necessarily saying run it with your normal CI pipelines, maybe just run it for every release.
This doesn't seem to be particularly relevant to this issue (unless I'm misunderstanding). If you want apiv2 to be compatible with the docker api there's unfortunately a lot of annoying ego (and industry capturing) decisions to deal with and overcome. |
|
it matters because several of the tests fail because of it. again, so many of the tests use functions that we won't support I wonder how to make any of it applicable. |
|
A friendly reminder that this issue had no activity for 30 days. |
|
@gartnera Did you ever look any further into this? Even if we could get a subset of the tests to work, it would be a big step forward. |
|
No I haven't done anything more. I reran the tests this morning against master, here's the results: Looks like there was decent improvement in |
|
@gartnera Have you continued to work on this? |
|
@gartnera Would be nice to see how we have progressed. |
|
Looks like most of the docker-py tests fail now with errors like:
Worked around that by editing Now I get these results: There are six more passing tests (71 -> 77) since the last time I tested. The most prominent errors looks like this: This appears to be because podman doesn't pull images if they don't exist. To reproduce:
I also can't run containers if I don't explicitly set the command which I suspect is the cause of other test failures: Version info: |
|
@jwhonce Is our Docker version too old? |
|
|
|
A friendly reminder that this issue had no activity for 30 days. |
|
You could close if if you'd like. I'll try and file some more specific issues. I reran the tests today, the output is below. Doesn't look like there's been much movement over the past month or two. |
|
My understanding of the issues here is that a lot of docker-py tests rely on swarm and some network features, and rely on docker.io as the only registry. If docker-py wants to make their tests work with Podman or a subset, then we can do that. For now, we have been adding docker-py tests to podman but not the entire test suite. If a community member wants to take this on it would be appreciated, but the core team has decided this is not a priority. |
|
Still not a priority. |
|
I have the same issue as in #5386 (comment) Using Ansible's Intrigued by #5386 (comment) I tried to find out. Running {
"Platform": {
"Name": "linux/amd64/fedora-32"
},
"Components": [
{
"Name": "Podman Engine",
"Version": "2.0.6",
"Details": {
"APIVersion": "1.0.0",
"Arch": "amd64",
"BuildTime": "2020-09-01T19:26:51Z",
"Experimental": "true",
"GitCommit": "",
"GoVersion": "go1.14.6",
"KernelVersion": "5.4.0-47-generic",
"MinAPIVersion": "1.0.0",
"Os": "linux"
}
}
],
"Version": "2.0.6",
"ApiVersion": "1.0.0",
"MinAPIVersion": "1.0.0",
"GitCommit": "",
"GoVersion": "go1.14.6",
"Os": "linux",
"Arch": "amd64",
"KernelVersion": "5.4.0-47-generic",
"Experimental": true,
"BuildTime": "2020-09-01T19:26:51Z"
}It seems Podman actually reports it "only" supports |
|
I've separately reported this in #7682. |
|
Well a fix for this has been merged, does this make this any better? |
|
@gartnera Could you give us another run? |
|
If it's of any help ... I just ran the integration tests myself. 🙂 Distro: Ubuntu 20.04 |
|
Hey there, It happens for basic commands like |
I don't know if you're aware of the test filtering features pytest has. You can test specific files of the suite, or even filter them with expressions using You can maintain all those settings in a |
|
Interested in opening a PR to do this? |
|
This is how my I set @rhatdan Is there a rough list of (API) features/aspects that Podman is not going to replicate? e.g.
|
|
@riyad There are some outstanding endpoints like |
|
Because @matejvasek asked over in another PR: I've put together some scripts and documentation on how to run the docker-py integration tests against Podman. I hope someone finds it useful. 😃 |
|
@riyad nice, I'll try it out. |
|
I hope all the issues I'm opening aren't getting on your nerves 😅 |
|
@riyad never! But we'd love it if you sent in a PR or two too! 😃 |
|
A friendly reminder that this issue had no activity for 30 days. |
|
@riyad Is this still an issue? |
|
IMHO this issue can be closed. 👍 If I'm allowed to reinterpret the original issue: it was steering Podman's API efforts towards something that could be described as "ecosystem compatibility". With all the effort put in fixing issues uncovered by the docker-py test suite as well as the docker-compose compatibility I'd attest that this matter is taken seriously. 😄 With regard to the docker-py test suite I've put together https://github.com/riyad/podman-integration-test-utils as a way to document and automate running docker-py's integration tests against Podman's API service. It also has some rudimentary tools (and data) to display and track the progress in API compatibility. To summarize Podman's progress ... is quite impressive to be honest:
These are the "conservative" numbers: you'll generally get +2 if using crun and there're 1-3 tests which are flaky. |
github-actions
bot
added
the
locked - please file new issue/PR
/kind feature
Description
Consider using the docker-py integration tests to measure the progress on the apiv2 development effort. This is what the output of the tests look like with
dockerd:This is what the output of the tests look like with
podman system service -t 0:There's clearly a long way to go on development. This would also help potential users understand what is supported by podman apiv2 and perhaps even help scope future development efforts.
The text was updated successfully, but these errors were encountered: