Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integration tests for docker mode #335

Merged
merged 16 commits into from
Dec 26, 2019
Merged

Conversation

simar7
Copy link
Member

@simar7 simar7 commented Dec 18, 2019

This PR introduces integrations tests for docker mode.

Addresses #325

Signed-off-by: Simarpreet Singh [email protected]

@simar7 simar7 requested a review from knqyf263 December 18, 2019 05:15
@simar7 simar7 self-assigned this Dec 18, 2019
@simar7 simar7 changed the title Integration tests docker engine Integration tests for docker mode Dec 18, 2019
Copy link
Collaborator

@knqyf263 knqyf263 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost looks good to me! You will add a setting of CircleCI in a dedicated PR, right? Or, we can add it to this PR.

integration/docker_engine_test.go Outdated Show resolved Hide resolved
integration/docker_engine_test.go Outdated Show resolved Hide resolved
integration/docker_engine_test.go Outdated Show resolved Hide resolved
integration/docker_engine_test.go Outdated Show resolved Hide resolved
integration/docker_engine_test.go Outdated Show resolved Hide resolved
@simar7 simar7 force-pushed the integration-tests-docker-engine branch from 5680cd8 to 0bc1c75 Compare December 19, 2019 03:08
@simar7 simar7 force-pushed the integration-tests-docker-engine branch 4 times, most recently from 04c8e0d to f875ef8 Compare December 19, 2019 03:36
@simar7 simar7 force-pushed the integration-tests-docker-engine branch from f875ef8 to 8a9293a Compare December 19, 2019 03:39
@simar7 simar7 requested a review from knqyf263 December 19, 2019 04:10
.circleci/config.yml Outdated Show resolved Hide resolved
integration/docker_engine_test.go Outdated Show resolved Hide resolved
@simar7
Copy link
Member Author

simar7 commented Dec 23, 2019

@knqyf263 I added four different versions of docker setups here 923bd79 keep in mind each of them takes on average ~4min to complete. I picked these versions because they are evenly spaced in the supported versions list. Maybe we can add more?

Keep in mind we're only limited to 4x jobs running in parallel on circleci. Currently it's 1 for unit test and 4 for integration-tests that get run on each push. Also there's a 2500 credit limit (not sure how this actually boils down).

@simar7
Copy link
Member Author

simar7 commented Dec 23, 2019

Interestingly enough you can see some of the docker versions currently fail with the version of the client we're using 😰 https://circleci.com/workflow-run/61cb32c7-de96-4cdc-a34a-5b12b50d92cc

One example:

--- FAIL: TestRun_WithDockerEngine (0.27s)
    docker_engine_test.go:59: 
        	Error Trace:	docker_engine_test.go:59
        	Error:      	Received unexpected error:
        	            	Error response from daemon: client version 1.39 is too new. Maximum supported API version is 1.38
        	Test:       	TestRun_WithDockerEngine
        	Messages:   	happy path, valid image path, alpine:3.10

@simar7 simar7 requested a review from knqyf263 December 23, 2019 23:03
@knqyf263
Copy link
Collaborator

Hmm... It's weird as the test passes with 17.03.0-ce. I wonder if the maximum client is not implemented in 17.03.0-ce. So, our integration tests may work only with 18.09.03 in CircleCI. For now, it seems better that we give up running integration tests with multiple Docker Engine and just run them with 18.09.03.

@knqyf263 knqyf263 merged commit 77f1abc into master Dec 26, 2019
@knqyf263 knqyf263 deleted the integration-tests-docker-engine branch December 26, 2019 10:48
GuaoGuao pushed a commit to GuaoGuao/trivy that referenced this pull request Jun 24, 2020
* wip: run trivy after adding image

Signed-off-by: Simarpreet Singh <[email protected]>

* integration: Add an integration test for docker mode

Signed-off-by: Simarpreet Singh <[email protected]>

* integration: Add error checks for docker_engine_test

Signed-off-by: Simarpreet Singh <[email protected]>

* circleci: add specific docker version

Signed-off-by: Simarpreet Singh <[email protected]>

* circleci: add specific docker version

Signed-off-by: Simarpreet Singh <[email protected]>

* docker_engine_test: Add a sad path

Signed-off-by: Simarpreet Singh <[email protected]>

* circleci: Add docker_version by param

Signed-off-by: Simarpreet Singh <[email protected]>

* circleci: Add more docker versions

Signed-off-by: Simarpreet Singh <[email protected]>

* test(integration): remove old docker versions

* chore(ci): add requires

Co-authored-by: Teppei Fukuda <[email protected]>
liamg pushed a commit that referenced this pull request Jun 7, 2022
* wip: run trivy after adding image

Signed-off-by: Simarpreet Singh <[email protected]>

* integration: Add an integration test for docker mode

Signed-off-by: Simarpreet Singh <[email protected]>

* integration: Add error checks for docker_engine_test

Signed-off-by: Simarpreet Singh <[email protected]>

* circleci: add specific docker version

Signed-off-by: Simarpreet Singh <[email protected]>

* circleci: add specific docker version

Signed-off-by: Simarpreet Singh <[email protected]>

* docker_engine_test: Add a sad path

Signed-off-by: Simarpreet Singh <[email protected]>

* circleci: Add docker_version by param

Signed-off-by: Simarpreet Singh <[email protected]>

* circleci: Add more docker versions

Signed-off-by: Simarpreet Singh <[email protected]>

* test(integration): remove old docker versions

* chore(ci): add requires

Co-authored-by: Teppei Fukuda <[email protected]>
josedonizetti pushed a commit to josedonizetti/trivy that referenced this pull request Jun 24, 2022
…apters

test: add tests for aws sns tf adapters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants