Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove gcr.io/k8s-skaffold repository from examples #3368

Merged

Conversation

corneliusweig
Copy link
Contributor

@corneliusweig corneliusweig commented Dec 11, 2019

Description

All examples reference images in the gcr.io/k8s-skaffold/ repository. The underlying reason for this is that Skaffold uses these examples in its integration tests, and pushes the built images to the registry.
However, for regular Skaffold users the repository is not writable, so they need to modify the image names to try out the examples. The goal of this PR is to improve the usability of examples for Skaffold users. Removing the gcr.io/k8s-skaffold image prefix will allow to run examples with just the --default-repo option and without having to modify the skaffold.yaml.

User facing changes

Images in examples no longer reference the gcr.io/k8s-skaffold repository.

Before

Users could not use the --default-repo option to push to their own registries.

After

Users can use the --default-repo option to push to their own registries.

Next PRs.

n/a

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you
review them:

  • Includes unit tests
  • Mentions any output changes.
  • Adds documentation as needed: user docs, YAML reference, CLI reference.
  • Adds integration tests if needed.

Open question
Should the README to run examples include instructions for running with the --default-repo option?

Reviewer Notes

  • The code flow looks good.
  • Unit test added.
  • User facing changes look good.

@codecov
Copy link

codecov bot commented Dec 11, 2019

Codecov Report

Merging #3368 into master will increase coverage by 0.18%.
The diff coverage is n/a.

Impacted Files Coverage Δ
pkg/skaffold/schema/defaults/defaults.go 94.25% <0%> (ø) ⬆️
pkg/skaffold/deploy/deploy_mux.go 89.13% <0%> (ø)
pkg/skaffold/deploy/util.go 79.16% <0%> (+0.21%) ⬆️
pkg/skaffold/initializer/init.go 63.63% <0%> (+0.22%) ⬆️
pkg/skaffold/deploy/kubectl.go 65.87% <0%> (+2.16%) ⬆️
pkg/skaffold/deploy/labels.go 13.41% <0%> (+3.65%) ⬆️
pkg/skaffold/util/tar.go 57.47% <0%> (+4.59%) ⬆️

@balopat balopat added the kokoro:run runs the kokoro jobs on a PR label Dec 11, 2019
@kokoro-team kokoro-team removed the kokoro:run runs the kokoro jobs on a PR label Dec 11, 2019
@corneliusweig corneliusweig force-pushed the w/examples-default-repo branch 3 times, most recently from 8bc58fd to 856e0b5 Compare December 14, 2019 13:57
@balopat balopat added the kokoro:run runs the kokoro jobs on a PR label Dec 18, 2019
@kokoro-team kokoro-team removed the kokoro:run runs the kokoro jobs on a PR label Dec 18, 2019
@balopat
Copy link
Contributor

balopat commented Dec 18, 2019

Not sure what's happening with the integration tests on travis, I'm looking...

@balopat
Copy link
Contributor

balopat commented Dec 18, 2019

So, make integration-in-kind fails locally with a bunch of errors, but I'm not sure why it's not showing up in Travis. Werid. However, I recommend running it locally and fixing them:

e.g.:

--- FAIL: TestDeployWithInCorrectConfig (0.04s)
    deploy_test.go:139: failed without saying the reason: You either need to:
        run [skaffold deploy] with [--images TAG] for each pre-built artifact
        or [skaffold run] instead, to let Skaffold build, tag and deploy artifacts.
        time="2019-12-18T14:26:01Z" level=fatal msg="no tag provided for image [skaffold-example]"
--- FAIL: TestDebug (682.06s)
    --- FAIL: TestDebug/kubectl (378.99s)
        util.go:152: Timed out waiting for pods [nodejs npm python3 go] ready in namespace skaffold2w448
    --- FAIL: TestDebug/kustomize (303.07s)
        util.go:152: Timed out waiting for pods [nodejs npm python3 go] ready in namespace skaffold6zslc
--- FAIL: TestBuildDeploy (6.35s)
    deploy_test.go:56: expected to find a tag for leeroy-web but found none 
=== RUN   TestSetGlobalDefaultRepo
time="2019-12-18T14:14:26Z" level=info msg="[skaffold config --default-repo gcr.io/k8s-skaffold set default-repo REPO2 -c /tmp/config144104431 --global]"
--- FAIL: TestSetGlobalDefaultRepo (0.02s)
    helper.go:180: skaffold config: exit status 1, 
    panic.go:406: 
        
        Examples:
          # Mark a registry as insecure
          skaffold config set insecure-registries <insecure1.io>
        
          # Globally set the default image repository
          skaffold config set default-repo <myrepo>
        
          # Disable pushing images for a given Kubernetes context
          skaffold config set --kube-context <mycluster> local-cluster true
        
        Options:
          -c, --config='': Path to Skaffold config
          -g, --global=false: Set value for global config
          -k, --kube-context='': Kubectl context to set values against
        
        Usage:
          skaffold config set [flags] [options]
        
        Use "skaffold options" for a list of global command-line options (applies to all commands).
        
        time="2019-12-18T14:14:26Z" level=fatal msg="unknown flag: --default-repo"
=== RUN   TestSetDefaultRepoForContext
time="2019-12-18T14:14:26Z" level=info msg="[skaffold config --default-repo gcr.io/k8s-skaffold set default-repo REPO1 -c /tmp/config128428752 -k test-context]"
--- FAIL: TestSetDefaultRepoForContext (0.02s)
    helper.go:180: skaffold config: exit status 1, 
    panic.go:406: 
        
        Examples:
          # Mark a registry as insecure
          skaffold config set insecure-registries <insecure1.io>
        
          # Globally set the default image repository
          skaffold config set default-repo <myrepo>
        
          # Disable pushing images for a given Kubernetes context
          skaffold config set --kube-context <mycluster> local-cluster true
        
        Options:
          -c, --config='': Path to Skaffold config
          -g, --global=false: Set value for global config
          -k, --kube-context='': Kubectl context to set values against
        
        Usage:
          skaffold config set [flags] [options]
        
        Use "skaffold options" for a list of global command-line options (applies to all commands).
        
        time="2019-12-18T14:14:26Z" level=fatal msg="unknown flag: --default-repo"
--- FAIL: TestBuild (143.53s)
    --- PASS: TestBuild/docker_build (10.08s)
    --- PASS: TestBuild/git_tagger (0.41s)
    --- PASS: TestBuild/sha256_tagger (0.37s)
    --- FAIL: TestBuild/dateTime_tagger (0.25s)
        build_test.go:116: expected image 'simple-build:2019-12-18' not present
    --- PASS: TestBuild/envTemplate_tagger (0.27s)
    --- PASS: TestBuild/custom (131.84s)
    --- PASS: TestBuild/--tag_arg (0.31s)
=== RUN   TestExpectedBuildFailures
=== RUN   TestExpectedBuildFailures/jib_is_too_old
time="2019-12-18T14:13:55Z" level=info msg="[skaffold build --default-repo gcr.io/k8s-skaffold -p old-jib]"
--- PASS: TestExpectedBuildFailures (6.73s)
    --- PASS: TestExpectedBuildFailures/jib_is_too_old (6.73s)
=== RUN   TestCacheAPITriggers
time="2019-12-18T14:14:02Z" level=info msg="[skaffold build --default-repo gcr.io/k8s-skaffold]"
time="2019-12-18T14:14:06Z" level=info msg="Ran in 3.748449s"
time="2019-12-18T14:14:06Z" level=info msg="Namespace: skaffold5wb8v"
time="2019-12-18T14:14:06Z" level=info msg="[skaffold dev --namespace skaffold5wb8v --default-repo gcr.io/k8s-skaffold --rpc-port 1569]"
time="2019-12-18T14:14:06Z" level=fatal msg="exiting dev mode because first deploy failed: loading images into kind nodes: unable to load image with kind: gcr.io/k8s-skaffold/skaffold-example:0cab72aed4f13163a9223e10df3f8a559fd687d5205aa2d2082d5b476409418f: exit status 1"
time="2019-12-18T14:14:26Z" level=info msg="Ran in 20.0125735s"
--- FAIL: TestCacheAPITriggers (23.87s)
    rpc_test.go:372: waiting for connection...
    rpc_test.go:372: waiting for connection...
    rpc_test.go:372: waiting for connection...
    rpc_test.go:372: waiting for connection...
    rpc_test.go:372: waiting for connection...
    rpc_test.go:372: waiting for connection...
    rpc_test.go:372: waiting for connection...
    rpc_test.go:372: waiting for connection...
    rpc_test.go:372: waiting for connection...
    rpc_test.go:372: waiting for connection...
    rpc_test.go:372: waiting for connection...
    rpc_test.go:372: waiting for connection...
    rpc_test.go:372: waiting for connection...
    rpc_test.go:372: waiting for connection...
    rpc_test.go:372: waiting for connection...
    rpc_test.go:372: waiting for connection...
    rpc_test.go:372: waiting for connection...
    rpc_test.go:372: waiting for connection...
    rpc_test.go:372: waiting for connection...
    rpc_test.go:372: waiting for connection...
    rpc_test.go:374: error retrieving event log: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp :1569: connect: connection refused"
=== RUN   TestConfigListForContext
time="2019-12-18T14:14:26Z" level=info msg="[skaffold config --default-repo gcr.io/k8s-skaffold list -c testdata/config/config.yaml -k test-context]"
--- FAIL: TestConfigListForContext (0.02s)
    config_test.go:31: skaffold config: exit status 1, 
    panic.go:406: 
        
        Options:
          -a, --all=false: Show values for all kubecontexts
          -c, --config='': Path to Skaffold config
          -k, --kube-context='': Kubectl context to set values against
        
        Usage:
          skaffold config list [flags] [options]
        
        Use "skaffold options" for a list of global command-line options (applies to all commands).
        
        time="2019-12-18T14:14:26Z" level=fatal msg="unknown flag: --default-repo"
=== RUN   TestConfigListForAll
time="2019-12-18T14:14:26Z" level=info msg="[skaffold config --default-repo gcr.io/k8s-skaffold list -c testdata/config/config.yaml --all]"
--- FAIL: TestConfigListForAll (0.02s)
    config_test.go:41: skaffold config: exit status 1, 
    panic.go:406: 
        
        Options:
          -a, --all=false: Show values for all kubecontexts
          -c, --config='': Path to Skaffold config
          -k, --kube-context='': Kubectl context to set values against
        
        Usage:
          skaffold config list [flags] [options]
        
        Use "skaffold options" for a list of global command-line options (applies to all commands).
        
        time="2019-12-18T14:14:26Z" level=fatal msg="unknown flag: --default-repo"

Copy link
Contributor

@balopat balopat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

integration tests are failing still

@corneliusweig
Copy link
Contributor Author

Sorry, I still didn't have time to look into this. I'll do this as soon as possible.

Interesting though that kokoro is happy 🤔

@balopat
Copy link
Contributor

balopat commented Dec 19, 2019

Sorry, I still didn't have time to look into this. I'll do this as soon as possible.

Interesting though that kokoro is happy

Yeah only GCP_ONLY tests run on kokoro

@corneliusweig corneliusweig force-pushed the w/examples-default-repo branch 2 times, most recently from 22747ef to 4f6807d Compare December 22, 2019 20:42
@@ -262,6 +276,15 @@ func (b *RunBuilder) cmd(ctx context.Context) *exec.Cmd {
return cmd
}

func isCoreCommand(command string) bool {
switch command {
case "build", "debug", "delete", "deploy", "dev", "generate-pipeline", "render", "run":
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not super-happy with this approach, as it mirrors what is already configured on the cobra command.

An alternative could be to create a NewSkaffoldCommand and filter valid arguments based on the cobra configuration. If this makes sense this should be done in a follow-up PR.

Copy link
Contributor

@balopat balopat Jan 7, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about using the cobra command to determine directly if the command has the given flags? Something like:

c := cmd.NewSkaffoldCommand(os.Stdout, os.Stderr)
	var command *cobra.Command 
	for _, comm := range c.Commands() {
		if comm.Name() == b.command {
			command = comm 
                         break
		}
	}		
	if b.ns != "" && command.Flags().Lookup("namespace") != nil {
		args = append(args, "--namespace", b.ns)
	}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is what I was thinking too. Then I'll add it in this PR.

@corneliusweig
Copy link
Contributor Author

Finally the build is back to normal, sorry it took so long. I had trouble running the integration tests locally, as make integration-in-kind didn't show proper test results either.

@balopat balopat added kokoro:run runs the kokoro jobs on a PR needs-rebase labels Jan 7, 2020
Signed-off-by: Cornelius Weig <[email protected]>
Skaffold deploy with default-repo seems to be not working that way.
Flags such as --namespace or --default-repo are applied to "all" Skaffold commands, but this only applies to all core commands. Make the test helper a bit smarter to not forward unknown arguments.

Signed-off-by: Cornelius Weig <[email protected]>
When passing unknown flags to skaffold in integration tests, the tests rightfully fails. However, for simpler test setup it is desirable to unconditionally set some configurations. This change ensures that those flags will not break the integration tests.

Signed-off-by: Cornelius Weig <[email protected]>
@corneliusweig corneliusweig force-pushed the w/examples-default-repo branch from ab2a2d8 to 69ed501 Compare January 15, 2020 08:04
@corneliusweig
Copy link
Contributor Author

It's rebased now and I also adapted the new buildpack example. Let's see if everything stays green :)

@dgageot dgageot added kokoro:run runs the kokoro jobs on a PR and removed needs-rebase labels Jan 15, 2020
@kokoro-team kokoro-team removed the kokoro:run runs the kokoro jobs on a PR label Jan 15, 2020
Copy link
Contributor

@dgageot dgageot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! I'lll take another look after this is merged to see if we missed a few more cases

@corneliusweig
Copy link
Contributor Author

Oh dear.. there's another issue with a helm test. I need to take a look later today.

@corneliusweig
Copy link
Contributor Author

Ok, I couldn't wait to fix that right now. I really hope this was the last issue.

I think this is because the helm deployer does not correctly rewrite image names (possibly just for skaffold deploy).

@dgageot dgageot added the kokoro:run runs the kokoro jobs on a PR label Jan 15, 2020
@kokoro-team kokoro-team removed the kokoro:run runs the kokoro jobs on a PR label Jan 15, 2020
@dgageot
Copy link
Contributor

dgageot commented Jan 15, 2020

@corneliusweig bad news

--- FAIL: TestBuildKanikoWithExplicitRepo (4.43s)
    helper.go:192: skaffold build: exit status 1, Generating tags...
         - skaffold-default-repo -> skaffold-default-repo:v1.1.0-108-g29d51d5
        Checking cache...
         - skaffold-default-repo: Not found. Building
        Creating kaniko secret [default/e2esecret]...
        Building [skaffold-default-repo]...
        ERROR: logging before flag.Parse: E0115 18:02:33.875696       1 metadata.go:142] while reading 'google-dockercfg' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg
        ERROR: logging before flag.Parse: E0115 18:02:33.879152       1 metadata.go:159] while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url
        error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "index.docker.io/library/skaffold-default-repo:v1.1.0-108-g29d51d5": UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:library/skaffold-default-repo Type:repository] map[Action:push Class: Name:library/skaffold-default-repo Type:repository]]
    panic.go:563: time="2020-01-15T18:02:35Z" level=fatal msg="build failed: building [skaffold-default-repo]: waiting for pod to complete: condition error: pod already in terminal phase: Failed"

@corneliusweig
Copy link
Contributor Author

@dgageot 🤦‍♂️ well that one goes on me. Sorry for causing so much back-and-forth.

Would you care for another try?

@dgageot dgageot added the kokoro:run runs the kokoro jobs on a PR label Jan 15, 2020
@kokoro-team kokoro-team removed the kokoro:run runs the kokoro jobs on a PR label Jan 15, 2020
@dgageot dgageot dismissed balopat’s stale review January 16, 2020 06:18

Changes were made. It's all good

@dgageot dgageot merged commit ef6d3fb into GoogleContainerTools:master Jan 16, 2020
@corneliusweig corneliusweig deleted the w/examples-default-repo branch January 16, 2020 07:52
kelsk pushed a commit to kelsk/skaffold that referenced this pull request Apr 16, 2021
…ools#3368)

* Always use the default repository `gcr.io/k8s-skaffold` in tests

Signed-off-by: Cornelius Weig <[email protected]>

* Remove gcr.io/k8s-skaffold repository from image names

Signed-off-by: Cornelius Weig <[email protected]>

* Convert former default-repo test to explicit repo tests

Signed-off-by: Cornelius Weig <[email protected]>

* Adapt image names in examples

Signed-off-by: Cornelius Weig <[email protected]>

* Revert helm integration test

Skaffold deploy with default-repo seems to be not working that way.

* Restrict flags to core commands in test-helper

Flags such as --namespace or --default-repo are applied to "all" Skaffold commands, but this only applies to all core commands. Make the test helper a bit smarter to not forward unknown arguments.

Signed-off-by: Cornelius Weig <[email protected]>

* Adapt expectations to account for --default-repo=gcr.io/k8s-skaffold in integration tests

Signed-off-by: Cornelius Weig <[email protected]>

* Silently ignore unknown flags in integration tests

When passing unknown flags to skaffold in integration tests, the tests rightfully fails. However, for simpler test setup it is desirable to unconditionally set some configurations. This change ensures that those flags will not break the integration tests.

Signed-off-by: Cornelius Weig <[email protected]>

* Use default repo in buildpack example

Signed-off-by: Cornelius Weig <[email protected]>

* Revert helm-test: use fully qualified image name

* Change to fully qualified image in explicit repo integration test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants