Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

mounting kubectl from the host instead to installing in protokube #3550

Merged
merged 1 commit into from
Oct 10, 2017
Merged

mounting kubectl from the host instead to installing in protokube #3550

merged 1 commit into from
Oct 10, 2017

Conversation

chrislovecnm
Copy link
Contributor

@chrislovecnm chrislovecnm commented Oct 6, 2017 •
edited
Loading

So this will fix our protokube kubectl versioning issue. Kubectl is in on host, if we are on a master, and is always the right version, so let's use it! Refactored a bit to get the distro path for kubectl. Need to test on gossip. Set the path on protokube and mounted kubectl in /opt/kops/bin.

/approve

TODO

  • test gossip

Fixes #3518

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Oct 6, 2017
@chrislovecnm chrislovecnm added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 6, 2017
@chrislovecnm
Copy link
Contributor Author

chrislovecnm commented Oct 6, 2017 •
edited
Loading

ready to review, but I need to e2e

@KashifSaadat / @andrewsykim / @alrs you mind reviewing?

@KashifSaadat
Copy link
Contributor

I tried this out but get the following failure with protokube:

Oct 06 21:42:31 ip-10-0-0-185.eu-west-1.compute.internal systemd[1]: Starting Kubernetes Protokube Service...
Oct 06 21:42:31 ip-10-0-0-185.eu-west-1.compute.internal systemd[1]: Started Kubernetes Protokube Service.
Oct 06 21:42:31 ip-10-0-0-185.eu-west-1.compute.internal env[928]: time="2017-10-06T21:42:31.807995354Z" level=error msg="Handler for POST /v1.24/containers/81eca75664968a480dfd45e1553dd72071bdee09111f1b42d45020158b75ab52/start returned error: mkdir /usr/local/bin/kubectl: read-only file system"
Oct 06 21:42:31 ip-10-0-0-185.eu-west-1.compute.internal docker[1476]: /run/torcx/bin/docker: Error response from daemon: mkdir /usr/local/bin/kubectl: read-only file system.
Oct 06 21:42:31 ip-10-0-0-185.eu-west-1.compute.internal systemd[1]: protokube.service: Main process exited, code=exited, status=125/n/a
Oct 06 21:42:31 ip-10-0-0-185.eu-west-1.compute.internal systemd[1]: protokube.service: Unit entered failed state.
Oct 06 21:42:31 ip-10-0-0-185.eu-west-1.compute.internal systemd[1]: protokube.service: Failed with result 'exit-code'.

@chrislovecnm
Copy link
Contributor Author

Grumble - how the heck did it pass e2e...

@KashifSaadat
Copy link
Contributor

Do the E2E tests build a fresh protokube image and reference that for the build? Would be quite good to have, and same for nodeup.

@alrs
Copy link
Contributor

alrs commented Oct 7, 2017 via email

@chrislovecnm chrislovecnm changed the title mounting kubectl from the host instead to instaling in protokube mounting kubectl from the host instead to installing in protokube Oct 8, 2017
@k8s-ci-robot k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Oct 8, 2017
@chrislovecnm
Copy link
Contributor Author

@KashifSaadat e2e should be passing and I have tested it as well. I have not done testing with gossip yet. Yes, e2e tests everything, as committed, except for dns-controller (something we need to fix). I changed some path stuff so that paths won't mess with us on different distros.

@justinsb
Copy link
Member

justinsb commented Oct 9, 2017

A few nits, but LGTM. What we have to be careful of here is that the version of kubectl isn't necessarily the version of the manifest we're applying. But I don't think this trickiness is actually made any worse what you're doing here.

Want to make a pass over the code comments and then self-lgtm?

justinsb
justinsb reviewed Oct 9, 2017
View reviewed changes
images/protokube-builder/onbuild.sh
@@ -32,9 +32,4 @@ cp /src/.build/local/protokube /src/.build/artifacts/
make channels
cp /src/.build/local/channels /src/.build/artifacts/

# channels uses protokube
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ooops - that comment wasn't right :-)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SGTM

nodeup/pkg/model/context.go

// KubectlPath returns distro based path for kubectl
func (c *NodeupModelContext) KubectlPath() string {
kubeletCommand := "/usr/local/bin"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: rename kubeletComamnd to kubectlPath

nodeup/pkg/model/kubectl.go
@@ -52,7 +52,7 @@ func (b *KubectlBuilder) Build(c *fi.ModelBuilderContext) error {
}

t := &nodetasks.File{
Path: b.kubectlPath(),
Path: b.KubectlPath() + "/" + assetName,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: use filepath.Join

nodeup/pkg/model/protokube.go
// kubectl is downloaded an installed by other tasks
if t.IsMaster {
dockerArgs = append(dockerArgs, []string{
"-v", t.KubectlPath() + ":/opt/kops/bin:ro",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So docker can actually bind mount a file, which is nice because it avoids dragging in everything in the bin directory. Not sure whether I care, given we mount / -> /rootfs above...

A potential gotcha is that we have to be careful about shared libraries, but thankfully kubectl is statically linked.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, but we have a chance that protokube will start before kubectl is installed ... async tasks ...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah good point, and then we would hit the docker bug where it assumes a directory.

The tasks are technically ordered if we tell nodeup about our dependencies, but I do like your way better.

nodeup/pkg/model/protokube.go
if t.IsMaster {
dockerArgs = append(dockerArgs, []string{
"-v", t.KubectlPath() + ":/opt/kops/bin:ro",
"--env", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/kops/bin",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Probably better to put /opt/kops/bin at the start, except that we know that most of these dirs don't exist because we're in a container...

I guess the problem here is that channels calls kubectl, so we have to make sure it's on the path? If that's the case, can we have a comment, as otherwise the temptation will be to just change kubectl calls to be a fully-qualified path.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well ... I disagree. We need it to be at the end, so that other stuff does not get whacked. First, wins in bash?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK - we don't expect many copies of kubectl :-)

Can we have a comment though about why we don't just call kubectl with an absolute path, i.e. why we are messing with the PATH at all

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cause we do not want to make changes to channels :P Yes I will make a comment.

@justinsb
Copy link
Member

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 10, 2017
@k8s-github-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: justinsb

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

@k8s-github-robot k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 10, 2017
@k8s-github-robot
Copy link

/test all [submit-queue is verifying that this PR is safe to merge]

@k8s-github-robot
Copy link

Automatic merge from submit-queue.

@k8s-github-robot k8s-github-robot merged commit 2500ee0 into kubernetes:master Oct 10, 2017
@chrislovecnm chrislovecnm deleted the protokube-kubectl branch December 30, 2017 20:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@justinsb justinsb justinsb left review comments

Assignees

@justinsb justinsb

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

use the host kubectl in protokube
6 participants
@chrislovecnm @KashifSaadat @alrs @justinsb @k8s-github-robot @k8s-ci-robot