Experimental kind podman support for hive development. #982

dgoodwin · 2020-05-11T18:34:05Z

No description provided.

dgoodwin · 2020-05-11T18:48:05Z

hack/create-kind-podman-cluster.sh

+
+# create registry container unless it already exists
+reg_name='kind-podman-registry'
+reg_port='5001'


Should be 5000 I'm just testing along side docker at the moment.

dgoodwin · 2020-05-11T18:48:22Z

hack/create-kind-podman-cluster.sh

+
+sudo cp /root/.kube/config ~/.kube/${cluster_name}.kubeconfig
+sudo chown $USER ~/.kube/${cluster_name}.kubeconfig
+echo "Kubeconfig written to $HOME/.kube/${cluster_name}.kubeconfig"


This could probably be better...

dgoodwin · 2020-05-12T15:14:03Z

Current status, this gets hive running but we cannot launch an install, podman will not let us exec/fork to run openshift-install in our install pods, because the binary is copied into /output by the sidecar container and this is mounted with noexec.

Could we perhaps copy to /bin in installmanager? cc @joelddiaz

joelddiaz · 2020-05-12T15:16:55Z

Current status, this gets hive running but we cannot launch an install, podman will not let us exec/fork to run openshift-install in our install pods, because the binary is copied into /output by the sidecar container and this is mounted with noexec.

Could we perhaps copy to /bin in installmanager? cc @joelddiaz

I'd like to spend some time with the kind/docker setup to compare with how things are being brought up there compared to the kind/podman setup. But certainly moving the binary to mountpoint that isn't noexec will work around the immedate issue.

dgoodwin · 2020-05-12T17:53:49Z

Commit added to copy the binaries to /bin. This is working for me locally, I have an install running now.

csrwng · 2020-05-12T19:53:14Z

Wonder if we should have a copy retry:

time="2020-05-12T18:51:51Z" level=debug msg="checking for SSH private key" installID=2mqhtwnz
time="2020-05-12T18:51:51Z" level=info msg="unable to initialize host ssh key" error="cannot configure SSH agent as SSH_PRIV_KEY_PATH is unset or empty" installID=2mqhtwnz
time="2020-05-12T18:51:51Z" level=info msg="waiting for files to be available: [/output/openshift-install /output/oc]" installID=2mqhtwnz
time="2020-05-12T18:51:51Z" level=info msg="found file" installID=2mqhtwnz path=/output/openshift-install
time="2020-05-12T18:51:51Z" level=info msg="found file" installID=2mqhtwnz path=/output/oc
time="2020-05-12T18:51:51Z" level=info msg="all files found, ready to proceed" installID=2mqhtwnz
time="2020-05-12T18:51:51Z" level=error msg="error copying file /output/openshift-install to /bin/openshift-install" error="exit status 1"
time="2020-05-12T18:51:51Z" level=error msg="error waiting for binaries" error="exit status 1" installID=2mqhtwnz
time="2020-05-12T18:51:51Z" level=fatal msg="runtime error" error="exit status 1"

dgoodwin · 2020-05-20T12:05:42Z

I have added a commit which copies the binaries to /home/ where we have a homedir and permissions to write/run. This seems to be working fine.

However it looks like kind and or podman will be fixing the noexec in a future release. I would still propose we just merge this change to run from /home as it works fine and gets us kind+podman in the meantime.

dgoodwin · 2020-05-20T12:06:42Z

pkg/installmanager/installmanager.go

-
-	return destTarball, nil
-}
-


This was dead code.

dgoodwin · 2020-05-20T12:28:17Z

Reverted the 5001 port back to 5000.

staebler · 2020-05-20T12:57:00Z

However it looks like kind and or podman will be fixing the noexec in a future release. I would still propose we just merge this change to run from /home as it works fine and gets us kind+podman in the meantime.

+1
I always thought it a bit strange to run the binary from a folder called output anyway.

joelddiaz

this worked well for me. i look forward to being able to upgrade to fedora 32 when this merges...

dgoodwin · 2020-05-21T11:06:49Z

Multiple testers, I think we're good to go with this one.

hack/create-kind-podman-cluster.sh

pkg/installmanager/installmanager.go

staebler · 2020-05-21T12:47:15Z

pkg/installmanager/installmanager.go

 	fileList := []string{
 		filepath.Join(m.WorkDir, "openshift-install"),
 		filepath.Join(m.WorkDir, "oc"),
 	}
 	m.waitForFiles(fileList)
+
+	// copy each binary to /bin to avoid situations where the workdir may be mounted with noexec.
+	// (i.e. kind + podman)


Are you trying to say that "kind + podman" is the situation or an example of a situation?

Was originally meant to be an example. Clarifying a little bit.

staebler · 2020-05-21T12:50:57Z

pkg/installmanager/installmanager.go

@@ -466,12 +471,32 @@ func (m *InstallManager) waitForFiles(files []string) {
 	m.log.Infof("all files found, ready to proceed")
 }

-func (m *InstallManager) waitForInstallerBinaries() {
+func (m *InstallManager) waitForInstallerBinaries() error {


The name of this function should be changed. It is doing more than just waiting for binaries now. It would seem that its main purpose now is to copying the binaries to the needed location.

staebler · 2020-05-21T13:12:12Z

hack/create-kind-podman-cluster.sh

+# This script creates a kind cluster and configures it to use an insecure registry
+# running on the host OS.
+#
+# USAGE: sudo DEV_USER=myusername ./hack/create-kind-cluster-podman.sh [cluster_name]


Is the advice about using sudo and DEV_USER left over from a previous iteration?

Yes good catch thanks. Switched to in-script sudo so it could more easily get you a kubeconfig you could use as a non-root user.

/output is getting mounted with noexec in this environment and thus we quickly fail all provisions as we can't run the openshift-install binary. Copy the binaries to /home/hive as soon as we see they've been written to /output where we know we can exec them. Uses cp vs any Go copy just for simplicity and to avoid memory problems.

dgoodwin · 2020-05-21T18:32:29Z

Updated!

staebler

The changes look good to me. But I have not tested it personally where I know others have, so I will refrain from giving a /lgtm.

joelddiaz · 2020-05-21T19:12:31Z

/lgtm

openshift-ci-robot · 2020-05-21T19:12:45Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dgoodwin, joelddiaz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [dgoodwin,joelddiaz]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2020-05-21T21:08:37Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2020-05-21T21:34:32Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 11, 2020

openshift-ci-robot requested review from staebler and twiest May 11, 2020 18:34

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 11, 2020

dgoodwin commented May 11, 2020

View reviewed changes

joelddiaz mentioned this pull request May 12, 2020

docker vs podman emptyDir mount options (exec vs noexec) kubernetes-sigs/kind#1581

Closed

dgoodwin force-pushed the kind-podman branch from ebe2c72 to 98260cf Compare May 14, 2020 17:12

dgoodwin changed the title ~~WIP: Experimental kind podman support for hive development.~~ Experimental kind podman support for hive development. May 20, 2020

openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 20, 2020

dgoodwin commented May 20, 2020

View reviewed changes

pkg/installmanager/installmanager.go

return destTarball, nil

}

Copy link

Contributor Author

dgoodwin May 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was dead code.

dgoodwin force-pushed the kind-podman branch from 98260cf to 97da1d5 Compare May 20, 2020 12:27

joelddiaz reviewed May 20, 2020

View reviewed changes

staebler reviewed May 21, 2020

View reviewed changes

dgoodwin added 2 commits May 21, 2020 15:31

Experimental kind podman support for hive development.

d43e166

dgoodwin force-pushed the kind-podman branch from 97da1d5 to 587fb83 Compare May 21, 2020 18:32

staebler reviewed May 21, 2020

View reviewed changes

openshift-ci-robot assigned joelddiaz May 21, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label May 21, 2020

openshift-merge-robot merged commit caf770f into openshift:master May 22, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Experimental kind podman support for hive development. #982

Experimental kind podman support for hive development. #982

dgoodwin commented May 11, 2020

dgoodwin May 11, 2020

dgoodwin May 11, 2020

dgoodwin commented May 12, 2020

joelddiaz commented May 12, 2020

dgoodwin commented May 12, 2020

csrwng commented May 12, 2020

dgoodwin commented May 20, 2020

dgoodwin May 20, 2020

dgoodwin commented May 20, 2020

staebler commented May 20, 2020

joelddiaz left a comment

dgoodwin commented May 21, 2020

staebler May 21, 2020

dgoodwin May 21, 2020

staebler May 21, 2020

staebler May 21, 2020

dgoodwin May 21, 2020

dgoodwin commented May 21, 2020

staebler left a comment

joelddiaz commented May 21, 2020

openshift-ci-robot commented May 21, 2020

openshift-bot commented May 21, 2020

openshift-bot commented May 21, 2020

Experimental kind podman support for hive development. #982

Experimental kind podman support for hive development. #982

Conversation

dgoodwin commented May 11, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dgoodwin commented May 12, 2020

joelddiaz commented May 12, 2020

dgoodwin commented May 12, 2020

csrwng commented May 12, 2020

dgoodwin commented May 20, 2020

Choose a reason for hiding this comment

dgoodwin commented May 20, 2020

staebler commented May 20, 2020

joelddiaz left a comment

Choose a reason for hiding this comment

dgoodwin commented May 21, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dgoodwin commented May 21, 2020

staebler left a comment

Choose a reason for hiding this comment

joelddiaz commented May 21, 2020

openshift-ci-robot commented May 21, 2020

openshift-bot commented May 21, 2020

openshift-bot commented May 21, 2020