Allow for users to specify Golang and Docker versions and RPM repos #459

stevekuznetsov · 2016-06-09T18:50:03Z

In order to allow for better configuration of our AMI builds, four
new flags are added to the vagrant build-origin-base command:
--repofile: to specify repo files to install
--repo: to specify repositories to enable for Golang/Docker
--golang-version: to specify Golang version
--docker-version: to specify Docker version

Any repositories installed from files provided with --repofile
will be enabled only for the installation of Golang and Docker,
and will be enabled for those installs by default, unless the user
provides repositories with --repo to override the default.

The --{golang,docker}-version flags are used to construct package
names for the core package as well as for peripheral packages, for
instance golang and golang-src. Provide these versions with this
behavior in mind.

Signed-off-by: Steve Kuznetsov [email protected]

@danmcp PTAL impl, will test when we agree on it

stevekuznetsov · 2016-06-09T18:51:46Z

lib/vagrant-openshift/action/install_origin_base_dependencies.rb

+              })
+            @options[:reponames] << "dockerextra"
+
+            File.open("/etc/yum.repos.d/origin-deps-rhel7.repo", "w").write(%{


I don't see a good reason to be fetching this file from the web each time, but we can keep that. I like this approach as it's closer to the other hard-coded repo and because converting the wget to Ruby would mean we require 'open-uri' above and I'm not sure if that is totally ok (will that dep always be present?).

danmcp · 2016-06-10T17:35:24Z

lib/vagrant-openshift/action/install_origin_base_dependencies.rb

-fi
-          }, :timeout=>60*10, :verbose => false)
+          # Install base dependencies that we cannot continue without
+          sudo(@env[:machine], "~/install_dependencies.s", :timeout=>60*30, :verbose => false)


stevekuznetsov · 2016-06-13T21:08:43Z

I changed the behavior slightly. The flags are now:

$ vagrant build-origin-base --help
Usage: vagrant build-origin-base [vm-name]

        --golang.repofile [name]     Path to repository file to use when installing Golang.
        --golang.repo [name]         Name of repository to enable when installing Golang.
        --golang.version [version]   Install the specified Golang version. Leave empty to install latest available.
        --docker.repofile [name]     Path to repository file to use when installing Docker.
        --docker.repo [name]         Name of repository to enable when installing Docker.
        --docker.version [version]   Install the specified Docker version. Leave empty to install latest available.
    -h, --help                       Print this help

The behavior is to use the 'default' repos (which we only have for RHEL7) if no repofiles or reponames are given, or to use only the names of repos given in repofiles if they're given, or to use only the names given in reponames if they are given and no repofiles are given. Giving repofiles and reponames is fobidden.

We install latest from specified repos unless version is given.

To test we want to see:

vagrant build-origin-base installs Golang and Docker latest versions from origin-deps-rhel7 and dockerextra
vagrant build-origin-base --docker.version <ver> --docker.repo oso-rhui-rhel-server-extras installs whatever non-latest Docker version from oso-rhui-rhel-server-extras
vagrant build-origin-base --golang.version <ver> --golang.repo oso-rhui-rhel-server-releases-optional installs whatever non-latest Golang version from oso-rhui-rhel-server-releases-optional
vagrant build-origin-base --docker.repofile <file> --docker.version <ver> installs Docker version from the specified repo ... we could just copy the oso-rhui-rhel-server-extras repofile into a local file to use
vagrant build-origin-base --golang.repofile <file> --golang.version <ver> installs Golang version from the specified repo ... we could just copy the oso-rhui-rhel-server-releases-optional repofile into a local file to use

@danmcp thoughts?

stevekuznetsov · 2016-06-13T21:09:57Z

Without any flags, I've had the default setting (which is the same as what we had before) go until flake in e2e somewhere, so I'm fairly confident the resource file refactor is solid. Now just to test the logic.

danmcp · 2016-06-13T21:58:08Z

If you specify a golang and docker repo (or repofile) that are the same, will you dedup or add 2 of them?

vagrant build-origin-base installs Golang and Docker latest versions from origin-deps-rhel7 and dockerextra

We should change this case to pull in nothing extra. These should really be special cases already.

stevekuznetsov · 2016-06-14T11:24:26Z

If you specify a golang and docker repo (or repofile) that are the same, will you dedup or add 2 of them?

I'm not particularly interested in adding deduping logic -- what's the long-term support going to be like for this plugin? Once we move away from it as an organization, who's going to be using it? Do they need good UX?

. These should really be special cases already.

I like it. Makes the logic simpler.

stevekuznetsov · 2016-06-15T12:21:32Z

For posterity, now that default is special-case, previous default is achieved with:

$ vagrant build-origin-base --golang.repofile="${vagrant-openshift}/lib/vagrant-openshift/resources/origin-deps-rhel7.repo" \
                            --golang.version=1.6.2                                                                          \
                            --docker.repofile="${vagrant-openshift}/lib/vagrant-openshift/resources/dockerextra.repo"       \
                            --docker.repo=oso-rhui-rhel-server-releases                                                     \
                            --docker.version=1.10.3

danmcp · 2016-06-21T16:56:13Z

README.asciidoc

+==== Configuring Docker and Golang versions for tests
+
+If the version of OpenShift that is being tested requires a version
+of Golang or Docker that is not default in the RPM streams for the 


Don't reference AMI here or below.

danmcp · 2016-06-21T18:58:21Z

lib/vagrant-openshift/action/install_origin_rhel7.rb

@@ -58,6 +58,7 @@ def call(env)
 ADD vars/* /etc/yum/vars/
 ADD repos/* /etc/yum.repos.d/
 ADD certs/* /var/lib/yum/
+ADD keys/* /etc/pki/rpm-gpg/


What is this about?

This is actually exactly what I was talking about on the call - we're pulling later from repos that use these public keys for GPG package checks... but didn't have the keys

stevekuznetsov · 2016-07-06T19:22:32Z

@danmcp testing looks good:

origin/master

$ GOLANG_REPO='http://mirror.openshift.com/pub/openshift-origin/nightly/rhel-7/dependencies/x86_64/'
$ vagrant build-origin-base --golang.version=1.6.2            \
                            --golang.repourl="${GOLANG_REPO}" \
                            --docker.version=1.10.3
...
Jul 06 14:12:08   Installing : golang-1.6.2-1.el7.x86_64                                    3/3
...
Jul 06 14:12:32   Installing : docker-1.10.3-44.el7.x86_64                                  9/9

ose/master

$ GOLANG_REPO='http://mirror.openshift.com/pub/openshift-origin/nightly/rhel-7/dependencies/x86_64/'
$ vagrant build-origin-base --golang.version=1.6.2            \
                            --golang.repourl="${GOLANG_REPO}" \
                            --docker.version=1.10.3
...
Jul 06 14:16:07   Installing : golang-1.6.2-1.el7.x86_64                                    3/3
...
Jul 06 14:16:23   Installing : docker-1.10.3-44.el7.x86_64                                  9/9

ose/enterprise-3.0

$ vagrant build-origin-base --golang.version=1.4.2 \
                            --docker.version=1.8.2
...
Jul 06 14:17:22   Installing : golang-1.4.2-9.el7.x86_64                                    3/4
...
Jul 06 14:17:48   Installing : docker-1.8.2-10.el7.x86_64                                   2/2

ose/enterprise-3.1

$ vagrant build-origin-base --golang.version=1.4.2 \
                            --docker.version=1.8.2
...
Jul 06 14:20:12   Installing : golang-1.4.2-9.el7.x86_64                                    3/4
...
Jul 06 14:20:37   Installing : docker-1.8.2-10.el7.x86_64                                   2/2

ose/enterprise-3.2

$ GOLANG_REPO='http://mirror.openshift.com/pub/openshift-origin/nightly/rhel-7/dependencies/x86_64/'
$ vagrant build-origin-base --golang.version=1.6.2            \
                            --golang.repourl="${GOLANG_REPO}" \
                            --docker.version=1.9.1
...
Jul 06 14:21:50   Installing : golang-1.6.2-1.el7.x86_64                                    3/3
...
Jul 06 14:22:27   Installing : docker-1.9.1-40.el7.x86_64                                   4/4

$ GOLANG_REPO='http://mirror.openshift.com/pub/openshift-origin/nightly/rhel-7/dependencies/x86_64/'
$ vagrant build-origin-base --golang.version=1.6.2            \
                            --golang.repourl="${GOLANG_REPO}" \
                            --docker.version=1.10.3
...
Jul 06 14:23:42   Installing : golang-1.6.2-1.el7.x86_64                                    3/3
...
Jul 06 14:24:06   Installing : docker-1.10.3-44.el7.x86_64                                  9/9

stevekuznetsov · 2016-11-02T14:15:47Z

[test] me

stevekuznetsov · 2016-11-07T22:24:18Z

@danmcp I've taken the Docker and Golang installation completely out of build-origin-base since we can't do a re-install afterwords. Tests seem to be fine. I'll squash and make this pretty tomorrow, and we can get it installed Monday morning.

openshift-bot · 2016-11-07T23:31:06Z

Evaluated for vagrant openshift test up to 96a625d

openshift-bot · 2016-11-08T01:20:00Z

Vagrant OpenShift Test Results: SUCCESS (https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_vagrant_openshift/83/) (Base Commit: f06be13)

stevekuznetsov · 2016-11-08T14:33:28Z

Green tests, I'll squash and merge. @danmcp FYI

In order to allow each job building an AMI to be able to declare the Docker and Golang versions that should be present, the Docker and Golang install paths are separated out so that they live alone and are not called in the `build-origin-base` command but instead in specific `install-docker` and `install-golang` commands that can be run by a user to configfure the version of Docker or Golang installed on the system. These new commands have the following flags: --{golang,docker}.version to specify the version to install The following flags determine what repos are enabled during the installation process: --repourl to specify the URL of an RPM repo --repo to specify the name of an installed repo Any repositories installed from URLs provided with the above flags will be enabled *only* for the installation of Golang or Docker, and will be disabled thereafter. The `--{golang,docker}.version` flags are used to construct package names for the core package as well as for peripheral packages, for instance `golang` and `golang-src`. No leading dash is necessary to specify the version. For instance, a user would specify that they want `golang-1.6.2` by passing `--golang.version=1.6.2`. Repositories installed from URLs will have a .repo file created for them automatically. The name of the repository will be the given URL with all non-alphanumeric characters (/[^a-zA-Z0-9]/) stripped out. Then, the URL and name will be inserted into the following definition: ``` [#{reponame}] name=#{reponame} baseurl=#{url} enabled=1 gpgcheck=0 sslverify=0 sslclientcert=/var/lib/yum/client-cert.pem sslclientkey=/var/lib/yum/client-key.pem ``` As the sslclient{cert,key} are not used unless the RPM repository responds to a request with auth headers, these fields are ignored unless the repository needs them and therefore repositories that do not need them are not broken and it is safe to append them always. Signed-off-by: Steve Kuznetsov <[email protected]>

stevekuznetsov reviewed Jun 9, 2016
View reviewed changes

stevekuznetsov force-pushed the skuznets/branch-specific-builds branch from 0d29d7a to 876da65 Compare June 10, 2016 15:11

danmcp reviewed Jun 10, 2016
View reviewed changes

stevekuznetsov force-pushed the skuznets/branch-specific-builds branch 2 times, most recently from a78c558 to cce7a26 Compare June 10, 2016 17:43

stevekuznetsov force-pushed the skuznets/branch-specific-builds branch from 79ae3e2 to 47caf67 Compare June 14, 2016 16:47

stevekuznetsov force-pushed the skuznets/branch-specific-builds branch 9 times, most recently from 84603bf to 58b9c3f Compare June 21, 2016 15:27

danmcp reviewed Jun 21, 2016
View reviewed changes

stevekuznetsov force-pushed the skuznets/branch-specific-builds branch from 58b9c3f to e07bc0c Compare June 21, 2016 17:36

danmcp reviewed Jun 21, 2016
View reviewed changes

stevekuznetsov force-pushed the skuznets/branch-specific-builds branch from e07bc0c to 1165086 Compare July 6, 2016 19:09

openshift-bot added the needs-rebase label Aug 16, 2016

stevekuznetsov force-pushed the skuznets/branch-specific-builds branch 2 times, most recently from 8f7a349 to d2b19e0 Compare November 2, 2016 14:15

stevekuznetsov force-pushed the skuznets/branch-specific-builds branch 3 times, most recently from d9ce05a to 94db4ce Compare November 2, 2016 16:37

openshift-bot removed the needs-rebase label Nov 2, 2016

stevekuznetsov force-pushed the skuznets/branch-specific-builds branch 14 times, most recently from 20c9304 to 96a625d Compare November 7, 2016 22:23

stevekuznetsov force-pushed the skuznets/branch-specific-builds branch from 96a625d to 7ab8d5f Compare November 8, 2016 14:37

stevekuznetsov merged commit ca1e0b8 into openshift:master Nov 8, 2016

bparees mentioned this pull request Nov 8, 2016

Revert "Allow for users to specify Golang and Docker versions and RPM repos" #509

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow for users to specify Golang and Docker versions and RPM repos #459

Allow for users to specify Golang and Docker versions and RPM repos #459

stevekuznetsov commented Jun 9, 2016

stevekuznetsov Jun 9, 2016

danmcp Jun 10, 2016

stevekuznetsov commented Jun 13, 2016

stevekuznetsov commented Jun 13, 2016

danmcp commented Jun 13, 2016

stevekuznetsov commented Jun 14, 2016

stevekuznetsov commented Jun 15, 2016 •

edited

Loading

danmcp Jun 21, 2016 •

edited

Loading

danmcp Jun 21, 2016

stevekuznetsov Jun 21, 2016

stevekuznetsov commented Jul 6, 2016 •

edited

Loading

stevekuznetsov commented Nov 2, 2016

stevekuznetsov commented Nov 7, 2016

openshift-bot commented Nov 7, 2016

openshift-bot commented Nov 8, 2016

stevekuznetsov commented Nov 8, 2016

Allow for users to specify Golang and Docker versions and RPM repos #459

Allow for users to specify Golang and Docker versions and RPM repos #459

Conversation

stevekuznetsov commented Jun 9, 2016

stevekuznetsov Jun 9, 2016

Choose a reason for hiding this comment

danmcp Jun 10, 2016

Choose a reason for hiding this comment

stevekuznetsov commented Jun 13, 2016

stevekuznetsov commented Jun 13, 2016

danmcp commented Jun 13, 2016

stevekuznetsov commented Jun 14, 2016

stevekuznetsov commented Jun 15, 2016 • edited Loading

danmcp Jun 21, 2016 • edited Loading

Choose a reason for hiding this comment

danmcp Jun 21, 2016

Choose a reason for hiding this comment

stevekuznetsov Jun 21, 2016

Choose a reason for hiding this comment

stevekuznetsov commented Jul 6, 2016 • edited Loading

origin/master

ose/master

ose/enterprise-3.0

ose/enterprise-3.1

ose/enterprise-3.2

stevekuznetsov commented Nov 2, 2016

stevekuznetsov commented Nov 7, 2016

openshift-bot commented Nov 7, 2016

openshift-bot commented Nov 8, 2016

stevekuznetsov commented Nov 8, 2016

stevekuznetsov commented Jun 15, 2016 •

edited

Loading

danmcp Jun 21, 2016 •

edited

Loading

stevekuznetsov commented Jul 6, 2016 •

edited

Loading