e2e: deploy Nomad from testing framework #6949
Conversation
tgross
force-pushed
the
e2e_framework_updates
branch
from
d9cae97 to
cd1db7a
Compare
| // TODO: these override each other. local_file > sha > version | ||
| // but we should assert at most 1 is set. | ||
| var fProvisionNomadLocalBinary = flag.String("nomad.local_file", "", | ||
| "provision this specific local binary of Nomad (ignored for no-op provisioning).") |
| @@ -50,6 +116,10 @@ type Environment struct { | |||
|
|
|||
| // New creates a Framework | |||
| func New() *Framework { | |||
| flag.Parse() | |||
There was a problem hiding this comment.
Our flag parsing is pretty complex (and there are TODOs to add more), it might be time to create a flag parser (which we can test), call parse on that and provide a resulting config object into New. Just a thought for later
e2e/framework/framework.go
Outdated
| Name: name, | ||
| ExpectConsul: s.Consul, | ||
| ExpectVault: s.Vault, | ||
| }) | ||
| if err != nil { | ||
| return false, fmt.Errorf("could not provision cluster for case: %v", err) | ||
| } | ||
| defer f.provisioner.DestroyCluster(info.ID) | ||
| defer f.provisioner.TearDownTestCase(t, info.ID) |
There was a problem hiding this comment.
defer inside a loop is often a bug (and linters warn against it) - previously DestroyCluster was a noop and so it didn't really matter, but now I'm thinking we actually want to tear down each test case after it runs, otherwise I think we're putting the provisioned cluster into a kind of accumulated state as each Case runs
There was a problem hiding this comment.
Yup, you're totally right. Will fix.
There was a problem hiding this comment.
Fixed in aa3b15d. I want to make sure we're still defering the teardown so that t.Fatal doesn't prevent the teardown step from happening, so I factored this inner loop out into a runCase function so that we run the deferred teardown at the end of that scope.
| if err != nil && opts.ExpectConsul { | ||
| return nil, err | ||
| if err != nil { | ||
| return nil, fmt.Errorf("expected Consul: %v", err) |
There was a problem hiding this comment.
i really like errors.Wrapf(err, "expected Consul") (from github.com/pkg/errors)
There was a problem hiding this comment.
I don't have a strong opinion on it, but it looks like we've avoided that in all but one spot in the Nomad codebase, so I'd like to keep the existing idiom if we can? IIRC there's some cool new error wrapping features in a newer golang, so maybe there's a codebase-wide improvement to be had once we get bumped to the right minimum version.
tgross
force-pushed
the
e2e_framework_updates
branch
from
aa3b15d to
c9ca2a9
Compare
The e2e framework instantiates clients for Nomad/Consul but the provisioning of the actual Nomad cluster is left to Terraform. The Terraform provisioning process uses `remote-exec` to deploy specific versions of Nomad so that we don't have to bake an AMI every time we want to test a new version. But Terraform treats the resulting instances as immutable, so we can't use the same tooling to update the version of Nomad in-place. This is a prerequisite for upgrade testing. This changeset extends the e2e framework to provide the option of deploying Nomad (and, in the future, Consul/Vault) with specific versions to running infrastructure. This initial implementation is focused on deploying to a single cluster via `ssh` (because that's our current need), but provides interfaces to hook the test run at the start of the run, the start of each suite, or the start of a given test case. The input JSON for configuring provisioning targets can be generated by Terraform (or written by hand). That will be under a separate PR.
tgross
force-pushed
the
e2e_framework_updates
branch
from
dadf620 to
539867f
Compare
|
I've rebased this and the target branch off master and did a full end-to-end test (including Windows) with #6948. Once tests pass I'll merge this and then open a PR with the combined branch vs master. |
|
I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions. |
The e2e framework instantiates clients for Nomad/Consul but the provisioning of the actual Nomad cluster is left to Terraform. The Terraform provisioning process uses
remote-execto deploy specific versions of Nomad so that we don't have to bake an AMI every time we want to test a new version. But Terraform treats the resulting instances as immutable, so we can't use the same tooling to update the version of Nomad in-place. This is a prerequisite for upgrade testing.This changeset extends the e2e framework to provide the option of deploying Nomad (and, in the future, Consul/Vault) with specific versions to running infrastructure. This initial implementation is focused on deploying to a single cluster via
ssh(because that's our current need), but provides interfaces to hook the test run at the start of the run, the start of each suite, or the start of a given test case.The input JSON for configuring provisioning targets can be generated by Terraform (or written by hand). That work is #6948