DevOps Surge!

[CEHENKLE]

One of the lessons I learned from the OpenSearch 1.0 release is that the current development automation is not where we want it to be. There are, for example, too many manual processes that happen behind closed doors. Setting up good infrastructure is hard, but every hour spent automating tasks is rapidly paid back in full when everyone is able to build faster and release with greater confidence.

To get us caught up project where it needs to be, I’m going to shift some folks over (surge) on to working on developer operations (DevOps) == a DevOps Surge!

Principles

1. Security is the top priority.
2. Continuous Integration (CI) makes development faster, not slower.
3. Outputs can be trusted, as all CI is automated and easily reproducible by anyone, and produces signed artifacts.
4. Logs and outputs of all CI is accessible and visible to anyone.
5. Anyone can contribute to CI and preview results before changes are merged.
6. CI infrastructure is not provider locked, and can be quickly rebuilt on another provider if needed.
7. CI scripts run the same commands that developers use to build on their desktops.
8. CI infrastructure scales horizontally to accommodate parallel execution of builds and tests without being reworked.
9. OpenSearch plugins commit to project-wide infrastructure decisions.

High Level Goals:

You don’t have to work at AWS to produce a full distribution of OpenSearch!

• Too many parts of the build and bundle process are not reproducible outside of AWS. The goal is to make it so that anyone in the community can create a fresh build at any time.

The OpenSearch project team can produce a release without relying on any AWS-internal collaboration tools (for document sharing, conducting meetings, etc)!

• We have tests and tests frameworks that give us confidence our code is being exercised. The output of those tests need to be easily inspect visible and regularly produced so we don’t have to chase people to find out the state of the build. Today we have to poke each plug in to create a build, verify it and then manually assemble it.

The OpenSearch project team has a repeatable process to create the bits we ship!

• There are no manual steps in the creation or assembly of the distribution.

The OpenSearch project team has a path to objectively demonstrate that the bits we ship work the way we expect them to!

• We may not have all plugins at 100% test coverage for all types of testing initially, but we have hooks in place to do all the types of testing we are doing manually today (unit, integration, performance, backwards compatibility, upgrade)

To accomplish this work, we’ll be tracking two different projects:

• OpenSearch Continuous Build Deployment and Publishing
• OpenSearch Testing Automation

and the key meta issues to follow are:

• GA release process
• Bundle build
• Components build

Since a huge part of this effort is moving things out into the public, it’ll be harder for folks outside of AWS to contribute until the first round of work is done. But I’d love to hear your thoughts on our approach and I look forward to being able to collaborate more fully.

Thanks,
/C

[elfisher]

Let me know if I should open a separate issue/proposal for this. It would be really interesting to see if longer-term we can work towards having build logic for distributions (deb, arch, etc...) live in independent repos and have those build processes incorporated into the overall build/publishing process.

[bbarani]

@elfisher build logic to create a distribution is already present in a dedicated release folder. Is there a rationale to move them to separate repos?

[elfisher]

@elfisher build logic to create a distribution is already present in a dedicated release folder. Is there a rationale to move them to separate repos?

My thinking is more modularity to allow for independent development iteration outside of the main repo where we assemble all the build steps.

[CEHENKLE]

As we've been working through issues in the repos, we've settled in on this heuristic that I wanted to capture here:

If we're taking actions on a single repo, we'll use github actions.
If we're taking actions across multiple repos (building, running tests) we'll use Jenkins.

We're also trying to make sure we're making sure we're keeping separation of concerns for the different workflows.

[peternied]

[Triage] @bbarani can you track this completion of the surge and update this issue?

[dblock]

Closing.