Introduce end-to-end performance test bed

[tigrannajaryan]

The goal is to a create a test best - controlled environment and tools for conducting performance tests for the OpenTelemetry Agent, including reproducible short-term benchmarks, long-running stability tests and maximum load stress tests.

The test bed must allow specifying a configuration of the test, the load that needs to be generated, and be able to run and produce machine and human readable results. It should be possible to run the test bed locally on a developer machine or on a cloud.

Here is a more detailed proposal:
https://docs.google.com/document/d/1omU06mBYGY0slT1yojttn9BCyp18pHaRjkkYZrY8H4Q/edit#heading=h.9pln30mjg237

[tigrannajaryan]

@flands can you assign this to me?

[tigrannajaryan]

Thanks, @SergeyKanzhelev

[tigrannajaryan]

@bogdandrutu @SergeyKanzhelev I have the first version of the performance test bed implemented and running. The repo is here https://github.com/Omnition/opentelemetry-service-testbed and the execution results look like this https://github.com/Omnition/opentelemetry-service-testbed/blob/master/tests/results/TESTRESULTS.md

We would like to contribute this to OpenTelemetry. The only question is if it possible for me and one more developer from Omnition to be assigned as Maintainers on this particular repo. We plan to continue to actively improve the test bed and want to make sure we will not be blocked during the active development phase (which can happen if someone else outside the company needs to approve the PRs). Do you think it is possible to do?

[SergeyKanzhelev]

@tigrannajaryan repos looks to be private. If those are generic enoguh - we should definitely pick those up.

I assume you want a separate repository, not this one. Every repo is bootstrapped with a couple of maintainers. Ideally from different companies. So the standard process of becoming approver and maintainer can work. As we are in early stages we are expediting this process. So it is absolutely possible to make a repo and assign maintainers quickly. However, I'd suggest to get support from anybody from other company to have a "healthy" repository.

[tigrannajaryan]

@SergeyKanzhelev Sorry, I have just made it public if you want to have a look.
Let me see if there is anyone else from other companies interested in being an additional maintainer.

[SergeyKanzhelev]

@bogdandrutu @@songy23 any of you interested?

[songy23]

Feel free to add me, or we can just reuse the code owners in this repo.

[tigrannajaryan]

Thanks @songy23 :-)

@SergeyKanzhelev would it be enough to bootstrap the testbed with me, @pjanotti (also from Omnition) and @songy23 as maintainers?

[SergeyKanzhelev]

Absolutely. I see no problems with this. I can set everything up later tonight or in the morning tomorrow.

[tigrannajaryan]

Thanks @SergeyKanzhelev

Let me know if there is anything that I can help with. You can either fork our repo and I will delete our copy or I can transfer ownership of our repo to open-telemetry Github org. Whichever you prefer.

[SergeyKanzhelev]

You can send PR here: https://github.com/open-telemetry/opentelemetry-service-testbed

I will make you maintainers once PR will be approved in community repo. Just in case anybody have concerns

[tigrannajaryan]

@SergeyKanzhelev here is the PR: https://github.com/open-telemetry/opentelemetry-service-testbed/pull/2
Thank you.

[bogdandrutu]

QQ: Why this cannot live in the same repo with the service?

[tigrannajaryan]

@bogdandrutu It can.

The only reason I wanted it to be separate is to be able to get maintainer rights, which I thought would be easier for this small subset of codebase. The reason I want to have maintainer rights is that we (at Omnition) are currently very actively investing into the testbed and adding end-to-end tests in order to improve our confidence in the Service and have automated track record of performance improvements we work on.

I am not a maintainer on the Service repo, and it would make the development of the testbed much slower if I needed to wait for other maintainers to review and give approval.

If you think it is possible for me apply and become a maintainer of the Service repo that would solve the problem.

[SergeyKanzhelev]

It will be great if eventually tests would include some unusual combinations of arguments like span with a lot of links and such. And those can be used by other repositories. At least this was my expectation for creating a separate repo =)

If permissions is the only problem - we should fix it and ask maintainers to iterate faster

[tigrannajaryan]

I think from technical perspective it can work with either approach, in the same repo or separate.

From organizational perspective it probably makes a little more sense to have the tests in the same repo since it is highly likely that the same people work the Service will also work on the test bed.

My only concern is the ability to move at sufficient velocity. I have internal deadlines which will be impossible to achieve if the PR review turnaround is a few days (which is the typical time I have experienced so far). If we are not ready to move at a higher pace the alternate approach is for us to keep the testbed as an Omnition repo, iterate on it fast internally and then contribute the end result as a turnkey testbed.

I am happy with either approach, just let me know how you would like to move forward.

[SergeyKanzhelev]

@bogdandrutu you are one of maintainers on service. Can you please decide on how to move forward here.

[bogdandrutu]

We should stay with this repo.

[bogdandrutu]

@tigrannajaryan when you are ready please move the code here when you feel ready.