Make it easier to run custom agent builds with elastic-package

[fearful-symmetry]

So, unless I'm completely skipping over something, it's not particularly easy to connect an elastic-package stack to an instance of elastic-agent you built yourself. If you run the agent independently of docker, you have to tinker with the global agent settings so the agent has a viable elasticsearch/fleet endpoint, which breaks agents running inside docker.

There also doesn't seem to be a particularly elegant way to connect the docker stack to a custom-built elastic-agent image.

Ideally, there should be some kind of flag that does most of this work:
elastic-package stack up --build-agent

Something like this shouldn't be too complicated, as it just needs to:

• Build a new copy of the elastic-agent docker container
• spin up an elastic stack orchestrated with that container

[mtojek]

Originally the elastic-package was intended to be a tool for building integrations not developing elastic-agent, but in general it might be helpful for that too.

We may consider different options here.

[sayden]

I have also found the same need. I needed to do feature X in an Integration but it turns out that I need to update first the Metricbeat code for whatever reason.

That Metricbeat code can be tested in Metricbeat, but it is only relevant when used from an Integration for whatever reason.

I have only found this situation once, though. In any other situation I can be confident that "if it works in Metricbeat, it will work as a Integration"

Ideally for Metricbeat based integrations (happy rainbow dreaming without considering complexity) it will use a Metricbeat binary that I can pass with an argument, no need to compile it too.