[Elastic Agent V2]: Add support for service based components (aka. Endpoint)

[aleksmaus]

Describe the enhancement:

Add support for service based components (aka. Endpoint)
This works should be wired in here:
https://github.com/elastic/elastic-agent/blob/feature-arch-v2/pkg/component/runtime/runtime.go#L62

The implementation in progress addresses the basic operations such install/start/stop/uninstall and periodic service status check.

We need to collaborate with the endpoint team probably in order to figure how to get more detailed status from the endpoint.
Are we going to establish gRPC connection between Endpoint and the Agent? How do we pass the connection info to the Endpoint service? etc.

[brian-mckinney]

Hi @aleksmaus. This is likely on my plate, but I am unsure of what is needed from endpoint. Can you provide some more detail about what you are looking for and what we (Endpoint) need to get done?

cc @softengchick

[aleksmaus]

Hi @aleksmaus. This is likely on my plate, but I am unsure of what is needed from endpoint. Can you provide some more detail about what you are looking for and what we (Endpoint) need to get done?

cc @softengchick

@brian-mckinney I'm not too familiar how the comms between elastic agent and the endpoint is established at the moment. If you know more, let me know. At this point I have the wrapper that can install/uninstall/start/stop/check status of the service.

Looking at how the comms creds are passed to the service in V1, looks like it's done here:

elastic-agent/internal/pkg/core/plugin/service/app.go

Line 334 in 9e1ccc9

lis, err := net.Listen("tcp", fmt.Sprintf("127.0.0.1:%d", a.credsPort))

Is this still valid?

Looks like the proto has been extended a bit
from

elastic-agent/pkg/core/server/server.go

Line 543 in 9e1ccc9

connInfo := &proto.ConnInfo{

to (added some Services)
https://github.com/elastic/elastic-agent/blob/feature-arch-v2/pkg/component/runtime/runtime_comm.go#L116

From the Endpoint perspective it would need to adopt V2 protocol additions.

[AndersonQ]

folks, I don't know if you have seen this issue: #761
tl;dr: when chanching the output, the elastic-agent doesn't restart endpoint-security and tries to another, that never starts as the grpc port is already take by the 1st endpoint-security.

I believe it'd be good to ensure we have a test or manual test scenario covering that to ensure we don't accidentally carry over that issue to V2

[aleksmaus]

@brian-mckinney I opened the draft PR, will need to test more and probably work with you to integrate Endpoint. I tested with the latest Endpoint binary it was connecting and picking up the creds, but I was not getting any status updates back.
Need to dig more in the agent (maybe something else is not working there) but I suspect the endpoint would need to be updated as well to provide the new status updates on V2 protocol.

[aleksmaus]

I believe it'd be good to ensure we have a test or manual test scenario covering that to ensure we don't accidentally carry over that issue to V2

The implementation is mostly new code, has unit tests coverage for different pieces, more tests should be implemented probably as a part of e2e integration tests.

Looking at the latest existing V1 code, it seems like it should close the creds listener port correctly when the agent stops the "application". Would be nice to have e2e coverage for V1 already.

[jlind23]

Closing this as per PR status