[Meta] elastic-agent-shipper journey to GA

[leehinman]

Fill in checklists below with issues

Checklist to achieve experimental status

• end to end acknowledgment / at least once delivery guarantee with memory queue ([Elastic Agent] Implement the shipper PersistedIndex API in the Filebeat shipper input beats#35266)
• ensure meta data isn't overwritten (shipper should not replace input, etc.) #309
• verify events ingested through shipper are same as without going through shipper (Add document equivalency test case to end to end tests #290) relates to (Investigate way to prove events ingested with & without shipper are functionally the same #257)
• elastic-agent can start filebeat as shipper (Support the shipper as an instance of Filebeat configured to use the shipper protocol input elastic-agent#2521)
• shipper monitoring (Update the shipper monitoring implementation to be a variant of Beat monitoring elastic-agent#2580) and ([Elastic Agent] Implement input metrics for the shipper Filebeat input beats#35267)
• performance tests for filebeat reading from file and sending to shipper using memory queue and writing to elasticsearch have been performed (related Onboard the shipper into the automated ingest performance testing framework #266)

Checklist to achieve beta status

• support global processors
• Elasticsearch V2 output
• diskqueue is beta (see [Meta] disk queue journey to GA #118)
• performance use cases are finalized
• performance is at least close (~90%) of Beats under agent with own output
• tests exist for memory and IO usage, and comparisons to Beats under agent with own output
• can be selected as output in fleet UI
• startup, shutdown, input, output & performance issues can all be debugged with only the data from the elastic-agent diagnostics command.
• handle policy updates & queued events

Checklist to achieve ga status

• elastic-agent-shipper is default output in fleet UI
• performance is as good as current Beats under agent for all performance use cases
• disk queue is ga (see [Meta] disk queue journey to GA #118)
• output automatic tuning finalized

Previous

Below is what we had when elastic-agent-shipper was a separate repo, with all new code.
Keeping for historic reasons.

Checklist to achieve experimental status

• disk queue is experimental (see [Meta] disk queue journey to GA #118)
• Implement an MVP of the Elasticsearch output #10
• Update the shipper to work with the elastic-agent #185
• support for selectors for index / data_streams
• performance testing framework exists
• performance tests for filebeat reading from file and sending to shipper using memory queue and writing to elasticsearch have been performed
• Investigate way to prove events ingested with & without shipper are functionally the same #257
• Fix mage test:integration #244
• Fix elasticsearch output adding bulk index item #253
• Fix so shipper doesn't stall while sending via elasticsearch output #255
• include @timestamp in events sent via elasticsearch output #256

Checklist to achieve beta status

• disk queue is beta (see [Meta] disk queue journey to GA #118)
• [meta] Implement the kafka output #116
• Implement the Logstash output #129
• beat processors are available
• at least once delivery guarantee
• handle policy updates & queued events
• start output automatic tuning
• global metrics
• input metrics
• processor metrics
• output metrics
• startup, shutdown, input, output & performance issues can all be debugged with only the data from the elastic-agent diagnostics command.
• performance use cases are finalized
• performance is at least close (~90%) of Beats under agent with own output
• tests exist for memory and IO usage, and comparisons to Beats under agent with own output
• A misbehaving input should not be able to flood the shipper and prevent publishing the events of other inputs #213
• can be selected as output in fleet UI
• tests to ensure that payloads are equivalent to those from beats output
• Ensure that the story around retries is solid and consistent

Checklist to achieve ga status

• disk queue is ga (see [Meta] disk queue journey to GA #118)
• output automatic tuning finalized
• support for global processors
• elastic-agent-shipper is default output in fleet UI
• performance is as good as current Beats under agent for all performance use cases

[cmacknz]

In either the experimental or beta criteria we need an item to track that the shipper is debuggable using only the information collected by the agent diagnostics command.

Checklist to achieve beta status

• performance testing framework exists
• performance use cases are finalized

Maybe this is implicit in the two items above, but I think we really want to know how the performance of the agent with the shipper compares to the performance of the agent without shipper before we can recommend anyone use it as a beta.

I think I would rather see "performance is as good as current Beats under agent for all performance use cases" as a Beta requirement to set expectations properly for ourselves, we don't want to pursue this only at the end. If there is some unexpected challenge here we can defer it from the Beta criteria later, but ideally we can make the shipper a performance improvement.

Checklist to achieve ga status

• support for global processors

I don't think we need global processors to be GA, because this is a completely new feature. This could be done at any time.

• Output automatic tuning finalized

What does "finalized" mean here? We may want to be cautious about coupling the shipper GA criteria to a GA-able implementation of automatic output tuning. Ideally we can include this though, it is likely necessary to avoid annoying configuration migrations (for existing workers and bulk_max_size configurations).

[leehinman]

In either the experimental or beta criteria we need an item to track that the shipper is debuggable using only the information collected by the agent diagnostics command.

Added.

[leehinman]

Maybe this is implicit in the two items above, but I think we really want to know how the performance of the agent with the shipper compares to the performance of the agent without shipper before we can recommend anyone use it as a beta.

Moved the "performance as good as curent Beats under agent" up to beta

[leehinman]

I don't think we need global processors to be GA, because this is a completely new feature. This could be done at any time.

I'm in favor of moving this post GA. The reason it on the list is because we don't seem to have a list of features for MVP, so I was going off the assumption that all of the ones listed in the design doc would be needed for GA.

[leehinman]

• Output automatic tuning finalized

What does "finalized" mean here? We may want to be cautious about coupling the shipper GA criteria to a GA-able implementation of automatic output tuning. Ideally we can include this though, it is likely necessary to avoid annoying configuration migrations (for existing workers and bulk_max_size configurations).

I was thinking "finalized" would be the user facing portion, so if we need to change the configuration parameters we can up to this point, but after this we have to worry about configuration migration. Maybe it would be better to rename this to something like "finalize configuration options for GA"?

[cmacknz]

Maybe it would be better to rename this to something like "finalize configuration options for GA"?

Agreed, let's make that change to clarify this.

[faec]

support for selectors for index / data_streams

This is listed in beta but to me it seems like we might want it for experimental, or at least some partial solution -- today the shipper can only target a single hardcoded Elasticsearch index. We could easily make that single index configurable, but it would still be a single fixed index. Targeting multiple indices with a single shipper would likely require updates to the support library (I've created an issue for the main technical dependency here).

I'm not sure how we expect people to use the experimental releases, but to me it seems like sending all inputs from all sources to a single fixed index would rule out an awful lot of use cases, even for testing.

Overall the question of index / data stream selection could use a lot more clarity... I gather that at some point the output data streams will all be managed through agent, but I'm not sure we have a definite plan how that will happen. Maybe "Event index / datastream can be derived from the agent policy" should be its own item on the checklist, since getting that information from upstream is a separate process than just supporting selectors internally?

[leehinman]

Overall the question of index / data stream selection could use a lot more clarity... I gather that at some point the output data streams will all be managed through agent, but I'm not sure we have a definite plan how that will happen. Maybe "Event index / datastream can be derived from the agent policy" should be its own item on the checklist, since getting that information from upstream is a separate process than just supporting selectors internally?

The gRPC Event has the datastream field. https://github.com/elastic/elastic-agent-shipper-client/blob/a7eedbe6bd6c711eac7ee1b2f7d7cf6ea03155be/api/messages/publish.proto#L56-L63 Is that sufficient for index / data stream selection?

[leehinman]

This is listed in beta but to me it seems like we might want it for experimental, or at least some partial solution -- today the shipper can only target a single hardcoded Elasticsearch index. We could easily make that single index configurable, but it would still be a single fixed index. Targeting multiple indices with a single shipper would likely require updates to the support library (I've created an issue for the main technical dependency here).

Moved it to experimental. From the comments on #202 it looks like targeting multiple indexes should work.