Support the shipper as an instance of Filebeat configured to use the shipper protocol input

[cmacknz]

This is the companion issue for elastic/beats#35135 where we will implement a shipper gRPC input for Filebeat.

Currently the shipper is implemented with the elastic-agent-shipper binary defined in https://github.com/elastic/elastic-agent-shipper. We want to simplify the shipper implementation to a shipper protocol input in Filebeat.

When the shipper feature flag is set for an output in an agent policy, the agent should start an instance of Filebeat. This is the configuration used to enable the shipper for reference:

outputs:
  default:
    type: elasticsearch
    hosts: https://localhost:9200
    shipper:
       enabled: true

For now, we can continue to send the same set of input and output units to the Filebeat based shipper that we were sending to the previous implementation of the shipper. This may change based on the design in elastic/beats#35135.

At a minimum, the Filebeat spec file will need to be modified to declare that it supports the new shipper gRPC input and also that it implements the data shipper as is currently done in https://github.com/elastic/elastic-agent-shipper/blob/main/elastic-agent-shipper.spec.yml#L2-L21:

shippers:
  - name: shipper
    description: "Elastic Agent Shipper"
    platforms:
      - linux/amd64
      - linux/arm64
      - darwin/amd64
      - darwin/arm64
      - windows/amd64
      - container/amd64
      - container/arm64
    outputs:
      - elasticsearch
      - kafka
      - logstash
      - redis
    command: &command
      args:
        - "-E"
        - "logging.level=info"

After this change the agent will see two binaries declaring that the implement the shipper, Filebeat and elastic-agent-shipper. We should remove the shipper support block above from the elastic-agent-shipper repository to avoid this conflict.

We will remove the elastic-agent-shipper binary from the agent packaging and release jobs separately once we have proven that the Filebeat based approach works.

[jlind23]

Closing this as it is not relevant anymore.
cc @ycombinator