Add e2e testsuite for Cloud managed Elastic Agent setup

[simitt]

Once the cloud specific logic is added to the elastic-agent container command (elastic/beats#23697) new e2e tests for this "hybrid" docker image are necessary.

What to test

• Legacy mode: APM Server is started according to apm-server.yml file if present and Elastic Agent is started with Fleet Server
• Otherwise: Elastic Agent is started with Fleet Server and APM Server

The Elastic Agent takes care of the Kibana setup if necessary, the Fleet Server setup for the cloud managed policy and enrolling itself via Fleet Server.

The setup takes care via ENV variables and config files. Details to follow when the above linked issue is implemented.

[EricDavisX]

note we have relating tickets for general Fleet Server support which are being worked now. @simitt will you be the best POC to pair with to work through this? The Engg Productivity team typically can help start off a project and do training / first-few-scenarios to help test. @mdelapenya is our first POC typically.

So, we'll have some building blocks already available, but we'll probably need a lot more specifics pulled out of feature tickets or re-posted as to the specific APIs we need to cover or which policies and what to query in the Kibana API or ES index to validate it is working as desired.

[jalvz]

I am available for help here too

[simitt]

More details on the expected setup (describing how to configure via files rather than ENV):

Providing config files with according information:

• config/apm-server.yml (containing all necessary credentials and URLs)
• config/credentials.yml

fleet_server:
  #cert: ""
  #cert_key: ""
  elasticsearch:
    host: ""
    username: ""
    password: ""
    #ca: ""

kibana:
  fleet:
    host: ""
    username: ""
    password: ""
    #ca: ""

• config/fleet-setup.yml

fleet:
  enroll: true
  force: false
  insecure: true
  url: "<fleet-server-url>"
fleet_server:
  enable: true
  host: "0.0.0.0" 
  port: "8220"
kibana:
  fleet:
    setup: true

What should be covered:

by providing following ENV

ELASTIC_AGENT_CLOUD=true 
APM_SERVER_PATH=<your-path>/apm-server/ 
STATE_PATH=<your-path/elastic-agent> 
HOME_PATH=<your-path> 
CONFIG_PATH=<your-path>/config 
LOGS_PATH="<your-path>/logs/" 
DATA_PATH="<your-path>/data/" 
HTTPPROF=":5066" 

Testing that APM Server standalone (not managed as integration) gets set up.

• should extract the APM Server from the Elastic Agent into the given directory,
• write APM Server logs to the given logs path
• enable the expvar endpoint at the given port
• startup APM Server and ensure it is reachable
• create fleet server policy and enroll elastic agent into the policy
• not certain whether other tests already cover certificates; if not also add tests for the trusted setup.

Test that Kibana/Fleet gets set up, a policy is created and the Elastic Agent is successfully enrolled

• write Elastic Agent json logs to the STATE_PATH

When changing ELASTIC_AGENT_CLOUD=false or not providing an APM_SERVER_PATH the standalone apm-server test that the APM Server is not extracted nor started

Test that when neither the fleet-setup.yml file nor the credentials.yml file exists the elastic-agent is not started.

Updated 2021/04/21 with description of current functionality

@jalvz are you available to pick this up?

[jalvz]

Yes, I was waiting on #900, but maybe this can be worked on in parallel.

[simitt]

Yeah that makes sense to first sort out any Fleet Server issues. Thanks!

[simitt]

@jalvz I updated the description above according to current functionality.

In alignment with https://github.com/elastic/cloud-assets/pull/589 we should also cover that a managed, non default policy is created during the setup to which the elastic-agent enrolls. This is already supported by Kibana, but hasn't been merged to the cloud code yet; there is one open beats issue elastic/beats#25133 we have to solve before this can also be tested.

[simitt]

Update to #890 (comment):
The mentioned PRs have been merged and backported. The only issue left open is that the preconfigured policy does not require to be the default policy elastic/kibana#97673.
https://github.com/elastic/cloud-assets/blob/7.13/stackpack/kibana/config/kibana.yml#L157-L176. shows how a preconfigured, managed, default fleet server policy can be configured via kibana.yml.