Kibana Fleet setup failed: http POST request to .... fails: Unauthorized

[mtojek]

Hi Team,

Today I faced another flaky error while starting the elastic-package stack up --version 8.0.0-SNAPSHOT:

Attaching to elastic-package-stack_fleet-server_1
�[36mfleet-server_1               |�[0m Performing setup of Fleet in Kibana
�[36mfleet-server_1               |�[0m 
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Unauthorized: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"Unauthorized"}
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Unauthorized: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"Unauthorized"}
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Unauthorized: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"Unauthorized"}
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Unauthorized: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"Unauthorized"}
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Unauthorized: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"Unauthorized"}
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Unauthorized: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"Unauthorized"}
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Unauthorized: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"Unauthorized"}
...
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Unauthorized: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"Unauthorized"}
�[36mfleet-server_1               |�[0m Error: http POST request to http://kibana:5601/api/fleet/setup fails: Unauthorized: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"Unauthorized"}
�[36mfleet-server_1               |�[0m For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.0/fleet-troubleshooting.html

�[36melastic-agent_1              |�[0m Kibana fetch policy failed: http GET request to http://kibana:5601/api/fleet/agent_policies fails: Unauthorized: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"Unauthorized"}
�[36melastic-agent_1              |�[0m Kibana fetch policy failed: http GET request to http://kibana:5601/api/fleet/agent_policies fails: Unauthorized: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"Unauthorized"}
�[36melastic-agent_1              |�[0m Kibana fetch policy failed: http GET request to http://kibana:5601/api/fleet/agent_policies fails: Unauthorized: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"Unauthorized"}
�[36melastic-agent_1              |�[0m Kibana fetch policy failed: http GET request to http://kibana:5601/api/fleet/agent_policies fails: Unauthorized: <nil>. Response: {"statusCode":401,"error":"Unauthorized","message":"Unauthorized"}
�[36melastic-agent_1              |�[0m Error: unable to find policy named ""
�[36melastic-agent_1              |�[0m For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.0/fleet-troubleshooting.html

Failed build is here: https://beats-ci.elastic.co/blue/organizations/jenkins/Ingest-manager%2Felastic-package/detail/PR-633/10/pipeline

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[mtojek]

Ok, so with this pull request I confirm that this is a breaking issue, which broke down our master.

We definitely shouldn't push changes this way as it will slow us all down (integrations developers impacted). I'm leaving it here for further investigation. cc @jlind23

[mdelapenya]

@mtojek I was able to reproduce the error described here even with last version of the elastic-agent AND this version of the stack 8.1.0-c1a942c7-SNAPSHOT:

$ elastic-package stack up --daemon --verbose --version 8.1.0-c1a942c7-SNAPSHOT --services elasticsearch,fleet-server,kibana -p default

2021/12/29 12:22:19  WARN CommitHash is undefined, in both /Users/mdelapenya/.elastic-package/version and the compiled binary, config may be out of date.
2021/12/29 12:22:19 DEBUG Enable verbose logging
2021/12/29 12:22:19 DEBUG Distribution built without a version tag, can't determine release chronology. Please consider using official releases at https://github.com/elastic/elastic-package/releases
Boot up the Elastic stack
Using profile /Users/mdelapenya/.elastic-package/profiles/default.
Remember to load stack environment variables using 'eval "$(elastic-package stack shellinit)"'.
Packages from the following directories will be loaded into the package-registry:
- built-in packages (package-storage:snapshot Docker image)
2021/12/29 12:22:19 DEBUG running command: /usr/local/bin/docker-compose version --short
2021/12/29 12:22:20 DEBUG Determined Docker Compose version: <nil>, the tool will use Compose V1
2021/12/29 12:22:20 DEBUG running command: /usr/local/bin/docker-compose -f /Users/mdelapenya/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack build elasticsearch elasticsearch_is_ready fleet-server fleet-server_is_ready kibana kibana_is_ready
elasticsearch uses an image, skipping
elasticsearch_is_ready uses an image, skipping
kibana uses an image, skipping
kibana_is_ready uses an image, skipping
fleet-server uses an image, skipping
fleet-server_is_ready uses an image, skipping
2021/12/29 12:22:20 DEBUG running command: /usr/local/bin/docker-compose version --short
2021/12/29 12:22:21 DEBUG Determined Docker Compose version: <nil>, the tool will use Compose V1
2021/12/29 12:22:21 DEBUG running command: /usr/local/bin/docker-compose -f /Users/mdelapenya/.elastic-package/profiles/default/stack/snapshot.yml -p elastic-package-stack up -d elasticsearch elasticsearch_is_ready fleet-server fleet-server_is_ready kibana kibana_is_ready
Creating network "elastic-package-stack_default" with the default driver
Creating elastic-package-stack_package-registry_1 ... done
Creating elastic-package-stack_elasticsearch_1    ... done
Creating elastic-package-stack_kibana_1                 ... done
Creating elastic-package-stack_elasticsearch_is_ready_1 ... done
Creating elastic-package-stack_kibana_is_ready_1        ... done
Creating elastic-package-stack_fleet-server_1           ... done

ERROR: for fleet-server_is_ready  Container "9886b52aa93f" is unhealthy.
ERROR: Encountered errors while bringing up the project.
Error: booting up the stack failed: running docker-compose failed: running command failed: running Docker Compose up command failed: exit status 1

On the other hand, running the very same command with a stack version from 5 days ago (which we know is correct from testing standpoint), 8.1.0-dbc834fd-SNAPSHOT:

$ elastic-package stack up --daemon --verbose --version 8.1.0-dbc834fd-SNAPSHOT --services elasticsearch,fleet-server,kibana -p default
...

then the execution succeeds.

This makes me think something changed in Kibana 5 days ago

[michel-laterman]

Correct me if I'm wrong, but the ES instance used in elastic-package stack up has elastic/changeme as the default credentials, correct?
in 7.x, when setting up Kibana the agent to attempts to get credentials (basic auth or token) from the env with elastic/changeme as defaults
when I removed basic auth from the agent/fleet it broke this assumption. So our options are:

1. revert the removal of basic auth for agent->kibana communications (https://github.com/elastic/beats/pull/29458/files#diff-1f8e1e2079524931fa8f3cf121667a25baa2b998557136500d5e7596b46e9187L488); just to fix kibana fleet setup
2. alter elastic-package to generate and use a service_token for the agent

I'm not sure if 1 will completely resolve all the issues but it may be easier to do then option 2 (which is more correct)
cc @jlind23 @ruflin

[blakerouse]

There is another option that I suggest and that is to only add the ability for the container subcommand to use the username/password and generate the service token.

I think without doing that the whole docker-compose up for any users gets more complicated.

[jlind23]

Must be solved by elastic/kibana#122320

[mtojek]

Hi @michel-laterman @jlind23,

I'm afraid we might be missing something as the compose boot-up still fails (see: elastic/elastic-package#640 and pipeline) for 8.0.0-SNAPSHOT.

Is there something else that needs to be changed in the elastic-package?

[jlind23]

@michel-laterman it seems that you forgot to validate the backport PR, am I correct?

cc @mtojek

[michel-laterman]

correct, the backport has been merged

[mtojek]

I'm afraid we have to wait until the unified release process succeeds eventually.

[mtojek]

Status update on the release pipeline

The unified snapshot release pipeline is still red, so we couldn't validate if the fix works.

[michel-laterman]

@mtojek are we able to close this issue?

[mtojek]

Absolutely, let me close it. Thanks for your help!