Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed

[mtojek]

Hi Team,

we're facing issues with Elastic Agent with elastic-package stack up --version 8.1.0-SNAPSHOT -v -d in 8.1.0-SNAPSHOT. Would you mind taking a look? It's blocking for Integrations developers using the newest stack (Kubernetes development).

elastic-agent:

Attaching to elastic-package-stack_elastic-agent_1
�[36melastic-agent_1              |�[0m Requesting service_token from Kibana.
�[36melastic-agent_1              |�[0m Created service_token named: token-1641976304637
�[36melastic-agent_1              |�[0m Error: unable to find enrollment token named "Default" in policy "Default policy"
�[36melastic-agent_1              |�[0m For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.1/fleet-troubleshooting.html

(source)

fleet-server:

Attaching to elastic-package-stack_fleet-server_1
�[36mfleet-server_1               |�[0m Requesting service_token from Kibana.
�[36mfleet-server_1               |�[0m Created service_token named: token-1641976248071
�[36mfleet-server_1               |�[0m Performing setup of Fleet in Kibana
�[36mfleet-server_1               |�[0m 
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^[a-z][A-Za-z0-9]*$ (found '.fleet');: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^... (truncated)
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^[a-z][A-Za-z0-9]*$ (found '.fleet');: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^... (truncated)
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^[a-z][A-Za-z0-9]*$ (found '.fleet');: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^... (truncated)
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^[a-z][A-Za-z0-9]*$ (found '.fleet');: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^... (truncated)
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^[a-z][A-Za-z0-9]*$ (found '.fleet');: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^... (truncated)
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^[a-z][A-Za-z0-9]*$ (found '.fleet');: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^... (truncated)
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^[a-z][A-Za-z0-9]*$ (found '.fleet');: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^... (truncated)
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^[a-z][A-Za-z0-9]*$ (found '.fleet');: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^... (truncated)
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^[a-z][A-Za-z0-9]*$ (found '.fleet');: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^... (truncated)
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^[a-z][A-Za-z0-9]*$ (found '.fleet');: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^... (truncated)
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^[a-z][A-Za-z0-9]*$ (found '.fleet');: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^... (truncated)
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^[a-z][A-Za-z0-9]*$ (found '.fleet');: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^... (truncated)
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^[a-z][A-Za-z0-9]*$ (found '.fleet');: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^... (truncated)
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^[a-z][A-Za-z0-9]*$ (found '.fleet');: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^... (truncated)
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^[a-z][A-Za-z0-9]*$ (found '.fleet');: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^... (truncated)
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^[a-z][A-Za-z0-9]*$ (found '.fleet');: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^... (truncated)
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^[a-z][A-Za-z0-9]*$ (found '.fleet');: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^... (truncated)
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^[a-z][A-Za-z0-9]*$ (found '.fleet');: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^... (truncated)
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^[a-z][A-Za-z0-9]*$ (found '.fleet');: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^... (truncated)
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^[a-z][A-Za-z0-9]*$ (found '.fleet');: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^... (truncated)
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^[a-z][A-Za-z0-9]*$ (found '.fleet');: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^... (truncated)
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^[a-z][A-Za-z0-9]*$ (found '.fleet');: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^... (truncated)
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^[a-z][A-Za-z0-9]*$ (found '.fleet');: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^... (truncated)
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^[a-z][A-Za-z0-9]*$ (found '.fleet');: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^... (truncated)
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^[a-z][A-Za-z0-9]*$ (found '.fleet');: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^... (truncated)
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^[a-z][A-Za-z0-9]*$ (found '.fleet');: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^... (truncated)
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^[a-z][A-Za-z0-9]*$ (found '.fleet');: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^... (truncated)
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^[a-z][A-Za-z0-9]*$ (found '.fleet');: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^... (truncated)
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^[a-z][A-Za-z0-9]*$ (found '.fleet');: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^... (truncated)
�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^[a-z][A-Za-z0-9]*$ (found '.fleet');: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^... (truncated)
�[36mfleet-server_1               |�[0m Error: http POST request to http://kibana:5601/api/fleet/setup fails: Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^[a-z][A-Za-z0-9]*$ (found '.fleet');: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^... (truncated)
�[36mfleet-server_1               |�[0m For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.1/fleet-troubleshooting.html

(source)

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[ruflin]

This seems to be a Kibana / Elasticsearch issue / change. The error comes from here: https://github.com/elastic/kibana/blob/6693ef371f887eca639b09c4c9b15701b4ebabd4/x-pack/plugins/fleet/server/services/api_keys/enrollment_api_key.ts#L226 And I think the relevant error is:

Validation Failed: 1: An application name prefix must match the pattern ^[a-z][A-Za-z0-9]*$ (found '.fleet');: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"Impossible to create an api key: action_request_validation_exception: [action_request_validation_exception] Reason: Validation Failed: 1: An application name prefix must match the pattern ^.

What is the An application name prefix must match the pattern in this context? Is this validate by Elasticearch or Kibana? Where did this change?

[ruflin]

Here is the code on the Elasticsearch side that throws the error: https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/privilege/ApplicationPrivilege.java#L128 But it seems the pattern to match did not change recently?

[nchaulet]

Looks like they added some validation in ES elastic/elasticsearch#82049
We should change the application name in fleet for these API keys, actually changing from .fleet to fleet should work https://github.com/elastic/kibana/blob/main/x-pack/plugins/fleet/server/services/api_keys/enrollment_api_key.ts/#L216

[michel-laterman]

@nchaulet, are you handling the change in Kibana? Do we need to do any changes for the agent/fleet-server?

[ruflin]

If we change from .fleet to fleet, will that have any effect on API keys created before?

@ywangd For awareness that your PR around validation broke Fleet. I wonder if we did something wrong before of this is a breaking change.

[nchaulet]

@nchaulet, are you handling the change in Kibana? Do we need to do any changes for the agent/fleet-server?

Yes I can do the change, this should not affect previously created API key, (I am just going to test to be sure)

[ywangd]

If we change from .fleet to fleet, will that have any effect on API keys created before?

@ywangd For awareness that your PR around validation broke Fleet. I wonder if we did something wrong before of this is a breaking change.

@ruflin Thanks for the ping. It is a bug fix rather than breaking change. But technically any bug fix is likely a "breaking change". The validation has existed for long but it was never applied when creating API keys. ES does not allow application name (of application privilege) to begin with a dot. This is true when creating the actual application privilege, meaning the following does not work:

PUT _security/privilege
{
  ".fleet": {
    "all": {
      "application": ".fleet",
      "name": "all",
      "actions": [ "all:some/thing" ]
    }
  }
}

That is there is no way to define such application privilege. Before elastic/elasticsearch#82049, you can create an API key that contains invalid application name (.fleet). But it is not useful since such application privilege cannot be defined within ES. This means the API key won't actually work because there is nothing for it to check against. Existing API keys with invalid application name will not report error but they are not useful either.

In summary, I'd recommend Fleet to change the API key creation. Or is it a BWC problem?

EDIT:
I'd like to clarify above statement of "... the API key won't actually work ..." since I realise the usage of enrollment API keys is only for authenticating and no authorization is needed. In that case, it will continue to work even with the invalid application name.

[ruflin]

Thanks for the details @ywangd Is my understanding correct that the above means API keys created in previous versions of Elasticsearch with .fleet as application will keep working after the upgrade?

[mtojek]

Let me open the issue, as the fixed application hasn't been released as a Docker image yet.

I'm keeping an eye on it here: elastic/elastic-package#643, but I'm afraid that we're failing with the unified release process.

[ywangd]

Is my understanding correct that the above means API keys created in previous versions of Elasticsearch with .fleet as application will keep working after the upgrade?

Yes since enrollment API keys are used only for authentication purpose and intended to have no privileges at all. They should keep working.

[jlind23]

@mtojek are those docker images available now?

[mtojek]

@jlind23 Yes, I think we have already received the correct images. Feel free to close the issue.