Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

grpc.max_message_size in Fleet policy is ignored #3782

Open
faec opened this issue Nov 16, 2023 · 3 comments
Open

grpc.max_message_size in Fleet policy is ignored #3782

faec opened this issue Nov 16, 2023 · 3 comments
Labels
bug Something isn't working Team:Elastic-Agent Label for the Agent team

Comments

@faec
Copy link
Contributor

faec commented Nov 16, 2023

The agent.grpc.max_message_size setting in a Fleet policy is ignored by the Agent.

To reproduce, create a policy setting max_message_size:

POST kbn:/api/fleet/agent_policies?sys_monitoring=true
{
  "name": "rpc message size test",
  "description": "",
  "namespace": "default",
  "monitoring_enabled": [
    "logs",
    "metrics"
  ],
  "inactivity_timeout": 1209600,
  "is_protected": false,
  "overrides": {
    "agent": {
      "grpc": {
        "max_message_size": 1000
      }
    }
  }
}

Install an agent with the resulting policy and notice that everything works fine even though 1000 bytes is certainly too low for a functioning agent. Alternately, once #3781 is merged, set any value you like and observe that it is ignored and replaced with Agent's default of 104857600.

@faec faec added bug Something isn't working Team:Elastic-Agent Label for the Agent team labels Nov 16, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

@faec
Copy link
Contributor Author

faec commented Nov 16, 2023

After digging some more I'm not sure this is exactly a "bug" so much as a confusingly documented missing feature. It looks like the intended way to set max_message_size is to edit the elastic-agent.yml for the installed Agent and change that parameter, and that it is intentionally ignored in Fleet policies.

This has confused some users as well as some Agent engineers 😄 We should probably either support this in policies or more clearly document that this parameter is local-only.

@cmacknz
Copy link
Member

cmacknz commented Nov 22, 2023

If we pick increasing max_message_size as the solution to #2460 then we should support it from Fleet to make it easy to change.

One other possible complication is that the max message size can apply to both the client and server sides and I think the existing parameter only affects the server side. Probably we need to provide this to clients as well.

See https://pkg.go.dev/google.golang.org/grpc#MaxCallRecvMsgSize also https://github.com/grpc/grpc-go/blob/287c47355e154ac2193bf743c886656cd48e753c/clientconn.go#L98

const (
	defaultClientMaxReceiveMessageSize = 1024 * 1024 * 4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working Team:Elastic-Agent Label for the Agent team
Projects
None yet
Development

No branches or pull requests

3 participants