Beats started by agent do not respect the allow_older_versions: true configuration flag

[cmacknz]

Beats started by agent do not respect the allow_older_versions: true flag added to the output configuration. Beats by default will not connect to older versions of Elasticsearch, unless allow_older_versions: true appears in the output configuration.

Fleet managed agents will also refuse to enroll with older versions of Elasticsearch, but it is possible to configure a standalone agent to write to an older version. In this case the agent will run but the Beats will refuse to connect with the following error:

{"log.level":"error","@timestamp":"2023-01-10T17:54:57.248Z","message":"Failed to connect to backoff(elasticsearch(https://agent-8-5-3.es.us-central1.gcp.cloud.es.io:443)): Connection marked as failed because the onConnect callback failed: Elasticsearch is too old. Please upgrade the instance. If you would like to connect to older instances set output.elasticsearch.allow_older_versions to true. ES=8.5.3, Beat=8.6.0","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":150,"file.name":"pipeline/client_worker.go"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}

Setting allow_older_versions: true in the output section of the agent policy should resolve this but it does not. This may be because the flag isn't available on this code path when the callback is first registered:

beats/libbeat/cmd/instance/beat.go

Lines 910 to 939 in b2a1310

	// checkElasticsearchVersion registers a global callback to make sure ES instance we are connecting
	// to is at least on the same version as the Beat.
	// If the check is disabled or the output is not Elasticsearch, nothing happens.
	func (b *Beat) checkElasticsearchVersion() {
	if b.isConnectionToOlderVersionAllowed() {
	return
	}
	_, _ = elasticsearch.RegisterGlobalCallback(func(conn *eslegclient.Connection) error {
	esVersion := conn.GetVersion()
	beatVersion, err := libversion.New(b.Info.Version)
	if err != nil {
	return err
	}
	if esVersion.LessThanMajorMinor(beatVersion) {
	return fmt.Errorf("%w ES=%s, Beat=%s", elasticsearch.ErrTooOld, esVersion.String(), b.Info.Version)
	}
	return nil
	})
	}
	func (b *Beat) isConnectionToOlderVersionAllowed() bool {
	config := struct {
	AllowOlder bool `config:"allow_older_versions"`
	}{false}
	_ = b.Config.Output.Config().Unpack(&config)
	return config.AllowOlder
	}

To reproduce this:

1. Build or obtain the latest version of the agent, e.g. 8.6.0.
2. Create an Elasticsearch deployment for the previous minor version e.g. 8.5.3
3. Configure a standalone agent to write to to the 8.5.3 Elasticsearch endpoint, with the allow_older_versions: true flag set on the output. For example:

outputs:
  default:
    type: elasticsearch
    hosts: [https://agent-8-5-3.es.us-central1.gcp.cloud.es.io:443]
    username: "elastic"
    password: <REDACTED>
    allow_older_versions: true

5. Observe that the Beats refuse to connect to Elasticsearch and the allow_older_versions: true flag is configured on the generated Beat configuration in the components directory of the elastic agent diagnostics bundle. Example:

outputs:
  elasticsearch:
    allow_older_versions: true
    bulk_max_size: 50
    hosts:
    - https://agent-8-5-3.es.us-central1.gcp.cloud.es.io:443
    password: <REDACTED>
    type: elasticsearch
    username: elastic

For example configuring

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[blakerouse]

Elastic Agent itself also has checks to ensure that the version of Fleet Server is greater than the Elastic Agent. Do we really want to allow the beats to talk to an older version of elasticsearch?

[cmacknz]

Yes Fleet agents can't even be enrolled in this setup. The Beats themselves can currently be configured to do this, so this is a blocker for some people to switch to a standalone agent instead. This was originally reported by some of our Solution architects working with users trying to do exactly this.

[alexsapran]

I am facing a similar issue when trying to set up Agent for some benchmark experiments I am doing.
The reasoning behind my use case is that I want to test a build of an agent that is more recent in version than the monitoring cluster I have.
This monitoring cluster collects all the monitoring data from all the experiments, running the latest version.

[belimawr]

A monkey patch to unblock @alexsapran: https://github.com/belimawr/beats/tree/allow-older-verisons-hardcoded