Clarify sampling error logging

[graphaelli]

Start apm-server @ f3458ec with the default configuration plus:

  sampling:
    tail:
      enabled: true
      interval: 10s
      policies:
      - default:
        sample_rate: 0.1

When connecting to a basic licensed Elasticsearch instance, it logs (with logging.json=false):

INFO	map[file.line:468 file.name:instance/beat.go]	apm-server start running.	{"ecs.version": "1.6.0"}
INFO	[config]	map[file.line:116 file.name:config/sampling.go]	Falling back to elasticsearch output for tail-sampling	{"ecs.version": "1.6.0"}
INFO	[beater]	map[file.line:54 file.name:apm-server/main.go]	creating transaction metrics aggregation with config: {Enabled:true Interval:1m0s MaxTransactionGroups:10000 HDRHistogramSignificantFigures:2}	{"ecs.version": "1.6.0"}
INFO	[beater]	map[file.line:69 file.name:apm-server/main.go]	creating service destinations aggregation with config: {Enabled:true Interval:1m0s MaxGroups:10000}	{"ecs.version": "1.6.0"}
INFO	[beater]	map[file.line:133 file.name:apm-server/main.go]	Created tail-sampling data stream index template	{"ecs.version": "1.6.0"}
INFO	[sampling]	map[file.line:46 file.name:[email protected]/logger.go]	All 0 tables opened in 0s
	{"ecs.version": "1.6.0"}
INFO	[sampling]	map[file.line:46 file.name:[email protected]/logger.go]	Replaying file id: 0 at offset: 0
	{"ecs.version": "1.6.0"}
INFO	[sampling]	map[file.line:46 file.name:[email protected]/logger.go]	Replay took: 5.186µs
	{"ecs.version": "1.6.0"}
INFO	[publisher_pipeline_output]	map[file.line:143 file.name:pipeline/output.go]	Connecting to backoff(elasticsearch(http://localhost:9200))	{"ecs.version": "1.6.0"}
INFO	[esclientleg]	map[file.line:314 file.name:eslegclient/connection.go]	Attempting to connect to Elasticsearch version 8.0.0-SNAPSHOT	{"ecs.version": "1.6.0"}
INFO	[license]	map[file.line:51 file.name:licenser/es_callback.go]	Elasticsearch license: Basic	{"ecs.version": "1.6.0"}
INFO	[license]	map[file.line:102 file.name:apm-server/main.go]	Checking license for tail-based sampling	{"ecs.version": "1.6.0"}
INFO	[license]	map[file.line:35 file.name:licenser/check.go]	License is active for Platinum	{"ecs.version": "1.6.0"}
ERROR	[publisher_pipeline_output]	map[file.line:154 file.name:pipeline/output.go]	Failed to connect to backoff(elasticsearch(http://localhost:9200)): Connection marked as failed because the onConnect callback failed: invalid license found, requires a basic or a valid trial license and received Basic	{"ecs.version": "1.6.0"}

The conflicting messages regarding license requirements and those found are confusing. Ideally the last line would communicate a platinum license requirement.

Possibly worth a separate issue, starting without a reachable elasticsearch results in a quick exit without explanation.

[axw]

Thanks for opening the issue.

apm-server/x-pack/apm-server/main.go

Lines 95 to 99 in 667e03d

	// FIXME(axw) each time licenser.Enforce is called an additional global
	// callback is registered with Elasticsearch, which fetches the license
	// and checks it. The root command already calls licenser.Enforce with
	// licenser.BasicAndAboveOrTrial. We need to make this overridable to
	// avoid redundant checks.

😬

[stuartnelson3]

when on a basic license:

{"log.level":"error","@timestamp":"2022-01-11T12:02:36.374+0100","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: invalid license level Basic: tail-based sampling requires license level Platinum","service.name":"apm-server","ecs.version":"1.6.0"}

when elasticsearch cannot be connected to:

{"log.level":"error","@timestamp":"2022-01-11T11:52:54.332+0100","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: error getting Elasticsearch licensing information: dial tcp 127.0.0.1:9200: connect: connection refused","service.name":"apm-server","ecs.version":"1.6.0"}