Send APM Server config to Kibana #5424

stuartnelson3 · 2021-06-09T06:58:21Z

Motivation/summary

Send a sanitized and flattened version of the apm-server config to kibana on
startup.

Closes #5377

Checklist

Update CHANGELOG.asciidoc
Documentation has been updated

For functional changes, consider:

Is it observable through the addition of either logging or metrics?
Is its use being published in telemetry to enable product improvement?
Have system tests been added to avoid regression?

How to test these changes

Start up kibana, elasticsearch and apm-server. Verify that the apm-server
config is indexed in elasticsearch.

Related issues

Depends on elastic/kibana#100657

stuartnelson3 · 2021-06-09T07:05:03Z

I'm removing anything involving ESConfig or Kibana, is there anything else that should be removed? apm-server.ssl.keyphrase, instrumentation.api_key, and instrumentation.secret_token all look like candidates.

Question regarding the flattened format: Do we want to do any sort of lower camelcasing for the config values? Or just leave them with their default formatting, ie.

{
  "APIKeyConfig.Enabled": false,
  "APIKeyConfig.LimitPerMin": 100,
  "AgentConfigs": null,
  "Aggregation.ServiceDestinations.Enabled": true,
  "Aggregation.ServiceDestinations.Interval": 60000000000,
  "Aggregation.ServiceDestinations.MaxGroups": 10000,
  "Aggregation.Transactions.Enabled": true,
  "Aggregation.Transactions.HDRHistogramSignificantFigures": 2,
  "Aggregation.Transactions.Interval": 60000000000,
  "Aggregation.Transactions.MaxTransactionGroups": 10000,
  "AugmentEnabled": true,
  "DataStreams.Enabled": false,
  "DefaultServiceEnvironment": "",
  "Expvar.Enabled": false,
  "Expvar.URL": "/debug/vars",
  "Host": "localhost:8200",
  "Register.Ingest.Pipeline.Enabled": true,
  "Register.Ingest.Pipeline.Overwrite": false,
  "Register.Ingest.Pipeline.Path": "ingest/pipeline/definition.json",
  "ResponseHeaders": null,
  "RumConfig.AllowHeaders": [],
  "RumConfig.AllowOrigins": [
    "*"
  ],
  "RumConfig.AllowServiceNames": null,
  "RumConfig.Enabled": false,
  "RumConfig.EventRate.Limit": 300,
  "RumConfig.EventRate.LruSize": 1000,
  "RumConfig.ExcludeFromGrouping": "^/webpack",
  "RumConfig.LibraryPattern": "node_modules|bower_components|~",
  "RumConfig.ResponseHeaders": null,
  "RumConfig.SourceMapping.Cache.Expiration": 300000000000,
  "RumConfig.SourceMapping.Enabled": true,
  "RumConfig.SourceMapping.IndexPattern": "apm-*-sourcemap*",
  "SelfInstrumentation.APIKey": "",
  "SelfInstrumentation.Enabled": false,
  "SelfInstrumentation.Environment": "",
  "SelfInstrumentation.Hosts": null,
  "SelfInstrumentation.Profiling.CPU.Duration": 10000000000,
  "SelfInstrumentation.Profiling.CPU.Enabled": false,
  "SelfInstrumentation.Profiling.CPU.Interval": 60000000000,
  "SelfInstrumentation.Profiling.Heap.Enabled": false,
  "SelfInstrumentation.Profiling.Heap.Interval": 60000000000,
  "SelfInstrumentation.SecretToken": "",
  "ShutdownTimeout": 5000000000,
  "TLS": null,
  "WriteTimeout": 30000000000
}

apmmachine · 2021-06-09T07:12:07Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #5424 updated
Start Time: 2021-06-23T11:31:42.354+0000
Duration: 44 min 15 sec
Commit: 81d0be4

Test stats 🧪

Test	Results
Failed	0
Passed	6165
Skipped	119
Total	6284

Trends 🧪

axw · 2021-06-10T04:29:27Z

I'm removing anything involving ESConfig or Kibana, is there anything else that should be removed? apm-server.ssl.keyphrase, instrumentation.api_key, and instrumentation.secret_token all look like candidates.

None of these can be set in ESS, but to be on the safe side I think it would be a good idea to remove:

apm-server.ssl.*
apm-server.instrumentation.*
*.elasticsearch.*
instrumentation.*

@simitt will that work?

Question regarding the flattened format: Do we want to do any sort of lower camelcasing for the config values? Or just leave them with their default formatting, ie.

I think we should be sending the libbeat config input, rather than the apm-server config into which it is unmarshaled. The raw libbeat config is accessible in

apm-server/beater/beater.go

Line 135 in 7f55bc8

rawConfig *common.Config

simitt · 2021-06-10T07:26:50Z

None of these can be set in ESS, but to be on the safe side I think it would be a good idea to remove:
apm-server.ssl.*
apm-server.instrumentation.*
*.elasticsearch.*
instrumentation.*

Following ssl related settings need to be synced: apm-server.ssl.enabled, apm-server.ssl.certificate, apm-server.ssl.key. Removing the other settings is fine.

I think we should be sending the libbeat config input, rather than the apm-server config into which it is unmarshaled.

Why is that? On the Kibana side we need to define a list of supported config options (hardcoded).

axw · 2021-06-10T07:31:18Z

Why is that? On the Kibana side we need to define a list of supported config options (hardcoded).

A couple of reasons:

If we refactor the config structs (e.g. rename fields), then the JSON will change. This should be stable for the rest of 7.x, right? We could add json:"..." struct tags, but we already have a defined stable config format so may as well use that.
The config that's being marshalled at the moment includes defaults, which I don't think should be transferred?

simitt · 2021-06-10T07:41:41Z

@axw good points; +1

indicates we're running in ece/ess

kibana/send_config.go

ogupte · 2021-06-21T15:19:05Z

In the Kibana PR, I'm defining only the configs which map to input vars in the package policy:

https://github.com/elastic/kibana/pull/102682/files#diff-ef993cd3669a97d649974bea6f79a1c2406abbf7ca7f1cc0d86cda4df7d2c5adR231-R254

{
  'apm-server.host': 'host',
  'apm-server.secret_token': 'secret_token',
  'apm-server.rum.enabled': 'enable_rum',
  'apm-server.default_service_environment': 'default_service_environment',
  'apm-server.rum.allow_service_names': 'rum_allow_service_names',
  'apm-server.rum.allow_origins': 'rum_allow_origins',
  'apm-server.rum.allow_headers': 'rum_allow_headers',
  'apm-server.rum.event_rate.limit': 'rum_event_rate_limit',
  'apm-server.rum.event_rate.lru_size': 'rum_event_rate_lru_size',
  'apm-server.max_event_size': 'max_event_bytes',
  'apm-server.capture_personal_data': 'capture_personal_data',
  'apm-server.max_header_size': 'max_header_bytes',
  'apm-server.idle_timeout': 'idle_timeout',
  'apm-server.read_timeout': 'read_timeout',
  'apm-server.shutdown_timeout': 'shutdown_timeout',
  'apm-server.write_timeout': 'write_timeout',
  'apm-server.max_connections': 'max_connections',
  'apm-server.expvar.enabled': 'expvar_enabled',
  'output.elasticsearch.ssl.enabled': 'tls_enabled',
  'output.elasticsearch.ssl.certificate': 'tls_certificate',
  'output.elasticsearch.ssl.key': 'tls_key',
  'output.elasticsearch.ssl.supported_protocols': 'tls_supported_protocols',
  'output.elasticsearch.ssl.cipher_suites': 'tls_cipher_suites',
  'output.elasticsearch.ssl.curve_types': 'tls_curve_types',
}

Anything that isn't included in this map will be displayed to the user as unsupported.

stuartnelson3 · 2021-06-22T08:24:13Z

@ogupte we've made a couple changes to the input vars but haven't pushed them to the package policy yet. There's one final change for 7.14 in #5410, and then we'll pushed an updated version.

The config should look like the example you posted in your comment, ie. with the full path for each config. The example I posted above was from only marshalling the things nested under apm-server, but the full raw config will be marshaled.

simitt

generally looks good, left minor comments

simitt · 2021-06-22T20:23:03Z

kibana/send_config.go

+		// TODO: What sort of response will we get?
+		if resp.StatusCode > http.StatusOK {
+			return fmt.Errorf("bad response %s", resp.Status)
+		}


Maybe cover the scenario where Kibana < 7.14 and therefore doesn't support the endpoint. (log and stop retrying)

simitt · 2021-06-22T20:27:25Z

kibana/send_config_test.go

+  protocol: "https"
+  username: "elastic"
+  password: "changeme"
+  worker: 1


instrumentation and kibana are not covered here; can you also cover some config option that doesn't start with apm-server to check it is sent.

simitt · 2021-06-23T07:56:24Z

I just realized that the config options only contain the apm-server.* options, and not the output.*, setup.*, etc related config options, and that the options are not prefixed with apm-server.* anymore at this point (for example apm-server.ssl is only ssl at the point of flattening).

Since only apm-server options are supported as input vars in the apm integration, I guess that is fair, but we need to add some note to the APM migration UI that none of the other config options is ever supported.

The config options are currently sent without the apm-server.* prefix. The flattenAndClean check assumes that the whole configuration (including output.*, instrumentation.*), etc. is there and that the server specific settings are prefixed with apm-server.* which is not the case. Please change the logic and the testcase accordingly. When sending the options to Kibana, the apm-server prefix should be added, to make it straight forward on the Kibana side.

axw · 2021-06-23T08:00:26Z

If we need to, I believe it is still possible to get the parent config: cast the *common.Config to *ucfg.Config, and use the latter's Parent method: https://pkg.go.dev/github.com/elastic/go-ucfg#Config.Parent

This raw config only has apm-server values, but casting and then accessing the parent has the full paths for the received config.

simitt

Apart from a minor comment on the testcase, changes LGTM now.

kibana/send_config_test.go

mergify · 2021-06-23T11:27:44Z

This pull request is now in conflicts. Could you fix it @stuartnelson3? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b apm-config-to-kibana upstream/apm-config-to-kibana
git merge upstream/master
git push upstream apm-config-to-kibana

(cherry picked from commit 1920f95) # Conflicts: # beater/beater.go # changelogs/head.asciidoc

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

simitt · 2021-07-09T13:51:13Z

Tested this on an ECE instance.

(cherry picked from commit 1920f95) # Conflicts: # changelogs/head.asciidoc # kibana/send_config.go # kibana/send_config_test.go

flatten, clean, and marshal

209c376

Merge branch 'master' into apm-config-to-kibana

5486a67

stuartnelson3 added the v7.14.0 label Jun 9, 2021

stuartnelson3 added 2 commits June 10, 2021 10:23

check for ELASTIC_AGENT_CLOUD env var

8e5d4fc

indicates we're running in ece/ess

send filtered raw config

54c184d

ogupte reviewed Jun 20, 2021

View reviewed changes

kibana/send_config.go Outdated Show resolved Hide resolved

Merge branch 'master' into apm-config-to-kibana

a3d3449

stuartnelson3 added 3 commits June 22, 2021 10:33

use an example raw config

09517f5

update config path

b44a6d7

golint

1d508b1

stuartnelson3 marked this pull request as ready for review June 22, 2021 15:01

stuartnelson3 requested a review from a team June 22, 2021 15:01

update changelog

bd0b194

simitt reviewed Jun 22, 2021

View reviewed changes

stuartnelson3 added 3 commits June 23, 2021 10:35

Merge branch 'master' into apm-config-to-kibana

35fc440

report parent config

02170f9

This raw config only has apm-server values, but casting and then accessing the parent has the full paths for the received config.

Merge branch 'master' into apm-config-to-kibana

bfadd17

simitt approved these changes Jun 23, 2021

View reviewed changes

kibana/send_config_test.go Outdated Show resolved Hide resolved

fix test config nesting

73c2b67

Merge branch 'master' into apm-config-to-kibana

81d0be4

stuartnelson3 merged commit 1920f95 into elastic:master Jun 23, 2021

stuartnelson3 deleted the apm-config-to-kibana branch June 23, 2021 12:17

mergify bot pushed a commit that referenced this pull request Jun 23, 2021

Send APM Server config to Kibana (#5424)

2f96f7f

(cherry picked from commit 1920f95) # Conflicts: # beater/beater.go # changelogs/head.asciidoc

mergify bot mentioned this pull request Jun 23, 2021

[7.x] Send APM Server config to Kibana (backport #5424) #5526

Closed

stuartnelson3 mentioned this pull request Jun 24, 2021

update api endpoint and body format #5536

Merged

simitt added a commit to simitt/apm-server that referenced this pull request Jun 28, 2021

fix backport of elastic#5424

c1a1c34

simitt mentioned this pull request Jun 28, 2021

fix backport of https://github.com/elastic/apm-server/pull/5526 #5555

Merged

axw pushed a commit that referenced this pull request Jun 29, 2021

fix backport of #5424 (#5555)

b33edda

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

stuartnelson3 added the test-plan label Jun 29, 2021

axw mentioned this pull request Jul 6, 2021

config is now rooted under "schema" key #5544

Merged

simitt added the test-plan-ok label Jul 9, 2021

mergify bot pushed a commit that referenced this pull request Jul 9, 2021

Send APM Server config to Kibana (#5424)

b441f9d

(cherry picked from commit 1920f95) # Conflicts: # changelogs/head.asciidoc # kibana/send_config.go # kibana/send_config_test.go

mergify bot mentioned this pull request Jul 9, 2021

[7.14] Send APM Server config to Kibana (backport #5424) #5658

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Send APM Server config to Kibana #5424

Send APM Server config to Kibana #5424

stuartnelson3 commented Jun 9, 2021 •

edited

Loading

stuartnelson3 commented Jun 9, 2021 •

edited

Loading

apmmachine commented Jun 9, 2021 •

edited

Loading

Build stats

Test stats 🧪

Trends 🧪

axw commented Jun 10, 2021

simitt commented Jun 10, 2021

axw commented Jun 10, 2021

simitt commented Jun 10, 2021

ogupte commented Jun 21, 2021 •

edited

Loading

stuartnelson3 commented Jun 22, 2021

simitt left a comment

simitt Jun 22, 2021

simitt Jun 22, 2021

simitt commented Jun 23, 2021

axw commented Jun 23, 2021

simitt left a comment

mergify bot commented Jun 23, 2021

simitt commented Jul 9, 2021

Send APM Server config to Kibana #5424

Send APM Server config to Kibana #5424

Conversation

stuartnelson3 commented Jun 9, 2021 • edited Loading

Motivation/summary

Checklist

How to test these changes

Related issues

stuartnelson3 commented Jun 9, 2021 • edited Loading

apmmachine commented Jun 9, 2021 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

axw commented Jun 10, 2021

simitt commented Jun 10, 2021

axw commented Jun 10, 2021

simitt commented Jun 10, 2021

ogupte commented Jun 21, 2021 • edited Loading

stuartnelson3 commented Jun 22, 2021

simitt left a comment

Choose a reason for hiding this comment

simitt Jun 22, 2021

Choose a reason for hiding this comment

simitt Jun 22, 2021

Choose a reason for hiding this comment

simitt commented Jun 23, 2021

axw commented Jun 23, 2021

simitt left a comment

Choose a reason for hiding this comment

mergify bot commented Jun 23, 2021

simitt commented Jul 9, 2021

stuartnelson3 commented Jun 9, 2021 •

edited

Loading

stuartnelson3 commented Jun 9, 2021 •

edited

Loading

apmmachine commented Jun 9, 2021 •

edited

Loading

ogupte commented Jun 21, 2021 •

edited

Loading