[APM] Add data access plugin #162367

sorenlouv · 2023-07-20T23:05:43Z

Closes #161906
Related to: https://github.com/elastic/observability-dev/discussions/2787 (internal)

This add a new plugin apm_data_access that contains the APM query targets (indices to query for APM data).
This plugin can be consumed by apm and any other plugin, making it possible for other plugins to know about the configured APM query targets.

Example:

APM query targets can be specified in kibana[.dev].yml using xpack.apm.indices.{dataset}: some-index-* for instances:

xpack.apm.indices.transaction: apm-*

See all config options on: https://www.elastic.co/guide/en/kibana/current/apm-settings-kb.html#general-apm-settings-kb

Query targets can also be specified via the UI (and persisted in a saved object) via the settings page: /app/apm/settings/apm-indices

Retrieving the query targets
Query targets can be retrieved from other plugins via getApmIndices:

const apmIndices = await plugins.apmDataAccess.setup.getApmIndices(savedObjects.client);

TODO:

Add SO client and fetch space aware index config (see https://github.com/sqren/kibana/blob/4d7f626da020c156500f8bf52bf8861cde54b4f1/x-pack/plugins/apm/server/routes/settings/apm_indices/get_apm_indices.ts#L32-L44)
Add simple APM client for querying apm data

apmmachine · 2023-07-20T23:05:57Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

roshan-elastic · 2023-07-31T16:56:24Z

Thanks so much for jumping on this @sqren @neptunian !

sorenlouv · 2023-08-12T13:58:54Z

@neptunian This PR is now ready to test out. Can you try consuming the apmDataAccess plugin and see if you can retrieve the apm indices from infra or another plugin where you need it?

Add apmDataAccess as required plugin

https://github.com/elastic/kibana/blob/ac0454896aebe072dbd831247a6c7bba4e4c75b7/x-pack/plugins/apm/kibana.jsonc#L11-L12

Add ApmDataAccessPluginSetup and ApmDataAccessPluginStart to the plugin dependency types:

https://github.com/elastic/kibana/blob/ac0454896aebe072dbd831247a6c7bba4e4c75b7/x-pack/plugins/apm/server/types.ts#L80-L82

https://github.com/elastic/kibana/blob/ac0454896aebe072dbd831247a6c7bba4e4c75b7/x-pack/plugins/apm/server/types.ts#L105-L107

Consume apmDataAccess

https://github.com/elastic/kibana/blob/146648a4688c5d98e99eb494284d54b5fbcb7f33/x-pack/plugins/apm/server/routes/apm_routes/register_apm_server_routes.ts#L113-L115

x-pack/plugins/apm_data_access/tsconfig.json

elasticmachine · 2023-08-14T10:17:28Z

Pinging @elastic/apm-ui (Team:APM)

gbamparop · 2023-08-14T10:19:57Z

x-pack/plugins/apm_data_access/server/index.ts

+export const config: PluginConfigDescriptor<APMDataAccessConfig> = {
+  deprecations: ({ renameFromRoot }) => [
+    // deprecations due to removal of apm_oss plugin
+    renameFromRoot('apm_oss.transactionIndices', 'xpack.apm.indices.transaction', {


Do we still have to use xpack.apm.* as the new keys here? Can these be replaced with xpack.apm_data_access.*?

Yes, this will log a visible warning to the user if they are using apm_oss.transactionIndices. In that case they should be recommended to use xpack.apm.indices.transaction - not xpack.apm_data_access.indices.transaction.

For now apm_data_access is purely an implementation detail, and not something end users should know about (maybe that changes in the future but I'd like to keep this change invisible for now).

@sqren I'm getting an error unless I use xpack.apm_data_access.indices.*

[config validation of [xpack.apm].indices]: definition for this key is missing

@neptunian What's in your kibana.yml?

Just a copy paste of my remote oblt cluster config

xpack.apm.indices.sourcemap and xpack.apm.indices.onboarding were missing in the config definitions. I re-added them which should fix the issue

x-pack/plugins/apm/server/index.ts

yngrdyn · 2023-08-16T08:53:01Z

...ver/routes/alerts/rule_types/transaction_duration/register_transaction_duration_rule_type.ts

@@ -76,7 +74,8 @@ const ruleTypeConfig = RULE_TYPES_CONFIG[ApmRuleType.TransactionDuration];
 export function registerTransactionDurationRuleType({
  alerting,
  ruleDataClient,
-  config$,
+  getApmIndices,


Why not passing directly the indices instead of the function? Couldn't we get the indices on plugin start and save them as a resource for the routes?

I considered it, but decided to load the indices lazily for two reasons:

registerTransactionDurationRuleType is called when kibana starts. Retrieving the apm indicies at this point would mean that if the indices were update via APM Settings, the change would not be reflected until Kibana was restarted.

Because registerTransactionDurationRuleType is called during startup it is called with the privileges of the internal kibana user and not with the privileges of the rule execution user. I don't think this is a problem in practice but it feels slightly wrong.

yngrdyn · 2023-08-16T11:05:15Z

x-pack/test/apm_api_integration/tests/settings/apm_indices/apm_indices.spec.ts

yngrdyn

LGTM 📦

…fix'

…ocs'

…fix'

…-fix'

…fix'

This reverts commit 8f3abab9d44614b78e67a85f0492cbc10be7d0a0.

kibana-ci · 2023-08-22T09:19:10Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: eec4f34

Failed CI Steps

APM Cypress Tests #2

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`apm`	48	29	-19
`apmDataAccess`	-	9	+9
total			-10

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`apm`	120	118	-2

Unknown metric groups

API count

id	before	after	diff
`apm`	48	29	-19
`apmDataAccess`	-	9	+9
total			-10

ESLint disabled line counts

id	before	after	diff
`apmDataAccess`	-	1	+1

Total ESLint disabled count

id	before	after	diff
`apmDataAccess`	-	1	+1

History

💔 Build #152031 failed af8bb345fa7cc7f380cd1f0ba28c23f81944edcc
💔 Build #151946 failed f8ea7ef843b529f42c63290c7b6c39c5c7e4990d
💚 Build #151775 succeeded 60cba7b8c3b1ad2df9777543571800475bb78150
💔 Build #151724 failed 8f3abab9d44614b78e67a85f0492cbc10be7d0a0
💔 Build #150990 failed 2df92fe6f1c664b1ec72f3154d637dd297e54104

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

#164750) Closes #164695 Uses [apm_data_access plugin](#162367) in place of statically defined indices to extract services --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Kevin Lacabane <[email protected]>

sorenlouv force-pushed the add-apm-data-access-plugin branch 3 times, most recently from dc9a777 to 04795e5 Compare July 20, 2023 23:50

sorenlouv force-pushed the add-apm-data-access-plugin branch from 951db70 to ac04548 Compare August 11, 2023 20:28

sorenlouv marked this pull request as ready for review August 11, 2023 22:20

sorenlouv requested a review from a team as a code owner August 11, 2023 22:20

sorenlouv added release_note:skip Skip the PR/issue when compiling release notes v8.11.0 labels Aug 12, 2023

gbamparop reviewed Aug 14, 2023

View reviewed changes

x-pack/plugins/apm_data_access/tsconfig.json Outdated Show resolved Hide resolved

botelastic bot added the Team:APM All issues that need APM UI Team support label Aug 14, 2023

gbamparop reviewed Aug 14, 2023

View reviewed changes

sorenlouv force-pushed the add-apm-data-access-plugin branch from 2e019c7 to faf6c9e Compare August 14, 2023 13:06

neptunian self-requested a review August 14, 2023 17:58

neptunian reviewed Aug 15, 2023

View reviewed changes

x-pack/plugins/apm/server/index.ts Show resolved Hide resolved

sorenlouv force-pushed the add-apm-data-access-plugin branch from d839e7a to 85d88e4 Compare August 16, 2023 08:19

yngrdyn reviewed Aug 16, 2023

View reviewed changes

x-pack/test/apm_api_integration/tests/settings/apm_indices/apm_indices.spec.ts Outdated

Copy link

Contributor

yngrdyn Aug 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️

yngrdyn approved these changes Aug 16, 2023

View reviewed changes

sorenlouv enabled auto-merge (squash) August 20, 2023 23:44

sorenlouv force-pushed the add-apm-data-access-plugin branch 2 times, most recently from e3aa6ee to 60cba7b Compare August 21, 2023 09:37

sorenlouv added release_note:skip Skip the PR/issue when compiling release notes and removed release_note:skip Skip the PR/issue when compiling release notes labels Aug 21, 2023

sorenlouv force-pushed the add-apm-data-access-plugin branch 2 times, most recently from f8ea7ef to af8bb34 Compare August 21, 2023 17:00

[APM] Add data access plugin

959e319

sorenlouv and others added 21 commits August 22, 2023 09:56

Fix tsconfig

e78e252

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

8364235

…fix'

[CI] Auto-commit changed files from 'node scripts/generate codeowners'

7ffd393

[CI] Auto-commit changed files from 'node scripts/build_plugin_list_d…

8827532

…ocs'

Fixes

976b5c4

Fix tests

ab27b17

Move saved object type to apmDataAccess plugin

516f84f

Delete js files

278c0c6

Remove missing folders from tsconfig

800ff75

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

0c6390b

…fix'

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

ba243d0

…-fix'

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

92f9809

…fix'

Fix i18n

534a995

Fix i18n

692c1b4

Add xpack.apm.indices.onboarding and xpack.apm.indices.sourcemap

6b7509d

Added api test

b813312

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

ec20d68

…fix'

Use APMIndices instead of APMDataAccessConfig

ee4e2dc

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

2efa7a4

…fix'

Use start lifecycle instead of setup

40bf18c

Revert "Use start lifecycle instead of setup"

a8c4e9c

This reverts commit 8f3abab9d44614b78e67a85f0492cbc10be7d0a0.

sorenlouv force-pushed the add-apm-data-access-plugin branch from af8bb34 to a8c4e9c Compare August 22, 2023 07:57

Fix type

eec4f34

sorenlouv merged commit 7df1cee into elastic:main Aug 22, 2023

kibanamachine added the backport:skip This commit does not require backporting label Aug 22, 2023

sorenlouv deleted the add-apm-data-access-plugin branch August 22, 2023 13:50

neptunian mentioned this pull request Aug 24, 2023

[Asset Manager] services endpoint to use apm data access plugin #164695

Closed

klacabane mentioned this pull request Sep 4, 2023

[Asset Manager] update services endpoint to use apm data access plugin #164750

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[APM] Add data access plugin #162367

[APM] Add data access plugin #162367

sorenlouv commented Jul 20, 2023 •

edited

Loading

apmmachine commented Jul 20, 2023

roshan-elastic commented Jul 31, 2023

sorenlouv commented Aug 12, 2023 •

edited

Loading

elasticmachine commented Aug 14, 2023

gbamparop Aug 14, 2023 •

edited

Loading

sorenlouv Aug 14, 2023 •

edited

Loading

neptunian Aug 15, 2023 •

edited

Loading

sorenlouv Aug 15, 2023 •

edited

Loading

neptunian Aug 15, 2023

sorenlouv Aug 15, 2023

yngrdyn Aug 16, 2023 •

edited

Loading

sorenlouv Aug 16, 2023 •

edited

Loading

yngrdyn Aug 16, 2023

yngrdyn left a comment

kibana-ci commented Aug 22, 2023

API count

ESLint disabled line counts

Total ESLint disabled count

[APM] Add data access plugin #162367

[APM] Add data access plugin #162367

Conversation

sorenlouv commented Jul 20, 2023 • edited Loading

Example:

apmmachine commented Jul 20, 2023

🤖 GitHub comments

roshan-elastic commented Jul 31, 2023

sorenlouv commented Aug 12, 2023 • edited Loading

elasticmachine commented Aug 14, 2023

gbamparop Aug 14, 2023 • edited Loading

Choose a reason for hiding this comment

sorenlouv Aug 14, 2023 • edited Loading

Choose a reason for hiding this comment

neptunian Aug 15, 2023 • edited Loading

Choose a reason for hiding this comment

sorenlouv Aug 15, 2023 • edited Loading

Choose a reason for hiding this comment

neptunian Aug 15, 2023

Choose a reason for hiding this comment

sorenlouv Aug 15, 2023

Choose a reason for hiding this comment

yngrdyn Aug 16, 2023 • edited Loading

Choose a reason for hiding this comment

sorenlouv Aug 16, 2023 • edited Loading

Choose a reason for hiding this comment

yngrdyn Aug 16, 2023

Choose a reason for hiding this comment

yngrdyn left a comment

Choose a reason for hiding this comment

kibana-ci commented Aug 22, 2023

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Public APIs missing comments

Public APIs missing exports

API count

ESLint disabled line counts

Total ESLint disabled count

History

sorenlouv commented Jul 20, 2023 •

edited

Loading

sorenlouv commented Aug 12, 2023 •

edited

Loading

gbamparop Aug 14, 2023 •

edited

Loading

sorenlouv Aug 14, 2023 •

edited

Loading

neptunian Aug 15, 2023 •

edited

Loading

sorenlouv Aug 15, 2023 •

edited

Loading

yngrdyn Aug 16, 2023 •

edited

Loading

sorenlouv Aug 16, 2023 •

edited

Loading