Enable synthetic source for Elastic-Agent datastreams #9826

belimawr · 2024-05-09T16:37:21Z

Proposed commit message

This commit enables the synthetic source for all Elastic-Agent datastreams except for cloudbeat_logs and cloud_defend_logs that are not compatible with synthetic source because they define a decision_id field as text.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

Update format-version to 3.1.4 in a separated PR

How to test this PR locally

Build the integration and bring up a stack with it
Go to Fleet, there should be no error/warning from Kibana.
Go to DevTools
Run the following query: GET _component_template/logs-elastic_agent.*@package

Ensure all component templates contain:

         "_source": {
           "mode": "synthetic"
         },

~~## Related issues~~
~~## Screenshots~~

This commit enables the synthetic source for all Elastic-Agent datastreams except for `cloudbeat_logs` and `cloud_defend_logs` that are not compatible with synthetic source because they define a `decision_id` field as text.

cmacknz · 2024-05-09T19:02:03Z

packages/elastic_agent/manifest.yml

 description: Collect logs and metrics from Elastic Agents.
 type: integration
 format_version: 1.0.0
 license: basic
 categories: ["elastic_stack"]
 conditions:
  kibana:
-    version: "^8.11.2"
+    version: "^8.15.0"


Do we need to be careful not to publish this too far ahead of the release of 8.15.0? 8.14.0 isn't even out yet. Presumably we couldn't bump the minor version if we need to add features for another reason if this goes first?

@jsoriano what's the best practice here? Do we need to mark this as a prerelease version?

What is the motivation to use synthetic source in this package?

So far synthetic source is only GA for TSDB indexes. APM has been using at least since 8.7.0 on metrics data streams. Around this version also Fleet starting enabling it by default on all TSDB indexes.
But I don't think we are using it for logs data streams anywhere, so it can be a bit risky to enable it here unless there are strong reasons.

If we want to enable it in any case here, I think we should release it as prerelease, adding a prerelease tag to the version, maybe even bumping to a new major, so setting a version like 2.0.0-beta1. This would also allow to keep 1.x for backports to GA versions till synthetic source is GA, or at least more tested, in logs data streams.

What is the motivation to use synthetic source in this package?

Significantly reducing the storage footprint of the agent monitoring logs data. This is the storage cost for an agent doing nothing useful, merely existing.

Agree it needs to be pre-release, since synthetic source on logs indices is still technical preview.

Per the PM for synthetic source, this is a good candidate for using synthetic _source prior to GA, given that these are internal logs, we do not maintain a contract for them, and we are not blocked by any synthetic source limitations.

Ok, makes sense to try this first in an internal feature. Is there a target version to release this feature as GA for all indexes?

strawgate · 2024-05-09T19:03:22Z

it was pointed out to me that synthetic source for logs indices is in technical preview. I'll chat with the ES team and see how bad of an idea this is from their perspective

elastic-sonarqube · 2024-05-09T19:11:54Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

jsoriano · 2024-05-09T21:29:19Z

packages/elastic_agent/data_stream/apm_server_logs/manifest.yml

+      _source:
+        mode: synthetic


I think this syntax is not valid in recent versions of the spec. Please use the following syntax, Fleet is aware of the source_mode setting and can handle it in a safer way.

Suggested change

_source:

mode: synthetic

source_mode: synthetic

Interesting, it worked well on all my manual testing. But I'll update and test again.

Yes, this enables synthetic mode, but Fleet may not be aware of it. There is some logic in Fleet to allow to override this option by the user, not sure if this works with this syntax.
This behavior is described here: elastic/kibana#141211 (comment)

Older versions of the spec, like 1.0.0, were much less restrictive, and allowed things that could lead to problematic or ambiguous situations. In v2 and v3 we have been introducing restrictions to avoid these situations. For example restrict now what can be configured for index templates in the manifest.

You can find here some help for some common issues when upgrading packages to v2 or v3: https://github.com/elastic/elastic-package/blob/main/docs/howto/update_major_package_spec.md

packages/elastic_agent/manifest.yml

jsoriano · 2024-05-09T21:56:14Z

packages/elastic_agent/manifest.yml

 description: Collect logs and metrics from Elastic Agents.
 type: integration
 format_version: 1.0.0
 license: basic
 categories: ["elastic_stack"]
 conditions:
  kibana:
-    version: "^8.11.2"
+    version: "^8.15.0"


What is the motivation to use synthetic source in this package?

So far synthetic source is only GA for TSDB indexes. APM has been using at least since 8.7.0 on metrics data streams. Around this version also Fleet starting enabling it by default on all TSDB indexes.
But I don't think we are using it for logs data streams anywhere, so it can be a bit risky to enable it here unless there are strong reasons.

If we want to enable it in any case here, I think we should release it as prerelease, adding a prerelease tag to the version, maybe even bumping to a new major, so setting a version like 2.0.0-beta1. This would also allow to keep 1.x for backports to GA versions till synthetic source is GA, or at least more tested, in logs data streams.

jsoriano · 2024-05-09T21:58:52Z

packages/elastic_agent/data_stream/cloud_defend_logs/manifest.yml

+        decision_id:
+          type: text
+          store: true


Could this field be defined as a normal field under a fields file? I think these definitions are also not allowed on recent versions of the package spec.

Could this field be defined as a normal field under a fields file?

That was the first thing I tried, but for some reason it had no effect, so I tried here and it worked.

I confess I don't know much about the integrations development, so for things like that I usually go with a trial and error approach.

Could you share what you tried first?

Actually I don't see any package using store: true and this is not explicitly supported by the spec, so maybe this is the only way to do it, and not sure if this will work in v3, I will take a look to this.

Actually I don't see any package using store: true and this is not explicitly supported by the spec, so maybe this is the only way to do it, and not sure if this will work in v3, I will take a look to this.

This is indeed not supported in current spec, adding support for it in elastic/package-spec#748.

Could you share what you tried first?

I had replaced

integrations/packages/elastic_agent/data_stream/cloud_defend_logs/fields/fields.yml

Lines 1 to 2 in 5620eac

- name: decision_id

type: text

by

- name: decision_id type: text store: true

But it had no effect :/

If I understood correctly, elastic/package-spec#748 should make store: true allowed and working when used in fields.yml, is that right?

If I understood correctly, elastic/package-spec#748 should make store: true allowed and working when used in fields.yml, is that right?

Yes, I tried something like what you mention and indeed this doesn't work.

These changes are both needed:

Add support for store mapping property package-spec#748

[Fleet] Add support for mappings with store: true kibana#183390

elasticmachine · 2024-05-10T06:29:01Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

belimawr · 2024-05-13T21:24:36Z

Given the backport PR elastic/kibana#183313 I've lowered the Kibana version to 8.14.1

Co-authored-by: Jaime Soriano Pastor <[email protected]>

elasticmachine · 2024-05-13T21:35:37Z

💔 Build Failed

Buildkite Build
Commit: 005715c

Failed CI Steps

Check integrations elastic_agent

History

💚 Build #11316 succeeded 3a2eba7
💚 Build #11314 succeeded 1c39f94
💔 Build #11313 failed b6495ea2431d6310c5c531d375f7c3b32a2e4c61

## Summary Add support for the `store` mapping parameter in fields defined in packages. ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### Related issues - Support in packages added in elastic/package-spec#748. - Needed for cases where synthetic source mode is used, like in elastic/integrations#9826.

pierrehilbert · 2024-05-21T07:38:24Z

Given the backport PR elastic/kibana#183313 I've lowered the Kibana version to 8.14.1

Won't it be available from 8.14.0?

botelastic · 2024-06-20T07:47:09Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

botelastic · 2024-08-18T15:11:55Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

andrewkroh · 2024-08-20T01:55:09Z

packages/elastic_agent/changelog.yml

+  changes:
+    - description: Enable synthetic source for Elastic-Agent datastreams
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/42


Suggested change

link: https://github.com/elastic/integrations/pull/42

link: https://github.com/elastic/integrations/pull/9826

botelastic · 2024-09-19T02:37:38Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

botelastic · 2024-10-19T03:29:26Z

Hi! This PR has been stale for a while and we're going to close it as part of our cleanup procedure. We appreciate your contribution and would like to apologize if we have not been able to review it, due to the current heavy load of the team. Feel free to re-open this PR if you think it should stay open and is worth rebasing. Thank you for your contribution!

Enable synthetic source for Elastic-Agent datastreams

1c39f94

This commit enables the synthetic source for all Elastic-Agent datastreams except for `cloudbeat_logs` and `cloud_defend_logs` that are not compatible with synthetic source because they define a `decision_id` field as text.

belimawr force-pushed the synthetic-source branch from b6495ea to 1c39f94 Compare May 9, 2024 16:37

Add cloud-defend and cloudbeat

3a2eba7

cmacknz reviewed May 9, 2024

View reviewed changes

belimawr marked this pull request as ready for review May 9, 2024 19:18

belimawr requested a review from a team as a code owner May 9, 2024 19:18

jsoriano reviewed May 9, 2024

View reviewed changes

pierrehilbert added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team [elastic/elastic-agent-data-plane] label May 10, 2024

Update Kibana dependency to 8.14.1

9853b2b

Update packages/elastic_agent/manifest.yml

005715c

Co-authored-by: Jaime Soriano Pastor <[email protected]>

This was referenced May 14, 2024

Add support for store mapping property elastic/package-spec#748

Merged

[Fleet] Add support for mappings with store: true elastic/kibana#183390

Merged

botelastic bot added the Stalled label Jun 20, 2024

andrewkroh added the Integration:elastic_agent Elastic Agent label Jul 19, 2024

botelastic bot removed the Stalled label Jul 19, 2024

botelastic bot added the Stalled label Aug 18, 2024

andrewkroh reviewed Aug 20, 2024

View reviewed changes

botelastic bot removed the Stalled label Aug 20, 2024

botelastic bot added the Stalled label Sep 19, 2024

botelastic bot closed this Oct 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable synthetic source for Elastic-Agent datastreams #9826

Enable synthetic source for Elastic-Agent datastreams #9826

belimawr commented May 9, 2024 •

edited

Loading

cmacknz May 9, 2024

jsoriano May 9, 2024 •

edited

Loading

cmacknz May 9, 2024

strawgate May 10, 2024

jsoriano May 10, 2024

strawgate commented May 9, 2024

elastic-sonarqube bot commented May 9, 2024

jsoriano May 9, 2024

belimawr May 13, 2024

jsoriano May 14, 2024

jsoriano May 9, 2024 •

edited

Loading

jsoriano May 9, 2024

belimawr May 13, 2024

jsoriano May 14, 2024

jsoriano May 14, 2024

belimawr May 14, 2024

jsoriano May 14, 2024

elasticmachine commented May 10, 2024

belimawr commented May 13, 2024

elasticmachine commented May 13, 2024 •

edited

Loading

pierrehilbert commented May 21, 2024

botelastic bot commented Jun 20, 2024

botelastic bot commented Aug 18, 2024

andrewkroh Aug 20, 2024

botelastic bot commented Sep 19, 2024

botelastic bot commented Oct 19, 2024

	link: https://github.com/elastic/integrations/pull/42
	link: https://github.com/elastic/integrations/pull/9826

Enable synthetic source for Elastic-Agent datastreams #9826

Enable synthetic source for Elastic-Agent datastreams #9826

Conversation

belimawr commented May 9, 2024 • edited Loading

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Choose a reason for hiding this comment

jsoriano May 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

strawgate commented May 9, 2024

elastic-sonarqube bot commented May 9, 2024

Quality Gate passed

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsoriano May 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented May 10, 2024

belimawr commented May 13, 2024

elasticmachine commented May 13, 2024 • edited Loading

💔 Build Failed

Failed CI Steps

History

pierrehilbert commented May 21, 2024

botelastic bot commented Jun 20, 2024

botelastic bot commented Aug 18, 2024

Choose a reason for hiding this comment

botelastic bot commented Sep 19, 2024

botelastic bot commented Oct 19, 2024

belimawr commented May 9, 2024 •

edited

Loading

jsoriano May 9, 2024 •

edited

Loading

jsoriano May 9, 2024 •

edited

Loading

elasticmachine commented May 13, 2024 •

edited

Loading