Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove ecs.version field from mappings #11632

Merged
merged 1 commit into from
Sep 12, 2023
Merged

Remove ecs.version field from mappings #11632

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions apmpackage/apm/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,9 @@
- description: Explicitly set `event.ingested` in traces-apm.sampled
type: enhancement
link: https://github.com/elastic/apm-server/pull/11623
- description: Remove `ecs.version` from all data streams
type: enhancement
link: https://github.com/elastic/apm-server/pull/11632
- version: "8.10.0"
changes:
- description: Add permissions to reroute to dedicated datasets for logs, metrics and traces
Expand Down
2 changes: 1 addition & 1 deletion apmpackage/apm/data_stream/app_logs/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,4 @@ processors:
- pipeline:
name: observer_ids
- pipeline:
name: ecs_version
name: remove_ecs_version
2 changes: 0 additions & 2 deletions apmpackage/apm/data_stream/app_logs/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,8 +70,6 @@
name: device.model.identifier
- external: ecs
name: device.manufacturer
- external: ecs
name: ecs.version
- external: ecs
name: error.stack_trace
- external: ecs
Expand Down
2 changes: 1 addition & 1 deletion apmpackage/apm/data_stream/app_metrics/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ processors:
- pipeline:
name: observer_ids
- pipeline:
name: ecs_version
name: remove_ecs_version
- pipeline:
name: user_agent
- pipeline:
Expand Down
2 changes: 0 additions & 2 deletions apmpackage/apm/data_stream/app_metrics/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,8 +70,6 @@
name: device.model.identifier
- external: ecs
name: device.manufacturer
- external: ecs
name: ecs.version
- external: ecs
name: event.outcome
- external: ecs
Expand Down
2 changes: 1 addition & 1 deletion apmpackage/apm/data_stream/error_logs/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ processors:
- pipeline:
name: observer_ids
- pipeline:
name: ecs_version
name: remove_ecs_version
- pipeline:
name: user_agent
- pipeline:
Expand Down
2 changes: 0 additions & 2 deletions apmpackage/apm/data_stream/error_logs/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,8 +70,6 @@
name: device.model.identifier
- external: ecs
name: device.manufacturer
- external: ecs
name: ecs.version
- external: ecs
name: error.id
- external: ecs
Expand Down
2 changes: 1 addition & 1 deletion apmpackage/apm/data_stream/internal_metrics/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ processors:
- pipeline:
name: observer_ids
- pipeline:
name: ecs_version
name: remove_ecs_version
- pipeline:
name: user_agent
- pipeline:
Expand Down
2 changes: 0 additions & 2 deletions apmpackage/apm/data_stream/internal_metrics/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,8 +62,6 @@
name: destination.ip
- external: ecs
name: destination.port
- external: ecs
name: ecs.version
- external: ecs
name: event.outcome
- external: ecs
Expand Down
2 changes: 1 addition & 1 deletion apmpackage/apm/data_stream/rum_traces/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ processors:
- pipeline:
name: observer_ids
- pipeline:
name: ecs_version
name: remove_ecs_version
- pipeline:
name: user_agent
- pipeline:
Expand Down
2 changes: 1 addition & 1 deletion ...ata_stream/service_destination_interval_metrics/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ processors:
- pipeline:
name: observer_ids
- pipeline:
name: ecs_version
name: remove_ecs_version
- pipeline:
name: set_metrics
- remove:
Expand Down
2 changes: 0 additions & 2 deletions apmpackage/apm/data_stream/service_destination_interval_metrics/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
- external: ecs
name: agent.name
- external: ecs
name: ecs.version
- external: ecs
name: event.outcome
- external: ecs
Expand Down
2 changes: 1 addition & 1 deletion ...pm/data_stream/service_summary_interval_metrics/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ processors:
- pipeline:
name: observer_ids
- pipeline:
name: ecs_version
name: remove_ecs_version
- pipeline:
name: set_metrics
- remove:
Expand Down
2 changes: 0 additions & 2 deletions apmpackage/apm/data_stream/service_summary_interval_metrics/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
- external: ecs
name: agent.name
- external: ecs
name: ecs.version
- external: ecs
name: observer.hostname
- external: ecs
Expand Down
2 changes: 1 addition & 1 deletion ...ata_stream/service_transaction_interval_metrics/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ processors:
- pipeline:
name: observer_ids
- pipeline:
name: ecs_version
name: remove_ecs_version
- pipeline:
name: set_metrics
- remove:
Expand Down
2 changes: 0 additions & 2 deletions apmpackage/apm/data_stream/service_transaction_interval_metrics/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
- external: ecs
name: agent.name
- external: ecs
name: ecs.version
- external: ecs
name: observer.hostname
- external: ecs
Expand Down
2 changes: 1 addition & 1 deletion ...ackage/apm/data_stream/traces/elasticsearch/ingest_pipeline/apm_data_stream_migration.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ processors:
- pipeline:
name: observer_ids
- pipeline:
name: ecs_version
name: remove_ecs_version
- pipeline:
name: process_ppid
- script:
Expand Down
2 changes: 1 addition & 1 deletion apmpackage/apm/data_stream/traces/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ processors:
- pipeline:
name: observer_ids
- pipeline:
name: ecs_version
name: remove_ecs_version
- pipeline:
name: user_agent
- pipeline:
Expand Down
2 changes: 0 additions & 2 deletions apmpackage/apm/data_stream/traces/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,8 +70,6 @@
name: device.model.identifier
- external: ecs
name: device.manufacturer
- external: ecs
name: ecs.version
- external: ecs
name: event.outcome
- external: ecs
Expand Down
2 changes: 1 addition & 1 deletion ...ge/apm/data_stream/transaction_interval_metrics/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ processors:
- pipeline:
name: observer_ids
- pipeline:
name: ecs_version
name: remove_ecs_version
- pipeline:
name: set_metrics
- remove:
Expand Down
2 changes: 0 additions & 2 deletions apmpackage/apm/data_stream/transaction_interval_metrics/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,6 @@
name: cloud.service.name
- external: ecs
name: container.id
- external: ecs
name: ecs.version
- external: ecs
name: event.outcome
- external: ecs
Expand Down
11 changes: 5 additions & 6 deletions apmpackage/cmd/genpackage/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ var (
ecsVersion = flag.String("ecs", "", "ECS version (required)")
)

func generatePackage(pkgfs fs.FS, version, ecsVersion *version.V, ecsReference string) error {
func generatePackage(pkgfs fs.FS, version *version.V, ecsReference string) error {
// Walk files, performing some APM-specific validations and transformations as we go.
//
// We assume the target destination does not yet exist.
Expand Down Expand Up @@ -77,7 +77,7 @@ func generatePackage(pkgfs fs.FS, version, ecsVersion *version.V, ecsReference s
// Use `lifecycle.json` instead of e.g. `lifecycle.1m.json`
outputPath = strings.Replace(filepath.Join(*outputDir, p.Path), d.Name(), "lifecycle.yml", -1)
}
err := renderFile(pkgfs, path, outputPath, version, ecsVersion, ecsReference, p.Interval)
err := renderFile(pkgfs, path, outputPath, version, ecsReference, p.Interval)
if err != nil {
return err
}
Expand All @@ -86,7 +86,7 @@ func generatePackage(pkgfs fs.FS, version, ecsVersion *version.V, ecsReference s
})
}

func renderFile(pkgfs fs.FS, path, outputPath string, version, ecsVersion *version.V, ecsReference, interval string) error {
func renderFile(pkgfs fs.FS, path, outputPath string, version *version.V, ecsReference, interval string) error {
content, err := fs.ReadFile(pkgfs, path)
if err != nil {
return err
Expand All @@ -111,7 +111,7 @@ func renderFile(pkgfs fs.FS, path, outputPath string, version, ecsVersion *versi
}
content = buf.Bytes()
}
content, err = transformFile(path, content, version, ecsVersion, ecsReference, interval)
content, err = transformFile(path, content, version, ecsReference, interval)
if err != nil {
return fmt.Errorf("error transforming %q: %w", path, err)
}
Expand Down Expand Up @@ -176,7 +176,6 @@ func main() {
*ecsReference = fmt.Sprintf("git@v%s", *ecsVersion)
}
pkgVersion := version.MustNew(*pkgVersion)
ecsVersion := version.MustNew(*ecsVersion)

// Locate the apmpackage/apm directory.
_, file, _, ok := runtime.Caller(0)
Expand All @@ -188,7 +187,7 @@ func main() {
// Generate a completely rendered _source_ package, which can then be fed to
// `elastic-agent build` to build the final package for inclusion in package-storage.
log.Printf("generating integration package v%s in %q", pkgVersion.String(), *outputDir)
if err := generatePackage(os.DirFS(pkgdir), pkgVersion, ecsVersion, *ecsReference); err != nil {
if err := generatePackage(os.DirFS(pkgdir), pkgVersion, *ecsReference); err != nil {
log.Fatal(err)
}
}
20 changes: 10 additions & 10 deletions apmpackage/cmd/genpackage/pipelines.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,15 +36,15 @@ import (
// - ...
func getCommonPipeline(name string, version *version.V) []map[string]interface{} {
commonPipelines := map[string][]map[string]interface{}{
"observer_version": getObserverVersionPipeline(version),
"observer_ids": observerIDsPipeline,
"ecs_version": ecsVersionPipeline,
"user_agent": userAgentPipeline,
"process_ppid": processPpidPipeline,
"client_geoip": clientGeoIPPipeline,
"event_duration": eventDurationPipeline,
"set_metrics": setMetricsPipeline,
"event_ingested": eventIngestedPipeline,
"observer_version": getObserverVersionPipeline(version),
"observer_ids": observerIDsPipeline,
"remove_ecs_version": removeECSVersionPipeline,
"user_agent": userAgentPipeline,
"process_ppid": processPpidPipeline,
"client_geoip": clientGeoIPPipeline,
"event_duration": eventDurationPipeline,
"set_metrics": setMetricsPipeline,
"event_ingested": eventIngestedPipeline,
}
return commonPipelines[name]
}
Expand Down Expand Up @@ -103,7 +103,7 @@ var observerIDsPipeline = []map[string]interface{}{{
},
}}

var ecsVersionPipeline = []map[string]interface{}{{
var removeECSVersionPipeline = []map[string]interface{}{{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, even the previous ecsVersionPipeline was removing the field ecs.version. Does this mean we were never indexing ecs.version so it was not used in the mappings anyway?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ingest pipeline was there to remove the field from documents written by older versions of apm-server. The field would get added back into documents as a constant_keyword field.

I've left the removal there for now, but I don't think we'll keep that in elastic/elasticsearch#97546. We'll just dynamically map the field for older versions of apm-server - no big deal.

"remove": map[string]interface{}{
"field": "ecs", // remove ecs.version
"ignore_missing": true,
Expand Down
40 changes: 1 addition & 39 deletions apmpackage/cmd/genpackage/transform.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ const (
integrationName = "apm"
)

func transformFile(path string, content []byte, version, ecsVersion *version.V, ecsReference, interval string) ([]byte, error) {
func transformFile(path string, content []byte, version *version.V, ecsReference, interval string) ([]byte, error) {
if path == "manifest.yml" {
return transformPackageManifest(content, version)
}
Expand All @@ -43,9 +43,6 @@ func transformFile(path string, content []byte, version, ecsVersion *version.V,
if isIngestPipeline(path) {
return transformIngestPipeline(path, content, version)
}
if isECSFieldsYAML(path) {
return transformECSFieldsYAML(content, ecsVersion)
}
if path == "changelog.yml" {
return transformChangelog(content, version)
}
Expand All @@ -55,33 +52,6 @@ func transformFile(path string, content []byte, version, ecsVersion *version.V,
return content, nil
}

func transformECSFieldsYAML(content []byte, ecsVersion *version.V) ([]byte, error) {
var doc yaml.Node
if err := yaml.Unmarshal(content, &doc); err != nil {
return nil, err
}

var found bool
for _, fieldNode := range doc.Content[0].Content {
if yamlMapLookup(fieldNode, "name").Value != "ecs.version" {
continue
}
fieldNode.Content = append(fieldNode.Content,
&yaml.Node{Kind: yaml.ScalarNode, Value: "type"},
&yaml.Node{Kind: yaml.ScalarNode, Value: "constant_keyword"},
&yaml.Node{Kind: yaml.ScalarNode, Value: "value"},
&yaml.Node{Kind: yaml.ScalarNode, Value: ecsVersion.String()},
)
found = true
break
}
if !found {
return content, nil
}

return marshalYAML(&doc)
}

func transformBuildDependencies(content []byte, ecsReference string) ([]byte, error) {
var doc yaml.Node
if err := yaml.Unmarshal(content, &doc); err != nil {
Expand Down Expand Up @@ -223,14 +193,6 @@ func isDataStreamManifest(path string) bool {
return filepath.Base(dir) == "data_stream"
}

func isECSFieldsYAML(path string) bool {
dir, file := filepath.Split(path)
if file != "ecs.yml" {
return false
}
return filepath.Base(dir) == "fields"
}

func yamlMapLookup(n *yaml.Node, key ...string) *yaml.Node {
if n.Kind != yaml.MappingNode {
panic(fmt.Sprintf("expected node kind %v, got %v", yaml.MappingNode, n.Kind))
Expand Down
1 change: 1 addition & 0 deletions changelogs/head.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ https://github.com/elastic/apm-server/compare/8.10\...main[View commits]

[float]
==== Breaking Changes
- The `ecs.version` field has been removed from APM data streams {pull}11632[11632]

[float]
==== Deprecations
Expand Down
6 changes: 0 additions & 6 deletions systemtest/approvals/TestAgentConfig.approved.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,6 @@
"namespace": "default",
"type": "metrics"
},
"ecs": {
"version": "dynamic"
},
"labels": {
"etag": "dynamic"
},
Expand All @@ -34,9 +31,6 @@
"namespace": "default",
"type": "metrics"
},
"ecs": {
"version": "dynamic"
},
"labels": {
"etag": "dynamic"
},
Expand Down
Loading