Input v2 compatibility layer #19401

urso · 2020-06-25T18:59:21Z

Enhancement

What does this PR do?

This change introduces the input/v2/compat package, that provides a
compatibility layer for integration the input v2 API into existing beats
functionality.
The package provides a function for wrapping v2 inputs into a
cfgfile.RunnerFactory (used by autodiscovery, filebet module loading,
config file reloading, ....).

The current state of the full implementation can be seen here and sample inputs based on the new API.

The full list of changes will include:

Introduce v2 API interfaces
Introduce compatibility layer to integrate API with existing functionality
Introduce helpers for writing stateless inputs.
Introduce helpers for writing inputs that store a state between restarts.
Integrate new API with existing inputs and modules in filebeat.

Test coverage report:

gocov test ./... | gocov report
ok  	github.com/elastic/beats/v7/filebeat/input/v2/compat	0.009s	coverage: 94.6% of statements

github.com/elastic/beats/v7/filebeat/input/v2/compat/composed.go composeFactory.Create		 100.00% (11/11)
github.com/elastic/beats/v7/filebeat/input/v2/compat/compat.go	 factory.CheckConfig		 100.00% (4/4)
github.com/elastic/beats/v7/filebeat/input/v2/compat/compat.go	 factory.Create			 100.00% (4/4)
github.com/elastic/beats/v7/filebeat/input/v2/compat/composed.go composeFactory.CheckConfig	 100.00% (4/4)
github.com/elastic/beats/v7/filebeat/input/v2/compat/compat.go	 runner.Stop			 100.00% (3/3)
github.com/elastic/beats/v7/filebeat/input/v2/compat/compat.go	 runner.Start			 100.00% (3/3)
github.com/elastic/beats/v7/filebeat/input/v2/compat/compat.go	 RunnerFactory			 100.00% (1/1)
github.com/elastic/beats/v7/filebeat/input/v2/compat/composed.go Combine			 100.00% (1/1)
github.com/elastic/beats/v7/filebeat/input/v2/compat/compat.go	 @99:5				 80.00% (4/5)
github.com/elastic/beats/v7/filebeat/input/v2/compat/compat.go	 runner.String			 0.00% (0/1)
github.com/elastic/beats/v7/filebeat/input/v2/compat		 --------------------------	 94.59% (35/37)

Why is it important?

This PR only introduces the compatiblity layer which will allow use to use new input with existing Beats functionality, but also run new and old architecture in parallel.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~- [ ] I have made corresponding changes to the documentation~~
~~- [ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
~~- [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.~~

Related issues

Relates Filebeat input v2 API #15324

elasticmachine · 2020-06-25T18:59:23Z

Pinging @elastic/integrations-services (Team:Services)

elasticmachine · 2020-06-25T19:51:28Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [urso commented: jenkins run the tests please]
Start Time: 2020-06-29T18:34:22.514+0000
Duration: 70 min 30 sec

Test stats 🧪

Test	Results
Failed	0
Passed	543
Skipped	127
Total	670

Steps errors

Expand to view the steps failures

Name: Report to Codecov
- Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done
- Duration: 2 min 22 sec
- Start Time: 2020-06-29T19:30:14.657+0000
- log

urso · 2020-06-29T18:33:46Z

jenkins run the tests please

kvch · 2020-06-30T11:09:06Z

filebeat/input/v2/compat/compat.go

+		log.Infof("Input %v starting", name)
+		err := r.input.Run(
+			v2.Context{
+				ID:          "", // TODO: hmmm....


Are you going to resolve this TODO in an upcoming PR?

Answer: hmmm.... 😄

There is no common way of how IDs are defined/generated. Let me update this by computing a hash from the configuration object (some parts in libbeat do so).

The cursor based input manager allows you to configure an id setting per input that allows you to have 2 similar inputs with different ID. I think I will move the setting into this layer (will update this PR), so it can be applied to each input (ID will be added to structured logs + can be set by agent in the future).

sayden · 2020-06-30T11:28:44Z

filebeat/input/v2/compat/composed.go

+	fallback cfgfile.RunnerFactory
+}
+
+var _ cfgfile.RunnerFactory = composeFactory{}


What is this for? At least it does not clearly come to my mind so it may require a short comment to clarify.

Ah, this can be removed. I used to export the factory, but added the constructor that returns the expected interface type later. When implementing and exporting a concrete type you still might want the compiler to check that this concrete type implements the supposed interface correctly. Common pattern to do so is:

var _ <interface> = <struct type>{}

or (for interfaces on pointers):

var _ <interface> = (*<struct type>)(nil)

This declares an unused variable, but still triggers the type checker.

An example in the stdlib coming to mind is json.RawMessage.

The idea is to create the compiler error for the author modifying the type, and not the person using the type.

sayden · 2020-06-30T11:29:48Z

filebeat/input/v2/compat/composed.go

+// signal that the input type is unknown via v2.ErrUnknown.
+//
+// XXX: This RunnerFactory is used for compining the v2.Loader with the
+// existing RunnerFactory for inputs in Filebeat. The Combine function should be removed once the old RunnerFactory is removed.


Nit: XXX? Also typo in compining

elasticmachine · 2020-06-30T12:35:06Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [Pull request #19401 updated]
Start Time: 2020-06-30T13:10:03.841+0000
Duration: 30 min 1 sec

Test stats 🧪

Test	Results
Failed	0
Passed	543
Skipped	127
Total	670

Steps errors

Expand to view the steps failures

Name: Report to Codecov
- Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done
- Duration: 1 min 27 sec
- Start Time: 2020-06-30T13:35:46.457+0000
- log

urso · 2020-06-30T12:36:13Z

jenkins run the tests please

This change introduces the input/v2/compat package, that provides a compatibility layer for integration the input v2 API into existing beats functionality. The package provides a function for wrapping v2 inputs into a cfgfile.RunnerFactory (used by autodiscovery, filebet module loading, config file reloading, ....).

- fix typo - remove superfluous declaration - set input ID based on config hash or from setting if presen

…ne-beats * upstream/master: (105 commits) ci: enable packaging job (elastic#19536) ci: disable upstream trigger on PRs for the packaging job (elastic#19490) Implement memlog on-disk handling (elastic#19408) fix go.mod for PR elastic#19423 (elastic#19521) [MetricBeat] add param `aws_partition` to support aws-cn, aws-us-gov regions (elastic#19423) Input v2 stateless manager (elastic#19406) Input v2 compatibility layer (elastic#19401) [Elastic Agent] Fix artifact downloading to allow endpoint-security to be downloaded (elastic#19503) fix: ignore target changes on scans (elastic#19510) Add more helpers to pipeline/testing package (elastic#19405) Report dependencies in CSV format (elastic#19506) [Filebeat] Fix reference leak in TCP and Unix socket inputs (elastic#19459) Cursor input skeleton (elastic#19378) Add changelog. (elastic#19495) [DOC] Typo in Kerberos (elastic#19265) Remove accidentally commited unused NOTICE template (elastic#19485) [Elastic Agent] Support the install, control, and uninstall of Endpoint (elastic#19248) [Filebeat][httpjson] Add split_events_by config setting (elastic#19246) ci: disabling packaging job until we fix it (elastic#19481) Fix golang.org/x/tools to release1.13 (elastic#19478) ...

(cherry picked from commit c9a9bf5)

urso added review Filebeat Filebeat needs_backport PR is waiting to be backported to other branches. Project:Filebeat-Input-v2 Team:Services (Deprecated) Label for the former Integrations-Services team labels Jun 25, 2020

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 25, 2020

andresrc added [zube]: Inbox [zube]: In Review and removed [zube]: Inbox labels Jun 28, 2020

zube bot added [zube]: Inbox [zube]: In Review and removed [zube]: In Review [zube]: Inbox labels Jun 28, 2020

zube bot added [zube]: In Review [zube]: Inbox and removed [zube]: In Review [zube]: Inbox labels Jun 28, 2020

kvch reviewed Jun 30, 2020

View reviewed changes

sayden reviewed Jun 30, 2020

View reviewed changes

sayden approved these changes Jun 30, 2020

View reviewed changes

kvch approved these changes Jun 30, 2020

View reviewed changes

urso added 3 commits June 30, 2020 15:07

Review

cdf4fb1

- fix typo - remove superfluous declaration - set input ID based on config hash or from setting if presen

fix import order

54698d6

urso force-pushed the input-v2-compat-layer branch from 74ee468 to 54698d6 Compare June 30, 2020 13:08

urso merged commit c9a9bf5 into elastic:master Jun 30, 2020

urso deleted the input-v2-compat-layer branch June 30, 2020 14:03

zube bot added [zube]: Done and removed [zube]: In Review labels Jun 30, 2020

urso added the v7.9.0 label Jul 3, 2020

urso pushed a commit to urso/beats that referenced this pull request Jul 8, 2020

Input v2 compatibility layer (elastic#19401)

10b9d3d

(cherry picked from commit c9a9bf5)

urso mentioned this pull request Jul 8, 2020

Cherry-pick #19401 to 7.x: Input v2 compatibility layer #19728

Merged

3 tasks

urso removed the needs_backport PR is waiting to be backported to other branches. label Jul 8, 2020

urso pushed a commit that referenced this pull request Jul 8, 2020

Cherry-pick #19401 to 7.x: Input v2 compatibility layer (#19728)

e6547fb

(cherry picked from commit c9a9bf5)

andresrc removed the [zube]: Done label Jul 14, 2020

urso added the skip-test-plan label Jul 14, 2020

melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020

Input v2 compatibility layer (elastic#19401)

9a01df6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Input v2 compatibility layer #19401

Input v2 compatibility layer #19401

urso commented Jun 25, 2020 •

edited by andresrc

Loading

elasticmachine commented Jun 25, 2020

elasticmachine commented Jun 25, 2020 •

edited

Loading

Build stats

Test stats 🧪

urso commented Jun 29, 2020

kvch Jun 30, 2020

sayden Jun 30, 2020

urso Jun 30, 2020

sayden Jun 30, 2020

urso Jun 30, 2020

sayden Jun 30, 2020

elasticmachine commented Jun 30, 2020 •

edited

Loading

Build stats

Test stats 🧪

urso commented Jun 30, 2020

Input v2 compatibility layer #19401

Input v2 compatibility layer #19401

Conversation

urso commented Jun 25, 2020 • edited by andresrc Loading

What does this PR do?

Why is it important?

Checklist

Related issues

elasticmachine commented Jun 25, 2020

elasticmachine commented Jun 25, 2020 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Steps errors

urso commented Jun 29, 2020

kvch Jun 30, 2020

Choose a reason for hiding this comment

sayden Jun 30, 2020

Choose a reason for hiding this comment

urso Jun 30, 2020

Choose a reason for hiding this comment

sayden Jun 30, 2020

Choose a reason for hiding this comment

urso Jun 30, 2020

Choose a reason for hiding this comment

sayden Jun 30, 2020

Choose a reason for hiding this comment

elasticmachine commented Jun 30, 2020 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Steps errors

urso commented Jun 30, 2020

urso commented Jun 25, 2020 •

edited by andresrc

Loading

elasticmachine commented Jun 25, 2020 •

edited

Loading

elasticmachine commented Jun 30, 2020 •

edited

Loading