modelindexer: introduce go-elasticsearch indexer #5970

axw · 2021-08-19T05:08:42Z

Motivation/summary

Introduce an experimental option to index events using go-elasticsearch, bypassing libbeat. This only works when data streams are enabled. You can enable the experimental indexer with:

apm-server -E output.elasticsearch.experimental=true -E apm-server.data_streams.enabled=true

This approach aligns with how we want to evolve apm-server output tuning: moving away from the current libbeat queue and output configuration, simplifying config to be similar to how APM Agents are configured: bulk requests are now executed either after some time elapses (output.elasticsearch.flush_interval, defaults to 1s) or if a number of bytes is reached (output.elasticsearch.flush_bytes, defaults to 5MB).

We have temporarily forked the go-elasticsearch bulk-indexer. See comments in the code for rationale.

Some libbeat metrics are faked, so this output works with stack monitoring and other existing uses of libbeat metrics. We should consider removing "libbeat" from our metric names and standardising on those, so we are not locked into a particular technology.

Checklist

Update CHANGELOG.asciidoc
~~- [ ] Documentation has been updated~~

How to test these changes

Run apm-server with -E output.elasticsearch.experimental=true -E apm-server.data_streams.enabled=true
Send various events, check they are indexed successfully

Related issues

#6002

apmmachine · 2021-08-19T05:50:56Z

❕ Build Aborted

There is a new build on-going so the previous on-going builds have been aborted.

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Reason: Aborted from #19
Start Time: 2021-10-11T07:59:42.389+0000
Duration: 33 min 27 sec
Commit: 5084eba

Test stats 🧪

Test	Results
Failed	0
Passed	5688
Skipped	17
Total	5705

Steps errors

Expand to view the steps failures

`Print Message`

Took 0 min 0 sec . View more details here
Description: �[39;49m[INFO] Can not determine redirect link!!!�[0m

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/hey-apm : Run the hey-apm benchmark.
/package : Generate and publish the docker images.

axw · 2021-10-07T06:04:22Z

publish (libbeat):

go test -benchmem -bench=. -run=NONE -benchtime=5s -cpu=1,6,12
goos: linux
goarch: amd64
pkg: github.com/elastic/apm-server/publish
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkPublisher                329248             17501 ns/op            1931 B/op         19 allocs/op
BenchmarkPublisher-6              463827             11226 ns/op            1935 B/op         19 allocs/op
BenchmarkPublisher-12             360158             14351 ns/op            1935 B/op         19 allocs/op
PASS
ok      github.com/elastic/apm-server/publish   54.398s

modelindexer:

$ go test -v -benchmem -bench=. -benchtime=5s -run=NONE -cpu=1,6,12
goos: linux
goarch: amd64
pkg: github.com/elastic/apm-server/model/modelindexer
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkModelIndexer
BenchmarkModelIndexer            1253119              5272 ns/op            1303 B/op         12 allocs/op
BenchmarkModelIndexer-6          2137680              2537 ns/op            1241 B/op         12 allocs/op
BenchmarkModelIndexer-12         1768158              2898 ns/op            1259 B/op         12 allocs/op
PASS
ok      github.com/elastic/apm-server/model/modelindexer        25.672s

Introduce an experimental option to index events using go-elasticsearch, bypassing libbeat.

stuartnelson3

looking really good!

model/modelindexer/bulk_indexer.go

model/modelindexer/indexer.go

model/modelindexer/indexer_integration_test.go

model/modelindexer/indexer.go

marclop

Looks great!

beater/beater.go

beater/server_test.go

model/modelindexer/bulk_indexer.go

model/modelindexer/indexer.go

model/modelindexer/indexer_integration_test.go

Don't let flushing block Close. Add a context param to Close; when it is cancelled, cancel any ongoing flush attempts.

simitt

This loooks great!

Tested a bit with standalone+data streams and also when running managed by agents, didn't observe any obvious issues.

beater/beater.go

simitt · 2021-10-11T07:15:07Z

model/modelindexer/indexer.go

+	var eventsFailed int64
+	for _, item := range resp.Items {
+		for _, info := range item {
+			if info.Error.Type != "" || info.Status > 201 {


Not certain we need this, but libbeat did check for status < 500 per event, before calling an event nonIndexable and dropping it.

Maybe for later, libbeat just recently introduced sending events that can't be indexed to a dead letter index (if configured).

Yeah I think I'll defer these kinds of things, I'd like to get this merged so we can test out the basics first.

apmmachine · 2021-10-11T08:39:19Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-10-11T10:34:24.206+0000
Duration: 44 min 55 sec
Commit: a3003da

Test stats 🧪

Test	Results
Failed	0
Passed	6185
Skipped	18
Total	6203

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/hey-apm : Run the hey-apm benchmark.
/package : Generate and publish the docker images.

* modelindexer: introduce go-elasticsearch indexer Introduce an experimental option to index events using go-elasticsearch, bypassing libbeat. (cherry picked from commit aa4f76a) # Conflicts: # changelogs/head.asciidoc

… (#6317) * modelindexer: introduce go-elasticsearch indexer (#5970) * modelindexer: introduce go-elasticsearch indexer Introduce an experimental option to index events using go-elasticsearch, bypassing libbeat. (cherry picked from commit aa4f76a) # Conflicts: # changelogs/head.asciidoc * Delete head.asciidoc Co-authored-by: Andrew Wilkins <[email protected]>

stuartnelson3 · 2021-11-04T14:33:29Z

Confirmed with BC3

axw added v7.16.0 backport-7.x labels Aug 19, 2021

zube bot added the [zube]: In Review label Aug 20, 2021

simitt mentioned this pull request Aug 20, 2021

Create experimental output using go-elasticsearch #6002

Closed

axw force-pushed the publish-goelasticsearch branch from be2c558 to 2ca39f7 Compare August 23, 2021 09:29

axw mentioned this pull request Aug 23, 2021

Use @timestamp instead of timestamp.us and switch to type date_nanos #1678

Open

axw force-pushed the publish-goelasticsearch branch 2 times, most recently from 8943471 to 0aeebb2 Compare September 3, 2021 07:24

axw force-pushed the publish-goelasticsearch branch 4 times, most recently from a00c818 to 9ac9844 Compare September 15, 2021 10:02

axw mentioned this pull request Oct 4, 2021

publish: implement BatchProcessor; remove tracing #6243

Merged

1 task

axw force-pushed the publish-goelasticsearch branch 2 times, most recently from 266196b to 0320bae Compare October 7, 2021 04:51

axw force-pushed the publish-goelasticsearch branch 2 times, most recently from f2a07a3 to 66b59df Compare October 7, 2021 06:06

modelindexer: introduce go-elasticsearch indexer

6f7fb2b

Introduce an experimental option to index events using go-elasticsearch, bypassing libbeat.

axw force-pushed the publish-goelasticsearch branch from 66b59df to 6f7fb2b Compare October 7, 2021 06:13

axw marked this pull request as ready for review October 7, 2021 07:10

axw requested a review from a team October 7, 2021 07:10

stuartnelson3 reviewed Oct 7, 2021

View reviewed changes

Mpdreamz reviewed Oct 7, 2021

View reviewed changes

model/modelindexer/indexer.go Show resolved Hide resolved

marclop reviewed Oct 11, 2021

View reviewed changes

axw added 4 commits October 11, 2021 13:12

model/modelindexer: s/Bytes/Len/

acaf109

Check MaxRequests <= 0 for default

4bf8907

model/modelindexer: add context to Close

06a5a8a

Don't let flushing block Close. Add a context param to Close; when it is cancelled, cancel any ongoing flush attempts.

beater: add config:",inline" tag to ES config

5e6e369

simitt approved these changes Oct 11, 2021

View reviewed changes

axw added 3 commits October 11, 2021 15:59

beater: address review

5084eba

model/modelindexer: address review

8a790b8

beater: close model indexer on shutdown

bcc7c54

axw added 3 commits October 11, 2021 16:49

model/modelindexer: rate limit logging

2691c0f

model/modelindexer: fix lint

270a299

model/modelindexer: more doc comments

45902f6

stuartnelson3 approved these changes Oct 11, 2021

View reviewed changes

marclop approved these changes Oct 11, 2021

View reviewed changes

Mpdreamz approved these changes Oct 11, 2021

View reviewed changes

axw enabled auto-merge (squash) October 11, 2021 10:33

Merge branch 'master' into publish-goelasticsearch

a3003da

axw merged commit aa4f76a into elastic:master Oct 11, 2021

zube bot added [zube]: Done and removed [zube]: In Review labels Oct 11, 2021

mergify bot mentioned this pull request Oct 11, 2021

[7.x] modelindexer: introduce go-elasticsearch indexer (backport #5970) #6317

Merged

axw mentioned this pull request Oct 12, 2021

[APM] add option to use experimental elasticsearch output to integration policy editor elastic/kibana#114606

Closed

marclop added backport-skip Skip notification from the automated backport with mergify test-plan labels Oct 25, 2021

stuartnelson3 self-assigned this Nov 4, 2021

stuartnelson3 removed their assignment Nov 4, 2021

stuartnelson3 added the test-plan-ok label Nov 4, 2021

cmacknz mentioned this pull request May 27, 2022

Use the go-elasticsearch client for the Elasticsearch output elastic/elastic-agent-shipper#14

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

modelindexer: introduce go-elasticsearch indexer #5970

modelindexer: introduce go-elasticsearch indexer #5970

axw commented Aug 19, 2021 •

edited by zube bot

Loading

apmmachine commented Aug 19, 2021 •

edited

Loading

Build stats

Test stats 🧪

`Print Message`

axw commented Oct 7, 2021

stuartnelson3 left a comment

marclop left a comment

simitt left a comment

simitt Oct 11, 2021

axw Oct 11, 2021

apmmachine commented Oct 11, 2021 •

edited

Loading

Build stats

Test stats 🧪

stuartnelson3 commented Nov 4, 2021

modelindexer: introduce go-elasticsearch indexer #5970

modelindexer: introduce go-elasticsearch indexer #5970

Conversation

axw commented Aug 19, 2021 • edited by zube bot Loading

Motivation/summary

Checklist

How to test these changes

Related issues

apmmachine commented Aug 19, 2021 • edited Loading

❕ Build Aborted

Build stats

Test stats 🧪

Steps errors

Print Message

🤖 GitHub comments

axw commented Oct 7, 2021

stuartnelson3 left a comment

Choose a reason for hiding this comment

marclop left a comment

Choose a reason for hiding this comment

simitt left a comment

Choose a reason for hiding this comment

simitt Oct 11, 2021

Choose a reason for hiding this comment

axw Oct 11, 2021

Choose a reason for hiding this comment

apmmachine commented Oct 11, 2021 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

stuartnelson3 commented Nov 4, 2021

axw commented Aug 19, 2021 •

edited by zube bot

Loading

apmmachine commented Aug 19, 2021 •

edited

Loading

`Print Message`

apmmachine commented Oct 11, 2021 •

edited

Loading