feat(inputs.opensearch_query): Add opensearch_query input plugin #12393

NullOranje · 2022-12-13T20:54:32Z

Required for all PRs

Updated associated README.md.
Wrote appropriate unit tests.
Pull request title or commits are in conventional commit format

This plugin queries endpoints to obtain metrics from data stored in an OpenSearch cluster. The code is largely an update of the existing inputs.elasticsearch_query plugin.

The library in that plugin is incompatible with OpenSearch (and, in fact, all Elastic clusters >= 7.0.0). To resolve the issue, I made the following changes:

Use v7 of the high-level Elasticsearch API library olivere/elastic to process query requests and responses
Add the Amazon OpenSearch go client to support cluster queries

Future work should look at removing the deprecated olivere/elastic library.

powersj

Thanks for putting this up! I have two high-level concerns:

why does this need to be a second plugin?
most of this is copy and paste from inputs.elasticsearch_query. I'd rather not have this much duplication. how much of this could be handled with a config variable or if not abstracted into the common folder?

I have done a very quick first pass through the PR as well, I realize some of these things already exist in the current plugin, but I'd like to clean up anything new that comes in.

powersj · 2022-12-14T21:32:31Z

plugins/inputs/opensearch_query/README.md

+
+## OpenSearch Support
+
+This plugins is tested against OpenSearch 2.4.0.


Is it possible to test this against 1.3 to ensure some backwards compat before we accept this in?

I can add testing for this. I wasn't worried about it since the 1.3.x line is currently maintenance-only, and will be EOL December 31, 2023.

Thanks - I know someone is going to try and ask if it is supported so I'd rather find out now

I tested it against 1.3.7, and it ran fine. I've add this container to the test suite.

plugins/inputs/opensearch_query/README.md

plugins/inputs/opensearch_query/opensearch_query.go

plugins/inputs/opensearch_query/opensearch_query_test.go

plugins/inputs/opensearch_query/aggregation_query.go

powersj · 2022-12-14T22:11:50Z

CircleCI Pipeline — Could not find a usable config.yml, you may have revoked the CircleCI OAuth app.

I recently had a chat with CircleCI about this error. It is usually due to you having a CirlceCI account with an expired GitHub oauth. To resolve this, you need to log back into CircleCI with your usename/password if that is how you log in or if you use GitHub log, re-create your oauth/relogin with github.

That should regenerate your token and then allow you to push a commit or close and reopen this PR and tests should run.

I know the error message doesn't line up, and I have suggested them changing that, but that is what causes this error. And we do need tests to run :)

NullOranje · 2022-12-14T22:26:23Z

Thanks for putting this up! I have two high-level concerns:
1. why does this need to be a second plugin?

As I'm sure you're aware, Elasticsearch and Opensearch are divergent products. The existing plugin supports old versions of ES, and (1) I didn't want to break workflows for existing users, and (2) I wanted there to be a clean spot to capture the delta.

2. most of this is copy and paste from inputs.elasticsearch_query. I'd rather not have this much duplication. how much of this could be handled with a config variable or if not abstracted into the common folder?

The existing elasticsearch_query plugin doesn't support ES 7.10.2, which is what OpenSearch is based on. I used the olivere/elastic library for a quick win, since it already implements the structs for ES, but I would like to eventually move away from those, since the author has deprecated the library.

This submission was intended to be an in-kind update for OS. I have more updates I want to put forward, such as support for multi-field aggregations that aren't implemented in the existing elasticsearch_query plugin.

powersj · 2022-12-15T16:40:38Z

I used the olivere/elastic library for a quick win,

I appreciate this and I can understand how from your perspective this gives users what they need quickly. As a maintainer though, I now have to support two plugins that are nearly identical.

I recognize the possible need for a second plugin, see my comment in #11345, as well as the need to change the underlying client library, see #12099.

Ideally I would prefer us try to use the existing plugin and add an option for what version or interface we are connecting to as proposed in #12099 (this was for the elasticsearch input, but applies here as well). If that truly is not possible, then a new plugin using the supported library would be the next best option.

NullOranje · 2022-12-15T17:21:32Z

Thanks for the links to the issues. I certainly understand the drive to not maintain two separate plugins, given their similarity. My thought on this, though, is that the similarity will diverge over time. Elastic already makes big API changes between major versions, and they've been especially hostile towards Opensearch, for obvious reasons.

I'm not sure what the best course of action here is. Looking at the opensearch-project/opensearch-go library again, I think I refactor to eliminate the need for olivere/elastic. The painful part of this is handling the output, which is where olivere excelled. Given the limited scope of the responses, it might not be too bad.

Would it make sense then to try and have two separate plugins? I know having an API selection option would be appealing, but given the ever increasing divergence, I think we'd eventually end up maintaining two plugins under one package anyway.

powersj · 2022-12-15T20:13:07Z

Would it make sense then to try and have two separate plugins? I know having an API selection option would be appealing, but given the ever increasing divergence, I think we'd eventually end up maintaining two plugins under one package anyway.

Given your comments about the diverging of the two products it does sound like having two plugins does make sense. I would still want to see it using the new, supported library though.

Thanks!

go.sum

plugins/inputs/opensearch_query/README.md

plugins/inputs/all/opensearch_query.go

plugins/inputs/opensearch_query/aggregation_parser.go

plugins/inputs/opensearch_query/aggregation_query.go

plugins/inputs/opensearch_query/opensearch_query.go

… function; change metric name format

…; update README.md

NullOranje · 2023-01-23T20:24:55Z

Thanks for the feedback. I've left a couple questions in-line to hopefully tie up the loose ends on this PR.

plugins/inputs/opensearch_query/opensearch_query.go

srebhan

There are still a few comments open and I think the test here is not sufficient...

srebhan · 2023-01-30T14:59:30Z

plugins/inputs/opensearch_query/opensearch_query_test.go

+			err = o.Gather(&acc)
+			if (len(acc.Errors) > 0) && !tt.wantQueryResErr {
+				for _, err = range acc.Errors {
+					t.Errorf("OpensearchQuery.Gather() error: %v, wantQueryResErr %v", err, tt.wantQueryResErr)
+				}
+				return
+			}
+
+			require.NoError(t, err)


Hmm what if we tt.wantQueryResErr but do not get any?

Easy. You rewrite the test to properly check.

(len(acc.Errors) > 0) != tt.wantQueryResErr should fix that

…st all OpenSearch versions

srebhan

Looks good to me. Thanks for your work to make this fly @NullOranje!

srebhan · 2023-02-01T17:02:50Z

@powersj there has been some significant changes, so please take a look again!

Co-authored-by: Sven Rebhan <[email protected]>

powersj

Hi! We are almost there, just two questions hopefully easy :)

plugins/inputs/opensearch_query/opensearch_query.go

plugins/inputs/opensearch_query/README.md

plugins/inputs/opensearch_query/opensearch_query.go

… handling cleanup

telegraf-tiger · 2023-02-02T15:05:58Z

Download PR build artifacts for linux_amd64.tar.gz, darwin_amd64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

⚠️ This pull request increases the Telegraf binary size by 2.19 % for linux amd64 (new size: 164.5 MB, nightly size 161.0 MB)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_arm64.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz	windows_i386.zip
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz
		static_linux_amd64.tar.gz

powersj · 2023-02-02T15:41:43Z

Thank you for driving this!

NullOranje · 2023-02-02T15:43:14Z

My pleasure. Looking forward to putting it to use!

Nick McKinney added 6 commits December 13, 2022 15:03

Initial commit: add support for input/opensearch_query

7b0885b

Add TLS support

f45fb68

Fix bug getting field mappings on a wildcard index

07ee619

Finish renaming from Elasticsearch to Opensearch

cf8c210

Update README.md; update sample config

3a1495f

go mod tidy

e684460

telegraf-tiger bot added feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin plugin/input 1. Request for new input plugins 2. Issues/PRs that are related to input plugins labels Dec 13, 2022

Nick McKinney added 5 commits December 13, 2022 16:02

Update docs/LICENSE_OF_DEPENDENCIES.md

a96e9d3

Update README.md to conform

e1bb29e

Lint

958e8f7

Lint

1caabf5

Lint

4195041

powersj reviewed Dec 14, 2022

View reviewed changes

NullOranje added 3 commits December 14, 2022 19:28

Fix error handling; refactor tests to better check for errors; lint

214304b

Update LICENSE_OF_DEPENDENCIES.md per circleci make check-deps job

ab9651c

Use HTTPS for testing (demo certs; InsecureSkipVerify)

fabc895

Hipska suggested changes Dec 19, 2022

View reviewed changes

NullOranje added 7 commits December 20, 2022 11:10

Fix error handling; remove duplicated client initialization

c0d73d6

Add request structs for use with opensearchapi

c8d0650

(Broken) Refactor aggregation request to eliminate olivere library

205102b

Refactor to remove olivere/elastic library; rewrite metric collection…

3456787

… function; change metric name format

Update buildAggregationQuery test

3fc08f5

Remove unused code

9991916

Add support for nested values, such as from percentiles; update tests…

903bbf6

…; update README.md

make docs

b0da877

Hipska reviewed Jan 24, 2023

View reviewed changes

plugins/inputs/opensearch_query/opensearch_query.go Outdated Show resolved Hide resolved

Add support for secret stores

0560bc5

powersj assigned srebhan Jan 25, 2023

Nick McKinney added 4 commits January 26, 2023 08:05

Move test data out of globals

eb349d8

Cleanup test opensearch client

ab1a4ba

Merge remote-tracking branch 'upstream/master'

909a1bf

Update from master

cc3261a

srebhan reviewed Jan 30, 2023

View reviewed changes

NullOranje added 2 commits January 31, 2023 10:26

Fix test case

8d08405

Add OpenSearch 1.3.7 to test cases; bump Opensearch 2.x to 2.5.0 ; te…

372c425

…st all OpenSearch versions

srebhan approved these changes Feb 1, 2023

View reviewed changes

srebhan requested a review from powersj February 1, 2023 17:02

srebhan assigned powersj and unassigned srebhan Feb 1, 2023

Update plugins/inputs/opensearch_query/opensearch_query.go

33d42f4

Co-authored-by: Sven Rebhan <[email protected]>

powersj reviewed Feb 1, 2023

View reviewed changes

plugins/inputs/opensearch_query/opensearch_query.go Outdated Show resolved Hide resolved

plugins/inputs/opensearch_query/README.md Outdated Show resolved Hide resolved

Hipska suggested changes Feb 2, 2023

View reviewed changes

plugins/inputs/opensearch_query/opensearch_query.go Outdated Show resolved Hide resolved

plugins/inputs/opensearch_query/opensearch_query.go Outdated Show resolved Hide resolved

Removed unused function and associated test; updated README.md; error…

120d2e2

… handling cleanup

Hipska approved these changes Feb 2, 2023

View reviewed changes

Lint

5ddf598

powersj approved these changes Feb 2, 2023

View reviewed changes

powersj merged commit d35bd06 into influxdata:master Feb 2, 2023

powersj mentioned this pull request Feb 16, 2023

feat(outputs.opensearch): opensearch output plugin #11958

Merged

3 tasks

srebhan added this to the v1.26.0 milestone Jun 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(inputs.opensearch_query): Add opensearch_query input plugin #12393

feat(inputs.opensearch_query): Add opensearch_query input plugin #12393

NullOranje commented Dec 13, 2022

powersj left a comment

powersj Dec 14, 2022

NullOranje Dec 14, 2022

powersj Dec 15, 2022

NullOranje Jan 31, 2023

powersj commented Dec 14, 2022

NullOranje commented Dec 14, 2022

powersj commented Dec 15, 2022

NullOranje commented Dec 15, 2022

powersj commented Dec 15, 2022 •

edited

Loading

NullOranje commented Jan 23, 2023

srebhan left a comment

srebhan Jan 30, 2023

NullOranje Jan 31, 2023

srebhan left a comment

srebhan commented Feb 1, 2023

powersj left a comment

telegraf-tiger bot commented Feb 2, 2023

Artifact URLs

powersj commented Feb 2, 2023

NullOranje commented Feb 2, 2023


		## OpenSearch Support

		This plugins is tested against OpenSearch 2.4.0.

feat(inputs.opensearch_query): Add opensearch_query input plugin #12393

feat(inputs.opensearch_query): Add opensearch_query input plugin #12393

Conversation

NullOranje commented Dec 13, 2022

Required for all PRs

powersj left a comment

Choose a reason for hiding this comment

powersj Dec 14, 2022

Choose a reason for hiding this comment

NullOranje Dec 14, 2022

Choose a reason for hiding this comment

powersj Dec 15, 2022

Choose a reason for hiding this comment

NullOranje Jan 31, 2023

Choose a reason for hiding this comment

powersj commented Dec 14, 2022

NullOranje commented Dec 14, 2022

powersj commented Dec 15, 2022

NullOranje commented Dec 15, 2022

powersj commented Dec 15, 2022 • edited Loading

NullOranje commented Jan 23, 2023

srebhan left a comment

Choose a reason for hiding this comment

srebhan Jan 30, 2023

Choose a reason for hiding this comment

NullOranje Jan 31, 2023

Choose a reason for hiding this comment

srebhan left a comment

Choose a reason for hiding this comment

srebhan commented Feb 1, 2023

powersj left a comment

Choose a reason for hiding this comment

telegraf-tiger bot commented Feb 2, 2023

Artifact URLs

powersj commented Feb 2, 2023

NullOranje commented Feb 2, 2023

powersj commented Dec 15, 2022 •

edited

Loading