[Heartbeat] Support JSON expressions / validation of JSON arrays #28073

andrewvc · 2021-09-22T23:44:25Z

This PR greatly improves validation of JSON HTTP response bodies in heartbeat in two different ways:

Introduces a new notion of JSON expressions, using gval as an expression engine. This allows validation of complex data, including the use of JSON Path expressions within. As an example you can write an expression like: $.nodes[?(@name == "foo"][2].message == "hello" to traverse a JSON object containing a nodes key with nested objects, find all elements named "foo", select the third one, then see if its' message attribute equals "hello".
Supports handling APIs that return arrays of JSON objects without a top level object root. Previously, we only parsed JSON response that looked like an object, e.g. {"an": "object"}, we now support: [{"one": 1},{"two": 2}].
Deprecates the condition configuration in favor of this new expression syntax. The new syntax is more consistent and powerful, and should be less effort to maintain.

It also splits out the JSON checking code into separate files to cut down on the extreme length of the originals.

Fixes #27283

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Try the docs examples

elasticmachine · 2021-09-23T00:04:13Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-09-27T13:35:21.838+0000
Duration: 223 min 22 sec
Commit: 0e6a11b

Test stats 🧪

Test	Results
Failed	0
Passed	54135
Skipped	5328
Total	59463

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.

heartbeat/monitors/active/icmp/stdloop.go

andrewvc · 2021-09-23T02:12:16Z

@lowryxiao I noticed you spotted a bugfix earlier, if you have any feedback here it's appreciated as well!

andrewvc · 2021-09-23T02:12:49Z

heartbeat/docs/monitors/monitor-http.asciidoc

@@ -258,12 +258,6 @@ This monitor examines match successfully only when 'foo' or 'Foo' in body AND no
  name: Demo Service
  schedule: '@every 5s'
  urls: ["http://localhost:8080/demo/add"]
-  check.request:


This was redundant for this section, we're focusing on JSON here

andrewvc · 2021-09-23T02:15:30Z

heartbeat/monitors/active/http/check.go

@@ -219,53 +211,3 @@ func checkBody(positiveMatch, negativeMatch []match.Matcher) bodyValidator {
 		return nil
 	}
 }
-
-func checkJSON(checks []*jsonResponseCheck) (bodyValidator, error) {


moved to checkjson.go

andrewvc · 2021-09-23T02:15:43Z

heartbeat/monitors/active/http/check_test.go

@@ -196,142 +194,6 @@ func TestCheckBody(t *testing.T) {
 	}
 }

-func TestCheckJson(t *testing.T) {


moved to checkjson_test.go

andrewvc · 2021-09-23T02:18:31Z

heartbeat/monitors/active/http/checkjson.go

+				source:      check.Expression,
+			})
+		} else if check.Condition != nil {
+			cfgwarn.Deprecate("8.0.0", "JSON conditions are deprecated, use 'expression' instead.")


I can do a follow-up PR to actually delete this in master if we agree we should just move to expression

I don't have a strong opinion here, Do you see people using conditions for simpler use case?

elasticmachine · 2021-09-23T02:26:59Z

Pinging @elastic/uptime (Team:Uptime)

botelastic · 2021-09-23T02:27:05Z

This pull request doesn't have a Team:<team> label.

vigneshshanmugam

Overall looks good to me, Added small comments.

heartbeat/monitors/active/http/checkjson.go

vigneshshanmugam · 2021-09-23T22:36:56Z

heartbeat/monitors/active/http/checkjson.go

+				source:      check.Expression,
+			})
+		} else if check.Condition != nil {
+			cfgwarn.Deprecate("8.0.0", "JSON conditions are deprecated, use 'expression' instead.")


I don't have a strong opinion here, Do you see people using conditions for simpler use case?

heartbeat/monitors/active/http/checkjson.go

vigneshshanmugam · 2021-09-23T22:53:51Z

heartbeat/monitors/active/http/http_test.go

 		condition           common.MapStr
 		expectedErrMsg      string
 		expectedContentType string
 	}

 	testCases := []testCase{
 		{
-			"simple match",
+			"expression simple match",


do we want to add one more case when both condition and expression exists?

andrewvc · 2021-09-27T13:35:58Z

@vigneshshanmugam I've incorporated your PR feedback into 0e6a11b , I think it's a bit cleaner, esp. with the new config validator

vigneshshanmugam

LGTM 👍🏽

@name

) This PR greatly improves validation of JSON HTTP response bodies in heartbeat in two different ways: Introduces a new notion of JSON expressions, using gval as an expression engine. This allows validation of complex data, including the use of JSON Path expressions within. As an example you can write an expression like: $.nodes[?(@name == "foo"][2].message == "hello" to traverse a JSON object containing a nodes key with nested objects, find all elements named "foo", select the third one, then see if its' message attribute equals "hello". Supports handling APIs that return arrays of JSON objects without a top level object root. Previously, we only parsed JSON response that looked like an object, e.g. {"an": "object"}, we now support: [{"one": 1},{"two": 2}]. Deprecates the condition configuration in favor of this new expression syntax. The new syntax is more consistent and powerful, and should be less effort to maintain. It also splits out the JSON checking code into separate files to cut down on the extreme length of the originals. Fixes #27283 (cherry picked from commit 5ae6d63) # Conflicts: # go.mod

@name

…ion of JSON arrays (#28137) * [Heartbeat] Support JSON expressions / validation of JSON arrays (#28073) This PR greatly improves validation of JSON HTTP response bodies in heartbeat in two different ways: Introduces a new notion of JSON expressions, using gval as an expression engine. This allows validation of complex data, including the use of JSON Path expressions within. As an example you can write an expression like: $.nodes[?(@name == "foo"][2].message == "hello" to traverse a JSON object containing a nodes key with nested objects, find all elements named "foo", select the third one, then see if its' message attribute equals "hello". Supports handling APIs that return arrays of JSON objects without a top level object root. Previously, we only parsed JSON response that looked like an object, e.g. {"an": "object"}, we now support: [{"one": 1},{"two": 2}]. Deprecates the condition configuration in favor of this new expression syntax. The new syntax is more consistent and powerful, and should be less effort to maintain. It also splits out the JSON checking code into separate files to cut down on the extreme length of the originals. Fixes #27283 (cherry picked from commit 5ae6d63) # Conflicts: # go.mod * Fix merge conflict Co-authored-by: Andrew Cholakian <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

@name

…stic#28073) This PR greatly improves validation of JSON HTTP response bodies in heartbeat in two different ways: Introduces a new notion of JSON expressions, using gval as an expression engine. This allows validation of complex data, including the use of JSON Path expressions within. As an example you can write an expression like: $.nodes[?(@name == "foo"][2].message == "hello" to traverse a JSON object containing a nodes key with nested objects, find all elements named "foo", select the third one, then see if its' message attribute equals "hello". Supports handling APIs that return arrays of JSON objects without a top level object root. Previously, we only parsed JSON response that looked like an object, e.g. {"an": "object"}, we now support: [{"one": 1},{"two": 2}]. Deprecates the condition configuration in favor of this new expression syntax. The new syntax is more consistent and powerful, and should be less effort to maintain. It also splits out the JSON checking code into separate files to cut down on the extreme length of the originals. Fixes elastic#27283

andrewvc added 6 commits September 15, 2021 20:46

Checkpoint

efe1133

Array checkpoint

4c46978

Checkpoint

9fbe079

Checkpoint

430ee70

Checkpoint

aa43aba

Enhance testing

82313d4

andrewvc added enhancement v7.16.0 backport-v7.16.0 Automated backport with mergify labels Sep 22, 2021

andrewvc self-assigned this Sep 22, 2021

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Sep 22, 2021

lowryxiao reviewed Sep 23, 2021

View reviewed changes

heartbeat/monitors/active/icmp/stdloop.go Outdated Show resolved Hide resolved

andrewvc added 4 commits September 22, 2021 21:02

Tidy things up

8d72a0c

revert heartbeat.yml

55260d2

FMT / Update reference

f1785c3

Update changelog

9ba6ca0

andrewvc requested review from vigneshshanmugam and justinkambic September 23, 2021 02:10

andrewvc mentioned this pull request Sep 23, 2021

[Heartbeat] Monitor - Add ability for check an array of objects from a JSON payload response #27283

Closed

andrewvc commented Sep 23, 2021

View reviewed changes

Cleanup

74e1131

andrewvc marked this pull request as ready for review September 23, 2021 02:23

andrewvc requested a review from a team as a code owner September 23, 2021 02:23

andrewvc added the Team:obs-ds-hosted-services Label for the Observability Hosted Services team label Sep 23, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Sep 23, 2021

andrewvc added Heartbeat needs_team Indicates that the issue/PR needs a Team:* label labels Sep 23, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Sep 23, 2021

andrewvc added 2 commits September 23, 2021 08:23

Update modules / license

8ee27ba

Update x-pack ref yml

2e0f70b

vigneshshanmugam reviewed Sep 23, 2021

View reviewed changes

Incorporate PR feedback

0e6a11b

vigneshshanmugam approved these changes Sep 27, 2021

View reviewed changes

andrewvc merged commit 5ae6d63 into elastic:master Sep 27, 2021

andrewvc deleted the array-syntax branch September 27, 2021 20:03

mergify bot mentioned this pull request Sep 27, 2021

[7.x](backport #28073) [Heartbeat] Support JSON expressions / validation of JSON arrays #28137

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Heartbeat] Support JSON expressions / validation of JSON arrays #28073

[Heartbeat] Support JSON expressions / validation of JSON arrays #28073

andrewvc commented Sep 22, 2021 •

edited

Loading

elasticmachine commented Sep 23, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

andrewvc commented Sep 23, 2021

andrewvc Sep 23, 2021

andrewvc Sep 23, 2021

andrewvc Sep 23, 2021

andrewvc Sep 23, 2021

vigneshshanmugam Sep 23, 2021

elasticmachine commented Sep 23, 2021

botelastic bot commented Sep 23, 2021

vigneshshanmugam left a comment

vigneshshanmugam Sep 23, 2021

vigneshshanmugam Sep 23, 2021

andrewvc commented Sep 27, 2021

vigneshshanmugam left a comment

[Heartbeat] Support JSON expressions / validation of JSON arrays #28073

[Heartbeat] Support JSON expressions / validation of JSON arrays #28073

Conversation

andrewvc commented Sep 22, 2021 • edited Loading

Checklist

How to test this PR locally

elasticmachine commented Sep 23, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

andrewvc commented Sep 23, 2021

andrewvc Sep 23, 2021

Choose a reason for hiding this comment

andrewvc Sep 23, 2021

Choose a reason for hiding this comment

andrewvc Sep 23, 2021

Choose a reason for hiding this comment

andrewvc Sep 23, 2021

Choose a reason for hiding this comment

vigneshshanmugam Sep 23, 2021

Choose a reason for hiding this comment

elasticmachine commented Sep 23, 2021

botelastic bot commented Sep 23, 2021

vigneshshanmugam left a comment

Choose a reason for hiding this comment

vigneshshanmugam Sep 23, 2021

Choose a reason for hiding this comment

vigneshshanmugam Sep 23, 2021

Choose a reason for hiding this comment

andrewvc commented Sep 27, 2021

vigneshshanmugam left a comment

Choose a reason for hiding this comment

andrewvc commented Sep 22, 2021 •

edited

Loading

elasticmachine commented Sep 23, 2021 •

edited by jenkins-beats-ci bot

Loading