proofpoint_tap: improve clarity of agent config and fix pagination logic #11361
Conversation
efd6
added
enhancement
efd6
force-pushed
the
s5180-proofpoint_tap
branch
from
9816952 to
16c8244
Compare
There was no check for the pagination termination state, which for this API is an empty array of events. Add that check.
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
efd6
changed the title
| default: |- | ||
| [[- $start := (now (parseDuration "-{{initial_interval}}")) -]] | ||
| [[- $hour := (parseDuration "1h") -]] | ||
| [[- $end := ($start .Add $hour) -]] |
There was a problem hiding this comment.
This needs to use a prefix style call like .Add $start $hour.
If this type of problem always logs an message, then we should add it to https://github.com/elastic/elastic-package/blob/07fb3a2eab67291c015a63c8d50ffe42d8391550/internal/testrunner/runners/system/tester.go#L104-L124.
There was a problem hiding this comment.
In the logs we have
{
"@timestamp": "2024-10-08T14:30:31.642Z",
"component": {
"binary": "filebeat",
"dataset": "elastic_agent.filebeat",
"id": "httpjson-default",
"type": "httpjson"
},
"ecs.version": "1.6.0",
"error": {
"message": "template: :3:13: executing \"\" at <$start>: can't give argument to non-function $start"
},
"id": "httpjson-proofpoint_tap.clicks_blocked-d1a045a5-2476-4a83-90c7-26b5e0cb5a95",
"input_source": "http://svc-proofpoint_tap:8080/v2/siem/clicks/blocked",
"input_url": "http://svc-proofpoint_tap:8080/v2/siem/clicks/blocked",
"log": {
"source": "httpjson-default"
},
"log.level": "debug",
"log.logger": "input.httpjson-cursor",
"log.origin": {
"file.line": 111,
"file.name": "httpjson/value_tpl.go",
"function": "github.com/elastic/beats/v7/x-pack/filebeat/input/httpjson.(*valueTpl).Execute.func2"
},
"message": "template execution failed",
"service.name": "filebeat",
"target": "interval"
}
There was a problem hiding this comment.
This is surprising. The issue is not the call order, this is just not possible with this mechanism. I'll explore further. The key docs are
The name of a niladic method of the data, preceded by a period, such as .Method The result is the value of invoking the method with dot as the receiver, dot.Method(). Such a method must have one return value (of any type) or two return values, the second of which is an error. If it has two and the returned error is non-nil, execution terminates and an error is returned to the caller as the value of Execute. Method invocations may be chained and combined with fields and keys to any depth: .Field1.Key1.Method1.Field2.Key2.Method2 Methods can also be evaluated on variables, including chaining: $x.Method1.Field
The key word here is "niladic".
There was a problem hiding this comment.
Well, this is harder than it really should be, https://go.dev/play/p/3wt50n_Fy2o.
|
This is not a new problem from this PR, but I noticed that tracer logs were not being enabled during testing because of this: input: httpjson
service: proofpoint_tap
vars:
+ enable_request_tracer: true
url: http://{{Hostname}}:{{Port}}
principal: xxxx
secret: xxxx
data_stream:
vars:
preserve_original_event: true
- enable_request_tracer: true
|
|
@andrewkroh I took a look at the system tests to see if there was a way to extend them at least over the pagination logic, but I don't see a way to do this with stream as it exists with any sane approach. I do have an approach though. |
🚀 Benchmarks reportTo see the full report comment with |
|
LGTM. I would like to do a manual test tomorrow so I can peek at the agent log and tracer log. |
💚 Build Succeeded
History
cc @efd6 |
|
|
Package proofpoint_tap - 1.24.0 containing this change is available at https://epr.elastic.co/search?package=proofpoint_tap |
Proposed commit message
Checklist
changelog.ymlfile.Author's Checklist
How to test this PR locally
Related issues
Screenshots