[receiver/awss3receiver]: Add ingest progress notifications via OpAMP #33047
Conversation
Use OpAMP custom messages to report progress of start/end time based ingest from an S3 bucket.
| component: awss3receiver | ||
|
|
||
| # A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`). | ||
| note: 'Add support for monitoring the progress of ingesting data from an S3 bucket via OpAMP custom messages.' |
There was a problem hiding this comment.
wow. That's the first time I see this done. Can you use obsreport to report metrics first?
| | `stop_time` | The time to stop retrieving data in RFC3339 format. | | ||
| | `ingest_time` | The time of the data currently being ingested in RFC3339 format. | | ||
| | `failure_message` | Error message if `ingest_status` is "failed". | |
There was a problem hiding this comment.
Is there a spec for this as part of opAmp or maybe entities?
There was a problem hiding this comment.
There is no spec for this and that is the reason why I'm using a Custom Capability.
| return nil | ||
| } | ||
|
|
||
| func (n *opampNotifier) Shutdown(_ context.Context) error { |
There was a problem hiding this comment.
Why are you exporting Shutdown, Start and SendStatus? Are all 3 part of an interface?
There was a problem hiding this comment.
They are part of the statusNotifier interface defined in the same file. Possibly overkill at this stage, I was allowing for other methods of notification (ie Webhook, file ...)
|
I believe it is worth a good SIG convo: |
|
I want to echo @atoulme's questions. This can be done via OpAMP, but I wonder if it should be done via OpAMP. Can the receiver generate metrics or traces that describe what happens instead? Is there a reason it has to go via OpAMP? I think this is something for Collector maintainers to decide: how should components report progress details of what they do. One thing to watch out for is that OpAMP has no built-in concept of batching. If the rate of custom messages is very high it may turn out to be inefficient (probably more so for plain http transport than for WS transport). |
|
@atoulme / @tigrannajaryan - Fair questions - though I'm not looking for metrics as the intention is to use the collector as one component of a larger app to retrieve data from S3 back into an observability platform. As a result I'm looking at being able to report progress to a user so they know when the requested trace has been ingested. I'll look at adding obsreport metrics as suggested while this is up for discussion. |
|
@adcharre it would be great to learn more about your use case and perhaps OpAMP is the right solution for it. |
|
We're discussing this here as well: https://cloud-native.slack.com/archives/C02J58HR58R/p1715714974462069 |
| the time of the data being ingested when the failure occurred. | ||
| If the ingest process completes successfully a status message with `ingest_status` set to "completed" is sent. |
There was a problem hiding this comment.
Docs writer here. Just a quick edit for clarity. Thanks!
| If during the processing of the data an error occurs a status message with `ingest_status` set to "failed" status with | |
| the time of the data being ingested when the failure occurred. | |
| If the ingest process completes successfully a status message with `ingest_status` set to "completed" is sent. | |
| If an error occurs during the processing of the data, a status message with `ingest_status` set to "failed" is sent, along with the time the ingestion failure occurred. | |
| If the ingest process completes successfully, a status message with `ingest_status` set to "completed" is sent. |
|
This PR was marked stale due to lack of activity. It will be closed in 14 days. |
|
Closed as inactive. Feel free to reopen if this PR is still being worked on. |
Description: Add support for monitoring the progress of ingesting data from an S3 bucket via OpAMP custom messages.
This change would allow an appropriate UI to display the state of ingested.
Link to tracking Issue: 30750
Testing: Additional unit tests added. Full stack testing using a built collector and the example OpAMP server provided in the PR.
Documentation: README.md updated to include details of the configuration changes and details of the OpAMP Custom Capability and message format.