Snowbridge v3 #1068
Open
Snowbridge v3 #1068
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for snowplow-docs ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
|
Note that we may wish to hold off on merging until such a time as we're ready to announce |
pondzix
reviewed
Nov 8, 2024
docs/destinations/forwarding-events/snowbridge/concepts/failure-model/index.md
Outdated
Show resolved
Hide resolved
|
|
||
| A transient failure is a failure which we expect to succeed again on retry. For example some temporary network error, or when we encounter throttling. Typically you would configure a short backoff for this type of failure. When we encounter a transient failure, we keep processing the rest of the data as normal, under the expectation that everyhting is operating as normal. The failed data is retried after a backoff. | ||
|
|
||
| A setup failure is one which we don't expect to be immediately resolved, for example an incorrect address, or an invalid API Key. Typically you would configue a long backoff for this type of failure, under the assumption that the issue needs to be fixed with either a configuration change or a change to the target itself (eg. permissions need to be granted). When we encounter a setup error, we stop attempting to process any data, and the whole app waits for the backoff period before trying again. Setup errors will be retried 5 times, before the app crashes. |
There was a problem hiding this comment.
we stop attempting to process any data
We don't do anything explicit to stop processing in this case. Right now setup is pretty much like transient, but with much longer backoff. In the future we might add monitoring/alerts/health toggle for setup errors, but it's not there now.
In practice, if you mark your HTTP response as setup error in config, it probably means nothing gets through and we indeed 'stop' processing anything. But there is no code in Snowbridge that would say stop pulling from source now, we hit setup error!.
Theoretically it's possible to have both: setup and transient simultaneously. Then it means your setup error probably shouldn't be configured as setup error.
|
|
||
| A setup failure is one which we don't expect to be immediately resolved, for example an incorrect address, or an invalid API Key. Typically you would configue a long backoff for this type of failure, under the assumption that the issue needs to be fixed with either a configuration change or a change to the target itself (eg. permissions need to be granted). When we encounter a setup error, we stop attempting to process any data, and the whole app waits for the backoff period before trying again. Setup errors will be retried 5 times, before the app crashes. | ||
|
|
||
| As of v3.0.0, only the http target can be configured to return setup errors, via the response rules feature - configuration details for response rules can be found in [the http target configuration section](/docs/destinations/forwarding-events/snowbridge/configuration/targets/http/index.md). For all other targets, all errors returned will be considered transient, and behaviour can be configured using the `tranisent` block of the retry configuration. |
There was a problem hiding this comment.
Suggested change
| As of v3.0.0, only the http target can be configured to return setup errors, via the response rules feature - configuration details for response rules can be found in [the http target configuration section](/docs/destinations/forwarding-events/snowbridge/configuration/targets/http/index.md). For all other targets, all errors returned will be considered transient, and behaviour can be configured using the `tranisent` block of the retry configuration. | |
| As of v3.0.0, only the http target can be configured to return setup errors, via the response rules feature - configuration details for response rules can be found in [the http target configuration section](/docs/destinations/forwarding-events/snowbridge/configuration/targets/http/index.md). For all other targets, all errors returned will be considered transient, and behaviour can be configured using the `transient` block of the retry configuration. |
docs/destinations/forwarding-events/snowbridge/configuration/targets/http/index.md
Outdated
Show resolved
Hide resolved
|
|
||
| `jq` runs a jq command on the message data, and outputs the result of the command. While jq supports multi-element results, commands must output only a single element - this single element can be an array data type. | ||
|
|
||
| The provided command must return a boolean result. `false` filters the message out, `true` keeps it. |
There was a problem hiding this comment.
I guess this line shouldn't be here as it's only for filter, right?
There was a problem hiding this comment.
Correct. Done. Ty.
|
@pkutaj any reason this was closed, or was it done in error? |
|
That was surely my mistake, thanks Piotr for re-opening! |
…e-model/index.md Co-authored-by: Piotr Poniedziałek <[email protected]>
|
Rebased on main |
stanch
requested changes
Jan 20, 2025
docs/destinations/forwarding-events/snowbridge/concepts/failure-model/index.md
Outdated
Show resolved
Hide resolved
| @@ -34,6 +38,8 @@ Writes of oversized messages to the failure target will be recorded with 'Oversi | |||
|
|
|||
| In the unlikely event that Snowbridge encounters data which is invalid for the target destination (for example empty data is invalid for pubsub), it will create a [generic error failed event](/docs/fundamentals/failed-events/index.md#generic-error), emit it to the failure target, and ack the original message. | |||
|
|
|||
| As of version 3.0.0, the http target may produce invalid failures. This occurs when the a POST request body cannot be formed, when the templating feature's attempts to template data result in an error, or when the response conforms to a response rules configuration which sepcifies that the failure is to be treated as invalid. You can find more details in the [configuration section](/docs/destinations/forwarding-events/snowbridge/configuration/targets/http/index.md). | |||
There was a problem hiding this comment.
“invalid failures” sounds like the failures themselves are invalid (which I assume is not the case?)
Maybe may produce “invalid event” failures?
docs/destinations/forwarding-events/snowbridge/configuration/monitoring/index.md
Outdated
Show resolved
Hide resolved
docs/destinations/forwarding-events/snowbridge/configuration/monitoring/index.md
Outdated
Show resolved
Hide resolved
docs/destinations/forwarding-events/snowbridge/configuration/retries/index.md
Outdated
Show resolved
Hide resolved
docs/destinations/forwarding-events/snowbridge/configuration/targets/http/index.md
Outdated
Show resolved
Hide resolved
...orwarding-events/snowbridge/configuration/transformations/builtin/reusable/_jqDescription.md
Outdated
Show resolved
Hide resolved
...orwarding-events/snowbridge/configuration/transformations/builtin/reusable/_jqDescription.md
Outdated
Show resolved
Hide resolved
...ns/forwarding-events/snowbridge/configuration/transformations/builtin/reusable/_jqHelpers.md
Outdated
Show resolved
Hide resolved
...ns/forwarding-events/snowbridge/configuration/transformations/builtin/reusable/_jqHelpers.md
Outdated
Show resolved
Hide resolved
…e-model/index.md Co-authored-by: Nick <[email protected]>
…onitoring/index.md Co-authored-by: Nick <[email protected]>
…onitoring/index.md Co-authored-by: Nick <[email protected]>
…etries/index.md Co-authored-by: Nick <[email protected]>
…etries/index.md Co-authored-by: Nick <[email protected]>
…argets/http/index.md Co-authored-by: Nick <[email protected]>
…ransformations/builtin/reusable/_jqDescription.md Co-authored-by: Nick <[email protected]>
…ransformations/builtin/reusable/_jqHelpers.md Co-authored-by: Nick <[email protected]>
…ransformations/builtin/reusable/_jqHelpers.md Co-authored-by: Nick <[email protected]>
…etries/index.md Co-authored-by: Nick <[email protected]>
…etries/index.md Co-authored-by: Nick <[email protected]>
…etries/index.md Co-authored-by: Nick <[email protected]>
…argets/http/index.md Co-authored-by: Nick <[email protected]>
…argets/http/index.md Co-authored-by: Nick <[email protected]>
…argets/http/index.md Co-authored-by: Nick <[email protected]>
…argets/http/index.md Co-authored-by: Piotr Poniedziałek <[email protected]>
colmsnowplow
commented
Jan 20, 2025
| @@ -34,6 +38,8 @@ Writes of oversized messages to the failure target will be recorded with 'Oversi | |||
|
|
|||
| In the unlikely event that Snowbridge encounters data which is invalid for the target destination (for example empty data is invalid for pubsub), it will create a [generic error failed event](/docs/fundamentals/failed-events/index.md#generic-error), emit it to the failure target, and ack the original message. | |||
|
|
|||
| As of version 3.0.0, the http target may produce invalid failures. This occurs when the a POST request body cannot be formed, when the templating feature's attempts to template data result in an error, or when the response conforms to a response rules configuration which sepcifies that the failure is to be treated as invalid. You can find more details in the [configuration section](/docs/destinations/forwarding-events/snowbridge/configuration/targets/http/index.md). | |||
There was a problem hiding this comment.
Suggested change
| As of version 3.0.0, the http target may produce invalid failures. This occurs when the a POST request body cannot be formed, when the templating feature's attempts to template data result in an error, or when the response conforms to a response rules configuration which sepcifies that the failure is to be treated as invalid. You can find more details in the [configuration section](/docs/destinations/forwarding-events/snowbridge/configuration/targets/http/index.md). | |
| As of version 3.0.0, the http target may produce 'invalid' type failures. This occurs when the a POST request body cannot be formed, when the templating feature's attempts to template data result in an error, or when the response conforms to a response rules configuration which sepcifies that the failure is to be treated as invalid. You can find more details in the [configuration section](/docs/destinations/forwarding-events/snowbridge/configuration/targets/http/index.md). |
There was a problem hiding this comment.
@stanch this work?
…ransformations/builtin/reusable/_jqDescription.md Co-authored-by: Nick <[email protected]>
colmsnowplow
commented
Jan 20, 2025
...orwarding-events/snowbridge/configuration/transformations/builtin/reusable/_jqDescription.md
Outdated
Show resolved
Hide resolved
…ransformations/builtin/reusable/_jqDescription.md
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR to update docs for Snowbridge v3
Things to check:
@pondzix - I would appreciate your paying close attention to whether everything is accurate. Specifically I'm not 100% sure of the part that states that setup errors mean we'll stop processing all data for the retry period. Is that accurate?
@stanch - in addition to a normal PR review, I have marked some features as beta, wdyt about this? Happy to discuss to explain!