Skip to content

Commit

Permalink
runtime: add materialize-kafka to ser_policy
Browse files Browse the repository at this point in the history
Kafka has a default maximum message size of 1MiB.

This adds `materialize-kafka` to the list of connectors that use `ser_policy`,
which should generally ensure that documents with individual extremely large
string, array, or object fields don't exceed the message size limit for Kafka.

This simple heuristic may not always be 100% what users desire, but in general
it seems better to have the connector at least sort-of-work rather than crash in
a way that would require a more complicated fix (possibly a derivation) to best
support data exploration / PoC use cases.
  • Loading branch information
williamhbaker authored and jgraettinger committed Jan 21, 2025
1 parent d04668a commit 79e05ac
Showing 1 changed file with 1 addition and 0 deletions.
1 change: 1 addition & 0 deletions crates/runtime/src/materialize/task.rs
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ impl Task {
// See go/runtime/materialize.go:135
let ser_policy = if [
"ghcr.io/estuary/materialize-bigquery",
"ghcr.io/estuary/materialize-kafka",
"ghcr.io/estuary/materialize-snowflake",
"ghcr.io/estuary/materialize-redshift",
"ghcr.io/estuary/materialize-sqlite",
Expand Down

0 comments on commit 79e05ac

Please sign in to comment.