v4.45.0-rc1

For installation instructions check out the getting started guide.

NOTE: This is a release candidate, you can download a binary from this page.

Added

• aws_sqs now has a max_outstanding field to prevent unbounded memory usage. (@rockwotj)
• avro scanner now emits metadata for the Avro schema it used along with the schema fingerprint. (@rockwotj)
• Field content_type added to the amqp_1 output. (@timo102)
• kafka_franz, ockam_kafka, redpanda, redpanda_common, redpanda_migrator now support fetch_max_wait configuration field.
• snowpipe_streaming now supports interpolating table names (@rockwotj)
• snowpipe_streaming now supports interpolating channel names (@rockwotj)
• snowpipe_streaming now supports exactly once delivery using offset_token (@rockwotj)

Fixed

• The code and file fields on the javascript processor docs no longer erroneously mention interpolation support. (@mihaitodor)
• The postgres_cdc now correctly handles null values. (@rockwotj)
• Fix an issue in aws_sqs with refreshing in-flight message leases which could prevent acks from processed. (@rockwotj)

The full change log can be found here.

v4.44.0

For installation instructions check out the getting started guide.

Added

• Go API: New public/license package added to allow custom programmatic instantiations of Redpanda Connect to run enterprise license components. (@Jeffail)

Fixed

• gcp_bigquery output with parquet format no longer returns errors incorrectly. (@rockwotj)
• postgres_cdc input now allows quoted identifiers for the table names. (@mihaitodor, @rockwotj)

The full change log can be found here.

v4.43.1

For installation instructions check out the getting started guide.

Fixed

• Trial Redpanda Enterprise licenses are now considered valid. (@Jeffail)
• The redpanda_migrator_bundle output now skips schema ID translation when translate_schema_ids: false and schema_registry is configured. (@mihaitodor)

The full change log can be found here.

v4.43.0

For installation instructions check out the getting started guide.

Changed

• The pg_stream input has been renamed to postgres_cdc. The old name will continue to function as an alias. (@rockwotj)
• The postgres_cdc input no longer emits mode metadata and instead snapshot reads set operation metadata to be read instead of insert. (@rockwotj)

Fixed

• The redpanda_migrator_bundle output no longer attempts to translate schema IDs when a schema registry is not configured. (@mihaitodor)

The full change log can be found here.

v4.42.0

For installation instructions check out the getting started guide.

Added

• Add support for spanner driver to SQL plugins. (@yufeng-deng)
• Add support for complex database types (JSONB, TEXT[], INET, TSVECTOR, TSRANGE, POINT, INTEGER[]) for pg_stream input. (@le-vlad)
• Add support for Parquet files to bigquery output (@rockwotj)
• (Benthos) New exists operator added to the cache processor. (@mihaitodor)
• New CLI flag redpanda-license added as an alternative way to specify a Redpanda license. (@Jeffail)

Fixed

• Fixed pg_stream issue with discrepancies between replication and snapshot streaming for UUID type. (@le-vlad)
• Fixed avro scanner bug introduced in v4.25.0. (@mihaitodor)

Changed

• The redpanda_migrator output now registers destination schemas with all the subjects associated with the source schema ID extracted from each message. (@mihaitodor)
• Enterprise features will now only run when a valid Redpanda license is present. More information can be found at the licenses getting started guide. (@Jeffail)

The full change log can be found here.

v4.42.0-rc1

For installation instructions check out the getting started guide.

Added

• Add support for spanner driver to SQL plugins. (@yufeng-deng)
• Add support for complex database types (JSONB, TEXT[], INET, TSVECTOR, TSRANGE, POINT, INTEGER[]) for pg_stream input. (@le-vlad)
• Add support for Parquet files to bigquery output (@rockwotj)
• (Benthos) New exists operator added to the cache processor. (@mihaitodor)
• New CLI flag redpanda-license added as an alternative way to specify a Redpanda license. (@Jeffail)

Fixed

• Fixed pg_stream issue with discrepancies between replication and snapshot streaming for UUID type. (@le-vlad)
• Fixed avro scanner bug introduced in v4.25.0. (@mihaitodor)

Changed

• The redpanda_migrator output now registers destination schemas with all the subjects associated with the source schema ID extracted from each message. (@mihaitodor)
• Enterprise features will now only run when a valid Redpanda license is present. More information can be found at the licenses getting started guide. (@Jeffail)

The full change log can be found here.

v4.41.0

For installation instructions check out the getting started guide.

Added

• Field max_records_per_request added to the aws_sqs output. (@Jeffail)

Fixed

• (Benthos) Fixed an issue where running a CLI with a custom environment would cause imported templates to be rejected. (@Jeffail)

Changed

• The -cgo suffixed docker images are no longer built and pushed along with the regular images. This decision was made due to low demand, and the unacceptable cadence with which the image base (Debian) receives security updates. It is still possible to create your own CGO builds with the command CGO_ENABLED=1 make TAGS=x_benthos_extra redpanda-connect. (@Jeffail)

The full change log can be found here.

v4.40.0

For installation instructions check out the getting started guide.

Added

• New pg_stream input supporting change data capture (CDC) from PostgreSQL. (@le-vlad)
• Field metadata_max_age added to the redpanda_migrator_offsets output. (@mihaitodor)
• Field kafka_timestamp_ms added to the kafka, kafka_franz, redpanda, redpanda_common and redpanda_migrator outputs. (@mihaitodor)
• (Benthos) New Bloblang method timestamp. (@mihaitodor)
• (Benthos) New benchmark processor. (@ooesili)

Fixed

• Addresses an issue where snowflake_streaming could create more channels than configured. (@rockwotj)

Changed

• The snowflake_streaming output with schema_evolution.enabled set to true can now autocreate tables. (@rockwotj)
• Fields translate_schema_ids and schema_registry_output_resource added to the redpanda_migrator output. (@mihaitodor)
• Fields backfill_dependencies and input_resource added to the schema_registry output. (@mihaitodor)
• The schema_registry input and output and the schema_registry_encode and schema_registry_decode processors now use the github.com/twmb/franz-go/pkg/sr SchemaRegistry client. (@mihaitodor)
• Metadata field kafka_timestamp_ms added to the kafka, kafka_franz, redpanda, redpanda_common and redpanda_migrator inputs now contains a unix timestamp with millisecond precision. (@mihaitodor)
• Metadata field kafka_timestamp removed from the kafka, kafka_franz, redpanda, redpanda_common and redpanda_migrator inputs. (@mihaitodor)

The full change log can be found here.

v4.39.0

For installation instructions check out the getting started guide.

Added

• New timeplus input. (@ye11ow)
• New snowflake_streaming output. (@rockwotj)
• Redpanda Connect will now use an optional /etc/redpanda/connector_list.yaml config to determine which connectors are available to run. (@Jeffail)
• (Benthos) Field follow_redirects added to the http processor. (@ooesili)
• New CLI flag --secrets added. (@Jeffail)
• New CLI flag --disable-telemetry added. (@Jeffail)
• New experimental spicedb watch input. (@simon0191)
• New redpanda_common input and output. (@Jeffail)
• New redpanda input and output. (@Jeffail)
• New snowflake_streaming output. (@rockwotj)

Fixed

• The kafka, kafka_franz and redpanda_migrator outputs no longer waste CPU for large batches. (@rockwotj)

Changed

• The aws_sqs output field url now supports interpolation functions. (@rockwotj)
• (Benthos) CLI --set flags can now mutate array values indexed from the end via negative integers. E.g. --set 'foo.-1=meow' would set the last index of the array foo to the value of meow. (@Jeffail)

The full change log can be found here.

v4.39.0-rc1

For installation instructions check out the getting started guide.

NOTE: This is a release candidate, you can download a binary from this page.

Added

• New timeplus input. (@ye11ow)
• New snowflake_streaming output. (@rockwotj)
• Redpanda Connect will now use an optional /etc/redpanda/connector_list.yaml config to determine which connectors are available to run. (@Jeffail)
• (Benthos) Field follow_redirects added to the http processor. (@ooesili)
• New CLI flag --secrets added. (@Jeffail)
• New CLI flag --disable-telemetry added. (@Jeffail)

Changed

• The aws_sqs output field url now supports interpolation functions. (@rockwotj)
• (Benthos) CLI --set flags can now mutate array values indexed from the end via negative integers. E.g. --set 'foo.-1=meow' would set the last index of the array foo to the value of meow. (@Jeffail)

The full change log can be found here.