Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dekaf: Fix endpoint config parsing #1841

Merged
merged 1 commit into from
Dec 18, 2024
Merged

dekaf: Fix endpoint config parsing #1841

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 18 additions & 3 deletions crates/dekaf/src/connector.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
use anyhow::{bail, Context};
use proto_flow::materialize;
use proto_flow::{flow::materialization_spec, materialize};
use schemars::JsonSchema;
use serde::{Deserialize, Serialize};
use std::collections::BTreeMap;
Expand Down Expand Up @@ -38,6 +38,7 @@ pub struct DekafConfig {
/// tombstones with null values, and "Header" emits then as a kafka document
/// with empty string and `_is_deleted` header set to `1`. Setting this value
/// will also cause all other non-deletions to have an `_is_deleted` header of `0`.
#[serde(default)]
pub deletions: DeletionMode,
}

Expand Down Expand Up @@ -70,8 +71,22 @@ pub async fn unary_materialize(
) -> anyhow::Result<materialize::Response> {
use proto_flow::materialize::response::validated;
if let Some(mut validate) = request.validate {
serde_json::de::from_str::<DekafConfig>(&validate.config_json)
.context("validating endpoint config")?;
match materialization_spec::ConnectorType::try_from(validate.connector_type)? {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just as an fyi, you could also use

let ConnectorType::Dekaf = ConnectorType::try_from(validate.connector_type)? else {
  bail!("...");
};

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But then I wouldn't have the parsed proto_flow::flow::materialization_spec::ConnectorType, which can give me a human-readableas_str_name() instead of just proto_flow::materialize::request::Validate:: connector_type: i32, which is much less useful in an error

materialization_spec::ConnectorType::Dekaf => {}
other => bail!("invalid connector type: {}", other.as_str_name()),
};

let parsed_outer_config =
serde_json::from_str::<models::DekafConfig>(&validate.config_json)
.context("validating dekaf config")?;

let _parsed_inner_config = serde_json::from_value::<DekafConfig>(
parsed_outer_config.config.to_value(),
)
.context(format!(
"validating dekaf endpoint config for variant {}",
parsed_outer_config.variant
))?;

// Largely copied from crates/validation/src/noop.rs
let validated_bindings = std::mem::take(&mut validate.bindings)
Expand Down
Loading