Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

📝 add docs for how to add normalization #21563

Merged
merged 3 commits into from
Jan 19, 2023
Merged

📝 add docs for how to add normalization #21563

Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
cbbf172
add docs
pedroslopez Jan 18, 2023
d23ca3b
add schema link
pedroslopez Jan 18, 2023
dd980df
update based on feedback
pedroslopez Jan 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions docs/connector-development/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,32 @@ The steps for updating an existing connector are the same as for building a new
3. Add any needed docs updates
4. Create a PR to get the connector published

## Adding normalization to a connector

In order to enable normalization for a destination connector, you'll need to set some fields on the destination definitions entry for the connector. This is done in the `airbyte-config/init/src/main/resources/seed/destination_definitions.yaml` file.

### New connectors
If you're adding normalization to a new connector, you'll need to first add a destination definitions entry:
1. Add a new connector definition in `airbyte-config/init/src/main/resources/seed/destination_definitions.yaml`. You can copy an existing entry and modify it to match your connector, generating a new UUID for the `destinationDefinitionId`.
2. Run the command `./gradlew :airbyte-config:init:processResources` to generate the seed spec yaml files, and commit the changes to the PR. See [this readme](https://github.com/airbytehq/airbyte/tree/a534bb2a8f29b20e3cc7c52fef1bc3c34783695d/airbyte-config/specs) for more information.

### Add normalization fields

Once you have a destination definitions entry, you'll need to add a `normaliationConfig` field to enable normalization.

Here's an example of normalization fields being set to enable normalization for the Postgres destination:

```yaml
normalizationConfig:
normalizationRepository: airbyte/normalization
normalizationTag: 0.2.25
normalizationIntegrationType: postgres
```

For more information about what these fields mean, see the [NormalizationDestinationDefinitionConfig](https://github.com/airbytehq/airbyte/blob/master/airbyte-config/config-models/src/main/resources/types/NormalizationDestinationDefinitionConfig.yaml) schema.

The presence of these fields will enable normalization for the connector, and determine which docker image will run.

## Publishing a connector

Once you've finished iterating on the changes to a connector as specified in its `README.md`, follow these instructions to ship the new version of the connector with Airbyte out of the box.
Expand Down