Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Regexrouter support update delete #234

Open
wants to merge 7 commits into
base: master
Choose a base branch
from

Conversation

FreCap
Copy link

@FreCap FreCap commented Sep 15, 2022

This MR implements a similar support to changing topics/schemas like in the JDBCSinkConnector + regexRouter
e.g. https://stackoverflow.com/a/55290401

Example to move all tables from a schema to another.
Slightly different code can be used to change names to tables.

   schema_mapping = { "original" : "new_bq_dataset" }

    for (from_schema, to_schema) in schema_mapping.items():
        transformer_name = f"schemaMapping_{from_schema}"
        config_json.update({
            "transforms": config_json['transforms'] + "," + transformer_name,
            #  extracting the table name
            f"transforms.{transformer_name}.type": "org.apache.kafka.connect.transforms.RegexRouter",
            f"transforms.{transformer_name}.regex": f"^[^.]+\\.{from_schema}\\.(.*)",
            #    "transforms.route.regex": "([^.]+)\\.([^.]+)\\.([^.]+)",

            # using the `:` as divisor as the dataset from the topic name
            # if the topic is in the form <dataset>:<tableName>.
            f"transforms.{transformer_name}.replacement": to_schema + ":$1",
        })

fcapponi added 7 commits July 18, 2022 19:00
Updating to version 2.6.0 required to use the original topic
Updating to version 2.7.0 required to use the original topic
Updating to version 2.7.0 required to use the original topic
Updating to version 2.7.0 required to use the original topic
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant