Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add post_process to SQLStream.get_records to match RESTStream functionality #1674

Closed
BuzzCutNorman opened this issue May 5, 2023 · 0 comments · Fixed by #1681
Closed
Labels
kind/Feature New feature or request valuestream/SDK

Comments

@BuzzCutNorman
Copy link
Contributor

Feature scope

Taps (catalog, state, stream maps, etc.)

Description

I would like to propose that the code from RESTStream.get_records which invokes the method post_process be added to SQLStream.get_records. This enables SQL Tap developers to work with records before they are sent off to a Target. One example use would be the encoding of records values.

Current RESTStream.get_records:

        for record in self.request_records(context):
            transformed_record = self.post_process(record, context)
            if transformed_record is None:
                # Record filtered out during post_process()
                continue
            yield transformed_record

Current SQLStream.get_records:

        with self.connector._connect() as conn:
            for record in conn.execute(query):
                yield dict(record._mapping)

Proposed SQLStream.get_records:

        with self.connector._connect() as conn:
            for record in conn.execute(query):
                transformed_record = self.post_process(dict(record._mapping))
                if transformed_record is None:
                    # Record filtered out during post_process()
                    continue
                yield transformed_record
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/Feature New feature or request valuestream/SDK
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant