Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a simple plugin system for writing data to external destinations #143

Closed
JCZuurmond opened this issue Apr 12, 2023 · 4 comments · Fixed by #173
Closed

Create a simple plugin system for writing data to external destinations #143

JCZuurmond opened this issue Apr 12, 2023 · 4 comments · Fixed by #173

Comments

@JCZuurmond
Copy link

The same as in #141, though, now for writing data to external destinations.

Motivation

The reasoning is similar to reading. Sometimes the data has to be presented in a different system. For example, I use jdbc tables in Spark to write data to a SQL server. The SQL server might be connected to a dashboard tool, which can be preferred over connecting the Spark cluster.

The general approach for the plugins is fantastic! Users could implement anything; maybe they want to send an email with a report summarizing the data quality of a data model.

@JCZuurmond JCZuurmond changed the title Create a simple plugin system for **writing** data to external destinations Create a simple plugin system for writing data to external destinations Apr 12, 2023
@armsepehr
Copy link

We need this feature and curious about that

@jwills
Copy link
Collaborator

jwills commented Apr 25, 2023

@armsepehr tell me more about what you need here? There are several different ways to approach this and I haven't hit upon one that I'm happy with yet, so this context would be really helpful for me in figuring it out.

(FWIW, I'm racing to get issue #147 done asap, so I think the write-side plugins will have to wait for a post-1.5.0 release)

@armsepehr
Copy link

We read information from the file and then do the transformation pipeline on data. Finally, we would like to store transformed data in another db such as Clickhouse, AlloyDB or other relational dbs. I've watched out #141 and think about the external destination to write data.

@jwills
Copy link
Collaborator

jwills commented Apr 26, 2023

okay, that makes sense and fits with how I was thinking about this stuff-- thanks! It's going to involve some major surgery tho, so I'm not going to get to it until I get the 1.5.0 stuff out the door; sorry about that!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants