Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

implement sql alchemy destination #21

Closed
rudolfix opened this issue Jun 15, 2022 · 0 comments · Fixed by #1734
Closed

implement sql alchemy destination #21

rudolfix opened this issue Jun 15, 2022 · 0 comments · Fixed by #1734
Assignees
Labels
destination Issue related to new destinations enhancement New feature or request

Comments

@rudolfix
Copy link
Collaborator

rudolfix commented Jun 15, 2022

Background
We can handle many rdbmses with a single SQLAlchemy destination. In this ticket we want to implement a generic loader and test it with two engines: SQLLite and MySQL

Requirements

  1. implement bulk inserts if possible
  1. sql alchemy has identifier length in each dialect
  2. now destination capabilities may be dynamically modified. Use the connection string (that contains engine+dialect) to synthesize it.
  3. if needed add sql_alchemy_caps to destination caps with additional settings (ie. if bulk inserts are supported or JSON fields)
  • allow user to provide own sql_alchemy_caps
  1. Use SQLALchemy to create and reflect databases. Implement sql_client interface (but probably form scratch).
  2. write dispositions:
  • implement append and replace write disposition
  • merge in next ticket (I’m not even sure it is possible. But we are using only INSERT and DELETE so maybe?)

Tests:

  • most of the tests that abstract sql_client and job_client should pass right away (ie. pipeline tests, state sync, drop command)
  • sql_client tests should also pass
@rudolfix rudolfix self-assigned this Jun 15, 2022
@rudolfix rudolfix moved this from Todo to Planned in dlt core library Jan 28, 2024
@rudolfix rudolfix removed their assignment Jan 28, 2024
@rudolfix rudolfix moved this from Planned to Todo in dlt core library Apr 22, 2024
@rudolfix rudolfix moved this from Todo to Planned in dlt core library Aug 13, 2024
@rudolfix rudolfix added the enhancement New feature or request label Aug 13, 2024
@rudolfix rudolfix added the destination Issue related to new destinations label Aug 13, 2024
@rudolfix rudolfix changed the title [loader] implement alchemy based generic loader implement sql alchemy destination Aug 13, 2024
@rudolfix rudolfix moved this from Planned to In Progress in dlt core library Aug 14, 2024
@steinitzu steinitzu mentioned this issue Aug 23, 2024
7 tasks
@github-project-automation github-project-automation bot moved this from In Progress to Done in dlt core library Sep 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
destination Issue related to new destinations enhancement New feature or request
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

2 participants