Add initial cdc options for sql databases

[kartik4949]

Description

#1614

Related Issues

Checklist

• Is this code covered by new or existing unit tests or integration tests?
• Did you run make unit-testing and make integration-testing successfully?
• Do new classes, functions, methods and parameters all have docstrings?
• Were existing docstrings updated, if necessary?
• Was external documentation updated, if necessary?

Additional Notes or Comments

[blythed]

How does this get injected into the class? From CFG?

[kartik4949]

uhm .. good question
no its not injected in CFG

[blythed]

So, is the strategy to use the current CDC code, but simply with a new producer?

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (34830a7) 80.33% compared to head (87828d8) 67.42%.
Report is 1374 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #1643       +/-   ##
===========================================
- Coverage   80.33%   67.42%   -12.92%     
===========================================
  Files          95      118       +23     
  Lines        6602     8371     +1769     
===========================================
+ Hits         5304     5644      +340     
- Misses       1298     2727     +1429     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[kartik4949]

So, is the strategy to use the current CDC code, but simply with a new producer?

@blythed exactly!

[jieguangzhou]

Great work.

I think we should add a simple integration test to test the fault tolerance of CDC tasks, such as the behavior after obtaining the latest batch of data and predicting errors in certain two data.

It doesn’t have to be added now, it can be used as a TODO item.

[blythed]

Looking really good!

Some questions:

How do we activate this class via configuration?

Can we document the developer journey to setting up this class to work
with some SQL database?

[kartik4949]

Looking really good!

Some questions:

How do we activate this class via configuration?

Can we document the developer journey to setting up this class to work with some SQL database?

@blythed
Sure, I can add a documentation as separate pr

so following is basic usage:

from superduperdb import superduper

db = superduper('SQL/URI')

db.cdc.start()

#or

table = Table('my_table')
strategy = PollingStrategy(type='incremental', frequency=0.5, auto_incremental_field='id_field')

'''
# Here type could be either 
`incremental` meaning user has a incremental field in table

or 
`join_id` meaning user does not have an incremental field in table and we create separate metadata table where we store processed ids and `ANTI left join` on user table.
'''

db.cdc.listen(on=table, strategy=strategy)

[blythed]

Great work.