Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replicate data from external source to internal source with a Plugin #72

Open
Brayden opened this issue Jan 23, 2025 · 5 comments · May be fixed by #75
Open

Replicate data from external source to internal source with a Plugin #72

Brayden opened this issue Jan 23, 2025 · 5 comments · May be fixed by #75
Labels
💎 Bounty enhancement New feature or request

Comments

@Brayden
Copy link
Member

Brayden commented Jan 23, 2025

Is your feature request related to a problem? Please describe.
StarbaseDB instances support by default an internal database (SQLite offered by the Durable Object) as well as an optional external data source. External data sources can be powered in one of two ways, both by providing values in the wrangler.toml file of the project.

  • Outerbase API Key
  • Connection details of the database
Image

Describe the solution you'd like
What would be beneficial for some use cases is the ability to bring in an external data source (e.g. a Postgres on Supabase) and have a pull mechanism where data can be brought into the internal DO SQLite so that the instance serves as a close-to-edge replica that can be queried alternatively to querying the Supabase Postgres instance.

Describe alternatives you've considered

  • Considering the pull vs push mechanism. A pull mechanism seems to be a better global solution where a push mechanism would be required to live elsewhere on a per provider basis.

Additional context

  • Might be beneficial for users to be able to define in the plugin what intervals data should be pulled at
  • Might be beneficial to allow users to define which tables should have data pulled into it (perhaps not all tables need replicated)
  • Likely need a way to know for each table what the last queried items were so you can do append-only type polling for new data. Does a user need to define a column to base this on (e.g. id or created_at columns perhaps)?
@Brayden Brayden added the enhancement New feature or request label Jan 23, 2025
@Brayden
Copy link
Member Author

Brayden commented Jan 24, 2025

/bounty $250

Copy link

algora-pbc bot commented Jan 24, 2025

💎 $250 bounty • Outerbase (YC W23)

Steps to solve:

  1. Start working: Comment /attempt #72 with your implementation plan
  2. Submit work: Create a pull request including /claim #72 in the PR body to claim the bounty
  3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Thank you for contributing to outerbase/starbasedb!

Add a bountyShare on socials

Attempt Started (GMT+0) Solution
🟢 @onyedikachi-david Jan 24, 2025, 10:18:25 PM #75

@onyedikachi-david
Copy link

onyedikachi-david commented Jan 24, 2025

/attempt #72

Algora profile Completed bounties Tech Active attempts Options
@onyedikachi-david 14 bounties from 7 projects
TypeScript, Python,
JavaScript & more
Cancel attempt

Copy link

algora-pbc bot commented Jan 24, 2025

💡 @onyedikachi-david submitted a pull request that claims the bounty. You can visit your bounty board to reward.

@itsxdamdam
Copy link

@Brayden is this still open?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
💎 Bounty enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants