Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

services/horizon: Research Spike - Using unpacked txmeta for reingestion #4474

Closed
Tracked by #4571
Shaptic opened this issue Jul 21, 2022 · 0 comments
Closed
Tracked by #4571

Comments

@Shaptic
Copy link
Contributor

Shaptic commented Jul 21, 2022

With all of the unpacked txmeta for pubnet through July 2022 on S3, we can leverage it to do reingestion without Captive Core. There are a few steps to this spike:

  • Keeping the buckets updated: There's a new ledger every 5 seconds. The exporter tool supports an "online" mode to continually upload meta as it sees new ledgers after it finishes exporting the initial range, but we've yet to use this capability. There are open questions here such as where should it run? How much will it cost? What's the latency? etc. Maybe this isn't necessary for an MVP and we can just launch a Captive Core for the latest bit of history.

  • Reingesting: Then, there's the actual reingestion. There's already a LedgerBackend that supports fetching from the buckets (see HistoryArchiveBackend, whose name is a misnomer). Can we hook up Horizon's reingestion routine to use it? What's the new performance like? What are the resource requirements? etc.

@Shaptic Shaptic mentioned this issue Jul 21, 2022
7 tasks
@Shaptic Shaptic changed the title exp/lighthorizon: Research Spike - Using unpacked txmeta for reingestion services/horizon: Research Spike - Using unpacked txmeta for reingestion Jul 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants