You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
With all of the unpacked txmeta for pubnet through July 2022 on S3, we can leverage it to do reingestion without Captive Core. There are a few steps to this spike:
Keeping the buckets updated: There's a new ledger every 5 seconds. The exporter tool supports an "online" mode to continually upload meta as it sees new ledgers after it finishes exporting the initial range, but we've yet to use this capability. There are open questions here such as where should it run? How much will it cost? What's the latency? etc. Maybe this isn't necessary for an MVP and we can just launch a Captive Core for the latest bit of history.
Reingesting: Then, there's the actual reingestion. There's already a LedgerBackend that supports fetching from the buckets (see HistoryArchiveBackend, whose name is a misnomer). Can we hook up Horizon's reingestion routine to use it? What's the new performance like? What are the resource requirements? etc.
The text was updated successfully, but these errors were encountered:
Shaptic
changed the title
exp/lighthorizon: Research Spike - Using unpacked txmeta for reingestion
services/horizon: Research Spike - Using unpacked txmeta for reingestion
Jul 21, 2022
With all of the unpacked txmeta for pubnet through July 2022 on S3, we can leverage it to do reingestion without Captive Core. There are a few steps to this spike:
Keeping the buckets updated: There's a new ledger every 5 seconds. The exporter tool supports an "online" mode to continually upload meta as it sees new ledgers after it finishes exporting the initial range, but we've yet to use this capability. There are open questions here such as where should it run? How much will it cost? What's the latency? etc. Maybe this isn't necessary for an MVP and we can just launch a Captive Core for the latest bit of history.
Reingesting: Then, there's the actual reingestion. There's already a
LedgerBackend
that supports fetching from the buckets (seeHistoryArchiveBackend
, whose name is a misnomer). Can we hook up Horizon's reingestion routine to use it? What's the new performance like? What are the resource requirements? etc.The text was updated successfully, but these errors were encountered: