horizon: Merge master into horizon-protocol-19 branch #4299
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add a readme for the xdr package describing how to generate xdr. I frequently forget and have to figure it out each time. This might help someone else too. The examples use docker so that it is a quick copy/paste for most people. We use docker frequently so most of us have docker. There's a good chance that less of us have ruby setup in a good state locally. I also added the ref to the Rakefile because I also forget how to specify a different source location for downloading XDR and if the ref was always specified it would be easier to just edit it.
Cache TOMLs to during scraping to some degree. Caching is local to each routine that is scraping and not shared across those routines, so it is possible to still get duplicate requests. Cache usage is logged. The asset scraper may end up repeatedly scraping the same issuer's TOML many times if the issuer has issued many assets. This is not great since any scraper should attempt to respect the resources of hosts as much as possible. The cache is in-memory and per routine because the cache is only required temporarily, and to introduce an external cache, such as redis, would be overkill. Cache usage is logged so that it can be inspected and understood.
Use log fields in logs in the asset ingestion job of Ticker and include TOML url in logs. To make filtering and analyzing logs easier.
Change the maximum size of the toml cache used during asset ingestion in Ticker to a single toml, and sort the assets by their TOML URLs when loading. If we sort assets by their TOML URLs we only need to store one toml in the cache, the last one. This reduces the amount of memory required.
Signed-off-by: Satyam Zode <[email protected]> Co-authored-by: Satyam Zode <[email protected]>
It's the first time I get into this code and the naming has made it hard to understand. So, I decided to make the naming more understandable.
Shaptic
approved these changes
Mar 21, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.