Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: find endpoints asynchronously, cache errors #419

Merged
merged 3 commits into from
Nov 9, 2023

Conversation

rvagg
Copy link
Contributor

@rvagg rvagg commented Nov 8, 2023

Goals:

  1. Make endpoint lookup parallel across all SPs
  2. Don't error unless we don't have any endpoints—but log an error
  3. Cache the error case too because it's not cheap and with a misbehaving SP it'll slow down every call for a particular piece

Critique of my design decisions welcome of course, there's a bunch of ways this could be done, I made some choices.

Copy link

codecov bot commented Nov 8, 2023

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Comparison is base (2d540e3) 74.02% compared to head (dd7d1a7) 74.08%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #419      +/-   ##
==========================================
+ Coverage   74.02%   74.08%   +0.06%     
==========================================
  Files         149      150       +1     
  Lines        9823     9880      +57     
==========================================
+ Hits         7271     7320      +49     
- Misses       1802     1807       +5     
- Partials      750      753       +3     
Files Coverage Δ
api/api.go 75.07% <100.00%> (+1.31%) ⬆️
retriever/endpointfinder/options.go 100.00% <100.00%> (ø)
retriever/endpointfinder/endpointfinder.go 93.67% <89.79%> (-0.78%) ⬇️

... and 2 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@rvagg rvagg force-pushed the rvagg/parallel-endpoint-lookup-cache-errors branch from a00fec2 to 889cd19 Compare November 8, 2023 11:08
retriever/endpointfinder/endpointfinder.go Outdated Show resolved Hide resolved
retriever/endpointfinder/endpointfinder.go Outdated Show resolved Hide resolved
api/api.go Outdated Show resolved Hide resolved
@rvagg rvagg force-pushed the rvagg/parallel-endpoint-lookup-cache-errors branch from 2585c02 to 8664bbf Compare November 8, 2023 21:33
@rvagg
Copy link
Contributor Author

rvagg commented Nov 8, 2023

@gammazero PTAL I've switched to an options pattern for this but also slightly changed the parallelism such that it only forks a goroutine if there's something to look up, so the common call here will just bounce right through and collect values from the cache, no goroutine, no channels. Can you sanity check the logic for me please?

Copy link
Collaborator

@gammazero gammazero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with one question.

retriever/endpointfinder/endpointfinder.go Outdated Show resolved Hide resolved
@rvagg rvagg force-pushed the rvagg/parallel-endpoint-lookup-cache-errors branch from 8a7bad5 to dd7d1a7 Compare November 9, 2023 01:16
@gammazero gammazero merged commit ead1a94 into main Nov 9, 2023
@gammazero gammazero deleted the rvagg/parallel-endpoint-lookup-cache-errors branch November 9, 2023 18:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants