GRPC ingester config, code gen, wiring and docs #147
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Partial merge of: #82
This PR focuses on merging the foundational parts with low risk. The new ingester component and server are disabled by default.
GRPC Ingest Protocl Documentation
This is a component that is responsible for ingesting and processing data.
It exposes a GRPC API for Osmosis node clients to send data to be ingested.
Note that, as of right now the protocol is syncronous where each GRPC call happens in sequence. However, from the
node perspective it is processed in a separate goroutine, letting the node continue producing blocks. The node
acquires a lock on the block processing so that the interaction is not affected by synching.
This is not a concern since, when the node is caught up, the block time is approximately 4.5 seconds while entire
protocol is capped at 1.5 seconds.
Currently, we push all pool data into SQS every processed block. As we lower block time, we will introduce a mechanism for
pushing the pool data only for the modified pools. This will allow us to drastically lower the protocol interaction from 1.5 seconds.
Alternative methods will include making the protocol more asyncronous but this will require introducing more complex
locking mechanisms which are overkill today.