Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GRPC ingester config, code gen, wiring and docs #147

Merged
merged 1 commit into from
Mar 21, 2024

Conversation

p0mvn
Copy link
Member

@p0mvn p0mvn commented Mar 21, 2024

Partial merge of: #82

This PR focuses on merging the foundational parts with low risk. The new ingester component and server are disabled by default.

GRPC Ingest Protocl Documentation

This is a component that is responsible for ingesting and processing data.

It exposes a GRPC API for Osmosis node clients to send data to be ingested.

sequenceDiagram
    participant NodeIngester
    Note over SQS,NodeIngester: SQS Ingest RPC Communication
    Note over NodeIngester: Acquires lock on block processing
    NodeIngester->>SQS: StartBlockProcess(height, taker_fees)
    Note over SQS: SQS starts processing height
    SQS-->>NodeIngester: StartBlockProcessReply
    NodeIngester->>SQS: ProcessChainPools(stream)
    NodeIngester-->>SQS: ProcessChainPools.Send(pools_data_chunk)
    Note over SQS: SQS transforms and loads pools into ingester cache
    NodeIngester-->>SQS: ProcessChainPools.Send(pools_data_chunk)
    Note over SQS: SQS transforms and loads pools into ingester cache
    NodeIngester->>SQS: ProcessChainPools.CloseSend()
    
    NodeIngester->>SQS: EndBlockProcess(EndBlockProcessRequest)
    Note over SQS: SQS commits all state into SQS repositories, making them available for clients.
    SQS-->>NodeIngester: EndBlockProcessReply
    Note over NodeIngester: Releases lock on block processing
Loading

Note that, as of right now the protocol is syncronous where each GRPC call happens in sequence. However, from the
node perspective it is processed in a separate goroutine, letting the node continue producing blocks. The node
acquires a lock on the block processing so that the interaction is not affected by synching.

This is not a concern since, when the node is caught up, the block time is approximately 4.5 seconds while entire
protocol is capped at 1.5 seconds.

Currently, we push all pool data into SQS every processed block. As we lower block time, we will introduce a mechanism for
pushing the pool data only for the modified pools. This will allow us to drastically lower the protocol interaction from 1.5 seconds.

Alternative methods will include making the protocol more asyncronous but this will require introducing more complex
locking mechanisms which are overkill today.

@p0mvn p0mvn force-pushed the roman/ingest-refactor-foundation branch from 59d1d29 to deacfe6 Compare March 21, 2024 04:33
@p0mvn p0mvn changed the title ingest refactor foundation GRPC ingester config, code gen and wiring Mar 21, 2024
@p0mvn p0mvn changed the title GRPC ingester config, code gen and wiring GRPC ingester config, code gen, wiring and docs Mar 21, 2024
@p0mvn p0mvn requested a review from niccoloraspa March 21, 2024 04:35
"enable-overwrite-routes-cache": true
"grpc-ingester":{
"enabled": false,
"max-receive-msg-size-bytes": 16777216,
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: 16 KB. We can't leave comments. Once we implement yml config, will be able to transcribe this better

@p0mvn
Copy link
Member Author

p0mvn commented Mar 21, 2024

Merging per discussions with Nicco. Will have a walkthrough on full completion

@p0mvn p0mvn merged commit 8cfba90 into v23.x Mar 21, 2024
4 of 6 checks passed
@p0mvn p0mvn deleted the roman/ingest-refactor-foundation branch March 21, 2024 19:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant