[IND-402] Add cache for order updates for stateful orders.

[vincentwschau]

Changelist

First PR to track order updates for stateful orders.
Adds a cache to hold order update messages for stateful orders.
The cache consists of:

• ZSET storing the order id of the stateful order updated in the order update, mapped to the time when the order update was cached, used to remove old order updates from the cache periodically
• HSET storing the encoded order update mapped to the order id, to query the cache for the order update given the order id

The ZSET is used to query the cache for the order ids for order updates stored in the cache along with the cached timestamp.
Lua scripts are added for both adding and removing order updates from the cache.

Test Plan

Unit tests added.

Author/Reviewer Checklist

• If this PR has changes that result in a different app state given the same prior state and transaction list, manually add the state-breaking label.
• If this PR isn't state-breaking but has changes that modify behavior in PrepareProposal or ProcessProposal, manually add the label proposal-breaking.
• If this PR is one of many that implement a specific feature, manually label them all feature:[feature-name].
• If you wish to for mergify-bot to automatically create a PR to backport your change to a release branch, manually add the label backport/[branch-name].
• Manually add any of the following labels: refactor, chore, bug.

[coderabbitai]

Walkthrough

The changes introduce a stateful order updates cache using Redis. The cache supports adding and removing order updates, and retrieving old updates based on a timestamp. The implementation includes new utility functions, Lua scripts for Redis operations, and tests to ensure the cache behaves as expected.

Changes

File(s)	Summary
.../base/src/utils.ts	Added a new utility function hasDefinedProperties to check if an object has any defined properties.
.../redis/__tests__/caches/stateful-order-updates-cache.test.ts	Introduced tests for the new stateful order updates cache.
.../redis/src/caches/scripts.ts	Added two new Lua scripts for adding and removing stateful order updates.
.../redis/src/caches/stateful-order-updates-cache.ts	Implemented functions for adding and removing order updates from the cache, and retrieving old updates.
.../redis/src/index.ts	Added a new export statement for the StatefulOrderUpdatesCache module.
.../redis/src/scripts/add_stateful_order_update.lua	Added a Lua script to add a stateful order update to the cache.
.../redis/src/scripts/remove_stateful_order_update.lua	Added a Lua script to remove a stateful order update from the cache.
.../redis/src/types.ts	Introduced a new interface StatefulOrderUpdateInfo to represent stateful order update information.

---

Tips

Chat with CodeRabbit Bot (@coderabbitai)

• If you reply to a review comment from CodeRabbit, the bot will automatically respond.
• To engage with CodeRabbit bot directly around the specific lines of code in the PR, mention @coderabbitai in your review comment
• Note: Review comments are made on code diffs or files, not on the PR overview.
• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Commands (invoked as PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger a review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai help to get help.

Note: For conversation with the bot, please use the review comments on code diffs or files.

CodeRabbit Configration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• The JSON schema for the configuration file is available here.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/coderabbit-overrides.json

[linear]

IND-402 Fix Stateful order placement with no fill

context

Stateful orders that are not filled do not get added to the order book.

Issue:
1. Orders aren't set to be "resting on the book" until an OrderUpdate message is sent for the message. After receiving an OrderPlace message the order does not have it's size added to the order book.
2. OrderUpdate messages aren't processed if an OrderPlace message was not previously processed for the order.
3. For stateful orders
- the OrderPlace message is sent after ender processes the order
- the OrderUpdate message is sent by the node after the stateful order is placed
4. The OrderUpdate message is received before the OrderPlace, due to it being sent from a different service.

Potential Solution:

• keep track of all OrderUpdate messages for stateful orders that don't update an order in a separate cache similar to the CanceledOrdersCache

• when a stateful order placement is ingested by ender, check if there's a corresponding OrderUpdate message, and if so, send it to vulcan

• the difference in time between the OrderPlace and OrderUpdate should be a most a few blocks, so keep only the last 1 minute of OrderUpdate messages in the cache

[coderabbitai]

Review Status

Actionable comments generated: 5
Configuration used: CodeRabbit UI

Commits

Files that changed from the base of the PR and between 31281e2 and b966802.

Files selected for processing (8)

• indexer/packages/base/src/utils.ts (1 hunks)
• indexer/packages/redis/tests/caches/stateful-order-updates-cache.test.ts (1 hunks)
• indexer/packages/redis/src/caches/scripts.ts (2 hunks)
• indexer/packages/redis/src/caches/stateful-order-updates-cache.ts (1 hunks)
• indexer/packages/redis/src/index.ts (1 hunks)
• indexer/packages/redis/src/scripts/add_stateful_order_update.lua (1 hunks)
• indexer/packages/redis/src/scripts/remove_stateful_order_update.lua (1 hunks)
• indexer/packages/redis/src/types.ts (1 hunks)

Files skipped from review due to trivial changes (1)

• indexer/packages/redis/src/types.ts

Additional comments (Suppressed): 11

indexer/packages/base/src/utils.ts (1)

• 1-14: The function hasDefinedProperties is well implemented and follows best practices. It checks if an object has any defined properties and returns a boolean value accordingly. This function can be useful in various scenarios where it's necessary to check if an object has any meaningful data before proceeding with further operations.

indexer/packages/redis/src/index.ts (1)

• 12-12: The new StatefulOrderUpdatesCache is being exported. Ensure that the implementation of this cache is thread-safe and that it handles potential race conditions correctly. Also, verify that the cache eviction policy aligns with the application's requirements.

indexer/packages/redis/src/scripts/add_stateful_order_update.lua (1)

• 1-16: The Lua script for adding a stateful order update to the Redis cache seems to be logically correct. It uses the ZADD command to add the order ID and timestamp to the ZSET and the HSET command to add the order ID and the encoded order update to the HSET. However, it's important to ensure that the statefulOrderId, statefulOrderUpdate, and timestamp are correctly provided and formatted before calling this script. Also, consider adding error handling to catch any potential Redis errors.

indexer/packages/redis/src/scripts/remove_stateful_order_update.lua (3)

• 1-4: Ensure that the keys for the ZSET and HSET are unique and do not collide with other keys in the Redis database. This is to prevent accidental overwriting or deletion of data.

• 23-27: The script checks if the timestamp of the stateful order update is less than the maximum timestamp to remove. However, the comment on line 24 seems to contradict the condition on line 25. Please verify if the condition is correct.

• 39-41: The script deletes the order update from the HSET and returns the old order update. Ensure that this operation is atomic to prevent race conditions where another process might try to access the order update while it's being deleted.

indexer/packages/redis/src/caches/scripts.ts (2)

• 63-64: The addition of addStatefulOrderUpdateScript and removeStatefulOrderUpdateScript is consistent with the PR summary. Ensure that the corresponding Lua scripts add_stateful_order_update.lua and remove_stateful_order_update.lua are correctly implemented and tested.

• 74-75: The newly added scripts addStatefulOrderUpdateScript and removeStatefulOrderUpdateScript are included in the allLuaScripts array. This is necessary for the scripts to be loaded onto the Redis client. Ensure that these scripts are being used appropriately in the rest of the codebase.

indexer/packages/redis/__tests__/caches/stateful-order-updates-cache.test.ts (2)

• 58-108: The test cases for removeStatefulOrderUpdate are well written and cover different scenarios. They test the removal of an existing order update, the case where the timestamp is lower, and the removal of a non-existing order.

• 111-163: The test cases for getOldOrderUpdates are also well written and cover different scenarios. They test the retrieval of order updates older than a certain threshold and the retrieval of multiple order updates.

indexer/packages/redis/src/caches/stateful-order-updates-cache.ts (1)

• 108-127: The getOldOrderUpdates function retrieves order updates that were added to the cache before a specified timestamp. It uses the zRangeByScoreAsync helper function to retrieve the order IDs and timestamps from the ZSET. The function then maps the raw results to an array of StatefulOrderUpdateInfo objects. This function is well implemented and does not require any changes.