Prod Release 03/11/24 #601
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
darunrs
commented
Mar 8, 2024
darunrs
commented
- chore: Remove legacy code chore: Remove legacy code #576
- Feat: Support Wildcard and * contract filters feat: Support Wildcard and * contract filters #567
- feat: Cache Streamer Message from Block Streamer feat: Cache Streamer Message from Block Streamer #582
- fix: Increase timeout of VM run function fix: Increase timeout of VM run function #584
- 541 update the frontend to dis allow use of this filter fix: update middleware for CORS #583
- feat: Cache Database Credentials feat: Cache Database Credentials #585
- feat: fixed reload button, unmount grid feat: fixed reload button, unmount grid #592
- feat: Support Adding of Owners through add_user feat: Support Adding of Owners through add_user #590
- feat: Reduce Runner Logs and DB Writes feat: Reduce Runner Logs and DB Writes #589
- fix: Block Streamer not updating last published block fix: Block Streamer not updating last published block #596
- fix: Cap Block Streamer Caching and Merge Redis Impl fix: Cap Block Streamer Caching and Merge Redis Impl #599
Removes: - Coordinator V1 - SQS Runner - Relevant GH actions
added the * filter for contracts.
A valid contract is defined as
```
function validateContractId(accountId) {
accountId = accountId.trim();
const isLengthValid = accountId.length >= 2 && accountId.length <= 64;
if (!isLengthValid) return false;
//test if the string starts with a '*.' and remove it if it does
const isWildCard = WILD_CARD_REGEX.test(accountId);
isWildCard ? accountId = accountId.slice(2) : null;
//test if rest of string is valid accounting for/not isWildCard
const isRegexValid = CONTRACT_NAME_REGEX.test(accountId);
return isRegexValid;
}
note: early termination for performance
```
The function take in account all **invalid** contract names on
[nomicon](https://nomicon.io/DataStructures/Account)
Please view test for more details.
Regex Broken down Below:
CONTRACT_NAME_REGEX:
`/^(([a-z\d]+[-_])*[a-z\d]+(\.([a-z\d]+[-_])*[a-z\d]+)*\.([a-z\d]+)|([a-z\d]+))$/`
1. **^**: Asserts the start of the string.
2. **(**: Begins a capturing group.
3. **([a-z\d]+[-_])***: Matches a sequence of lowercase letters or
digits, possibly followed by hyphens or underscores. The sequence may
repeat zero or more times.
4. **[a-z\d]+**: Matches a sequence of one or more lowercase letters or
digits.
5. **(\.([a-z\d]+[-_])*[a-z\d]+)***: Allows for zero or more occurrences
of a dot . followed by another sequence.
6. **\.([a-z\d]+)**: Matches a dot followed by a sequence of lowercase
letters or digits.
7. **|**: OR operator.
8. **([a-z\d]+)**: Matches a standalone sequence of lowercase letters or
digits.
9. **)$**: Asserts the end of the string.
WILD_CARD_REGEX:
`/\*\./`
1. **\***: Matches a literal asterisk.
2. **\.**: Matches a literal dot.
EDIT: Added more test please review
Added jest to support function testing.
It seems there's a problem getting Babel and the VM to work smoothly
together. I'm going to dig deeper into it. Right now, I'm adjusting the
test file to recreate the functions instead of using the usual
require/import method, which seems to cause trouble with the VM code. It
does mean there's some duplicated code in the testing files, but the
test work.
Some indexers such as eduohe.near/access_keys potentially exceeds the timeout of the VM while running. This may be due to latency of DB calls, which is the heaviest parts of the code. We are planning to drive down the latency of these calls, but this is a stopgap to allow indexers to continue running in the mean time.
Looks like the middleware file was pushed on a dev branch and was ignored when testing for CORS options. Fix: removed- frontend/src/pages/middleware.js out CORS is created from nextjs looking at next.config.js which is a sibling of package.json
Currently, Runner fetches DB credentials every invocation of runFunction because provisioning of the user's DB occurs in Runner. As such, we couldn't be sure if we could fetch credentials for the user's DB from Hasura or not. The fetch is slow and increases the latency of any indexers which leverage context.db. I updated the code to cache the credentials locally after the first successful fetch. This allows all future requests to succeed as these credentials will never change after they are set. The changes were aimed to be minimal as we intend to migrate provisioning out of Runner. In fact, the changes aid in the effort as the credentials fetch is done through the Provisioner class rather than separately in DmlHandler.
Fixed the reload button, added unmounting of the grid. IMO - we really don't need a library(gridjs) for a logging table with simple functions as this is the only area we use it as well. Happy to create an ticket to migrate for a todo in the future. Otherwise demo here: https://www.loom.com/share/c35ca732212e48f2ab11d17238f39812
There is no way to add someone as Owner without a migration. I've updated the add_user function to accept a role as a parameter, allowing addition of Owners without migration.
Runner logs excessively, both to the machine, and to postgres. Most of these logs are not necessary or contain too much information. The plan is to reduce logs the following ways. 1. Only output failure log for first failure (Instead of every 10 seconds) 2. Stop outputting "Running function on X block" logs to GCP 3. Stop outputting "Success: Ran function on X block" logs to GCP 4. Remove DB logs for hasura calls 5. Remove context.db logging To support 1, Runner needs a way to know when a failure was the first. So, it needs to remember the state of the Runner locally. This is a good opportunity to introduce a refactor to stop setting of Runner status every loop. We can instead only write when the status changes. In addition, there is some overlap with this epic. So, to orient these changes towards our intentions there, I will also introduce a rudimentary log level into Runner. This value will be INFO by default, and the extra logs will be at the DEBUG level. This ensures these logs are automatically skipped while also adding some backbone for expected future implementation.
Block Streamer was not updating last published block in redis due to a bug.
Redis memory usage is pegged at 100% due to Block Streamer backfilling * contract filter indexers. In addition, Redis implementation of Coordinator and Block Streamer have diverged. Coordinator is also not able to make successful calls to Redis. While not a targeted fix, the hope is that merging the implementation fixes the issue as Block Streamer is able to make calls to Redis. Before release, some manual intervention is necessary in dev and prod to update versions of indexers to the current, to prevent unwanted behavior from coordinator. I will do dev first, release the change, and then do prod before a prod release.
|
Requires manual update to ensure versions are correctly set in prod prior to release. There's a lot of changes in this and since there were a large quantity of bugs present and then fixed in this chunk of code, some care is needed to ensure prod remains in a stable state. |
|
Versions have been set in prod. We are good to go for prod release. I haven't noticed any odd behavior either. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.