-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Integrating TaskManager
into Discovery
#451
Integrating TaskManager
into Discovery
#451
Conversation
👇 Click on the image for a new way to code review
Legend |
Discovery
depends on TaskManager
TaskManager
into Discovery
We can limit the scope of discovery operations because we only care about:
We don't need to crawl transitive gestalts (friends of friends...). |
Discovery is likely to be push and pull at the same time. But right now everything in PK is pull-based cause it's easier to implement. So we can do something simple. At most we might have 1000 gestalts trusted, and each gestalt is likely a handful of nodes, say 5. Then at the most have 5000 nodes to check in with. We can say 1 day TTL to revisit each node. When the push-based gossip protocol is available, we can actually increase the TTL, and that TTL acts like the last-resort in the sense that if no update has occurred from there we can check in. It could be even simpler of we can expect that any given node of a gestalt should be up to date with their own gestalt's information. At this point in time, let's make it configurable and see how it goes. The goal here for this PR is not to make our discovery perfect UI/UX-wise, but to simply integrate the task manager system, simplify the code in discovery so it becomes stateless (with respect to task management), and also incorporate |
So |
ab75115
to
00ce3e1
Compare
General changes so far:
|
b5d7f5f
to
02912dd
Compare
Re-based on staging and a quick fix. Should pass CI now. |
The Discovery failures is due to
|
20c1eab
to
8408a8f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Some small changes and discussion needed.
b4842f4
to
2d004ee
Compare
This should be ready to merge pending CI. @CMCDragonkai did you still want to review? |
Yes give me a moment. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Initial review. Haven't gone into the discovery code yet.
Please link the next PR to here. Applying timedCancellable across the entire codebase. |
It is possible to remove these mocks from our tests: const mockedRefreshBucket = jest.spyOn(
NodeManager.prototype,
'refreshBucket',
); If we realise that what we need is to have 2-stage The
We can reuse the pattern of Then we also need to move // Adding seed nodes
for (const nodeIdEncoded in this.seedNodes) {
const nodeId = nodesUtils.decodeNodeId(nodeIdEncoded);
if (nodeId == null) never();
await this.nodeManager.setNode(
nodeId,
this.seedNodes[nodeIdEncoded],
true,
);
} The above code seems that it can just be done as part of Then in if (!lazy) {
await this.startSync();
} So this async startSync() {
await this.syncSeedNodes(); // Replaced name for `syncNodeGraph`
await this.setupRefreshBucketTasks();
} Can we then have an equivalent I guess it would whatever Atm, it looks like This would basically make Then in the await this.taskManager.start({ fresh, lazy: true });
await this.nodeManager.start({ lazy: true });
await this.nodeConnectionManager.start({
nodeManager: this.nodeManager
});
await this.nodeGraph.start({ fresh });
await this.discovery.start({ fresh });
await this.vaultManager.start({ fresh });
await this.notificationsManager.start({ fresh });
await this.sessionManager.start({ fresh });
if (lazy) {
await this.nodeManager.startSync();
await this.taskManager.startProcessing();
} Notice the What does it mean to do tests on PK agent without having it run background tasks? It seems that this means that it would avoid interference if we are trying to do a deterministic test. So imagine you want to test the PK agent to do a specific direct connection or a specific call, and you would prefer that these background tasks aren't running. Then it does make sense to have this ability of staging the second stage of start. So now we have 3/4 stages:
|
The |
@tegefaulkes can you create a new issue for discovery refactoring: #451 (comment) (needs TTL) - this can be done in separate PR And also a new issue for this: #451 (comment) - this can be done separate PR |
All review addressed now. I'll clean up the commit history again, merge, and create the new issues/PRs. |
- The parts of discovery that handles queuing has been removed. This includes the 'plug' promises, database levels and the discovery ID generator. - The logic of the queue where it handled the vertex was separated out into a new method. This logic included connecting to a vertex, getting the claims' data, iterating over the claims, verifying the claims and adding them to the gestaltGraph, and adding the new vertices to the list. - Adding a vertex to the list now consists of creating a singleton task. each task focuses on processing a single vertex. - I tried to cancel all tasks on `discovery.destroy()` but it would require the `taskManager` to be started at the time since we need to iterate over active tasks at the time. For now cancel the tasks when starting with fresh. - To avoid re-trying tasks, a `visitedVertices` set is used, this hasn't been updated. A more complex mechanism is need for this that has persistence. I think this will be left with the more complex discovery changes later. - Discovery tests have been updated. none removed or added so far. The change was just using `taskManager`. - Removed discovery utils and types. These are not needed anymore since we don't make use of a discovery ID generator. - Added locking on the vertex to avoid duplicating a task due to racing. Also catching and throwing the singleton reason, so we don't get any errors on tasks that were canceled for this reason. - Discovery tasks are canceled during fresh start and destroy. -`Discovery` stops any active tasks when stopping - Added a check for if `TaskManager` is still processing when calling stop on `Discovery` and `NodeManager`.
25f4ad3
to
6d8f606
Compare
Squashed and re-based on staging. |
Please link the new issues you've created here too. |
Issue #461 created to address #451 (comment). |
Issue #462 created to address further discovery refactoring. |
Description
This PR focuses on updating the
Discovery
domain to make use of theTaskManager
.Issues Fixed
Tasks
TaskManager
queuing.Discovery
since it's handled viaTaskManager
now.Final checklist