Threaded validation #513
Threaded validation #513
Conversation
…xecuted in separate threads
|
@binh-dam-ibigroup, Miles is to carry out some testing, but on the whole is happy to merge this into his working PR. If you don't mind, can you review please just in case I've missed something? |
| @@ -382,22 +414,6 @@ public void validate(MonitorableJob.Status status) { | |||
|
|
|||
| // This will persist the document to Mongo | |||
| this.mobilityDataResult = Document.parse(json); | |||
There was a problem hiding this comment.
The "regular" parsing step was previously persisting the this to mongo. If the mobility data parser finishes first, this still happens. If it finishes second, this fails to happen and the data isn't persisted! I think the solution is to add a mongo persistence line right here
There was a problem hiding this comment.
@miles-grant-ibigroup We need to be careful here because we don't want to create a new Mongo object if the other validator is expecting to do that. See ValidateFeedJob -> jobFinished(). Both validators need to be aware of the Mongo object state, so that either can create or update. Let me take a look. Good spot btw.
There was a problem hiding this comment.
@miles-grant-ibigroup This should now be addressed. Unit tests look good. Lemme know how you get on.
…of feed version in relation to
Bumps [mongodb-driver-sync](https://github.com/mongodb/mongo-java-driver) from 4.0.5 to 4.0.6. - [Release notes](https://github.com/mongodb/mongo-java-driver/releases) - [Commits](mongodb/mongo-java-driver@r4.0.5...r4.0.6) --- updated-dependencies: - dependency-name: org.mongodb:mongodb-driver-sync dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]>
…ongodb-driver-sync-4.0.6 chore(deps): bump mongodb-driver-sync from 4.0.5 to 4.0.6
…alidateMobilityDataFeedJob fail
| } else { | ||
| Persistence.feedVersions.replace(feedVersion.id, feedVersion); | ||
| } | ||
| feedVersion.persistFeedVersionAfterValidation(isNewVersion); |
There was a problem hiding this comment.
As written, it looks like the new version is shown in the UI (on the mobility-data-validator branch) after the first validator completes. Is that the desired effect?
There was a problem hiding this comment.
I think that's ok as long as we refresh the UI when the mobility data results come in, which is something we can do on the client.
|
I've noticed that on some large feeds the jobs are running sequentially? |
|
This is to be expected. Once both threads have started we are then effectively passing control to the JVM which then decides how much CPU each thread will get. If the first gets 100% CPU, the second will have to wait. |
Checklist
devbefore they can be merged tomaster)Description
Working on top of https://github.com/ibi-group/datatools-server/pull/512/files original and mobility data validation are now in their own separate threads.