-
Notifications
You must be signed in to change notification settings - Fork 77
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor Blueprint Mixins, create ScreenTaskRequired mixin #566
Conversation
Codecov Report
@@ Coverage Diff @@
## main #566 +/- ##
==========================================
+ Coverage 65.45% 66.32% +0.87%
==========================================
Files 79 81 +2
Lines 7297 7459 +162
==========================================
+ Hits 4776 4947 +171
+ Misses 2521 2512 -9
Continue to review full report at Codecov.
|
Overall looks good! Will re-review pending some of the changes we discussed, e.g. adding max_validations @ the blueprint level, as well as perhaps asserting that this value is necessary to re-emphasize to the user that there is a cost for these tasks |
mephisto/abstractions/blueprints/mixins/screen_task_required.py
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have a few copy and terminology related notes, but otherwise this looks good to me.
### `OnboardingRequired` | ||
This mixin allows for blueprints that require people to complete an onboarding task _before_ they're even able to start on their first task. Usually this is useful for providing task context, and then quizzing workers to see if they understand what's provided. Tasks using this mixin will activate onboarding mode for new `Worker`s whenever the `mephisto.blueprint.onboarding_qualification` hydra argument is provided. | ||
### `ScreenTaskRequired` | ||
This mixin allows for blueprints that require people to complete a _test_ version of the real task the first time a worker does the task. This allows you to validate workers on a run of the real task, either on your actual data (when providing `SharedTaskState.generate_screening_unit_data=False`) or on test data that you may more easily validate using (when providing a generator to `SharedTaskState.generate_screening_unit_data`). The tasks should be the same as your standard task, just able to be easily validated. You **do pay** for screening tasks, and as such we ask you set `mephisto.blueprint.max_screening_units` to put a cap on how many screening units you want to launch. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Really like this API! ❤️
screening_data_factory
could be another possible name here... just throwing out ideas
Overview
Creates the first full-fledged
Blueprint
mixinScreenTaskRequired
, in a similar vein toOnboardingRequired
, but with clearer delineation for workflow-specific functionality. This PR also refactors both of these to follow the newBlueprintMixin
structure, which lets us standardize how workflows for blueprints can be mixed in. The goal here is to create a task that, for workers that haven't completed any of the task yet, you can put out a test task for which you can more easily validate the results to than a full task. This launches new tasks on the fly, such that you can have validation tasks be separate from the rest of the pool.Implementation
ValidationRequired
mixin, which defines behaviors to properly validate and initialize blueprints with this functionality. Defines some logic hooks that Mephisto's ops layer can use to interact with this workflow. Also defines utility methods that are useful in user scripts for those utilizing this workflow.TaskRun
to ignore special units (for gold, validators, etc) when determining eligible units for workersTaskLauncher
along to theSupervisor
to be able to handle launching validator units. This smells but I aim to address this kind of over-coupling in the coming refactor of theSupervisor
.Supervisor
to run validation tasks for new workers on blueprints that haveValidationRequired
. Ensures that these are expired when abandoned, to not pass to others.TaskLauncher
that creates and launches the special validator units.Considerations
unit_index
. I think this is a decent way to go about this, but I wonder if mixins should also have considerations for review scripts? Likely need to update theDataBrowser
to handle these better in the future.Blueprint
file, but I think these should go somewhere better. Will break out before mergingTesting
In internal