Review script tooling

[JackUrb]

Overview

Introduces the examine_utils script tools, which provides a command-line based workflow for doing review of worker's work. Now it only requires that downstream users implement a function for the print view for their task, and then we can work through the process of reviewing from the command line, including functionality to soft-reject and soft-block, reject and hard block, and approve and grant a positive qualification. It also has a version that allows you to just view a slice of the incoming data.

Usage

All documentation is in the script:

Review mode:

>> python review_results.py 
Do you want to (r)eview, or (e)xamine data? Default examine. Can put e <end> or e <start> <end> to choose how many to view
r
Input task name: objects-interaction-task-pilot-1
If you'd like to soft-block workers, you'll need a block qualification. Leave blank otherwise.
Enter block qualification: collect_narrations_task_block
If you'd like to qualify high-quality workers, you'll need an approve qualification. Leave blank otherwise.
Enter approve qualification: collect_narrations_approved
Starting review with following params:
Task name: objects-interaction-task-pilot-1
Blocking qualification: collect_narrations_task_block
Approve qualification: collect_narrations_approved
Press enter to continue... 
You will be reviewing actual tasks with this flow. Tasks that you either Accept or Pass will be paid out to the worker, while rejected tasks will not. Passed tasks will be specially marked such that you can leave them out of your dataset. 
You may enter the option in caps to apply it to the rest of the units for a given worker.
When you pass or reject a task, the script gives you an option to disqualify the worker from future tasks by assigning a qualification. If provided, this worker will no longer be able to work on tasks where the set --block-qualification shares the same name you provided above: collect_narrations_task_block

You may use this script to establish a qualified worker pool by granting the provided approve qualification collect_narrations_approved to workers you think understand the task well. This will be provided as an option for workers you (A)pprove all on. Future tasks can use this qual as a required qualification, as described in the common qualification flows document.
**************
You should only reject tasks when it is clear the worker has acted in bad faith, and didn't actually do the task. Prefer to pass on tasks that were misunderstandings.
**************

Reviewing for worker XXXXXXXX, (1/5), Previous (First time worker!) (total remaining: 298)
-------------------
Worker: XXXXXXXX
Unit: XXXX
Duration: XXXX
Status: XXXXX
<my task view for the specific task I'm running>

Do you want to accept this work? (a)ccept, (r)eject, (p)ass: p
Do you want to soft block this worker? (y)es/(n)o: n
Reviewing for worker XXXXXXXX, (2/5), Previous (First time worker!) (total remaining: 297)
-------------------
Worker: XXXXXXXX
Unit: XXXX
Duration: XXXX
Status: XXXXX
<my task view for the specific task I'm running>

Do you want to accept this work? (a)ccept, (r)eject, (p)ass: p
Do you want to soft block this worker? (y)es/(n)o: y
Reviewing for worker XXXXXXXX, (3/5), Previous (First time worker!) (total remaining: 296)
-------------------
Worker: XXXXXXXX
Unit: XXXX
Duration: XXXX
Status: XXXXX
<my task view for the specific task I'm running>

Do you want to accept this work? (a)ccept, (r)eject, (p)ass: P
Do you want to soft block this worker? (y)es/(n)o: y
Reviewing for worker XXXXXXXX, (4/5), Previous (First time worker!) (total remaining: 295)
-------------------
Worker: XXXXXXXX
Unit: XXXX
Duration: XXXX
Status: XXXXX
<my task view for the specific task I'm running>

Reviewing for worker XXXXXXXX, (5/5), Previous (First time worker!) (total remaining: 294)
-------------------
Worker: XXXXXXXX
Unit: XXXX
Duration: XXXX
Status: XXXXX
<my task view for the specific task I'm running>

Reviewing for worker XXXXXXXX, (1/5), Previous (4 | 0(0) / 4) (total remaining: 275)
-------------------
Worker: XXXXXXXX
Unit: XXXX
Duration: XXXX
Status: XXXXX
<my task view for the specific task I'm running>

Do you want to accept this work? (a)ccept, (r)eject, (p)ass: a
Reviewing for worker XXXXXXXX, (2/5), Previous (4 | 0(0) / 4) (total remaining: 274)

Examine mode (redacted details):

Do you want to (r)eview, or (e)xamine data? Default examine. Can put e <end> or e <start> <end> to choose how many to view
e 2 
Input task name: XXXXXXXXX
-------------------
Worker: XXXXXXXX
Unit: XXXX
Duration: XXXX
Status: XXXXX
<my task view for the specific task I'm running>

-------------------
Worker: XXXXXXXX
Unit: XXXX
Duration: XXXX
Status: XXXXX
<my task view for the specific task I'm running>

[codecov-commenter]

Codecov Report

Merging #526 (5058d99) into master (4aff7a8) will increase coverage by 0.03%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #526      +/-   ##
==========================================
+ Coverage   65.58%   65.61%   +0.03%     
==========================================
  Files          78       78              
  Lines        7195     7196       +1     
==========================================
+ Hits         4719     4722       +3     
+ Misses       2476     2474       -2     

Impacted Files	Coverage Δ
mephisto/abstractions/providers/mock/mock_unit.py	84.37% <100.00%> (+0.50%)	⬆️
...tractions/architects/channels/websocket_channel.py	82.55% <0.00%> (+2.32%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4aff7a8...5058d99. Read the comment docs.

[pringshia]

This looks good. I like the idea of having these as utils. Thanks for the docs updates as well.

Aside:

I would love to explore in a separate PR (I may also submit a proposal) for possibilities for making the textual interface easier to grokk by using color and spacing (not quite as extensive as #388 but maybe there's some lower hanging fruit we can hit).

We may also want to establish a common design system for guidelines for our text UI and to make sure we stay consistent across different CLI interfaces we offer, but I'm getting ahead of myself here.

[JackUrb]

I would love to explore in a separate PR (I may also submit a proposal) for possibilities for making the textual interface easier to grokk by using color and spacing (not quite as extensive as #388 but maybe there's some lower hanging fruit we can hit).

Very much game for this, just recently discovered how nice tools are with a little bit of color, will want to revisit later.