Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a reusable app for task review #1058

Merged
merged 33 commits into from
Nov 30, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
c8faeb9
[WIP] Task review app - initial API commit
meta-paul Aug 25, 2023
5b5a733
[WIP] Task review app - moved unit_review to Mephisto DB and updated …
meta-paul Aug 29, 2023
1ddfb81
[WIP] Task review app - initial UI commit
meta-paul Aug 31, 2023
16e6e5d
[WIP] Task review app - added modal components, updated /stats endpoint
meta-paul Sep 2, 2023
03d4e8f
[WIP] Task review app - added getting units and stats data
meta-paul Sep 9, 2023
5a0fc23
[WIP] Task review app - implemented demo display of mnist task data
meta-paul Sep 13, 2023
1162f27
[WIP] Task review app - made transition between units and improved el…
meta-paul Sep 13, 2023
8f794bc
[WIP] Task review app - small fixes
meta-paul Sep 14, 2023
439bd46
[WIP] Task review app - more fixes
meta-paul Sep 18, 2023
86c8442
Task review app - Ported mnist app to load during task review
meta-paul Sep 21, 2023
4c291e3
Task review app - Added JSON viewer as fallback and fixed a bug
meta-paul Sep 25, 2023
96bff4b
Task review app - improved worker qualification endpoint
meta-paul Sep 25, 2023
75bf0e3
Task review app - linted code with Prettier
meta-paul Sep 26, 2023
c9489d9
Task review app - Fixed worker picking order during review
meta-paul Sep 27, 2023
259739a
Task review app - added creating Unit Review to Agent logic
meta-paul Oct 3, 2023
15689e2
Task review app - added a separate review-specific build for Task com…
meta-paul Oct 4, 2023
9318cac
Task review app - renamed 'tips' to 'bonus', updated docs
meta-paul Oct 4, 2023
ebe5cd5
removed deprecated mephisto client code
meta-paul Oct 5, 2023
1654887
Task review app - addressed PR review comments
meta-paul Oct 6, 2023
418c4c7
Fixed linting warnings
meta-paul Oct 6, 2023
2387d11
Added docs for dockerization support
meta-paul Oct 17, 2023
e7230c3
Addressed review comments
meta-paul Oct 18, 2023
88df9ad
Added a single script for launching TaskReview app server and client
meta-paul Oct 20, 2023
a8b8f40
Merge branch 'v1.2-dev' into make-task-review-server
meta-paul Oct 26, 2023
5b07bcc
Deprecated version of MNIST app without TaskReview support
meta-paul Oct 26, 2023
221bae3
Added step-by-step Mephisto guide
meta-paul Oct 27, 2023
6dfb39f
Added option to create new qualification during task review
meta-paul Nov 6, 2023
f2fe874
Tested TaskReview edge cases for Prolific
meta-paul Nov 6, 2023
ecfa55b
Added TaskReview API documentation
meta-paul Nov 6, 2023
9933658
Covered TaskReview app with unittests
meta-paul Nov 10, 2023
cc1c2e0
Fixed unittests for Prolific integration
meta-paul Nov 10, 2023
3494563
Added results export in JSON format for TaskReview app
meta-paul Nov 16, 2023
650326a
Fixed failing e2e Cypress tests
meta-paul Nov 25, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added ._data
Binary file not shown.
6 changes: 5 additions & 1 deletion .github/workflows/cypress-end-to-end-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ on:
branches: [main]
paths-ignore:
- "**.md"
env:
HYDRA_FULL_ERROR: 1
jobs:
# This job is made to setup path filtering, learn more about it here: https://github.com/facebookresearch/Mephisto/pull/857
changes:
Expand Down Expand Up @@ -273,7 +275,9 @@ jobs:
node-version: 16.16.0

- name: 🤖 Install Mephisto
run: pip install -e .
run: |
pip install -e .
pip install detoxify

- name: 🖋 Create data directory
run: mkdir -p ~/mephisto/data
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -55,3 +55,6 @@ aws_credentials
docker-compose.local.*
env.local
my-review-interface
docker/entrypoints/*.sh
!docker/entrypoints/server.mturk.sh
!docker/entrypoints/server.prolific.sh
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ You can find complete details about the project on our [docs website](https://me

[Get started in 10 minutes](https://mephisto.ai/docs/guides/quickstart)

[How to run projects](mephisto/README.md)

## Want to help?

Check out our [guidelines for contributing](https://github.com/facebookresearch/Mephisto/blob/main/CONTRIBUTING.md) and then take a look at some of our tagged issues: [good first issue](https://github.com/facebookresearch/Mephisto/labels/good%20first%20issue), [help wanted](https://github.com/facebookresearch/Mephisto/labels/help%20wanted).
Expand Down
4 changes: 2 additions & 2 deletions docker/docker-compose.dev.vscode.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@
version: "3"

services:
fb_mephisto_vscode:
container_name: fb_mephisto_vscode
mephisto_dc_vscode:
container_name: mephisto_dc_vscode
build:
context: ..
dockerfile: Dockerfile
Expand Down
4 changes: 2 additions & 2 deletions docker/docker-compose.dev.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@
version: "3"

services:
fb_mephisto:
container_name: fb_mephisto
mephisto_dc:
container_name: mephisto_dc
build:
context: ..
dockerfile: Dockerfile
Expand Down
3 changes: 2 additions & 1 deletion examples/parlai_chat_task_demo/hydra_configs/conf/base.yaml
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
#@package _global_

# Copyright (c) Meta Platforms and its affiliates.
# This source code is licensed under the MIT license found in the
# LICENSE file in the root directory of this source tree.

#@package _global_
defaults:
- /mephisto/blueprint: parlai_chat
- /mephisto/architect: local
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
#@package _global_

# Copyright (c) Meta Platforms and its affiliates.
# This source code is licensed under the MIT license found in the
# LICENSE file in the root directory of this source tree.

#@package _global_
defaults:
- base
mephisto:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
#@package _global_

# Copyright (c) Meta Platforms and its affiliates.
# This source code is licensed under the MIT license found in the
# LICENSE file in the root directory of this source tree.

#@package _global_
defaults:
- base
mephisto:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
#@package _global_

# Copyright (c) Meta Platforms and its affiliates.
# This source code is licensed under the MIT license found in the
# LICENSE file in the root directory of this source tree.

#@package _global_
defaults:
- base
mephisto:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
#@package _global_

# Copyright (c) Meta Platforms and its affiliates.
# This source code is licensed under the MIT license found in the
# LICENSE file in the root directory of this source tree.

#@package _global_
defaults:
- base
mephisto:
Expand Down
12 changes: 5 additions & 7 deletions examples/remote_procedure/mnist/README.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,9 @@
<!---
Copyright (c) Meta Platforms and its affiliates.
This source code is licensed under the MIT license found in the
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are licenses not required for .md?

LICENSE file in the root directory of this source tree.
-->

# Mephisto MNIST Model Evaluator demo

This task is a simple model evaluation demo, presenting the worker with three spaces to draw images in, running an MNIST model on them to predict an output, and then allowing the worker to assess and correct that output.

Introduced in [#677](https://github.com/facebookresearch/Mephisto/pull/677), some screenshots and discussion there.
Introduced in [#677](https://github.com/facebookresearch/Mephisto/pull/677), some screenshots and discussion there.

For more details how to write your own `webapp` (ReactJS application) see example in current directory.
Also, read [short documentation](../../../mephisto/client/review_app/README.md)
for the main Review App to understand how to add review part in your `webapp`.
23 changes: 0 additions & 23 deletions examples/remote_procedure/mnist/custom-review/.gitignore

This file was deleted.

92 changes: 0 additions & 92 deletions examples/remote_procedure/mnist/custom-review/README.md

This file was deleted.

42 changes: 0 additions & 42 deletions examples/remote_procedure/mnist/custom-review/package.json

This file was deleted.

Binary file not shown.
44 changes: 0 additions & 44 deletions examples/remote_procedure/mnist/custom-review/public/index.html

This file was deleted.

Binary file not shown.
Binary file not shown.

This file was deleted.

2 changes: 0 additions & 2 deletions examples/remote_procedure/mnist/custom-review/sample-data.csv

This file was deleted.

This file was deleted.

Loading
Loading