Skip to content

Commit

Permalink
docs(README): add dashboard visualization; improve description of Van…
Browse files Browse the repository at this point in the history
…illaReplayStrategy
  • Loading branch information
abrichr authored Jun 5, 2024
1 parent 81074c4 commit 6df4bb9
Showing 1 changed file with 25 additions and 12 deletions.
37 changes: 25 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -158,27 +158,37 @@ pointing the cursor and left or right clicking, as described in this

### Visualize

Visualize the latest recording you created by running the following command:
Quickly visualize the latest recording you created by running the following command:

```
python -m openadapt.visualize
```

This will open a scrollable window that looks something like this:
This will generate an HTML file and open a tab in your browser that looks something like this:

<img width="1512" alt="image" src="https://github.com/OpenAdaptAI/OpenAdapt/assets/774615/451dd467-20ae-4ce7-a3b4-f888635afe8c">
![image](https://github.com/OpenAdaptAI/OpenAdapt/assets/774615/5d7253b7-ae12-477c-94a3-b388e4f37587)

<img width="1511" alt="image" src="https://github.com/OpenAdaptAI/OpenAdapt/assets/774615/13264cf6-46c0-4413-a29d-59bdd040a32e">
For a more powerful dashboard, run:

```
python -m openadapt.app.dashboard.run
```

This will start a web server locally, and then open a tab in your browser that looks something like this:

For a browser-based visualization, run:
![image](https://github.com/OpenAdaptAI/OpenAdapt/assets/774615/48d27459-4be8-4b96-beb0-1973953b8a09)

For a desktop app-based visualization, run:

```
python -m openadapt.deprecated.visualize
python -m openadapt.app.visualize
```

This will open up a tab in your browser that looks something like this:
This will open a scrollable window that looks something like this:

![image](https://github.com/OpenAdaptAI/OpenAdapt/assets/774615/5d7253b7-ae12-477c-94a3-b388e4f37587)
<img width="1512" alt="image" src="https://github.com/OpenAdaptAI/OpenAdapt/assets/774615/451dd467-20ae-4ce7-a3b4-f888635afe8c">

<img width="1511" alt="image" src="https://github.com/OpenAdaptAI/OpenAdapt/assets/774615/13264cf6-46c0-4413-a29d-59bdd040a32e">

### Playback

Expand All @@ -190,12 +200,15 @@ python -m openadapt.replay NaiveReplayStrategy

Other replay strategies include:

- [`StatefulReplayStrategy`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/strategies/stateful.py): Proof-of-concept which uses the OpenAI GPT-4 API with prompts constructed via OS-level window data.
- [`VanillaReplayStrategy`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/strategies/vanilla.py): If AGI or GPT6 happens, this script should be able to suddenly do the work. --LunjunZhang
- [`VisualReplayStrategy`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/strategies/visual.py): Uses [Fast Segment Anything Model (FastSAM)](https://github.com/CASIA-IVA-Lab/FastSAM) to segment active window. Accepts an "instructions" parameter that is used to modify the recording, e.g.:
- [`StatefulReplayStrategy`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/strategies/stateful.py): Early proof-of-concept which uses the OpenAI GPT-4 API with prompts constructed via OS-level window data.
- (*)[`VisualReplayStrategy`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/strategies/visual.py): Uses [Fast Segment Anything Model (FastSAM)](https://github.com/CASIA-IVA-Lab/FastSAM) to segment active window.
- (*)[`VanillaReplayStrategy`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/strategies/vanilla.py): Assumes the model is capable of directly reasoning on states and actions accurately. With future frontier models, we hope that this script will suddenly work a lot better.


The (*) prefix indicates strategies which accept an "instructions" parameter that is used to modify the recording, e.g.:

```
python -m openadapt.replay VisualReplayStrategy --instructions "Multiply 9x5 instead of 6x8"
python -m openadapt.replay VanillaReplayStrategy --instructions "calculate 9-8"
```

See https://github.com/OpenAdaptAI/OpenAdapt/tree/main/openadapt/strategies for a complete list. More ReplayStrategies coming soon! (see [Contributing](#Contributing)).
Expand Down

0 comments on commit 6df4bb9

Please sign in to comment.