diff --git a/README.md b/README.md index 4aa767f3c..22f3c56a3 100644 --- a/README.md +++ b/README.md @@ -158,27 +158,37 @@ pointing the cursor and left or right clicking, as described in this ### Visualize -Visualize the latest recording you created by running the following command: +Quickly visualize the latest recording you created by running the following command: ``` python -m openadapt.visualize ``` -This will open a scrollable window that looks something like this: +This will generate an HTML file and open a tab in your browser that looks something like this: -image +![image](https://github.com/OpenAdaptAI/OpenAdapt/assets/774615/5d7253b7-ae12-477c-94a3-b388e4f37587) -image +For a more powerful dashboard, run: + +``` +python -m openadapt.app.dashboard.run +``` + +This will start a web server locally, and then open a tab in your browser that looks something like this: -For a browser-based visualization, run: +![image](https://github.com/OpenAdaptAI/OpenAdapt/assets/774615/48d27459-4be8-4b96-beb0-1973953b8a09) + +For a desktop app-based visualization, run: ``` -python -m openadapt.deprecated.visualize +python -m openadapt.app.visualize ``` -This will open up a tab in your browser that looks something like this: +This will open a scrollable window that looks something like this: -![image](https://github.com/OpenAdaptAI/OpenAdapt/assets/774615/5d7253b7-ae12-477c-94a3-b388e4f37587) +image + +image ### Playback @@ -190,12 +200,15 @@ python -m openadapt.replay NaiveReplayStrategy Other replay strategies include: -- [`StatefulReplayStrategy`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/strategies/stateful.py): Proof-of-concept which uses the OpenAI GPT-4 API with prompts constructed via OS-level window data. -- [`VanillaReplayStrategy`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/strategies/vanilla.py): If AGI or GPT6 happens, this script should be able to suddenly do the work. --LunjunZhang -- [`VisualReplayStrategy`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/strategies/visual.py): Uses [Fast Segment Anything Model (FastSAM)](https://github.com/CASIA-IVA-Lab/FastSAM) to segment active window. Accepts an "instructions" parameter that is used to modify the recording, e.g.: +- [`StatefulReplayStrategy`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/strategies/stateful.py): Early proof-of-concept which uses the OpenAI GPT-4 API with prompts constructed via OS-level window data. +- (*)[`VisualReplayStrategy`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/strategies/visual.py): Uses [Fast Segment Anything Model (FastSAM)](https://github.com/CASIA-IVA-Lab/FastSAM) to segment active window. +- (*)[`VanillaReplayStrategy`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/strategies/vanilla.py): Assumes the model is capable of directly reasoning on states and actions accurately. With future frontier models, we hope that this script will suddenly work a lot better. + + +The (*) prefix indicates strategies which accept an "instructions" parameter that is used to modify the recording, e.g.: ``` -python -m openadapt.replay VisualReplayStrategy --instructions "Multiply 9x5 instead of 6x8" +python -m openadapt.replay VanillaReplayStrategy --instructions "calculate 9-8" ``` See https://github.com/OpenAdaptAI/OpenAdapt/tree/main/openadapt/strategies for a complete list. More ReplayStrategies coming soon! (see [Contributing](#Contributing)).