From 606ef21f01566478492c1b574291f9fa35f59ce9 Mon Sep 17 00:00:00 2001 From: zhangqianze Date: Wed, 25 Dec 2024 11:04:16 +0800 Subject: [PATCH 1/7] feat: add extension doc --- docs/ten_agent/project_structure/extension.md | 29 +++++++++++++++++++ .../project_structure/property_json.md | 6 ++-- 2 files changed, 32 insertions(+), 3 deletions(-) create mode 100644 docs/ten_agent/project_structure/extension.md diff --git a/docs/ten_agent/project_structure/extension.md b/docs/ten_agent/project_structure/extension.md new file mode 100644 index 0000000000..2b24d37c6d --- /dev/null +++ b/docs/ten_agent/project_structure/extension.md @@ -0,0 +1,29 @@ +# Understand extension folder + +The `ten_packages/extension` folder contains the extension modules. Each extension module is a separate Python/Golang/C++ package. + +The extension folder name is often the same as the extension module name, while you can also find the extension module name in the `manifest.json` file under the extension folder. The module name shall be used in the `addon` property of the `property.json` file. + +Below is a sample structure of the extension folder: + +![Extension Folder Structure](https://github.com/TEN-framework/docs/blob/main/assets/png/extension_folder_struct.png?raw=true) + +## Extension Common Files + +- manifest.json: This file contains the metadata of the extension. It includes the extension name, version, properties, and apis (data, audio_frame, video_frame). +- property.json: This file contains the default properties of the extension. It is used to define the default configuration of the extension. + +## Python Extension + +- extension.py: This file contains the main logic of the extension. It usually contains the core implementation of the extension. +- requirements.txt: This file contains the Python dependencies required by the extension. Dependencies specified in this file will be installed automatically when you run `task use`. + +## Golang Extension + +- _extension.go: This file contains the main logic of the extension. It usually contains the core implementation of the extension. +- go.mod: This file contains the Go module definition. It specifies the module name and the dependencies of the extension. + +## C++ Extension + +- src/main.cc: This file contains the main logic of the extension. It usually contains the core implementation of the extension. +- BUILD.gn: This file contains the build configuration of the extension. It specifies the target name and dependencies of the extension. \ No newline at end of file diff --git a/docs/ten_agent/project_structure/property_json.md b/docs/ten_agent/project_structure/property_json.md index c8e5022c2e..0216eabc9e 100644 --- a/docs/ten_agent/project_structure/property_json.md +++ b/docs/ten_agent/project_structure/property_json.md @@ -35,7 +35,7 @@ The `property` section of a node contains the configuration of the extension. It } ``` -![Property JSON nodes](https://github.com/TEN-framework/docs/blob/main/assets/png/property_json_nodes.png?raw=true) +![Property JSON nodes](https://github.com/TEN-framework/docs/blob/main/assets/png/property_json_connections.png?raw=true) #### Read environment variables @@ -71,6 +71,6 @@ The `connections` section contains the list of connections between nodes. Each c } ``` -![Property JSON Connections](https://github.com/TEN-framework/docs/blob/main/assets/png/property_json_connections.png?raw=true) +![Property JSON Connections](https://github.com/TEN-framework/docs/blob/main/assets/png/property_json_nodes.png?raw=true) -In the above example, we are connecting `agora_rtc` extension to `deepgram_asr` extension. The `agora_rtc` extension is sending `pcm_frame` data to `deepgram_asr` extension. \ No newline at end of file +In the above example, we are connecting `agora_rtc` extension to `deepgram_asr` extension. The `agora_rtc` extension is sending `pcm_frame` data to `deepgram_asr` extension. From 7be66802fec1d1c22106de5ddae3c0ac8000a952 Mon Sep 17 00:00:00 2001 From: zhangqianze Date: Wed, 25 Dec 2024 11:06:37 +0800 Subject: [PATCH 2/7] feat: add extension folder doc --- docs/SUMMARY.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 131e51553d..40cd115e41 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -11,6 +11,7 @@ * [Getting Started](ten_agent/getting_started.md) * [Project Structure](ten_agent/project_structure/overview.md) * [Understand property.json](ten_agent/project_structure/property_json.md) + * [Understand extension folder](ten_agent/project_structure/extension.md) * [Run Demo](ten_agent/demo/quickstart.md) * [Run Playground](ten_agent/playground/quickstart.md) * [Configure Modules](ten_agent/playground/configure_modules.md) From 5240c46597178935755570fa40c9882f3ebac7ff Mon Sep 17 00:00:00 2001 From: zhangqianze Date: Wed, 25 Dec 2024 14:21:46 +0800 Subject: [PATCH 3/7] feat: add web server docs --- docs/SUMMARY.md | 1 + docs/ten_agent/project_structure/overview.md | 2 +- .../project_structure/property_json.md | 4 + docs/ten_agent/project_structure/server.md | 88 +++++++++++++++++++ 4 files changed, 94 insertions(+), 1 deletion(-) create mode 100644 docs/ten_agent/project_structure/server.md diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 40cd115e41..0c6465c648 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -12,6 +12,7 @@ * [Project Structure](ten_agent/project_structure/overview.md) * [Understand property.json](ten_agent/project_structure/property_json.md) * [Understand extension folder](ten_agent/project_structure/extension.md) + * [Understand web server](ten_agent/project_structure/server.md) * [Run Demo](ten_agent/demo/quickstart.md) * [Run Playground](ten_agent/playground/quickstart.md) * [Configure Modules](ten_agent/playground/configure_modules.md) diff --git a/docs/ten_agent/project_structure/overview.md b/docs/ten_agent/project_structure/overview.md index b6057c24e3..c26aad1b71 100644 --- a/docs/ten_agent/project_structure/overview.md +++ b/docs/ten_agent/project_structure/overview.md @@ -11,4 +11,4 @@ It contains the following important folders and files: - `property.json`: This file contains the orchestration of extensions. It is the main runtime configuration file. - `ten_packages/extension`: This folder contains the extension modules. Each extension module is a separate Python/Golang/C++ package. - `server`: This folder contains the web server code. It is responsible for handling the incoming requests and start/stop of agent processes. -- `playground`: This folder contains the UI code for the playground. It is a web-based interface to interact with the agent. \ No newline at end of file +- `playground`: This folder contains the UI code for the playground. It is a web-based interface to interact with the agent. diff --git a/docs/ten_agent/project_structure/property_json.md b/docs/ten_agent/project_structure/property_json.md index 0216eabc9e..3d04d3c807 100644 --- a/docs/ten_agent/project_structure/property_json.md +++ b/docs/ten_agent/project_structure/property_json.md @@ -7,6 +7,10 @@ This file contains the orchestration of extensions. It is the main runtime confi The `property.json` file contains the following orchestration info, +- Graphs +- Nodes +- Connections + ## Graphs The `predefined_graphs` property contains a list of available graphs. Each graph defines how agent should behave in a specific scenario. Each graph has a `name` property which is used to identify a graph. diff --git a/docs/ten_agent/project_structure/server.md b/docs/ten_agent/project_structure/server.md new file mode 100644 index 0000000000..1cb7428b4e --- /dev/null +++ b/docs/ten_agent/project_structure/server.md @@ -0,0 +1,88 @@ +# Web Server + +The web server is a simple Golang web server that serves the http requests. It is responsible for handling the incoming requests and starting/stopping agent processes. + +The playground/demo UI has been built to interact with the web server. You can also interact with the web server using curl or any other http client. + +## Request & Response Examples + +The server provides a simple layer for managing agent processes. The API resources are described below. + +### API Resources + + - [POST /start](#post-start) + - [POST /stop](#post-stop) + - [POST /ping](#post-ping) + +### POST /start + +This api starts an agent with given graph and override properties. The started agent will join into the specified channel, and subscribe to the uid which your browser/device's rtc use to join. + +| Param | Description | +| -------- | ------- | +| request_id | any uuid for tracing purpose | +| channel_name | channel name, it needs to be the same with the one your browser/device joins, agent needs to stay with your browser/device in the same channel to communicate | +| user_uid | the uid which your browser/device's rtc use to join, agent needs to know your rtc uid to subscribe your audio | +| bot_uid | optional, the uid bot used to join rtc | +| graph_name | the graph to be used when starting agent, will find in property.json | +| properties | additional properties to override in property.json, the override will not change original property.json, only the one agent used to start | +| timeout | determines how long the agent will remain active without receiving any pings. If the timeout is set to `-1`, the agent will not terminate due to inactivity. By default, the timeout is set to 60 seconds, but this can be adjusted using the `WORKER_QUIT_TIMEOUT_SECONDS` variable in your `.env` file. | + +Example: + +```bash +curl 'http://localhost:8080/start' \ + -H 'Content-Type: application/json' \ + --data-raw '{ + "request_id": "c1912182-924c-4d15-a8bb-85063343077c", + "channel_name": "test", + "user_uid": 176573, + "graph_name": "camera_va_openai_azure", + "properties": { + "openai_chatgpt": { + "model": "gpt-4o" + } + } + }' +``` + +### POST /stop + +This api stops the agent you started + +| Param | Description | +| -------- | ------- | +| request_id | any uuid for tracing purpose | +| channel_name | channel name, the one you used to start the agent | + +Example: + +```bash +curl 'http://localhost:8080/stop' \ + -H 'Content-Type: application/json' \ + --data-raw '{ + "request_id": "c1912182-924c-4d15-a8bb-85063343077c", + "channel_name": "test" + }' +``` + + +### POST /ping + +This api sends a ping to the server to indicate connection is still alive. This is not needed if you specify `timeout:-1` when starting the agent, otherwise the agent will quit if not receiving ping after timeout in seconds. + +| Param | Description | +| -------- | ------- | +| request_id | any uuid for tracing purpose | +| channel_name | channel name, the one you used to start the agent | + +Example: + +```bash +curl 'http://localhost:8080/ping' \ + -H 'Content-Type: application/json' \ + --data-raw '{ + "request_id": "c1912182-924c-4d15-a8bb-85063343077c", + "channel_name": "test" + }' +``` From 7ee92d19c3c6a533d519a04484382999db54b090 Mon Sep 17 00:00:00 2001 From: zhangqianze Date: Wed, 25 Dec 2024 14:25:47 +0800 Subject: [PATCH 4/7] feat: fix table of contents --- docs/ten_agent/project_structure/server.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/ten_agent/project_structure/server.md b/docs/ten_agent/project_structure/server.md index 1cb7428b4e..b34914bc2a 100644 --- a/docs/ten_agent/project_structure/server.md +++ b/docs/ten_agent/project_structure/server.md @@ -10,9 +10,9 @@ The server provides a simple layer for managing agent processes. The API resourc ### API Resources - - [POST /start](#post-start) - - [POST /stop](#post-stop) - - [POST /ping](#post-ping) +- [POST /start](#post-start) +- [POST /stop](#post-stop) +- [POST /ping](#post-ping) ### POST /start From f1097edd9a0d84cc1ec025ecb4bfbba62427fc58 Mon Sep 17 00:00:00 2001 From: zhangqianze Date: Wed, 25 Dec 2024 16:11:54 +0800 Subject: [PATCH 5/7] feat: add architecture flow --- docs/ten_agent/architecture_flow.md | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 docs/ten_agent/architecture_flow.md diff --git a/docs/ten_agent/architecture_flow.md b/docs/ten_agent/architecture_flow.md new file mode 100644 index 0000000000..789ae3fb7d --- /dev/null +++ b/docs/ten_agent/architecture_flow.md @@ -0,0 +1,11 @@ +# Architecture Flow + +The whole system contains multiple components that work together to provide a seamless experience. The main components are: + +- **TEN Agent App**: The main application that orchestrates the extensions and manages the data flow between them. It runs as a background process. Depending on the graph configuration, the agent starts the required extensions and handles the data flow between the them. +- **Front-end UI**: A web-based interface to interact with the agent. It allows users to configure the agent, start/stop the agent, and talk to the agent. +- **Web Server**: A simple Golang web server that serves the HTTP requests. It is responsible for handling the incoming requests and starting/stopping agent processes. It also passes parameter like `graph_name` to determine which graph to use. + +The flow of the system is as follows: + +![Architecture Flow](https://github.com/TEN-framework/docs/blob/main/assets/png/architecture_flow.png?raw=true) From 353e3b8d870f87732b349bc47fef30ee3346befc Mon Sep 17 00:00:00 2001 From: zhangqianze Date: Wed, 25 Dec 2024 16:13:28 +0800 Subject: [PATCH 6/7] feat: add into SUMMARY --- docs/SUMMARY.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 0c6465c648..b4a12d8ba2 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -9,6 +9,7 @@ * [Overview](ten_agent/overview.md) * [Getting Started](ten_agent/getting_started.md) +* [Architecture](ten_agent/architecture_flow.md) * [Project Structure](ten_agent/project_structure/overview.md) * [Understand property.json](ten_agent/project_structure/property_json.md) * [Understand extension folder](ten_agent/project_structure/extension.md) From a62f0eedea37b1099739c4807b1c483dd074ad35 Mon Sep 17 00:00:00 2001 From: zhangqianze Date: Wed, 25 Dec 2024 16:15:47 +0800 Subject: [PATCH 7/7] fix: fix typo --- docs/ten_agent/architecture_flow.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/ten_agent/architecture_flow.md b/docs/ten_agent/architecture_flow.md index 789ae3fb7d..a4ad356dc9 100644 --- a/docs/ten_agent/architecture_flow.md +++ b/docs/ten_agent/architecture_flow.md @@ -2,7 +2,7 @@ The whole system contains multiple components that work together to provide a seamless experience. The main components are: -- **TEN Agent App**: The main application that orchestrates the extensions and manages the data flow between them. It runs as a background process. Depending on the graph configuration, the agent starts the required extensions and handles the data flow between the them. +- **TEN Agent App**: The main application that orchestrates the extensions and manages the data flow between them. It runs as a background process. Depending on the graph configuration, the agent starts the required extensions and handles the data flow between them. - **Front-end UI**: A web-based interface to interact with the agent. It allows users to configure the agent, start/stop the agent, and talk to the agent. - **Web Server**: A simple Golang web server that serves the HTTP requests. It is responsible for handling the incoming requests and starting/stopping agent processes. It also passes parameter like `graph_name` to determine which graph to use.