Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow agent to have multiple backend instances #2562

Open
2 tasks
achimnol opened this issue Jul 25, 2024 · 0 comments
Open
2 tasks

Allow agent to have multiple backend instances #2562

achimnol opened this issue Jul 25, 2024 · 0 comments
Labels
comp:agent Related to Agent component urgency:1 If no other duties are available, volunteer to help.
Milestone

Comments

@achimnol
Copy link
Member

achimnol commented Jul 25, 2024

@fregataa's contribution of dummy agent backends (#1283) has allowed scalability tests for large-scale clusters.

Though, it is still cumbersome to actually configure and spawn hundreds of agents because we have to copy-and-edit agent.toml and execute ag start-server for hunderds times. Of course, we could automate this process by scripting, but it poses a psychological hurdle to start.

As an extension of #496, let's allow a single agent to host multiple agent backend instances from a single configuration file.

  • Allow Jinja templating of agent.toml to easily populate multiple backend configurations (including dummies).
    • Expand the structure of agent's local-config to handle multiple backend instances.
    • Store the rendered config file to /tmp (while parsing it in memory), so that it could be inspected by engineers and admins.
    • Let's keep backward-compatibility to existing agent.toml files.
  • Pass the agent ID when invoking the agent RPC calls and route the call to the designated backend instance.
    • From the manager's perspective, there are multiple agents sharing the same physical RPC endpoint.

For "real" backends, we need to resolve #496 first to get it actually working, but for "dummy" backends we can start right now.

  • In production, there won't be more than 2-3 backend instances per agent and only single one in most cases.
  • In scalability testing, there may be hundreds and thousands of backend instances multiplexed via asyncio in a single process and single thread.
@achimnol achimnol added type:feature Add new features comp:agent Related to Agent component urgency:1 If no other duties are available, volunteer to help. labels Jul 25, 2024
@achimnol achimnol added this to the 24.09 milestone Jul 25, 2024
@achimnol achimnol removed the type:feature Add new features label Oct 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
comp:agent Related to Agent component urgency:1 If no other duties are available, volunteer to help.
Projects
None yet
Development

No branches or pull requests

1 participant