Python: SK Multiagents - working for ChatCompletionAgents but not for AzureAssistantAgent #10084
Comments
zzulueta
changed the title
|
Hello @zzulueta could you give us some more information around:
Is there a stack trace? An error? Which Azure OpenAI api version are you using? I need to look more closely through your code, but my initial hunch is that there is a difference with how Azure is handling the Can you turn on some warning or error level debugging in your script, please, if you aren't seeing any errors propagated right now? You can add: import logging
logging.basicConfig(level=logging.DEBUG) |
moonbox3
added
python
github-actions
bot
changed the title
|
This is what came out: My Azure OpenAI api version is 2024-10-01-preview. My code for ChatCompletionAgents and AzureAssistantAgent identical. The only difference is how the Agents were individually created. Two ChatCompletionAgents work. But when one is converted into an AzureAssistantAgent upon creation, it just fails. |
|
Thanks for the additional context, @zzulueta. Let me spend some time on this, and I will get back to you ASAP. |
|
I haven't had a chance to dig into this more. But looking at the code, we should get more of an exception or a stack trace if we fail to select an agent: try:
selected_agent = await self.selection_strategy.next(self.agents, self.history.messages)
except Exception as ex:
logger.error(f"Failed to select agent: {ex}")
raise AgentChatException("Failed to select agent") from ex |
|
Sorry, where should I put this new code? |
|
This is the current code -- I was pointing out that we should get more of an error and we even chain the exception to the AgentChatException. |
|
@zzulueta any chance you can give me access to your image generator plugin (even private via GitHub)? I want to reproduce your issue and want to make sure I am looking at the same code as you. Thanks. |
|
@zzulueta in any case, I create some dummy plugins just to see how the selection strategy was. I'm not sure if when you are running the Azure Assistants code, you are running from one script right now, and if any code is commented while doing that, but the one thing that is important, and not shown in your code is how you create the kernel to add to the assistant. Are you making sure to add an It's something like this: def _create_kernel_with_chat_completion(service_id: str) -> Kernel:
kernel = Kernel()
kernel.add_service(AzureChatCompletion(service_id=service_id)) # without this line, the selection or termination function cannot run
return kernelHere's my import asyncio
import os
from typing import Annotated
from semantic_kernel.agents import AgentGroupChat
from semantic_kernel.agents.open_ai import AzureAssistantAgent
from semantic_kernel.agents.strategies import (
KernelFunctionSelectionStrategy,
KernelFunctionTerminationStrategy,
)
from semantic_kernel.connectors.ai.open_ai import AzureChatCompletion
from semantic_kernel.contents import AuthorRole, ChatMessageContent
from semantic_kernel.functions import KernelFunctionFromPrompt, kernel_function
from semantic_kernel.kernel import Kernel
MANAGER_NAME = "manager"
DALLE_NAME = "dalle"
VISION_NAME = "vision"
TERMINATION_KEYWORD = "END"
def _create_kernel_with_chat_completion(service_id: str) -> Kernel:
kernel = Kernel()
kernel.add_service(AzureChatCompletion(service_id=service_id))
return kernel
class GenerateImagePlugin:
@kernel_function(description="Generates an image based on the user's prompt.")
def generate_image(
self, prompt: Annotated[str, "The prompt for the image generation."]
) -> Annotated[str, "Returns the generated image."]:
return "An image of a cat."
class AnalyzeImagePlugin:
@kernel_function(description="Analyzes an image and provides feedback.")
def analyze_image(
self, image: Annotated[str, "The image to analyze."]
) -> Annotated[str, "Returns the analysis of the image."]:
return "The image is of a cat."
# AzureAssistantAgent - works in this code###
# Manager Assistant
async def main():
manager_kernel = _create_kernel_with_chat_completion(MANAGER_NAME)
manager_agent = await AzureAssistantAgent.create(
service_id=MANAGER_NAME,
kernel=manager_kernel,
name=MANAGER_NAME,
instructions=f"""
An image generated by {DALLE_NAME} and an analysis by {VISION_NAME} is considered as a single conversation.
You will monitor the conversation between the {DALLE_NAME} and the {VISION_NAME} and count the number of conversations.
Once four (4) conversations are completed, you will provide a termination message: {TERMINATION_KEYWORD}
You will not provide any other input or output to the conversation other than the termination message.
""",
)
# DALLE Assistant
dalle_assistant_kernel = _create_kernel_with_chat_completion(DALLE_NAME)
dalle_assistant_kernel.add_plugin(GenerateImagePlugin(), plugin_name="GenerateImagePlugin")
dalle_assistant_agent = await AzureAssistantAgent.create(
service_id=DALLE_NAME,
kernel=dalle_assistant_kernel,
name=DALLE_NAME,
instructions="""
As a premier AI specializing in image generation, you possess the expertise to craft precise visuals based on given prompts.
It is essential that you diligently generate the requested image, ensuring its accuracy and alignment with the user's specifications,
prior to delivering a response.
You will have access to the local file system to store the generated image.
You will generate an image based on the user's prompt and display it for review.
You will generate new images based on the feedback from the Vision Assistant.
""",
# execution_settings=settings,
)
# Vision Assistant
vision_assistant_kernel = _create_kernel_with_chat_completion(VISION_NAME)
vision_assistant_kernel.add_plugin(AnalyzeImagePlugin(), plugin_name="AnalyzeImagePlugin")
vision_assistant_agent = await AzureAssistantAgent.create(
service_id=VISION_NAME,
kernel=vision_assistant_kernel,
name=VISION_NAME,
instructions="""
As a leading AI expert in image analysis, you excel at scrutinizing and offering critiques to refine and improve images.
Your task is to thoroughly analyze an image, ensuring that all essential assessments are completed with precision
before you provide feedback to the user. You have access to the local file system where the image is stored.
You will analyze the image and provide a new prompt for Dall-e that enhances the image based on the criticism and analysis.
You will then instruct the Dall-e Assistant to generate a new image based on the new prompt.
""",
# execution_settings=settings,
)
selection_function = KernelFunctionFromPrompt(
function_name="selection",
prompt=f"""
Determine which participant takes the next turn in a conversation based on the most recent participant.
State only the name of the participant to take the next turn.
Choose only from these participants:
- {DALLE_NAME}
- {VISION_NAME}
- {MANAGER_NAME}
You will follow this sequence:
{DALLE_NAME} will generate an image based on the initial user prompt and display it for review.
{VISION_NAME} will analyze the image and provide a new prompt for {DALLE_NAME} to generate a new image based on the new prompt.
{MANAGER_NAME} will monitor the conversation between {DALLE_NAME} and {VISION_NAME} and count the number of conversations.
{DALLE_NAME} will generate an image based on the {VISION_NAME} prompt and display it for review.
{VISION_NAME} will analyze the image and provide a new prompt for {DALLE_NAME} to generate a new image based on the new prompt.
{MANAGER_NAME} will monitor the conversation between {DALLE_NAME} and {VISION_NAME} and count the number of conversations.
{DALLE_NAME} will generate an image based on the {VISION_NAME} prompt and display it for review.
{VISION_NAME} will analyze the image and provide a new prompt for {DALLE_NAME} to generate a new image based on the new prompt.
{MANAGER_NAME} will monitor the conversation between {DALLE_NAME} and {VISION_NAME} and count the number of conversations.
No participant should take more than one turn in a row.
History:
{{{{$history}}}}
""",
)
termination_function = KernelFunctionFromPrompt(
function_name="termination",
prompt=f"""
Determine if the conversation should be terminated based on the number of iterations.
If number of iterations is reached, respond with the termination keyword: {TERMINATION_KEYWORD}
RESPONSE:
{{{{$history}}}}""",
)
chat = AgentGroupChat(
agents=[dalle_assistant_agent, vision_assistant_agent, manager_agent],
selection_strategy=KernelFunctionSelectionStrategy(
function=selection_function,
kernel=_create_kernel_with_chat_completion("selection"),
result_parser=lambda result: str(result.value[0])
if result.value is not None
else DALLE_NAME or VISION_NAME,
agent_variable_name="agents",
history_variable_name="history",
),
termination_strategy=KernelFunctionTerminationStrategy(
agents=[manager_agent],
function=termination_function,
kernel=_create_kernel_with_chat_completion("termination"),
result_parser=lambda result: TERMINATION_KEYWORD in str(result.value[0]).lower(),
history_variable_name="history",
maximum_iterations=10,
),
)
is_complete: bool = False
while not is_complete:
user_input = input("User:> ")
if not user_input:
continue
if user_input.lower() == "exit":
is_complete = True
break
if user_input.lower() == "reset":
await chat.reset()
print("[Conversation has been reset]")
continue
await chat.add_chat_message(ChatMessageContent(role=AuthorRole.USER, content=user_input))
try:
async for response in chat.invoke():
print(os.linesep)
print(f"# {response.role} - {response.name or '*'}: '{response.content}'")
if chat.is_complete:
is_complete = True
break
except Exception as e:
print(f"Error: {e}")
break
finally:
await manager_agent.delete()
await dalle_assistant_agent.delete()
await vision_assistant_agent.delete()
if __name__ == "__main__":
asyncio.run(main())with the following sample output: |
|
@moonbox3 yes I assign an AzureChatCompletion service. I actually used your code there from other examples. from semantic_kernel.connectors.ai.open_ai.services.azure_chat_completion import AzureChatCompletion def _create_kernel_with_chat_completion(service_id: str) -> Kernel: |
|
@moonbox3 i added you as a collaborator to the whole notebook There are two notebooks: |
|
@zzulueta Thanks for adding me. I see there is a stack trace in your notebook -- I hadn't seen that before. The following bug was just recently fixed: Can you please try to upgrade your SK Python version to 1.18.1? I was able to run your code without issues, and the agent selection function is working without error. A couple of unsolicited thoughts for you, too:
|
|
If you continue to hit an issue after upgrading to 1.18.1, please re-open the issue. Otherwise, with your code, I've been able to successfully run it all and it's working on the latest SK Python code. |
|
@moonbox3 Thanks for the inputs. In the ChatCompletionAgent scenario, I added a tool for the Manager to count the number of images created. In addition, I modified everything to the AsyncAzureOpenAI. It works perfectly now even the image count. However, the AzureAssistantAgent scenario still does not work properly. The system was able to hand off from the Dalle Assistant to the Vision assistant, but there is a problem bringing the conversation back to the Dalle Assistant to create a new image. |
Can you please elaborate on this? What do you mean by "there is a problem?" An error? A stack trace? The model doesn't hand the conversation back to the Dalle assistant? |
|
You can try running the system. Here is my output below. As you can see, the Assistant should be selecting the Dalle Assistant with the new prompt. Unfortunately, the Vision assistant keeps on being called. Dalle Assistant Message: /workspaces/AzureAI/SemanticKernelAgents/temp.jpg Please review the image and let me know if there are any adjustments or additional details you'd like to include.' CritiqueWhile the image is visually striking, the colors might appear slightly oversaturated, potentially giving it an unreal aesthetic that might detract from the natural beauty of the scene. The central composition of the boat and the sun directly above may seem a bit too symmetric, which could be perceived as less dynamic. Enhanced Prompt for DALL-E"Create an image of an old wooden rowboat gently floating on a crystal clear mountain lake during sunset. The sky is a blend of soft pink, purple, and orange tones, reflecting subtly on the rippling water surface. Surrounding the lake are gentle slopes covered with wildflowers and sparse trees. The scene radiates a peaceful, almost ethereal quality, capturing the essence of a tranquil evening without oversaturation, incorporating a more dynamic asymmetrical composition that leads the viewer’s eye through the scene." The reflections in the water are also praisely executed, enhancing the overall calmness and depth of the scene. The sun, placed strategically on the horizon, serves as a focal point and balances the image by its positioning directly opposite the boat. However, the image could be criticized for perhaps being a bit too idealized in its portrayal of nature. The saturation and contrast might be seen as excessive, making the scene feel more like a painting than a photograph. This might detach the viewer from feeling a connection with what could be a real-world setting. Enhanced Prompt for DALL-E"With the above critiques in mind, generate an image of a solitary wooden boat on a smooth lake at sunset. Adjust the color palette to include softer and more natural shades of red, orange, and purple blending harmoniously into the darkening sky. The sun should be depicted just touching the horizon, softening its glare and enhancing the reflection on the water to depict a real-world scenario. Include subtle ripples in the lake to add a touch of realism, avoiding an overly smooth texture on the water surface. The composition should maintain a balanced distribution of elements with the boat tastefully offset rather than centered." "Create an image of an old wooden rowboat gently floating on a crystal clear mountain lake during sunset. The sky is a blend of soft pink, purple, and orange tones, reflecting subtly on the rippling water surface. Surrounding the lake are gentle slopes covered with wildflowers and sparse trees. The scene radiates a peaceful, almost ethereal quality, capturing the essence of a tranquil evening without oversaturation, incorporating a more dynamic asymmetrical composition that leads the viewer’s eye through the scene." I will now instruct the Dall-e Assistant to generate a new image based on this enhanced prompt.' This image depicts a beautifully serene scene with a solitary boat floating on what appears to be a smooth body of water. The setting sun forms a perfect circle on the horizon, casting a vibrant mix of red and orange hues across the sky and water. The reflections in the water are particularly striking, mirroring the intense colors of the sky and hinting at a slight ripple that adds texture to the scene. Criticism:
Enhanced Prompt for DALL-E Based on Criticism: "Create an image of a solitary boat floating on a calm lake at sunset with a slightly less saturated and more natural palette of reds, oranges, and purples. The sun should be placed at the intersection according to the rule of thirds for dynamic composition. Include a clear and detailed distant mountain range under a sky with realistic cloud formations, adding depth and contrast to the serene water that reflects a more subtly vibrant sunset." AuthorRole.ASSISTANT - VisionAssistant: 'Based on the analysis and critique, I have crafted a further refined prompt to enhance the image:"Generate an image of a solitary wooden boat on a smooth lake at sunset. Adjust the color palette to include softer and more natural shades of red, orange, and purple blending harmoniously into the darkening sky. The sun should be depicted just touching the horizon, softening its glare and enhancing the reflection on the water to depict a real-world scenario. Include subtle ripples in the lake to add a touch of realism, avoiding an overly smooth texture on the water surface. The composition should maintain a balanced distribution of elements with the boat tastefully offset rather than centered." I will now instruct the Dall-e Assistant to generate a new image based on this refined prompt.' The image depicts a serene sunset scene featuring a lone boat gently floating on a mirror-like body of water, under a deep red sky. The overall composition and the use of vibrant colors are visually striking, creating a sense of calm and isolation. Strengths:
Areas for Improvement:
Enhanced Prompt for DALL-E"Create an image of a small boat floating on a glass-like lake at sunset. The sky should be a gradient of deep red to orange, reflecting vividly on the water. Add more texture to the boat and the distant shoreline, emphasizing realism in the materials. Introduce subtle shadows on the boat and in the ripples of the water to enhance depth and dimensional contrast, maintaining a balance between realism and a tranquil, painterly aesthetic." "Create an image of a solitary boat floating on a calm lake at sunset with a slightly less saturated and more natural palette of reds, oranges, and purples. The sun should be placed at the intersection according to the rule of thirds for dynamic composition. Include a clear and detailed distant mountain range under a sky with realistic cloud formations, adding depth and contrast to the serene water that reflects a more subtly vibrant sunset." ' Strengths:
Areas for Improvement or Variation:
Enhanced Prompt for DALL-E:"Create an image of a small, traditional wooden boat gently floating on a calm lake during sunset. The lake reflects a vibrant mix of deep reds and oranges from the sunset sky, which features light cloud wisps and dynamic rays of sunlight piercing through. Add a slight ripple effect around the boat suggesting a soft breeze. Ensure the scene conveys a serene yet subtly dynamic atmosphere, rich in color with a balanced composition using the rule of thirds." This enhanced prompt suggests slight changes that maintain the serene mood while introducing elements that could make the scene even more engaging and visually diverse. "Create an image of a small boat floating on a glass-like lake at sunset. The sky should be a gradient of deep red to orange, reflecting vividly on the water. Add more texture to the boat and the distant shoreline, emphasizing realism in the materials. Introduce subtle shadows on the boat and in the ripples of the water to enhance depth and dimensional contrast, maintaining a balance between realism and a tranquil, painterly aesthetic." Let's generate the final image based on this enhanced prompt.' |
|
My advice is to play around with the selection criteria some more. Otherwise, just do a sequential selection strategy for your three agents, and when you get to the 4th agent, configure the termination to handle it. |
|
How do you do a Sequential selection strategy in SK? It was very easy with Autogen with the SelectorGroupChat and Swarm. There doesn't seem to be an equivalent in SK. |
|
@zzulueta the sequential strategy is configured by default (the selection strategy is optional, so if one is not configured we use the sequential strategy). |
|
@moonbox3 thanks! I must have missed this in the MS Learn documentation. |
I have a multi-agent system that works with the ChatCompletionAgent but does not work for AzureAssistantAgent as shown below. If I switch my agents to an AzureAssistantAgent version, the system does not want to handoff to the next agent. Any idea why?
The text was updated successfully, but these errors were encountered: