Continued conversation support

[Shulyaka]

Context

Some voice assistants have the continued conversation feature (aka follow-up mode), that allows to ask the user additional questions without the need of a wake word (or press of the button for push-to-talk clients such as HA mobile app).
We currently lack this functionality in Home Assistant, but it is a great usability feature.
This is probably related (in part) to #907

Proposed solution

1. Add new value for homeassistant.helpers.intent.IntentResponseType, i.e. FOLLOW_UP.
2. Add a new function to homeassistant.helpers.intent.IntentResponse class to set this value, i.e. async_set_follow_up(self, is_follow_up: bool)
3. For intent_script integration, we can add a new config option template and we can call async_set_follow_up in async_handle if that template evaluates to True.
4. Other intent providers can do the same thing.
5. Clients (websocket, API, service) should be able to receive the new response type value and act accordingly.

[jaymunro]

This is the kind of support/feature I'd like to see as well.

I'm using extended_open_conversation with OpenAI API and HA Cloud to control the home with natural conversations. I'm blown away with its interpretation of my requests and its choice of function with appropriate parameters for performing the actions.

It's gets even better when using the VA debug feature 'Run audio pipeline'. A full conversation of information requests and commands can be performed with full contextual awareness. Such as:
Is the car charging?
Ok, turn it on.
How much power is it using?
Is it using more than the solar?
Ok, limit it to use less than the solar please.

The LLM just selects the best HA entity it finds to handle the request and does it. The flow is worryingly natural. HA Cloud STT & TTS is so fast and natural that I can't stop demoing it to family & friends.

However, when moving to the normal Assist in the UI (not the debug) all contextual awareness is lost and the conversation is continually interrupted with having to say the wake word before each sentence.

A setting to allow continued conversation would be a great first step to eliminate the need for the wake word all the time. Naturally a timeout and/or an exit phrase(s) would be needed to exit continued conversation and stop listening. This could then lead to a method to enable contextual awareness of the conversation.

[emanuelbaltaretu]

Hi. I am in great need of this. Also, besides a support for continued conversation, a streaming feature would be great regarding speed, check this for context: jekalmin/extended_openai_conversation#58

[jaymunro]

I have this running now and a PR waiting for review. esphome/firmware#173

And a sample video of an interaction at esphome/firmware#174

Add a comment to the PR if you'd like to encourage the maintainers to notice it.

[emanuelbaltaretu]

Gave it a bump! I don't have the hardware sadly and i hope some time it will work for satellites or other gear too in hass

[witold-gren]

Guys, please check this place tetele/onju-voice-satellite#90 It is ready 😉 It is not super fast but easy to configure and works

[formatBCE]

That's clever solution.

But it looks band aid, where surgery needed. Conversation agent ideally should be able to determine lack of information and ask follow-up questions, not just launch listening after TTS finish.

[balloob]

@Shulyaka this proposal is missing a piece. When asking a follow-up question, what will the built-in Home Assistant integration match it against?

For example:

User: broadcast a message
System: what is your message
User: dinner time

User: open the garage door
System: Which garage door?
User: left

Something like #1143 (comment) could help, but it would need better matching architecture then.