Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wyoming: context in Transcript object is discarded #34

Open
OverloadUT opened this issue Nov 30, 2024 · 2 comments
Open

Wyoming: context in Transcript object is discarded #34

OverloadUT opened this issue Nov 30, 2024 · 2 comments
Assignees

Comments

@OverloadUT
Copy link

OverloadUT commented Nov 30, 2024

Observation:

The wyoming.asr.Transcript object takes a context parameter, but it does not end up serialized in to the Event, or deserialized on the other side, so the data is lost in the Wyoming communication layer.

Expected Behavior

The context passed to the Transcript constructor should end up transmitted to the server and available for use.

Note: I notice that the Transcribe object appears to have the same issue, although I'm not using that part of the code so I'm less confident about its intentions.

Relevant code:

@dataclass
class Transcript(Eventable):
    """Transcription response from ASR system"""

    text: str
    """Text transcription of spoken audio"""

    context: Optional[Dict[str, Any]] = None # <------ The context object is available at construction
    """Context for next interaction."""

    @staticmethod
    def is_type(event_type: str) -> bool:
        return event_type == _TRANSCRIPT_TYPE

    def event(self) -> Event:
        return Event(type=_TRANSCRIPT_TYPE, data={"text": self.text}) # <------ The context dict is not serialized

    @staticmethod
    def from_event(event: Event) -> "Transcript":
        assert event.data is not None
        return Transcript(text=event.data["text"]) # <-------- The context dict is not de-serialized

Side note for HA:

It would be great if Home Assistant would use this object to provide other important context for handling the intent. Right now it adds a conversation_id uuid, but the text "context" (previous conversational bits) as well as the device_id and agent_id would also be very valuable for providing more tailored intent handling.

Partially putting this here to validate that it's within the spirit of the Wyoming protocol to allow arbitrary data in the context dict, which could be implementation-specific, yeah?

@synesthesiam synesthesiam self-assigned this Dec 3, 2024
@synesthesiam
Copy link
Contributor

Thanks for bringing this up! Context ended up being unfinished since a workaround was found.
I definitely planned for context to contain arbitrary data, and to get copied between input/output events within each service.
I will get it finished here and then update it on the HA side 👍

@OverloadUT
Copy link
Author

Yay! I've got a draft PR open on the home-assistant side here: home-assistant/core#132235

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants