Expose media_player.async_browse_media as a service

[PeteRager]

The topic of this discussion is to consider extending the entity model to expose the existing media_player.async_browse_media method as a service call. This method call is currently exists and is used by the media browser through a websocket interface.

The change is to expose this as a service call with a return value, There is a draft pr here:
home-assistant/core#116452

The purpose is to allow automations to access the service to browse for media (just as you can do from the UI) and then use the return values to do other actions. As an example, here is an automation that gets a list of the albums by the Beatles and randomly plays one. This is using a Sonos music library.

alias: play_beatles
description: ""
trigger: []
condition: []
action:
  - service: media_player.browse_media
    metadata: {}
    data:
      media_content_id: A:ALBUMARTIST/Beatles
      media_content_type: album
    response_variable: albums
    target:
      entity_id: media_player.master_bedroom_2
  - variables:
      num_children: "{{ albums['media_player.master_bedroom_2']['children'] | length }}"
      random_album_index: "{{ range(1, num_children - 1) | random }}"
      random_album_media_id: >-
        {{
        albums['media_player.master_bedroom_2']['children'][random_album_index]['media_content_id']
        }}
  - service: media_player.play_media
    target:
      entity_id: media_player.master_bedroom_2
    data:
      media_content_id: "{{ random_album_media_id }}"
      media_content_type: album
    metadata: {}
mode: single

Here is a sample of the YAML returned when running the browse_media service from developer tools / services

media_player.master_bedroom_2:
  title: Beatles
  media_class: album
  media_content_type: album
  media_content_id: A:ALBUMARTIST/Beatles
  children_media_class: directory
  can_play: false
  can_expand: false
  thumbnail: null
  not_shown: 0
  children:
    - title: All
      media_class: artist
      media_content_type: artist
      media_content_id: A:ALBUMARTIST/Beatles/
      children_media_class: null
      can_play: true
      can_expand: true
      thumbnail: null
    - title: A Hard Day's Night
      media_class: album
      media_content_type: album
      media_content_id: A:ALBUMARTIST/Beatles/A%20Hard%20Day's%20Night
      children_media_class: null
      can_play: true
      can_expand: true
      thumbnail: >-
        http://192.168.50.40:1400/getaa?u=x-file-cifs://192.168.50.70/music/iTunes/iTunes%20Media/Music/The%20Beatles/A%20Hard%20Day's%20Night/01%20A%20Hard%20Day's%20Night%201.m4a&v=53
    - title: Abbey Road
      media_class: album
      media_content_type: album
      media_content_id: A:ALBUMARTIST/Beatles/Abbey%20Road
      children_media_class: null
      can_play: true
      can_expand: true
      thumbnail: >-
        http://192.168.50.40:1400/getaa?u=x-file-cifs://192.168.50.70/music/iTunes%20Music/The%20Beatles/Abbey%20Road/01%20Come%20Together.mp3&v=53

[allenporter]

I'm interested in seeing this as a potential option for browsing media for future input into multi-modal LLMs.

A couple thoughts:

• Can this should support media sources rather than media players only so that it can support a richer set of objects? The interface could take a string media content id and otherwise look similar to what is proposed.
• Can we limit the surface area of the response objects to just what is absolutely necessary for some initial use cases? The BrowseMedia is large-ish

[PeteRager]

• Can this should support media sources rather than media players only so that it can support a richer set of objects? The interface could take a string media content id and otherwise look similar to what is proposed.

It could. Right now, media sources do not have entities so we may want to expose them as a new entity type "media_source." to make them easy to find. The media browser for Sonos does not expose its library as a media source because they can only be played from Sonos. In that case the browse is through the media_player.

So I think we'd want both - media players and media sources to be able to expose the same service?

• Can we limit the surface area of the response objects to just what is absolutely necessary for some initial use cases? The BrowseMedia is large-ish

We can. One of the nice things about the prototype is it required no code change; just exposing the existing interface as a service. Such that all the existing media player browse data is available without having to update every integration.

If we want to provide less back, we should do this in a layer between the service call and the integration - such that it calls the existing interface in the integration and slims down the result?

[allenporter]

• Can this should support media sources rather than media players only so that it can support a richer set of objects? The interface could take a string media content id and otherwise look similar to what is proposed.

It could. Right now, media sources do not have entities so we may want to expose them as a new entity type "media_source." to make them easy to find. The media browser for Sonos does not expose its library as a media source because they can only be played from Sonos. In that case the browse is through the media_player.

So I think we'd want both - media players and media sources to be able to expose the same service?

I just happened to come across #675 which is an interesting discussion about why media sources aren't necessarily needed as entities, and I think the same may apply here. An entity isn't really needed for any other reason and its not polled and has no state given it's all user driven on demand. Maybe, then, it implies some divergence in how these are handled. I noticed the websockets appear to be separate so maybe any services (now actions) would be separate here too, but follow a similar pattern.

[peteS-UK]

We were discussing a media library search service for the squeezebox integration at home-assistant/core#127824 and @joostlek asked me to comment over here .....

Could you maybe explain your thoughts on the media_content_id structure. The media browser architecture is a browser unless I've missed something, so you're always navigating through a set of known content and their ids. For a service like this to be useful, wouldn't you need to know the media_content_id and the structure of that? I think that's end point specific, and in the case of the squeezebox integration, is a numeric string - e.g. an artist id might be 121423.

I think we might need async_search_media - something to which you could pass a search string and media content type. This I think would then let us expose a more useful service, but could also be implemented in the media browser UI itself, which would greatly increase useability of the browser - I have over a thousand of albums with 10's of thousands of tracks and the current browser ui without search isn't useable for me. Search would make a big difference to its usefulness IMHO.

[rajlaud]

Could you maybe explain your thoughts on the media_content_id structure. The media browser architecture is a browser unless I've missed something, so you're always navigating through a set of known content and their ids. For a service like this to be useful, wouldn't you need to know the media_content_id and the structure of that? I think that's end point specific, and in the case of the squeezebox integration, is a numeric string - e.g. an artist id might be 121423.

What about "search" as a media_content_type with a standard search format as the "media_content_id"? The results could then be displayed in the media browser or used in scripts if async_browse_media is exposed as a service as proposed here.

[peteS-UK]

Could you maybe explain your thoughts on the media_content_id structure. The media browser architecture is a browser unless I've missed something, so you're always navigating through a set of known content and their ids. For a service like this to be useful, wouldn't you need to know the media_content_id and the structure of that? I think that's end point specific, and in the case of the squeezebox integration, is a numeric string - e.g. an artist id might be 121423.

What about "search" as a media_content_type with a standard search format as the "media_content_id"? The results could then be displayed in the media browser or used in scripts if async_browse_media is exposed as a service as proposed here.

Wouldn't it be easier to integration owners to maintain if there was just a async_search_media function - less confusing than having to know a special standard format... Since there's no UI in the media browser to enter a search string, work would have to be done in that UI anyway.

[PeteRager]

I like this idea of extending the existing browse interface to support search and then expose the browse interface as an action.

[PeteRager]

I like the idea of search as an action. You are correct that the existing browse interface is a hierarchy that you navigate and get the content ids. If you specify no content id the integration will return its top level content ids. The use cases that i'd like to enable (play a random album by the Beatles) would work with either search or browse. Certainly there are a set of use cases that would not work well with browse (find all songs with the word love in it)

The advantage of the browse interface is that it is already implemented by many integrations and we'd just need to expose the existing interface as a service and then the existing implementations just work.

However, if the decision is to create a search interface. Id be happy to Implement that for Sonos.

[peteS-UK]

Hi @PeteRager . I think the benefit you have over some other integrations (inc. squeezebox) is that the structure of your media_content_id's are meaningful. So, you can find all the albums by "Beatles", whereas for squeezebox, I'd have to find all albums for albumid=123456. So, without a standard media_content_id structure, which we don't have since it varies by endpoint, I'm not sure how useful a generic service is - for the service to be generic, media_content_id structure would also need to be generic I think?

My thinking around search is that it's pretty much needed in the browser UI anyway, so perhaps defining the mechanism to add a search function could enable that in the UI, but also then be exposed as a service.

[PeteRager]

A simple approach could be for the top level browse to return a search item. It would have a media_type of search - this would be a new media type. The UI could then use this to determine if the integration supports media search, and then we can determine how to express search strings (content_ids) that search for media_types (e.g. search for an album, artists, tracks, etc.) You could pass the search content id to play_media to play a search result.

[peteS-UK]

But, we don't have a way to input the search string currently in the UI, so aren't we currently stuck anyhow?

[PeteRager]

The first step is to obtain architectural approval to add this the to media browse domain interfaces and expose as actions.

[emontnemery]

We discussed the proposal in the core team and the proposal is approved.
I've reopened + unlocked the implementation PR.

[PeteRager]

Thank you. I've updated the PR and it ready to start reviewing.

[MartinHjelmare]

Implemented in home-assistant/core#116452