Extend REST API to return service response data

[iamjackg]

Original content

Hey y'all,

When service responses were introduced, the /api/services/<domain>/<service> endpoint for the REST API was not updated to support this. Calling a service that returns responses using this endpoint currently results in a 500, with the actual error message only visible in the hass logs.

To put some more fire under us, now that weather forecast attributes have been deprecated, service calls are the only way to retrieve weather forecast data, but this doesn't work through the API. This leaves users with no way to retrieve it. See home-assistant/core#99820

I originally opened this PR to address this, and was kindly redirected here to have a conversation about it.

Current state

Currently, calling /api/services/<domain>/<service> returns a list of all the states that changed in the span of time it took for the service to finish executing. (This was always kinda kludgy, but that's a different conversation)

My draft solution

In order not to break backwards compatibility, my implementation added a return_response parameter. If the parameter is present in the query (?return_response) or in the JSON body ("return_response": true) the API call will return the service response data instead of the usual list of changed states.

If the service is SupportsResponse.ONLY and the parameter is not present, the API will return an error.
If the service is SupportsResponse.NONE and the parameter is present, the API will return an error.

Alternatives

@bdraco suggested creating an entirely new API endpoint for this, since the response format would be drastically different based on the presence of the parameter.

I'm not particularly committed to one choice or the other: either way the user will have to know in advance if the service supports returning data or not, because they will need to add the parameter or call another endpoint.

Thoughts?

First rewrite

Context

When service responses were introduced, the /api/services/<domain>/<service> endpoint for the REST API was not updated to support this. Calling a service that returns responses using this endpoint currently results in a 500, with the actual error message only visible in the Home Assistant logs.

Now that weather forecast attributes have been deprecated, service calls are the only way to retrieve weather forecast data, but this doesn't work through the API. This leaves users with no way to retrieve it. See home-assistant/core#99820 as an example.

Exposing this through the API is fairly uncontroversial, but deciding where and how to expose it has generated a bit of discussion.

The original proposal was to add a ?return_response query parameter or JSON key-value, which would cause the endpoint to return the response data instead of the state changes that occurred while running the service call. Various reviewers were concerned about changing the output format based on a parameter.

The original behaviour of returning all state changes (even ones unrelated to the service call) was already weird at best when it was first added. However, I'm sure plenty of people rely on it, so I don't want to get rid of it.

At the same time, creating an entirely new endpoint for this feels equally kludgy:

• /api/services_with_response_data/<domain>/<service> is a bit odd
• /api/services/<domain>/<service>/response_data is inconsistent, since there are currently no other API endpoints that introduce deeper paths to retrieve different data

Lastly, people might want to have access to both the response data and the set of state changes.

Proposal

Deprecate the current response format

We can follow whatever the current HASS deprecation timeline is. API calls in their current form will keep working exactly as they do now.

Introduce a new response format

"New" calls will return a response using the following format:

{
    "state_changes": [
        {
            "attributes": {},
            "entity_id": "sun.sun",
            "last_changed": "2016-05-30T21:43:32.418320+00:00",
            "state": "below_horizon"
        },
        {
            "attributes": {},
            "entity_id": "process.Dropbox",
            "last_changed": "22016-05-30T21:43:32.418320+00:00",
            "state": "on"
        }
    ],
    "service_response": {
        "forecast": [
          { 
            "datetime": "2024-04-10T08:57:12.151445-04:00",
            "condition": "rainy",
            "precipitation_probability": 30,
            "temperature": 18,
            "templow": 8
          },
          { 
            "datetime": "2024-04-11T08:57:12.151493-04:00",
            "condition": "rainy",
            "precipitation_probability": 0,
            "temperature": 20,
            "templow": 8
          }
        ]
    }
}

Behaviour during deprecation phase

• If the service is SupportsResponse.NONE
  • Calls to /api/services/<domain>/<service> will return a response in the current format (i.e. only a list of changed states)
• If the service is SupportsResponse.OPTIONAL
  • Calls to /api/services/<domain>/<service> will by default return a response in the current format (i.e. only a list of changed states)
  • Adding a ?return_response query parameter or JSON item will instead return the new format
• If the service is SupportsResponse.ONLY
  • Calls to /api/services/<domain>/<service> will return a response in the new format
  • (in its current state, this call simply returns a 500)

It's not pretty, but it will maintain the status quo until we feel comfortable with removing the old behaviour altogether.

Alternative: we introduce a new API version altogether, i.e. /api/v2/services/<domain>/<service>

Behaviour after deprecation

All API calls will return the new format, and service_response will be null if the service did not return a response.

Consequences

• People will have to be notified that this is changing (not sure what we do with REST API changes: the API is not even documented in the "normal" hass docs: it's in the developer docs)
• Behaviour in the interim will be slightly messy

Context

When service responses were introduced, the /api/services/<domain>/<service> endpoint for the REST API was not updated to support this. Calling a service that returns responses using this endpoint currently results in a 500, with the actual error message only visible in the Home Assistant logs.

Now that weather forecast attributes have been deprecated, service calls are the only way to retrieve weather forecast data, but this doesn't work through the API. This leaves users with no way to retrieve it. See home-assistant/core#99820 as an example.

Proposal

We will introduce a new endpoint: /api/services_with_response_data/<domain>/<service>.

Request format

The endpoint maintains the same request semantics as /api/services/<domain>/<service>, i.e. a POST request, and

You can pass an optional JSON object to be used as service_data.

Response format

If the called service is either SupportsResponse.OPTIONAL or SupportsResponse.ONLY, the endpoint will return a JSON object with the following format, which contains both state changes and response data from the service:

{
  "state_changes": [
    {
      "attributes": {},
      "entity_id": "sun.sun",
      "last_changed": "2016-05-30T21:43:32.418320+00:00",
      "state": "below_horizon"
    },
    {
      "attributes": {},
      "entity_id": "process.Dropbox",
      "last_changed": "22016-05-30T21:43:32.418320+00:00",
      "state": "on"
    }
  ],
  "service_response": {
    "weather.new_york_forecast": {
      "forecast": [
        {
          "condition": "clear-night",
          "datetime": "2024-04-22T20:45:55.173725-04:00",
          "precipitation_probability": 0,
          "temperature": null,
          "templow": 6.0
        },
        {
          "condition": "rainy",
          "datetime": "2024-04-23T20:45:55.173756-04:00",
          "precipitation_probability": 60,
          "temperature": 16.0,
          "templow": 4.0
        }
      ]
    }
  }
}

If the service is SupportsResponse.NONE, the endpoint will return a 400 Bad Request error and this JSON response:

{"message": "Service does not support response data."}

Consequences

• The documentation should be updated

[Klikini]

I was just bitten by this today (see "user story" below) and I was considering implementing it myself before I found your PR with all the work done already, so thank you!

As a REST API consumer, the separate endpoint vs. parameter issue doesn't matter that much, as long as this feature gets added. Though since that's what we're here to talk about, I do have some opinions to share:

1. Supporting the query parameter is unnecessary. This will always be a POST request, so adding "return_response": true to the JSON request body will always be possible.
2. If it makes the code cleaner, I could understand a separate endpoint like POST /api/services/<domain>/<service>/return_response, but it's just as verbose (and confusingly named; you're getting a response either way) as POST /api/services/<domain>/<service>?return_response.

User story: The removal of the weather forecast attribute in 2024.4 broke my wall-mounted weather forecast display. When I figured out why, I switched over to making a service call with the REST API, since my application is already using that. But then I found out that I can't get service response data with the REST API, and I really don't want to implement the WebSocket authentication handshake for such a simple application because (to my knowledge) HA doesn't support reading the API token from the pre-connection-upgrade Authorization header.

[sebsebseb1982]

I'm in the same situation ! My ePaper dashboard does not work anymore

[allenporter]

@bdraco suggested creating an entirely new API endpoint for this, since the response format would be drastically different based on the presence of the parameter.

So far, this discussion is just restating what is in the PR, so I think making an actual proposal here would be useful. Getting into another level of detail here: The existing API returns a list of state changes and the service response output is a dictionary (output format described in option 3 here #948). I agree its odd to change the format so much based on the input parameters.

Can you make a proposal for a new endpoint?

[allenporter]

This addresses what I was looking for, to leave the existing service api alone. Two small things:

• I believe this can target multiple entities, so it also needs to incorporate the change to service response values to return response values for each entity
• Naming is hard, but services_with_response_data seems like so i'd find something shorter or use paths.

Otherwise, this seems like the right direction.

[iamjackg]

Very good point about the multiple entities, great catch.

Regarding the path, I'm definitely open to other, catchier names (service_response?). Like I mentioned, deeper paths don't make sense and can put us in a tight spot if any actual nested property for services becomes a thing down the line.

[iamjackg]

Updated the example to correctly reflect the output format, including the entity name :)

For path, I'd probably recommend service_response or service_responses.

[Klikini]

@allenporter @iamjackg since it seems that adding a new endpoint is disallowed, could we reopen the discussion on making it a parameter? I know it's the "uglier" of the two solutions since it changes the output format, but anyone currently using the service call endpoint and expecting a list of changed states can continue to do so by omitting the parameter.

At the end of the day, I think we all just want to get this merged and have the functionality back, and it seems like the only way to do that is to make it a parameter instead of a new endpoint (for some reason nobody will share).

[sebsebseb1982]

Couldn't agree more !

[iamjackg]

So far, this discussion is just restating what is in the PR

Well, the PR was my proposal, but an alternative was suggested and I'm not a project maintainer, so I came here to ask which option is preferable. Seems like the preference is for creating a new endpoint judging from all the responses, so I'll edit this into a proposal for that.

[iamjackg]

I was thinking about this some more. While I agree that changing the format of the response based on the query parameter is not great, the original behaviour of returning all state changes (even ones unrelated to the service call) was already weird at best when it was first added. However, I'm sure plenty of people rely on it, so I don't want to get rid of it.

At the same time, creating an entirely new endpoint feels equally kludgy:

• /api/services_with_response_data/<domain>/<service> is a bit odd
• /api/services/<domain>/<service>/response_data is inconsistent, since there are currently no other API endpoints that introduce deeper paths to retrieve different data

Lastly, people might want to have access to both the response data and the set of state changes.

For these reasons, I'll propose a different change: deprecate the current format (with the usual HASS deprecation timeline), and introduce a new, more flexible response format that supports both state changes and service response data. I'll update the proposal accordingly.

[iamjackg]

Proposal updated.

[emontnemery]

The REST API has been superseded by the websocket API, and we don't want to add additional REST endpoints.
In the cases where the same functionality is offered by both the REST and websocket APIs, as is the case with service calls, we should deprecate the REST endpoint instead of adding new features to it.

If there's interest in an updated REST API, that should be offered as a custom integration.

[iamjackg]

Great, the last few weeks sure were a fantastic use of my time. I opened a documentation PR that adds a warning to the REST API page to let people know it's no longer supported. Please take a look when you have a second.

home-assistant/developers.home-assistant#2150

[iamjackg]

If you're going to call cURL, you can also call this CLI command instead.

I see where you're coming from, but in practice this is very often not the case. cURL ships with every distribution under the sun on basically every architecture and board, Python ≥ 3.9 does not, and/or might require extra permissions to be installed, not to mention having to deal with virtualenvs/modifying the system Python/using pipx. HTTP requests, on the other hand, are supported in one form or another on anything that can connect to the Internet. WebSockets are not. If I want to interact with Home Assistant from a teeny tiny embedded Linux device with limited storage and memory, chances are cURL (or, in a pinch, busybox's wget) is already there.

Again, I 100% understand where you're coming from, but these solutions are not equal. An HTTP REST API is the gold standard minimum common denominator for cross-platform interaction, and I'm really surprised that Home Assistant is giving that up.

[mcmanigle]

teeny tiny embedded Linux device

Agreed; the issue is not even so much a teeny Linux device, but a device too tiny to have Linux on it, like a microcontroller running Arduino or ESP-IDF or similar. Granted, there are web socket libraries, but your point that

An HTTP REST API is the gold standard minimum common denominator for cross-platform interaction

Is I think the right one... Based on the architecture discussion, is the entire REST API going away? This changes... a lot of custom devices that interact with HA, and a lot of my future plans personally.

[Klikini]

Yeah, the CLI tool is completely useless in an embedded environment without Python or the resources to make a subprocess call every time it needs to fetch something.

I’m really disappointed in the decision to deprecate the REST API, and I think it will come back around to hurt Home Assistant’s reputation in the long term. Hopefully this can be reconsidered in the future with a “REST API v2” once the necessary features are more stable and clearly defined.

And sure, it could be offered through HACS or something like that, but now any external tool that works with HA has to start its installation guide with “install HACS, then this other repo”.

It would be nice if the Websocket API at least supported authentication via a header in the initial request to simplify the initialization “handshake”.

[Klikini]

For anyone curious about what's involved in converting from the REST API to the WebSocket API just for making service calls that return response data, here's the bare minimum code changes required to adapt my weather dashboard. It's a lot, and it's not even good code that would be useful for other tasks!

WebSockets are designed for a different purpose than REST APIs, and it's great that Home Assistant offers both. I really hope they revisit the decision to have the WebSocket API deprecate supersede the REST API, as this decision would mean a lot of low-power devices would not be able to interact with Home Assistant's API anymore. If I needed to get the state of a single entity in my code, I would have to iterate over the ~1,000 entities in my Home Assistant instance to find it, which is far beyond the capabilities of the average microcontroller.

[Silergo]

But why not use something like home-assistant-cli? It can do all that, plus more. If you're going to call cURL, you can also call this CLI command instead.

I also once experimented with making this functionality in Deno. It was able to do quite some advanced stuff, like the CLI could block until a trigger fired, a state was obtained or it would create a notification and block until it was dismissed.

Because you can't install home-assistant-cli on hosting server. Only run php and java scripts.

[sebsebseb1982]

And no home-assistant-cli on Arduino :)

[Klikini]

I also have several iOS shortcuts that make REST API calls to get single entity states and render templates. The CLI and WS APIs would be useless for that.

[rytilahti]

So, just chiming in with my 0.02e. There is a proposal by @akx to allow making WS calls over the REST API (#959), which unfortunately did not receive any interest when it was created. If I'm reading correctly, having something like that would allow simpler interfacing with the WS API (e.g., using curl or other http clients), and solve the original issue and similar use cases rather amicably without further changes or maintenance requirements to the REST API, also in the future.

Any opinions on whether such approach would be acceptable? Discussing further on that should be done in the linked issue.

[Klikini]

I think that would be a great solution to this.

If the long-term plan is to remove the REST API, however, the get_states WebSocket command would need to be modified to take an optional parameter with an entity ID (or list of them) to filter the result as to not exceed the memory capacity of low-power devices.

[bdraco]

So, just chiming in with my 0.02e. There is a proposal by @akx to allow making WS calls over the REST API (#959), which unfortunately did not receive any interest when it was created. If I'm reading correctly, having something like that would allow simpler interfacing with the WS API (e.g., using curl or other http clients), and solve the original issue and similar use cases rather amicably without further changes or maintenance requirements to the REST API, also in the future.

Any opinions on whether such approach would be acceptable? Discussing further on that should be done in the linked issue.

Long term, thats probably a better solution if we want to continue to support a REST API as we don't want to have to maintain two APIs any more than we are already doing now. Most of the REST APIs are not used internally anymore, and we have been moving as much as possible to the websocket api since its so much faster and flexible for our use cases.

The flip side here is that we haven't documented many of the websocket apis because we want to have flexibility to change them without having to do long deprecations or relying on outside parties using them. We would have to move a bit slower and start documenting them if we wanted to do something like that which might be an unpopular position to take since it will generate significant more work.

[emontnemery]

We discussed this again in the core team and we've accepted the proposal. I've reopened + unlocked the implementation PR home-assistant/core#115046.

Note: This doesn't mean that the exact details of the API changes as described in this proposal are necessarily OK, let's work those details out in the review of the implementation PR.

[bdraco]

I'd prefer if this was a new endpoint per the original discussion above #1074 (comment) so we don't have different return types based on params

[iamjackg]

The comment means its good to move forward but the specific implementation still needs to be worked out.

Yes, that much was obvious: you will notice by the timestamps that I added that comment after discussing the specifics with @emontnemery in the PR. What he was asking me to implement was indeed not matching what the latest version of the architectural proposal suggested, which was surprising to me and clearly to you too since you're asking that we (again) reconsider the endpoint topic, hence my confusion.

You'll have to forgive my frustration, but there are many discordant voices (with different levels of authority) trying to pull this change in different directions, so I'm going to take a step back and watch you guys duke it out. I'll come back once you've made a decision.

[iamjackg]

@bdraco @emontnemery any updates?

[emontnemery]

I admit I completely dropped the ball here; when I wrote that we discussed and the proposal was approved I thought that meant allowing the behavior of the existing endpoint to be controlled by a query parameter, i.e. what's currently implemented by PR home-assistant/core#115046.
I did not notice the proposal had by then been changed from changing the existing endpoint to introducing a new endpoint.

Let me double check with some of the people who were part of that discussion how to proceed.

[emontnemery]

After checking, the conclusion is we do want to extend the existing endpoint, PR home-assistant/core#115046 is merged and included in yesterday's beta for Home Assistant Core 2024.8.0.

@iamjackg Can you update the description of this discussion please so it matches the implementation?

[blark]

Thank you for fixing this!