Smart Meter Texas Integration Failed to Connect

[CTLS]

The problem

I am attempting to setup the Smart Meter Texas integration, but upon trying to login I get "Failed To Connect"

Environment

• Home Assistant Core release with the issue: 0.115.2
• Last working Home Assistant Core release (if known):
• Operating environment (OS/Container/Supervised/Core): Supervised
• Integration causing this issue: Smart Meter Texas
• Link to integration documentation on our website: https://www.home-assistant.io/integrations/smart_meter_texas/

Problem-relevant configuration.yaml

Traceback/Error logs

Additional information

[bdraco]

Please post log messages if there are any.

[CTLS]

@bdraco Do I need to enable a specific level of logging? I don't see anything on the logs page on home assistant?

[bdraco]

You'll probably have to modify the code to get the full stack

--- a/homeassistant/components/smart_meter_texas/config_flow.py
+++ b/homeassistant/components/smart_meter_texas/config_flow.py
@@ -58,6 +58,7 @@ class ConfigFlow(config_entries.ConfigFlow, domain=DOMAIN):
             try:
                 info = await validate_input(self.hass, user_input)
             except CannotConnect:
+                _LOGGER.exception("Cannot connect")
                 errors["base"] = "cannot_connect"
             except InvalidAuth:
                 errors["base"] = "invalid_auth"

[CTLS]

The only thing I see so far is DEBUG (MainThread) [smart_meter_texas] Requesting login token

[CTLS]

Do I need to enable some sort of API for it to work? It isn't described in the docs, but I remember a while back there was a way of integrating smart meter texas but you had to email them to get api access.

[bdraco]

You shouldn't need to enable anything as it uses the site api. They do have some strict rate limits on number of logins so you might be hitting that.

[probot-home-assistant]

Hey there @grahamwetzler, mind taking a look at this issue as its been labeled with an integration (smart_meter_texas) you are listed as a codeowner for? Thanks!
_{^{(message by CodeOwnersMention)}}

[bdraco]

If its a ratelimit issue, I sent in a PR to detect these

grahamwetzler/smart-meter-texas#38

[grahamwetzler]

I just pushed out version 0.4.2 of smart_meter_texas with the new exception @bdraco.

I have noticed this log message a few times that I don't think I ever saw when I was running this in development. A JSON API shouldn't return text/html so I'm not sure what's going on.

Logger: homeassistant.components.smart_meter_texas
Source: helpers/update_coordinator.py:152
Integration: Smart Meter Texas (documentation, issues)
First occurred: 12:24:57 AM (3 occurrences)
Last logged: 12:30:10 PM
Error requesting Smart Meter Texas data: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://www.smartmetertexas.com/api/user/authenticate') 

[mrand]

I don't think it's a rate limit. I upgraded to 0.115 today for the first time and am getting the same problem when first adding the integration in HA.

Token request via curl works:

*   Trying 12.153.148.59...
* Connected to smartmetertexas.com (12.153.148.59) port 443 (#0)
* found 127 certificates in /etc/ssl/certs/ca-certificates.crt
* found 513 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
*        server certificate verification SKIPPED
*        server certificate status verification SKIPPED
*        common name: www.smartmetertexas.com (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: C=US,ST=New York,L=Armonk,O=International Business Machines Corporation,CN=www.smartmetertexas.com
*        start date: Tue, 15 Oct 2019 00:00:00 GMT
*        expire date: Thu, 14 Oct 2021 12:00:00 GMT
*        issuer: C=US,O=DigiCert Inc,CN=DigiCert SHA2 Secure Server CA
*        compression: NULL
* ALPN, server accepted to use http/1.1
> POST /api/user/authenticate HTTP/1.1
> Host: smartmetertexas.com
> User-Agent: curl/7.47.0
> Content-Type: application/json
> Accept:application/json
> Content-Length: 75
>
* upload completely sent off: 75 out of 75 bytes
< HTTP/1.1 200 OK
< Server: nginx
< Date: Sun, 20 Sep 2020 18:26:16 GMT
< Content-Type: application/json; charset=utf-8
< Content-Length: 678
< Connection: keep-alive
< X-Powered-By: Express
< Access-Control-Allow-Origin: *
< Cache-Control: no-store

BTW, thanks for the integration @grahamwetzler - I've been hoping someone would create one for the past six or so months!

[bdraco]

It stopped working for me as well, even after the changes. It seems like something else is going on here.

[bdraco]

Fixed in grahamwetzler/smart-meter-texas#39

[bdraco]

Well it was working with the changes and now it's stopped. If I use requests instead of aiohttp, it's working. Will do some more investigation later in the week, unfortunately I'm out of time tonight and have a busy mon/tue/wed so it will likely be a while until I get back to this

[mrand]

At the risk of putting my ignorance on display, could it be related to the certificate chain?

EDIT: oh, probably not - I see where you say on the PR where it worked 50 times.

[jjwBullfrog]

I seem to have the same issue. It initially worked and does every now and then, but mostly it gives invalid sensor. Thanks for building this component. Looking forward to it once the bugs are worked out.

[grahamwetzler]

I'm fairly confident the issue is the user agent we're sending. I'm trying to come up with a good way to rotate the user agent but it's not clear to me the best way to handle it.

[cmulk]

I had some (possibly) related issues with the user agent on a different project I worked on with the SmartMeterTexas API. The only way I could get it working consistently in the end was to use a browser user agent. Please take a look at cmulk/python_smartmetertx and see if that helps you any.

[bdraco]

This on has me stumped as I can see the response coming back down over the wire with strace, but aiohttp never processes it and hangs forever

[bdraco]

It looks like the traffic is explicitly being blocked

https://community.akamai.com/customers/s/article/Why-is-Akamai-blocking-me?language=en_US

[mrand]

Well, I originally wrote here that this wasn't working, but I happened to try again immediately after a reboot, and it added... so either I just got lucky, or maybe it's working?

[CTLS]

@mrand It still does not work for me so I assume you were lucky.

[mrand]

It actually isn't reliable. Sometimes I'm able to get readings, but most of the time there are problems.

So, here's what I had originally written in my previous comment: As much as it pains me to suggest it, I'm wondering if a PR should be made to the documentation page so that people don't waste their time? If so, I suppose the documentation should be hidden/removed altogether ("not currently working" warning banner seems strange to me).

[grahamwetzler]

I'm fine with that. I wasn't able to figure out why the requests are being detected and blocked. This feels like it should be fixable but it's a bit beyond me. I'll approve any PR on the library if someone wants to try.

[NateEaton]

Just tried adding this integration and got the Failed to Connect error message. I assume nothing has changed from the thread above?

[mrand]

Correct, although @grahamwetzler, is there any way to redo the accesses inside the integration to simulate/emulate cURL?

Unless you're afraid it will trigger some threshold stuff on their side, my examples at https://github.com/mrand/smart_meter_texas/tree/master/unofficial_api have been working for me.

[grahamwetzler]

It should be possible but I couldn't get it to work when I tried.

[github-actions]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.
Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍
This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

[mwjones1971]

Still not working as of HA Core 2021.5.4

[johnny5th]

Still not working as of core-2021.7.2. Unable to log in

[cbrooker]

Just tried now and it's not working in core-2021.8.0. Could this be related to the addition of a Security question which has to be answered when logging in from a new browser/user agent?

[grahamwetzler]

Unfortunately the library needs some work in order to get it working again. If anyone is able to fix it I will approve the PR.

https://github.com/grahamwetzler/smart-meter-texas

[elmigbot]

I tried a node-red implementation that is working. I noticed the base URL doesn't include the www subdomain, while this one does. I tried adding the www to node-red and it breaks (no response). Could that be the issue?

[meyerrj]

@elmigbot I just tested it locally and removed the www. from the BASE_URL constant... and sure enough, it worked. 😮

I was looking at my developer traces in the browser, and those calls had the www in the API calls without any issues... Weird, but it works!

[elmigbot]

Good news! Just a PR now?

[meyerrj]

I have submitted a PR for the library. 🤞

[excalbian]

Just tried adding this integration (after putting it off for months, the new energy features piqued my interest) and ran into the issue where it can’t connect 😢. Excited to find that a PR with a resolution is already in the works - please let me know if I can do anything to help!

[meyerrj]

@excalbian It was the same energy updates that reignited my interest in this as well. 😅

It currently looks like I may need to refactor some things to handle the SSL problems in a way that will work best with HA's storage. This would involve patching both the library to take in a SSL context variable and updating the HA plugin to create and manage the SSL context. You can refer to this for more details: #54493 (comment)

My current patches are a rough short-term solution at best, and it would be better to patch it so that it's more robust with these SSL validation issues.

I'm not sure how soon I'll have time to sit down and get some work done on it, as I'm patching in my free time. It might be later this week or the weekend if my schedule works out.

If you know python and want to help, please feel free to jump in and contribute as well! 😎

[cfuess84]

I tried to add it for the first time thought the integrations UI and I'm still getting "Failed to connect." Is there something I should be doing manually?

Edit:
I should add that I changed my logs setting look like below and I am not seeing anything when I get the error.

logger:
default: info
logs:
custom_components.hacs: debug
hacs: debug
queueman: debug
AIOGitHubAPI: debug

[jes1417]

I was able to add our account with no issue but the entity is not showing up in any of the energy dropdown selectors

[mrand]

I tried to add it for the first time thought the integrations UI and I'm still getting "Failed to connect." Is there something I should be doing manually?

This fix is in beta, slated to be released in 2021.9
If there are additional problems, it's best to discuss them in the forums or open a new issue since this bug is closed.

[NateEaton]

@meyerrj Thanks for putting this through. Good news is that with release of 2021.9, I can now log in and get data.

As @jes1417 noted, it doesn't show up as an option to select in the Energy dashboard config. However, I suspect that the solution may be to add the same thing I have in the Node-Red SMT solution I've used for the past year the utility meter integration:

utility_meter:

smt_energy_hourly:
source: sensor.smt_current_reading
cycle: hourly

In the meantime, I had seen something odd in that all the sudden the energy dashboard was showing a huge spike every day at midnight. I had left that running and set up in Energy. In looking at a four day graph of the reading from this integration, what I noticed is that I get readings every hour for about half the day and then it flatlines and then spikes at midnight. It dawned on me that what must be happening is that since I have this and the Node-Red solution running, I'm hitting SMT API too often and getting blocked until the next day, at which point the reading "catches up". I've disabled this integration until this evening when I can test adding utility_meter. If that works, I'll shut down the Node-Red flow for SMT.

Just thought I would share this for anyone else who had been using other solutions while waiting for this one to work again. If the utility_meter approach makes this integration's sensor visible for energy dashboard, I'll report that, too.

[jes1417]

@meyerrj Thanks for putting this through. Good news is that with release of 2021.9, I can now log in and get data.

As @jes1417 noted, it doesn't show up as an option to select in the Energy dashboard config. However, I suspect that the solution may be to add the same thing I have in the Node-Red SMT solution I've used for the past year the utility meter integration:

utility_meter:

smt_energy_hourly:
source: sensor.smt_current_reading
cycle: hourly

In the meantime, I had seen something odd in that all the sudden the energy dashboard was showing a huge spike every day at midnight. I had left that running and set up in Energy. In looking at a four day graph of the reading from this integration, what I noticed is that I get readings every hour for about half the day and then it flatlines and then spikes at midnight. It dawned on me that what must be happening is that since I have this and the Node-Red solution running, I'm hitting SMT API too often and getting blocked until the next day, at which point the reading "catches up". I've disabled this integration until this evening when I can test adding utility_meter. If that works, I'll shut down the Node-Red flow for SMT.

Just thought I would share this for anyone else who had been using other solutions while waiting for this one to work again. If the utility_meter approach makes this integration's sensor visible for energy dashboard, I'll report that, too.

I ended up getting it to work in the energy dashboard, the utility_meter integration was not needed as that resets monthly and the energy integration wants a total value for statistics, all I had to do was add a customize to the smart Texas integration.

sensor.electric_meter_xxxxxx
  state_class: total_increasing

That got it to work in the dashboard but I am also seeing the large spike at midnight and was trying to figure out why. What you said makes sense it must be catching up after being limited.

[jjwBullfrog]

Thanks for all the hard work. It's working for me now! I also tried to add to the Energy tab and it doesn't show up. I'll try modifying the device as shown above. HA does have a link asking to notify the developer if it doesn't show in the list.

https://developers.home-assistant.io/docs/core/entity/sensor#long-term-statistics.

[adamfeldman]

To enhance Smart Meter Texas to show in the Energy tab grid consumption dropdown, I have an open PR #55665 – it's in-progress on being reviewed and tested for a future HA release.

[meyerrj]

@jes1417 @NateEaton

The total is an increasing value, so unless you add additional energy tracking sensors to break down the totals hourly it will simply show you the current meter reading. https://www.home-assistant.io/integrations/utility_meter/#configuration

It's also important to remember that the data comes from a third party that collects the data from the meters, and there could easily be delays with the service provider. There are times where I've noticed a few hours where the meter reading value didn't change and then it suddenly jumps up hours later. Unfortunately there's really not a lot that can really be done about this right now.