Apache reverse proxy broken in upgrade to 0.38.x

[n0dyjeff]

Make sure you are running the latest version of Home Assistant before reporting an issue.

You should only file an issue if you found a bug. Feature and enhancement requests should go in the Feature Requests section of our community forum:

**Home Assistant release (hass --version): 0.38,2

**Python release (python3 --version): 3.4.2

**Component/platform: Frontend

**Description of problem: The reverse proxy I had set up in Apache stopped working with the upgrade to 0.38.1 (skipped 0.38.0) and is still failing in 0.38.2. The reverse proxy is set up in accordance with the Apache Configuration example, https://home-assistant.io/cookbook/apache_configuration/ and worked fine with prior releases.

Expected:

Problem-relevant configuration.yaml entries and steps to reproduce:

Traceback (if applicable):

Additional info:
Apache configuration (partial)

    ProxyPreserveHost On
    ProxyRequests Off

    <Location "/">
            AuthType Basic
            AuthName "Home Assistant"
            AuthBasicProvider file
            AuthUserFile {redacted}
            Require user {redacted}

            ProxyPass        "http:/{redacted}:8123/"
            ProxyPassReverse "http://{redacted}:8123/"
    </Location>

    <Location "/api/websocket">
            AuthType Basic
            AuthName "Home Assistant"
            AuthBasicProvider file
            AuthUserFile {redacted}
            Require user [redacted}

            ProxyPass ws://{redacted}:8123/api/websocket
            ProxyPassReverse ws://{redacted}:8123/api/websocket
    </Location>

[jceloria]

I recently updated myself and ran into a problem with multiple named virtualhosts for some reason. (not sure if its related to a recent apache upgrade either?).

I was able to get it working by moving away from name based virtualhosts (I'm not sure if it was a update to apache or what, I haven't had a lot of time to track it down). I was actually able to find a workaround using the following config (modifying where needed and just for now), could you possibly test this out just to see if its a similar problem?:

<VirtualHost <apache.ip>:443>
    ServerName <redacted.domain>

    SSLEngine on
    SSLCertificateFile /root/.acme.sh/<redacted.domain>/<redacted.domain>.cer
    SSLCertificateKeyFile /root/.acme.sh/<redacted.domain>/<redacted.domain>.key
    SSLCertificateChainFile /root/.acme.sh/<redacted.domain>/fullchain.cer
    SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown

    <Location />
        Include conf.d/openid-connect.options
    </Location>

    ProxyPreserveHost on
    ProxyRequests off
    ProxyPass /api/websocket ws://<hass.ip>:8123/api/websocket
    ProxyPassReverse /api/websocket ws://<hass.ip>:8123/api/websocket
    ProxyPass / http://<hass.ip>:8123/
    ProxyPassReverse / http://<hass.ip>:8123/
</VirtualHost>

[andriej]

Same with NGINX reverse proxy (config straight from HASS website)
Worked well with previous releases, HASS works well when bypassing the proxy via straight host:port URL.

[andriej]

Further investigation: my nginx-proxy works well from outside (like from my cellphone or anything else not connected to LAN), it just stopped for LAN/WiFi connections

[esstheevau]

I'm posting this here instead of creating a new issue since I think it might be related. Let me know if I should create a separate issue for this.

After upgrading to 0.38 I can no longer access my installation via browser.

Environment: HASS in docker container behind an NGINX reverse proxy. SSL using a Let's Encrypt certificate.

When opening the website I get to the password prompt, after entering the password the spinner is visible for about 10 seconds and login fails ("Unable to connect").

Strangely enough the web app on iOS (bookmark added to the home screen, webview launches in separate process) continues to work, however I cannot establish a connection using Safari or, (a freshly installed) Chrome instance on the same device.
Also, the connection can't be established from other devices (various Macs with different browsers in the same and other networks, some of which never accessed the HASS url before so I don't think it's a cache issue).

The HASS log file doesn't contain any errors or warnings, no failed login attempts are logged. Automations still work, so HASS seems to be up and running as usual.

The only thing which changed around the time I updated to HASS 0.38 was a certificate renewal. The certificate is valid and mapped read-only into the docker container.

Any ideas for what I could do to further debug the issue would be highly appreciated.

[subutux]

For nginx, have you tried creating a seperate location for the /api/websocket ? See http://nginx.org/en/docs/http/websocket.html for more info.

For example:

# add this to your server {}
location /api/websocket {
            proxy_pass http://[change-me-to-hass-location]/api/websocket;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
        }

Haven't tested this yet because I don't use a reverse proxy, but could be a logical explanation.

Strangely enough the web app on iOS (bookmark added to the home screen, webview launches in separate process) continues to work, however I cannot establish a connection using Safari or, (a freshly installed) Chrome instance on the same device.

The chances are that your "pinned" webapp in iOS is still an old version of the web ui, using the EventStream instead of websocket.

[esstheevau]

Thank you @subutux, that solved the issue for me!

Since my docker host is a Synology NAS I applied the patch from https://github.com/orobardet/dsm-reverse-proxy-websocket and everything is up and running again.

The only downside is that this probably won't be update-safe (when updating the Synology DSM) so I guess I'll have to figure out how to check the config file and apply the websocket fix automatically when needed. Also, I'm going to file a feature request with Synology.

[andriej]

I managed to fix without doing anything with my configuration.
All I had to do (on a problematic browsers) was to wipe cache and refresh the HASS frontend. :-)

[n0dyjeff]

I was also able to get things working using to wipe cache and refresh approach for my PC based browsers (Chrome and IE). However, nothing I do with the browsers on my iPhone seems to work (Chrome, Safari).

[grivers3]

Running HA 0.38.3. I am using apache reverse proxy. After clearing all site storage on Android it will work once. Using on the pc works once also. Reminds me of the caching issue from 4308. Cannot remember when it was fixed.

[grivers3]

A Few Chrome on Android observations:

Using Reverse Proxy for Authentication
• Works every time with incognito mode.
• When attempting to connect for the second time after clearing the site data, the screen finally goes to the HA login page. Need to clear data again.

Using an api_password instead of Reverse Proxy for Authentication
• It works, however the Remember check box does nothing and you have to enter the password every time.

[andriej]

Yeah, I have problems still :-(

[moskovskiy82]

Plus me in. Sophos Firewall. Looking at the web it's running nginx instance for web server protection. After recent update - same thing "unable to connect". All the cache is wiped up.

For the nginix /web/socket solution. Why was HASS working before the 0.38 update?

When trying to access the myexternalurl.com/api/webocket i get the lines below - that means the proxy is working out correctly.

No WebSocket UPGRADE hdr: None
 Can "Upgrade" only to "WebSocket".

Found nothing in Nginix logs except for

2017:02:19-22:11:19 home reverseproxy: id="0299" srcip="xxx.87.154.134" localip="zzz.36.60.118" size="66" user="-" host="xxx.87.154.134" method="GET" statuscode="400" reason="-" extra="-" exceptions="-" time="24555" url="/api/websocket" server="www.xxx" referer="-" cookie="-" set-cookie="-"
2017:02:19-22:11:43 home reverseproxy: id="0299" srcip="xxx.87.154.134" localip="zzz.36.60.118" size="0" user="-" host="xxx.87.154.134" method="-" statuscode="408" reason="-" extra="-" exceptions="-" time="13" url="-" server="www.xxx" referer="-" cookie="-" set-cookie="-"

[subutux]

Because 0.38 introduced websocket usage instead of an event stream: https://home-assistant.io/blog/2017/02/11/alert-appletv-mqtt-yeelight/#rewritten-frontend

[camrun91]

I was able to get the webfront working on all devices by changing my proxy to match the example configuration found here https://home-assistant.io/ecosystem/nginx/

[andriej]

The thing is - are you able to maintain all these devices working status after a few times.
I have same configuration and I can't. The only difference - I have a htpasswd auth in between, which was working fine in earlier releases.

[robbiet480]

Related issue #6138

[camrun91]

@andriej I am able to log into it from every device. I think the difference is that I have the API password set up within HASS.

[balloob]

This is not an issue with Home Assistant but with the config of your external server. If the example config in our docs is not correct, please submit a PR for that.

Home Assistant frontend is and will use websockets moving forward.

[andriej]

Closing issue won't solve it. Anyone have an idea how to solve the issue then?

[balloob]

Issue has been closed because it is not an issue in Home Assistant.

[ransom3910]

Since the problem is occurring with both Apache and NGINX which are independent of each other I am not sure I agree the problem is totally with the reverse proxy. That being said I have had no luck getting this to work so my only option is to remove the reverse proxy and expose HA directly to the internet which I would prefer not to do.

[hessu]

Same issue here, nginx doing basic auth + TLS + mapping for other services. I chose to put nginx there to do centralised authentication and to export various things in the private home network in a controlled fashion to the Internet.

nginx is supposed to proxy websockets just fine with the config (the HA example nginx config matches what is described here: http://nginx.org/en/docs/http/websocket.html).

For me, it works fine on Chrome (OS X), but not on iOS Safari.

The return code is 401 (not authorized).

From the nginx logs, it seems like Safari is not sending the Authorization (HTTP basic auth user/pass) information in the websocket requests, and the 401 is probably then coming from nginx, not HA:

1.2.3.30 - - [24/Feb/2017:06:08:34 +0000] "GET /api/websocket HTTP/1.1" 401 194 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 10_2_1 like Mac OS X) AppleWebKit/602.4.6 (KHTML, like Gecko) Version/10.0 Mobile/14D27 Safari/602.1"

Chrome does send the username, and it works, access.log shows the username:

2.3.4.50 - username [24/Feb/2017:06:14:56 +0000] "GET /api/websocket HTTP/1.1" 101 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36"

If I remove basic auth from the config, or disable it for /api/websocket requests (by adding a separate "location /api/websocket" block like in @subutux example #5954 (comment), iOS clients are happy, too. But then the API is not authenticated any more.

Googling around reveals that basic auth was not mentioned in older versions of the WebSocket documents. So, some browsers implementing an earlier version of the standard do not move the HTTP basic auth information over to the WebSocket request. It was added later on, implemented in later Chromium/Chrome versions, but apparently not in iOS Safari yet.

So, to support doing HTTP basic auth, /api/websocket requests need to be authorized with a different method. Maybe some cookie auth would already be done for them by HA?

[14rf31]

I think the solution by @subutux causes a security issue, if you have disabled the API password. Because I don't want two different password dialogues, I disables the API password. My solution is to set a cookie, if you are successfully authenticated. If no valid credentials have been provided, you will be redirected to a different path for authentication. I found this solution on this site: https://blog.pboehm.org/blog/2014/07/19/authentication-for-websockets/

I attached my current config (shortened).
nginx.txt

[subutux]

@14rf31 yeah my fault. Didn't thought it trough. Nice solution! It's a pitty that some browsers doesn't support auth over web sockets.. but indeed, this isn't a problem of home assistant!

[andriej]

The lua-workaround doesn't work for me. :-/

[subutux]

Maybe we should take this conversation to the forums because this is clearly not an issue wit home assistant.

[andriej]

Anyone solved the problem yet?
Sorry for asking here but I didn't find the thread on forums…
It's getting quite annoying with all Chrome's and WAF is going down :-(

[christcb03]

This also broke on an IIS reverse proxy. I had it set up extremely simple with no SSL and only api password for testing but since the 0.38 update it stopped working. Something definitely changed on HA that made all these stop working. Sadly I am not at all familiar enough with Python or HA to figure out what is going on.

[ransom3910]

It appears no proxies are working correctly with web sockets and Home Assistant. Probably not specific to Home Assistant but the reason it stopped working with .38 is Home Assistant was converted to use websockets. I ended up having to abandon the proxy in front of HA and just use port forwarding and a non standard port. I don't know enough about web sockets to fix the issue yet but continue to look for a solution. I will share if find one!

[murrayju]

Hey all, I had this same problem with my apache reverse proxy, but I figured it out. As others have hinted, HA started using websockets for the login, and those weren't being proxied. Here is what a working apache config looks like:

<VirtualHost *:80>
        ServerName "ha.whatever.com"

        ProxyPreserveHost On
        ProxyRequests off
        ProxyPass / http://localhost:8123/
        ProxyPassReverse / http://localhost:8123/
        ProxyPass /api/websocket ws://localhost:8123/api/websocket
        ProxyPassReverse /api/websocket ws://localhost:8123/api/websocket

        RewriteEngine on
        RewriteCond %{HTTP:Upgrade} =websocket [NC]
        RewriteRule /(.*)  ws://localhost:8123/$1 [P,L]
        RewriteCond %{HTTP:Upgrade} !=websocket [NC]
        RewriteRule /(.*)  http://localhost:8123/$1 [P,L]
</VirtualHost>

[andriej]

Anyone know how to rewrite this for nginx also?

[ScottCYoung]

I added the following to my previously working nginx configuration:

		# WebSocket Support
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection "upgrade";

After that, the auth stopped throwing 400 errors.

[tomikazi]

Thanks @murrayju! I think your solution should be used to update the official HASS docs for Apache Reverse Proxy setup.