Authentication fails when using DIGEST_AUTH from Apple iPhone and iPad

[jason-but]

Basic Infos

• This issue complies with the issue POLICY doc.
• I have read the documentation at readthedocs and the issue is not addressed there.
• I have tested that the issue is present in current master branch (aka latest git).
• I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it.
• I have filled out all fields below.

Platform

• Hardware: [All]
• Core Version: [2.4.1]
• Development Env: [Arduino IDE]
• Operating System: [All]

Settings in IDE

Irrelevant

Problem Description

I have built a web server using the ESP8266WebServer library and have come across an issue accessing the site via Apple products. I have tried using two different iPhone models (one 2 years old and one 3 years old) and two different iPad models (one 6 months old and one 3 years old) and all four devices indicate the same issues.

Problem Overview

If establishing a web site with NO authentication, everything works fine regardless of the browser or device being used

If establishing a web site with BASIC_AUTH authentication, everything works fine regardless of the browser or device being used

Establishing a web site with DIGEST_AUTH authentication does not function correctly. I have tried setting the authentications using:

requestAuthentication(DIGEST_AUTH);                          // use default realm and fail msg
requestAuthentication(DIGEST_AUTH, "realm");                 // use programmed realm and default fail msg
requestAuthentication(DIGEST_AUTH, "realm", "Message text"); // customise all parameters

For each of the three techniques above I have:

• Successfully accessed the website from a desktop PC/laptop using the Chrome browser
• Successfully accessed the website from an Android phone when ESP8266 running in STA mode
• Successfully accessed the website from an Android phone when ESP8266 running in AP mode
• Unable to access the website from an iPad/iPhone when ESP8266 running in STA mode
• Unable to access the website from an iPad/iPhone when ESP8266 running in AP mode

Apple products fail regardless of whether device is in AP or STA mode, the error message provided by HTTP_SERVER debugging is always that the hash response is incorrect

Apple products fail regardless of whether the iPhone/iPad is using the default Safari browser or the Chrome browser (although it appears that the Apple Chrome browser still uses the Safari libraries to submit qeb queries)

Code Evaluation

There is obviously an incorrect branch code that is taken based on the response Apple devices. Given the debug messages, it appears that the Digest_Auth branch of authenticate() in ESP8266WebServer.cpp IS being taken, however the incorrect response hash is being calculated.

Looking at the code, two possible options are being calculated

MD5("H1:nonce:nc:cnonce:auth:H2)
or
MD5(H1:nonce:H2)

However, the standard implies that the value of qop (which the library hard-codes to "auth") should be either "auth" or "auth-int"

The standard also implies that there are two possible calculations for H1 and H2 of:

H1

MD5(username:realm:password)
or
MD5(MD5(username:realm:password):nonce:cnonce)

H2

MD5(method:uri)
or
MD5(method:uri:MD5(entityBody))

Whereas the existing code only calculates the first possibility of either H1 or H2

It appears that perhaps Apple devices are requiring one of the three currently not coded options in the DIGEST authentication library and that the code needs to be modified to support this.

Potential Fixes

Supporting auth or auth-int in final response hash

Supporting "auth" or "auth-int" appears to require an extra check in line 194 and modifying line 195 to use the actual parameter value rather than the hardcoded ":auth:"

Supporting alternate H1 Calculation

Supporing the more complex H1 requires (after line 172) checking if the algorithm directive is "MD5-sess" and then:

• appending ":nonce:cnonce" to the original calculated H1
• running MD5 again

Supporting alternate H2 Calculation

Supporting the more complex H2 requires more thought. It appears the current code does not extract "entityBody" and so will have to be modified to support this

Conclusion

I am unsure which of these three changes are required to fix Apple brokenness

I note that if the issue is that the qop directive sent by Apple products is auth-int, then the solution will probably involve both fixing the H2 calculation AND modifying the code to calculate the final response hash

Given its current state, it is impossible to support digest authentcation on ESP8266WebServer for Apple devices accessing web sites

[jason-but]

If it helps, I've captured the debug output from the serial terminal below for one iPad

Request: /
 Arguments: 
username="admin", realm="realm", nonce="eff7fc53132a842b18fe54eef0314e67", uri="/", response="b52957641501c9dc93665a7032fddba4", opaque="1441c654f71ab4b73d0c9d56a1771b9f", cnonce="a8c5d880578345b43b40a299daeabd9b", nc=00000001, qop="auth"
Hash of user:realm:pass=4bb9f74448f3f0942b27283acdc7eff2
Hash of GET:uri=71998c64aea37ae77020c49c00f73fa8
The Proper response=812eb82cf02de08d05e340e92964c5f2

[Adam5Wu]

@jason-but Could you capture the debug output of the browser side, show exactly what headers are sent from the client?

BTW, MD5-sess was never correctly implemented in Mozilla, or Webkit.

• Mozilla had a 14 year old bug ticket they never fixed
• I filed a similar ticket for Chromium a couple of months ago
• IE/Edge had a working but unsafe implementation

It seems no major deployment of MD5-sess ever in the real world, so I wouldn't recommend to even think about MD5-sess, it is not worth the effort. :)

[jason-but]

I can try to capture the browser side if I can get some help. I don't know how to get debug info from an iPad and it's browser

Ack on the MD5-sess

Looking at my captured log above, I don't know if any of my three options are the problem as the "qop" variable is "auth" and not "auth-int"

I think you are right, we need to somehow see the intermediate calculations at the iPad to properly see "where" it is going wrong

[Adam5Wu]

Alternatively, can you alter the esp side source code and print the raw authorization header?

The serial output indicate that at least the authorization header does not contain "qop=auth".

Arduino/libraries/ESP8266WebServer/src/ESP8266WebServer.cpp

Line 40 in 5328a8b

static const char qop_auth[] PROGMEM = "qop=auth";

Arduino/libraries/ESP8266WebServer/src/ESP8266WebServer.cpp

Line 194 in 5328a8b

if(authReq.indexOf(FPSTR(qop_auth)) != -1) {

The question is, what exactly did the client send?

[jason-but]

So two separate comments here, one from a functioning Chrome Browser, followed by a non-functioning iPad.

First from Chrome, here is a Serial output from the ESP8266

New client
method: GET url: /json/config search: 
headerName: Host
headerValue: garagedoor.jaybee.net
headerName: Connection
headerValue: keep-alive
headerName: Authorization
headerValue: Digest username="admin", realm="realm", nonce="5b69f77a09b57d938d4ad594a66e246e", uri="/json/config", response="dec7912d2f3892c8160a3ecc6f53a1e8", opaque="d0f8cf2071a4ee9bfcac912ac31519d2", qop=auth, nc=00000001, cnonce="d84a164aa3663c27"
headerName: Upgrade-Insecure-Requests
headerValue: 1
headerName: User-Agent
headerValue: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36
headerName: Accept
headerValue: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
headerName: Accept-Encoding
headerValue: gzip, deflate
headerName: Accept-Language
headerValue: en-US,en;q=0.9
args: 
Request: /json/config
 Arguments: 
username="admin", realm="realm", nonce="5b69f77a09b57d938d4ad594a66e246e", uri="/json/config", response="dec7912d2f3892c8160a3ecc6f53a1e8", opaque="d0f8cf2071a4ee9bfcac912ac31519d2", qop=auth, nc=00000001, cnonce="d84a164aa3663c27"
Hash of user:realm:pass=4bb9f74448f3f0942b27283acdc7eff2
Hash of GET:uri=161650ef261ca379c529bab6e21cf46c
The Proper response=dec7912d2f3892c8160a3ecc6f53a1e8

[jason-but]

Now from the iPad

New client
method: GET url: /json/config search: 
headerName: Host
headerValue: garagedoor.jaybee.net
headerName: Authorization
headerValue: Digest username="admin", realm="realm", nonce="db77f837c97df385f24cf7f035e03f22", uri="/json/config", response="350b426973820774254a73f1273da602", opaque="307d123e7152064489768d76a3c1ecc8", cnonce="00579d4099e29d18671f1aa6d8e4d2ec", nc=00000001, qop="auth"
headerName: Accept-Encoding
headerValue: gzip, deflate
headerName: Accept
headerValue: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
headerName: Accept-Language
headerValue: en-au
headerName: Connection
headerValue: keep-alive
headerName: DNT
headerValue: 1
headerName: User-Agent
headerValue: Mozilla/5.0 (iPad; CPU OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F69 Safari/600.1.4
args: 
Request: /json/config
 Arguments: 
username="admin", realm="realm", nonce="db77f837c97df385f24cf7f035e03f22", uri="/json/config", response="350b426973820774254a73f1273da602", opaque="307d123e7152064489768d76a3c1ecc8", cnonce="00579d4099e29d18671f1aa6d8e4d2ec", nc=00000001, qop="auth"
Hash of user:realm:pass=4bb9f74448f3f0942b27283acdc7eff2
Hash of GET:uri=161650ef261ca379c529bab6e21cf46c
The Proper response=95dea70b18c23167eed7f1de1727280c

[jason-but]

In both cases, the remote browser seems to respond with "qop=auth" so my initial analysis proved to be incorrect

• from both captured debug logs, both have the correct username and realm values. Both also have equal values for the nc and qop parameters
• The parameters (both in the headers and the Authorization) have a different order
• The hashes are different, but that is to be expected given the unique nonces

However, looking more/very closely, the main difference appears to be:
Chrome qop=auth
iPad qop="auth"
With quotes
I wonder if the code at line 194

if(authReq.indexOf(FPSTR(qop_auth)) != -1) {

Is failing because of the quotes and therefore the wrong string is being calculated?

[Adam5Wu]

Yeap. Technically, I would say the fruity is in violation here, since RFC 2617 did not state quote is allowed in message-qop in section 3.2.2.
But it probably doesn't hurt to make the parser on ESP side more tolerant to benign errors like this.

[jason-but]

Just finished scanning RFC2617

Interestingly (see Section 3.2.1) it allows quotes in qop on the WWW-Authenticate Response Header from the server but does not (Section 3.2.2) on the Authorization Request Header from the browser

On my end, I am rather surprised this hasn't been caught earlier, the iPad/iPhone is not exactly a rare end-user device

My guess would be the bug fix would require checking for both instances at line 194, or alternatively pulling the parameter out earlier on (lines 148-152) and storing the value regardless of quotes or no

[boblemaire]

I have this working in some workaround code. If you change line 194 to recognize both instances, you will need to do the same at line 164 so that the _nc and _cnonce values are extracted.
Once that's done, it works fine.

[jason-but]

I presume you changed both conditionals to something like:

if((authReq.indexOf(FPSTR(qop_auth)) != -1) || (authReq.indexOf(PSTR("qop=\"auth\"")))) {

[boblemaire]

Yes. I defined another literal qop_auth_quote and used FPSTR, but same difference. I don't have the code to PR because I was just working through the issue on my way to writing a version of authorize that supports multiple outstanding sessions for multiple users. I implemented that as separate code as it really doesn't need to be in the class to work. But I'm satisfied that change fixes the IOS issue.

[jason-but]

Excellent
My use case doesn't require (or expect) multiple concurrent connections to the server so the issue of re-authenticating is something I've noticed during my testing, but didn't consider a huge issue.
Looking forward to seeing both fixes eventually committed

[devyte]

If somebody makes a pr, I can review it.

[mdombrowski13]

Do we know if a new build is out fixing this? Or when a fix will be released? I've verified that in the current ESP8266 arduino plugin I'm seeing this problem in all versions of Safari, iOS and Mac OS. When I run chrome or Firefox on the mac or iPhone it works fine though. The bug is bigger then an iOS bug.

[earlephilhower]

Reading @lukasostendorf's PR, it looks like it directly fixed this. Closing for now, but if there is still a problem with fruity computers, please do open a new issue so we can look at it.

[Hieromon]

@earlephilhower Hello, In my view, the esp32 arduino-core seems to have a similar problem. Please tell me if you have some information about this fix to the esp32 arduino-core.
Thank you.

[earlephilhower]

@Hieromon, the PR is here: https://github.com/esp8266/Arduino/pull/5506/files You can check the -32 core and see if a similar change would make sense. I haven't done much w/the -32, so can't comment directly.

[Hieromon]

@earlephilhower Always I surprise with your quick answer!
I'll check it. Thank you so much.