Upgrade to SDK 2.0.0

[djphoenix]

Fixes #1409, #1415, #1624, #1604

[marcelstoer]

I'm missing the update from axTLS to mbedTLS.

~/Data/NodeMCU/nodemcu-firmware (dev) > grep -r axTLS *
docs/en/modules/net.md:This specification is imposed by the [axTLS library](http://axtls.sourceforge.net/specifications.htm) used by the SDK. 
tools/makefile.sh:# * Neither the name of the axTLS project nor the names of its

Also, http://nodemcu.readthedocs.io/en/latest/en/modules/net/ lists all supported cipher suites for TLS 1.1 axTLS supports. I guess that needs to be updated as well?

What about IPv6? Espressif has confirmed on Twitter that it's included even though it's not in the release notes. We should mention that somewhere. WiFi module? net module?

[djphoenix]

Now only update of SDK included. I work on some tests with mbedtls, so this PR with old ssl module (axTLS-based).

[djphoenix]

Also cannot test IPv6 right now. Will work on it some later.

[marcelstoer]

Thanks Yuri. However, I personally would like this to be a "complete" PR i.e. including mbedTLS. It'd be odd to declare that NodeMCU be based on Espressif SDK 2.0 but retain axTLS. It would certainly provoke questions from our users as they would expect to find mbedTLS.

[djphoenix]

Agree, I will extend PR.
So will we use opensource version of mbedtls (like LWIP)? This may help us to optimize memory/flash footprint, and take more flexibility to end users (e.g. turn on/off debug messages).

[pjsg]

The lwip code that we are including doesn't have any evidence of IPv6 changes. If the SDK code does support IPv6, then we probably don't want to be including our version of lwip. My suspicion is that we want to dump the espconn layer first, then do the ipv6 work. [Dumping espconn will require a lot of TLS work]

[djphoenix]

I was reviewed mbedtls library that Espressif built... There are interesting points:

1. All sources included, with espconn<=>mbedtls layer.
2. Sources can be built into similar binary, and libmbedtls.a can be replaced with them.
3. mbedtls config that Espressif provided is not optimal for embedding... So, "thanks, Esp".

Actually I prefer to use source-based version, maybe with git submodule and config overrides. This way works for my own firmware template (RTOS-based). Also we provide more flexibility to endusers.

[pjsg]

I agree that building from source is the way to go -- it allows easier debugging for a start!

[djphoenix]

Was integrated mbedTLS... It works. Sometimes.

function test()
    local c = net.createConnection(net.TCP,1)
    c:on('connection',function(s)
        print('CONN')
        s:send(
        "GET / HTTP/1.0\n" ..
        "Host: google.com\n" ..
        "Connection: close\n" ..
        "\n"
        )
    end)
    c:on('dns',function(s,a)
        print('DNS',a)
    end)
    c:on('sent',function(s)
        print('SENT')
    end)
    c:on('reconnection',function(s)
        print('ERR')
    end)
    c:on('disconnection',function(s)
        print('CLSD')
    end)
    c:on('receive',function(s,p)
        print(p)
    end)
    c:connect(443,"google.com")
end
test()

This example works OK. But, for example, howsmyssl.com was not loaded - connection closed.
Need some testing...
// Also phoenix.dj loaded normally. Yay!

[devsaurus]

I've increased irom0 size for travis build and retriggered the build.

[jmattsson]

I really wish they'd release an RTOS2.0 SDK as well, because I'm not sure how to handle all these differences over on the dev-rtos branch >.< Maybe once they push the ESP32 SDK out the door... I can only hope.

It would seem there's room for some code reduction once mbedTLS is in, as it has SHA{256,384,512} support included. That should mean we could ditch that code from our crypto module and save a bit of flash.

[djphoenix]

Just updated SDK2.0.0 to 16_08_10. Good news
Now it looks like mbedTLS layer works properly (over espconn). This code executed as expected:

sock = net.createConnection(net.TCP, 1)
sock:on('connection', function(s)
  print('conn')
  sock:send(
  'GET /a/check HTTP/1.1\n' .. 
  'Host: www.howsmyssl.com\n' ..
  'Connection: close\n' ..
  '\n'
  )
end)
sock:on('receive', function(s,d)
  print(d)
end)
sock:on('disconnection', function(s) print('CLSD') end)
sock:connect(443, '104.196.5.74')

Output (for default config of mbedTLS):

HTTP/1.1 200 OK
Content-Length: 2440
Access-Control-Allow-Origin: *
Connection: close
Content-Type: application/json
Date: Mon, 24 Oct 2016 12:56:12 GMT
Strict-Transport-Security: max-age=631138519; includeSubdomains; preload

{"given_cipher_suites":["TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384","TLS_DHE_RSA_WITH_AES_256_GCM_SHA384","TLS_DHE_RSA_WITH_AES_256_CCM","TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384","TLS_DHE_RSA_WITH_AES_256_CBC_SHA256","TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA","TLS_DHE_RSA_WITH_AES_256_CBC_SHA","TLS_DHE_RSA_WITH_AES_256_CCM_8","TLS_ECDHE_RSA_WITH_CAMELLIA_256_GCM_SHA384","TLS_DHE_RSA_WITH_CAMELLIA_256_GCM_SHA384","TLS_ECDHE_RSA_WITH_CAMELLIA_256_CBC_SHA384","TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA256","TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA","TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256","TLS_DHE_RSA_WITH_AES_128_GCM_SHA256","TLS_DHE_RSA_WITH_AES_128_CCM","TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256","TLS_DHE_RSA_WITH_AES_128_CBC_SHA256","TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA","TLS_DHE_RSA_WITH_AES_128_CBC_SHA","TLS_DHE_RSA_WITH_AES_128_CCM_8","TLS_ECDHE_RSA_WITH_CAMELLIA_128_GCM_SHA256","TLS_DHE_RSA_WITH_CAMELLIA_128_GCM_SHA256","TLS_ECDHE_RSA_WITH_CAMELLIA_128_CBC_SHA256","TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA256","TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA","TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA","TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA","TLS_RSA_WITH_AES_256_GCM_SHA384","TLS_RSA_WITH_AES_256_CCM","TLS_RSA_WITH_AES_256_CBC_SHA256","TLS_RSA_WITH_AES_256_CBC_SHA","TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384","TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384","TLS_ECDH_RSA_WITH_AES_256_CBC_SHA","TLS_RSA_WITH_AES_256_CCM_8","TLS_RSA_WITH_CAMELLIA_256_GCM_SHA384","TLS_RSA_WITH_CAMELLIA_256_CBC_SHA256","TLS_RSA_WITH_CAMELLIA_256_CBC_SHA","TLS_ECDH_RSA_WITH_CAMELLIA_256_GCM_SHA384","TLS_ECDH_RSA_WITH_CAMELLIA_256_CBC_SHA384","TLS_RSA_WITH_AES_128_GCM_SHA256","TLS_RSA_WITH_AES_128_CCM","TLS_RSA_WITH_AES_128_CBC_SHA256","TLS_RSA_WITH_AES_128_CBC_SHA","TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256","TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256","TLS_ECDH_RSA_WITH_AES_128_CBC_SHA","TLS_RSA_WITH_AES_128_CCM_8","TLS_RSA_WITH_CAMELLIA_128_GCM_SHA256","TLS_RSA_WITH_CAMELLIA_128_CBC_SHA256","TLS_RSA_WITH_CAMELLIA_128_CBC_SHA","TLS_ECDH_RSA_WITH_CAMELLIA_128_GCM_SHA256","TLS_ECDH_RSA_WITH_CAMELLIA_128_CBC_SHA256","TLS_RSA_WITH_3DES_EDE_CBC_SHA","TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA","TLS_EMPTY_RENEGOTIATION_INFO_SCSV"],"ephemeral_keys_supported":true,"session_ticket_supported":true,"tls_compression_supported":false,"unknown_cipher_suite_supported":false,"beast_vuln":false,"able_to_detect_n_minus_one_splitting":false,"insecure_cipher_suites":{},"tls_version":"TLS 1.2","rating":"Probably Okay"}

(json at http://pastebin.com/QDBx974K)

Also loaded OK: google.com, yandex.ru, phoenix.dj, github.com

[djphoenix]

So MQTT over SSL works too:

m = mqtt.Client('nodemcu-test', 5, nil, nil, 1)
m:on('message', function(c,t,m)
  print('msg', t, m)
end)
m:connect('iot.eclipse.org', 8883, 1, 0, function(c, r)
  print('conn', r)
  c:subscribe('test/ndemcu', 1, function(c)
    print('sub')
  end)
  c:publish('test/ndemcu', 'it works', 1, 0, function(c)
    print('pub')
  end)
end)

[marcelstoer]

The CI build for pull requests failed because it needs more memory.

[djphoenix]

@devsaurus fixed it once before at this thread... Maybe needed to do it again. So irom size is not enough anyway, in local builds I always set it to at least 0xD0000

[marcelstoer]

After I merged the iram/irom fix I restarted the CI build for this PR -> ok now.

[djphoenix]

Okay, I think we're ready. Or you have another things that should be done before merge?

[marcelstoer]

Should be http://espressif.com/sites/default/files/sdks/esp8266_nonos_sdk_<version> instead of BBS.

[marcelstoer]

You missed the BBS URL in flash.md.

[djphoenix]

Fixed links

[djphoenix]

Any notes there?
/cc @marcelstoer @devsaurus @jmattsson

[marcelstoer]

I haven't had time yet to thoroughly test it yet but I'd like to see this merged sooner rather than later. I won't merge myself though unless I actually tested.

[devsaurus]

Did some smoke tests, looks good 👍

[pjsg]

Can we just merge this so that all the testing that we do is based on 2.0.0? Or is there something that we are waiting for?

[djphoenix]

+1 for merge it now from me. This is a key and huge PR that important to test as much as possible before get it to master.
Was played with this for some time, no issues was found.

[marcelstoer]

https://github.com/nodemcu/nodemcu-firmware/edit/dev/docs/en/modules/net.md lines 8:15 need to be adjusted. And so does https://github.com/nodemcu/nodemcu-firmware/edit/dev/tools/makefile.sh 4:6.

[djphoenix]

@marcelstoer Updated net.md. So, about makefile.sh - I not familiar with this one (was never used tools folder at all), how and what I should change?

[marcelstoer]

That stuff was added by @pjsg in 501bd1f. Lines 4-6 contain some axTLS disclaimer comment which is no longer relevant if axTLS isn't part of the firmware anymore.

[pjsg]

I haven't looked at the 2.0.0 documentation -- I hope that they changed the way that certificates worked. It was truly horrible. You can probably comment it out for now. I'll take a look....

[djphoenix]

As I can understand, this is enough to just remove it notice. So, this part done.
Any other picks?