Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updating the firmware trough local API #49

Closed
engelant opened this issue Aug 10, 2018 · 378 comments
Closed

Updating the firmware trough local API #49

engelant opened this issue Aug 10, 2018 · 378 comments

Comments

@engelant
Copy link

As I find it hard to find some information on if sombody managed to update the firmware of a tuya device utilizing any kind of OTA update, I thought this might be the right place to ask.

@codetheweb
Copy link
Owner

I haven't heard of anyone doing that, although I do know that at least one person has cracked their device open and directly flashed new firmware onto the board.

According to Tuya's docs (scroll/search for section 3.10), it may be possible.

If you manage to, let me know. 😃

@drushbrook
Copy link

drushbrook commented Dec 6, 2018

I was fortunate enough to get a device that needed a firmware update and hence captured the traffic from my router between the device and the outside world. Here is a run down of what it did - I have tried to identify my private information by substituting for the device id and <32 bit AES key> for the signing key of the communication between the atop gateway and the device.

I have attempted to replicate the ota process by running a local mqtt and webserver and redirecting locally - I have got as far as it downloading the 64 byte header of the Sonoff-Tasmota firmware.

MQTT - App Triggered MQTT Publish via Cloud which seems to kick off the OTA update
Topic: smart/device/in/
Message: 2.1cc684fc01148c5d8AtUW152t0tPFT2KItF9o+2iU4P+FRyQPpWiAlHQ0FJ+C1LFTkutfz2OboAvWd0ckWRauTKDdRG50K4uIOwyzITqiZgcdwgIVsiYuCks8w5G5V7+bGNnNy7lC3kV7GbQh
Decoded: {"data":{"gwId":""},"protocol":15,"s":1523713,"t":1544007190}

HTTP:
Post http://a.tuyaeu.com/gw.json?a=tuya.device.upgrade.get&gwId=&t=1544007189&v=4.0&sign=<32 bit aes key>
data=99BB3ABB69C97BCA62CCC4B0D50C9EF4

Response:
{"result":{"size":"825947","cdnUrl":"http://images.tuyaus.com/smart/firmware/upgrade/201810/1539875023-oem_esp_dltj_ug_1.1.0.bin","originalUrl":"http://s3-us-west-2.amazonaws.com/airtake-public-data/smart/firmware/upgrade/201810/1539875023-oem_esp_dltj_ug_1.1.0.bin","httpsUrl":"https://s3-us-west-2.amazonaws.com/airtake-public-data/smart/firmware/upgrade/201810/1539875023-oem_esp_dltj_ug_1.1.0.bin","version":"1.1.0","url":"http://s3-us-west-2.amazonaws.com/airtake-public-data/smart/firmware/upgrade/201810/1539875023-oem_esp_dltj_ug_1.1.0.bin","md5":"1142247a63a44dd97896c4fe6abd53f0"},"t":1544007191,"e":false,"success":true}

HTTP:
Post: http://a.tuyaeu.com/gw.json?a=s.gw.upgrade.updatestatus&gwId=&t=1544007191&sign=<32 bit aes key>
data=622AF6DE2E8048A841145B653500E9653CA71ED9200AD1CE8917A58AFA524E76

Response:
{"t":1544007193,"e":false,"success":true}

HTTP:
Header: Range: bytes=0-63
Get: http://s3-us-west-2.amazonaws.com/airtake-public-data/smart/firmware/upgrade/201810/1539875023-oem_esp_dltj_ug_1.1.0.bin

Partial Response:
UªUª1.1.03M#gMGM#'\½ªUªUê @À

... must perform some validation of the first 64 bytes of the firmware probably looking for the version number and it contains the sizing and offset for the next request ... offset in hex: 00064D47 (dec: 412999) and size in hex: 00064D14 (825947 [file length] - 412999 [offset] = 412948 [section])

HTTP:
Header: Range: bytes=412999-825946
Get: http://s3-us-west-2.amazonaws.com/airtake-public-data/smart/firmware/upgrade/201810/1539875023-oem_esp_dltj_ug_1.1.0.bin

Partial Response:
:
MQTT:
Topic: smart/device/out/
Message: {"protocol":16,"data":{"gwId":"","devId":"","progress":"0"}}

MQTT:
Topic: smart/device/out/
Message: {"protocol":16,"data":{"gwId":"","devId":"","progress":"1"}}

... a number of progress MQTT Publish...

MQTT:
Topic: smart/device/out/
Message: {"protocol":16,"data":{"gwId":"","devId":"","progress":"97"}}

... tuya device did a reboot ...

DNS Query: mq.gw.tuyaeu.com

MQTT Connect:
Client ID:
Will Topic: tuya/smart/will
Will Message: {"clientId":"","deviceType":"GATEWAY","message":"11","userName":""}
User Name:
Password: **********

DNS Query: a.tuyaeu.com

HTTP:
Post: http://a.tuyaeu.com/gw.json?a=s.gw.update&gwId=&t=9&v=2.0&sign=<32 bit aes key>
Form item: "data" = "F45D3AD1A13B58AD14FA7CB50C9CC1467B1812FE0E6DDE0DAE2F46A5F4DCA05AF12517176211F09B175F683D3C63C6A0ECE6726FD7C37634DE32CDBCD741CCDAC399BC7B3D6F6654D554E70B07D26A329ECD2ACB649724A686E54AB97B0098FB"

Response: {"t":1544007229,"e":false,"success":true}

@engelant
Copy link
Author

engelant commented Dec 7, 2018

This seems to be exactly what I was looking for, so thanks. You didn't happen to dump the original bin file, did you? I'll have a closer look at this the next weeks, since I'm a little busy these days.
afaik the signature is just a hash of some post vars from the request, salted with the device secret key. I'll have to look into my code from back in august, since unfortunatly a few months passed by and I don't remember everything. At first glance it looks like the bin is not encrypted, but X-mas is coming, so I'll find a few days to go into more detail I guess.

@drushbrook
Copy link

No I didn't dump the original bin file however I have a second device that is still needs to be updated. When I get time I will make an attempt - recently moved and need to find the soldering iron.

In the meantime I will continue to see if I can get the firmware header correct. Right now the device is downloading my header multiple times.

@nijave
Copy link

nijave commented Dec 10, 2018

Here's an older version of the firmware http://s3-us-west-2.amazonaws.com/airtake-public-data/smart/firmware/upgrade/201806/1528276183-esp_bsd_jl_plug_America_ug_1.0.5.bin

I think mine came with 1.0.0 or 1.0.1, though

@Zixim
Copy link

Zixim commented Dec 31, 2018

anything new on this ?

@SynAckFin
Copy link

The 1.1.0 firmware linked above has a header that is 51 bytes long and is then followed by the actual firmware image. Esptool reports the following about the firmware:

Image version: 2
Entry point: 40100004
4 segments
Segment 1: len 0x5b9c0 load 0x00000000 file_offs 0x00000008
Segment 2: len 0x066ac load 0x40100000 file_offs 0x0005b9d8
Segment 3: len 0x004dc load 0x3ffe8000 file_offs 0x0006208c
Segment 4: len 0x02788 load 0x3ffe84e0 file_offs 0x00062570
Checksum: a5 (valid)

What might work for OTA updating is replacing the tuya image with a tasmota one. You might also have to update the firmware version number in the header (located at bytes 4 - 8).

@SynAckFin
Copy link

SynAckFin commented Jan 6, 2019

The bad news is I've managed to brick my plug by forcing it to install
1539875023-oem_esp_dltj_ug_1.1.0.bin firmware.
The good news is that now I have an excuse to take it apart and try to fix it.
I guess the firmware wasn't intended for my plug.

In other news, it turns out the firmware file actually has 2 different firmwares in it.
Here is what I'm guessing the header contains:

55 aa 55 aa             - Prefix
31 2e 31 2e 30 00 00    - Version "1.1.0"
00 00 00 00             - Don't know (could be zero padding of version string)
00 00 00 02             - Number of firmwares in file
00 00 00 33             - File offset to first firmware
00 06 4d 14             - Length of first firmware
02 23 00 67             - Sum of all bytes in 1st firmware
00 06 4d 47             - File offset to second firmware
00 06 4d 14             - Length of second firmware
02 23 27 5c             - Sum of all bytes in 2nd firmware
00 00 05 bd             - Sum of all previous header bytes
aa 55 aa 55             - Suffix

@SynAckFin
Copy link

I can get the tuya plug to download and install the sonoff-basic firmware by constructing a binary using the above format header with 2 copies of the sonoff-basic appended. Unfortunately the plug no longer boots once its installed. I suspect that I'll have to get it to install some interim firmware and then get this to install sonoff-basic.
I've posted in the SonoffUsers group and hopefully somebody will have some ideas on how to proceed.

@Zixim
Copy link

Zixim commented Jan 8, 2019

I'm on the edge-of-my-seat here 😃
Did you try the minimal version of the firmware ?

@SynAckFin
Copy link

The minimal version is the same. At the moment I'm looking at what SonOTA did to see if I can adapt that.

@Zixim
Copy link

Zixim commented Jan 8, 2019

fwiw, there is also an Espurna core version which is only 299 KB, created to simplify OTA updates.
https://github.com/xoseperez/espurna/wiki/TwoStepUpdates

@SynAckFin
Copy link

**** SUCCESS ****
I used the stage 2 firmware from SonOTA (image_user2-0x81000.bin) and when the plug rebooted it acted as an AP with an SSID of "FinalStage". A bit of messing around with setting my wifi to connect to that SSID and setting up a HTTP server to let it download sonoff-basic (renamed to image_arduino.bin) and I now have the sonoff-basic firmware on my plug.

@Zixim
Copy link

Zixim commented Jan 8, 2019

can you do a detailed write-up ?
I'm aching to try this on one of the lightbulbs where I can not reach the serial pins without destroying the lamp.

@SynAckFin
Copy link

Will do.

@SynAckFin
Copy link

SynAckFin commented Jan 9, 2019

WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING
THIS IS EXPERIMENTAL, IT IS POSSIBLE THAT FOLLOWING THESE INSTRUCTIONS WILL TURN YOUR DEVICE INTO A PAPERWEIGHT
WARNING WARNING WARNING WARNING WARNING WARNING WARNING WARNING

Files
tuya.zip

Instructions for Updating Tuya device OTA

I did this on a linux machine running openSUSE 15.0 but there is no reason why it shouldn't work on other flavours of linux and should be easily adaptable to windows.

Prerequisites

  1. A web server where you can store the 2 firmwares
  2. A computer with WiFi running some flavour of linux
  3. A method of redirecting HTTP requests to a computer under your control
  4. A computer to receive the redirected HTTP requests.

NOTE
Although three computers are mentioned above everything can run on one computer provided that the web server is not listening to either port 80 or port 8080

Setting Everything up
Web Server
You need store the firmware tuya-sonoff.bin under a directory called ota. Download your target firmware and store it as ota/image_arduino.bin.
DO NOT USE sonoff-minimal.bin
I recommend using version 6.2.1 of sonoff.bin and if everything works you can then use its OTA mechanism install a different version.

Redirecting HTTP requests (the difficult bit)
If you are in Europe then your Tuya device probably makes HTTP request to a.tuyaeu.com. If you are in another part of the world then it could be a.tuyacn.com or a.tuyaus.com. You should be able to find out which one it is by snooping port 80 traffic from your device. Once you know what it is then you need to redirect these request to one of your computers.
I did this by using dnsmasq to return my address when the device did a DNS query for a.tuyaeu.com. An alternative would be to have a firewall redirect requests. Whatever method you use, the computer you are redirecting to must be able to lookup the correct address and access the remote site.

Running the Scripts
There are 2 scripts, tuyaproxy.pl and finalstage.sh.
In tuyaproxy.pl change the $WEBSERVER variable to point to the web server with the firmwares.
In finalstage.sh change DEVICE to your WiFi device (usually wlan0)
and WEBSERVER and PORT to your web server.
tuyaproxy.pl needs to run as root on the computer you are redirecting the http requests to.
finalstage.sh needs to run as root on the computer with WiFi

Installing the Firmware
Once both scripts are running, http requests are being redirected and both firmwares are on the web server then power off your device, power it on and go and have a cup of coffee.

After everything has finished follow the sonoff getting started instructions.

Acknowledgements
Thanks to Mirko Vogt for the original SonOTA. This uses the second stage firmware from that project.

@drushbrook
Copy link

drushbrook commented Jan 9, 2019 via email

@digiblur
Copy link

digiblur commented Jan 9, 2019

This definitely changes things! Have a plug I can flash the Tuya junk back on it and test with.

Don't have a full blown Linux build going except my media server but I think a wired raspberry Pi should work with this right since it has Wi-Fi capabilities as well?

@SynAckFin
Copy link

There is no reason a Pi wouldn't work.

@SynAckFin
Copy link

@Zixim
Copy link

Zixim commented Jan 9, 2019

In tuyaproxy.pl change the $WEBSERVER variable to point to the web server with the firmwares.
Shouldn't the port be edited to whatever port the webserver is listening on ?

@digiblur
Copy link

digiblur commented Jan 9, 2019

I feel like I am close.. I saw one plug request firmware with the log from the pl script but on the webserver I am not seeing any requests for the bin file though. I did a wget from that Pi just to make sure the bin was accessible and that works but I just don't see the plug asking for that bin yet.

@Zixim
Copy link

Zixim commented Jan 9, 2019

@digiblur did you edit line nr 8 in tuyaproxy.pl ?
my $WEBSERVER = "http://YourWebServer:8000/";

@SynAckFin
Copy link

If you add:
print("URL: $Firmware->{result}{url}\n");
to the pl file just before the while statement (about line 29) then it will output the url it sends to the device. You can then check this is valid using wget on the Pi.
Another thing you should check is that the Pi can reach the tuya site. Try:
ping a.tuyaeu.com
to check it can

@digiblur
Copy link

digiblur commented Jan 9, 2019

@SynAckFin so the Pi running the PL still needs to reach the real a.tuyaus.com? That would be my issue then as my whole network is redirecting.

@digiblur
Copy link

digiblur commented Jan 9, 2019

ahhh...I added that print and it was doing a ....server:8000// I removed the trailing / from the variable and the URL looks right now. Now to get it to request a firmware again. Patience...

@kueblc
Copy link
Collaborator

kueblc commented Jan 20, 2019

@kueblc Which bin of espurna are you using on your bulbs? link to the specific one? settings?

I tried a couple of the current prebuilt bins but I couldn't get color fully working so I'm back to a custom build of Tasmota

@codetheweb
Copy link
Owner

Sorry I've sat this discussion out so far, but it's not at all because I disagree with the approach being taken here. Fantastic work everyone, especially @kueblc and @SynAckFin.

A few questions:

  1. Is it possible to add a quickstart guide to the README of TuyOTA? I've never really used Perl before and I don't know if I have to install packages or do anything else before running it, something other new users will probably wonder as well.
  2. Once this firmware flashing process is perfected, do you see any reason to continue maintaining TuyAPI? While I would be slightly sad that my pet project is no longer needed, it sounds like the ready-made firmware alternatives offer an easier API as well as potentially greater reliability.

Let me know your thoughts, I'm eager to help in any way I can.

@meingraham
Copy link

meingraham commented Jan 21, 2019 via email

@kueblc
Copy link
Collaborator

kueblc commented Jan 21, 2019

And thank you @codetheweb (et al) I appreciate all your hard work as well. Huge amount of groundwork covered by this project. I only wish I had found it sooner!

Regarding 2, I know the feeling of code obsolescence. I was originally building MockTuyaCloud to sandbox my Tuya bulbs but now, I don't even need to! I'm sure there are still folks out there that can't/don't want to flash custom firmware but still want to integrate with their HA setup. In that way I think tuyapi should remain alive and well. There is a lot out there now that depends on it!

@SynAckFin
Copy link

Sorry I've sat this discussion out so far, but it's not at all because I disagree with the approach being taken here. Fantastic work everyone, especially @kueblc and @SynAckFin.

A few questions:

1. Is it possible to add a quickstart guide to the README of TuyOTA?  I've never really used Perl before and I don't know if I have to install packages or do anything else before running it, something other new users will probably wonder as well.

2. Once this firmware flashing process is perfected, do you see any reason to continue maintaining TuyAPI?  While I would be slightly sad that my pet project is no longer needed, it sounds like the ready-made firmware alternatives offer an easier API as well as potentially greater reliability.

Let me know your thoughts, I'm eager to help in any way I can.

I do need to add some decent documentation for it.
One of the objectives I had with the script was have minimal dependencies. As such it should run without the installation of any extra packages other than hostapd. I think every installation has perl and its core modules installed as part of the base install.

I think you should continue the project. Not everybody is comfortable with flashing custom firmware onto a device so they need a method of control using the built in firmware. Similarly, not every device has been flashed successfully. so those that fail (and are still usable) will find the TuyAPI very useful.

@kueblc
Copy link
Collaborator

kueblc commented Jan 23, 2019

As it turns out, another group has already implemented firmware flashing using a very similar technique to the one developed here. Check it out at https://github.com/ct-Open-Source/tuya-convert

Given these guys have most everything covered, I'm going to recommend this over https://github.com/kueblc/mocktuyacloud for firmware flashing purposes (they do pretty much exactly the same thing, just better organized). Funny how these things work out.

@Aterfax
Copy link

Aterfax commented Jan 23, 2019

Looks like they developed their stuff in parallel (time period) as here, though I suspect at this point TuyOTA developed by @SynAckFin beat them in their commit to github by a day.

Personally I used TuyOTA since it is basically a way more streamlined mocktuyacloud.

@SynAckFin you might want to consider updating the landing page for it? Great work all the same!

@AndyRPH
Copy link

AndyRPH commented Jan 23, 2019 via email

@drushbrook
Copy link

Yes. Tuya-Convert uses the Provisioning stage and can offer a device a firmware update - a device I couldn't get mocktuyacloud or tuyota to work with I got to the stage 1 loader. The stage 1 loader allows for the backup of the device firmware and to see if the flash mode is supported hence less likely to brick.
Unfortunately I bricked the device anyway because I didn't follow the instructions clearly. I thought you could skip the stage 2 loader and go straight to tasmota as alluded to on https://www.heise.de/ct/artikel/Tuya-Convert-Escaping-the-IoT-Cloud-no-solder-need-4284830.html

@codetheweb
Copy link
Owner

codetheweb commented Jan 24, 2019

Thanks for the feedback @SynAckFin and @kueblc; looking forward to seeing how these flashing solutions mature.

(By the way, @SynAckFin: I'm totally open to you moving TuyOTA to the TuyAPI Org if you want.)

@erwill2
Copy link

erwill2 commented Jan 24, 2019

Me too, Bricked 3 so far, but still excited to see how this matures. May try coverts firmware to see how it works. I have brought one back to life so far. Haven't really tried the other two. I like the option of making a firmware dump. Maybe that can be added to @Tuyota.

@Aterfax
Copy link

Aterfax commented Jan 24, 2019

As far as I know the only apparent difference is the choice in stage 1 firmware? The ability to backup the existing firmware certainly is nice.

@GeorgeIoak
Copy link

When you guys say that your devices are bricked does that mean that they do absolutely nothing? One of mine was acting oddly after the script attempted to run but I was able to hold the button and pair it back to the app. It wasn't as straight forward as that but the device was definitely showing signs that something was running because the LEDs would flash at different rates

@drushbrook
Copy link

I have bricked two devices:

  1. A lightbulb using the Tuya-Convert because from stage1 I went to stage3. According to the logs the firmware got applied however the light no longer turns on, doesn't create the so off AP and trying to reset by turning it off and on multiple times does nothing. I believe my only way to recover it is to update the firmware via serial.

  2. A smartplug that I used mocktuyacloud to flash. After flashing the sonota stage firmware it didn't turn back on, doesn't respond to the power button, no led. I believe it bricked because this device was at firmware 1.1.0 and had already received an ota firmware. This was the device I captured the logs from as per my first post in this issue. I believe the only way to recover is via serial.

@GeorgeIoak
Copy link

From my experience some of the smart plugs are relatively easy to open and then if you can solder some have just enough access to allow you to connect without having to remove the main PCB from the mains pins. I've posted some notes and pictures but now we have 4 repos and issues to keep track of so my head is spinning a bit.

What plug dies on you, maybe it's one of the ones I have.

@erwill2
Copy link

erwill2 commented Jan 24, 2019

Fortunately or Unfortunate the two I haven't messed with had very little or no glue holding the back together. Since I knew I might want to flash them I left them this way. Two of the three bricks had TYWE3S. I think the initial or second firmware failed. Wonder if it can be modified to work. I have only been dealing with pre compiled firmware. I used ESP8266Flasher.exe 32bit on TYWE3S and before flashing I grabbed some output of the chip using putty with baudrate of 77400. I'll post if if anyone things it would be useful.

@SynAckFin
Copy link

If you think you have a bricked device after using Tuya-Convert you probably do not. The firmware they use is sonoff-classic and when that wants an initial configuration it goes into WPS config mode. So switch the device on and go and push the little black button on your wifi router.

@drushbrook
Copy link

Nope unfortunately I got in when Tuya-Convert was using sonoff-minimal :(

@SynAckFin
Copy link

@drushbrook Its a Hail Mary but Tasmota also supports config via ESP8266 SmartConfig. I've never tried it but you never know....

@drushbrook
Copy link

Thanks @SynAckFin but still no luck. I will replace the bulb (they are only $15 each anyway) and do a serial autopsy on the bricked one but won't use it again - too dangerous in my opinion

@hikey
Copy link

hikey commented Jan 24, 2019

Device SHP-6
I am stuck on the FinalStage, tried around 50 times anything what i could do ?

REQ: Client closed connection while receiving request:
***** FinalStage Detected ******
etc.. etc.. etc...

Thanks !

@SynAckFin
Copy link

@hikey Can you raise an issue here please.

@drushbrook
Copy link

@SynAckFin WPS fixed one of my bulbs. THANK YOU!!!!

@manfredwilhelm
Copy link

Hi all,
i also use a Teckin Smartplug Model SP22. I've been able to flash the Tasmota Firmware: sonoff-sensors.bin

Program Version 6.4.1(sensors)
2018-12-24T14:45:55
2_4_2/2.2.1(cfd48f3)
0T00:00:27
131 at 0xF6000

in the recent posts i found that this SP22 should work when configured as a BlitzWolf SHP. In my case, it doesn't work, But i got it working (partially) as a Gosund SP1 v23. But no LED no Button. Voltage, Power, on/off is ok.
Is there any other Firmware you use to get it working as Blitzwolf?

Thanks in advance, Manfred

@Zixim
Copy link

Zixim commented Mar 10, 2019

@martinvana
Copy link

If you think you have a bricked device after using Tuya-Convert you probably do not. The firmware they use is sonoff-classic and when that wants an initial configuration it goes into WPS config mode. So switch the device on and go and push the little black button on your wifi router.

I can confirm that Tuya-Convert bricked the Feit Electric BPA800 bulb (Costco Sept 2019). When I removed the ESP8266 module to reflash it over serial port I logged what the poor thing was sending out (before the flashing). It was a loop of:
_no vtrust-flash found, reconnect after 1s
reconnect
scandone
no vtrust-flash found, reconnect after 1s
[notice]mqtt_client.c:576 gw wifi stat is:5
reconnect
scandone
...
Now fixed. Just need to figure out what template to use for Sonoff-Tasmota.

@kueblc
Copy link
Collaborator

kueblc commented Sep 17, 2019

That's not bricked, you just didn't have the vtrust-flash AP active. You could have put the device back into pairing mode to connect it to another network if you wanted. If it was bricked you would not have any meaningful serial output.

@martinvana
Copy link

That's not bricked, you just didn't have the vtrust-flash AP active. You could have put the device back into pairing mode to connect it to another network if you wanted. If it was bricked you would not have any meaningful serial output.
Sort of agree. Semantics :-) Not bricked but not usable unless one opens the device and spends an hour+ soldering and flashing through serial. I had the vtrust-flash AP active but all I was able to do is to bring it to this state. Power-cycling dance didn't bring it back to AP search mode. But who knows ;-) Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests