add new endpoints for app v2 #116

edenhaus · 2021-04-08T17:39:33Z

Adding the new endpoints for app version 2
Fixes #114
Fixes #107

Feel free to test it :). For testing purpose I have also created a new docker image edenhaus/bumper, which will be always up to date with this PR.

Testing instructions

The easiest ways to test is using the docker-compose example below. Please adjust it for your needs
Note: You need to recreate the certs as some new sub domains have been added. I have only tested the manual ways (see updated docs)
Currently only domains/urls for europe are included. Please comment if your app uses different ones

A docker-compose example can be found in the folder example
The docker-compose example has an nginx proxy, which redirects mqtt traffic on port 443 to port 8883
The redirection is required as the app v2+ and robots with a newer firmware are connecting to the mqtt server on port 433.

- new attribute model

edenhaus · 2021-04-16T14:21:57Z

Good news I was able to connect successfully with the app v2 to bumper with mitmproxy in between.
I was able to run some basic commands (auto clean, locate,...) but until bumper supports the whole app it will take some time.
Still some endpoints doesn't exist.

As the new version expects the MQTT server also on port 443, I have placed a nginx in the front.
See #103 and #102
Here the relevant part of the nginx config (Thanks @puddly):

stream {
        map_hash_bucket_size 64;

        map $ssl_preread_server_name $internalport {
                # redirect all requests, which contain "mq" in the SNI -> MQTT
                ~^.*(mq).*\.eco(vacs|user)\.(net|com)$    8883;

                # the rest of eco(user|vacs) requests
                ~^.*\.eco(vacs|user)\.(net|com)$          443;

                # mapping default to MQTT as the bots are connecting directly to the ip without SNI
                default                                   8883;
        }

        server {
                listen 443;
                ssl_preread  on;
                proxy_pass 172.19.200.10:$internalport;
        }

        server {
                listen 5223;
                proxy_pass 172.19.200.10:5223;
        }

        server {
                listen 8007;
                proxy_pass 172.19.200.10:8007;
        }

        server {
                listen 8883;
                proxy_pass 172.19.200.10:8883;
        }
}

I will create a complete docker-compose example in the next days.

Ecovacs uses now a lot of sub sub domains and for them the current certs are not valid. We need to find a way to generate them dynamically or find out all used subdomains

- add status

edenhaus · 2021-04-19T19:06:35Z

create_certs/Bumper_SAN.txt

+*.ecovacs.net
+*.ww.ecouser.net
+*.dc-eu.ww.ecouser.net


@bmartin5692 Can you please verify, which domains the app is using in the us? :)

If it's of any help, here are all of the DNS queries and TLS server names for requests the app makes when communicating with my older Deebot 901:

bigdata-international.ecovacs.com recommender.ecovacs.com gl-us-api.ecovacs.com api-app.ecouser.net mq-ww.ecouser.net

@puddly your urls are already covered by the wildcards domain above

Indeed they are. I'm located in the USA and and your v2 changes work fine with the latest version of the app on Android (thank you!), so either the app "upgrades" to different domains when communicating with newer-generation bots, or these existing domains in your PR are enough.

…all (.git,...)

d4rkd3v1l · 2021-05-16T08:04:47Z

Unfortunately I didn't get this working after following your instructions. But I have to admit, I'm not into docker at all. Previously I did setup bumper manually.

Now I always get this error, as far as I can tell, it may be related to the architecture? Is it correct, that the image was not build for ARM? I'm trying to use it on a raspi. So maybe that's the problem?!

bumper_1 | standard_init_linux.go:219: exec user process caused: exec format error
bumper_bumper_1 exited with code 1

I also had to change the nginx path to get rid of the following error.

nginx_1 | 2037/04/23 10:58:16 [emerg] 1#1: open() "/etc/nginx/nginx.conf" failed (2: No such file or directory)
nginx_1 | nginx: [emerg] open() "/etc/nginx/nginx.conf" failed (2: No such file or directory)

from

 - ./nginx/:/etc/nginx:ro # See config file below

to

 - /etc/nginx/:/etc/nginx:ro # See config file below

Don't quite understand this either, as I would have expect it to use exactly the same path for inside the container, no matter how my system is set up in the end... Isn't this the idea of docker?!

However, my "current issue" is the first error.

edenhaus · 2021-05-16T15:43:14Z

Unfortunately I didn't get this working after following your instructions. But I have to admit, I'm not into docker at all. Previously I did setup bumper manually.

Now I always get this error, as far as I can tell, it may be related to the architecture? Is it correct, that the image was not build for ARM? I'm trying to use it on a raspi. So maybe that's the problem?!

Please get familiar with docker and docker compose. You need to mount the folder containing the certs to the container. I also have provided only a image for x64, but you can create a image for the raspi yourself with the source code or run it manually without docker

When you certs are found, bumper will call a go program to generate it. As I‘m not familiar with go, I can‘t change the go code. Also a bug there exists #117. I only tested the manual way, which is described in the docs.

Don't quite understand this either, as I would have expect it to use exactly the same path for inside the container, no matter how my system is set up in the end... Isn't this the idea of docker?!

For your second issue I will update the docs above, but you should get familiar with docker first.
In the folder, where you have created the docker-compose.yaml, create a new folder named „nginx“ and within create a file nginx.conf and place there the whole Nginx configuration from above.

You don‘t docker to run bumper. You can still download the code of this pr and run it manually. Be aware that you need probably configure nginx manually as at least my deebot 950 is connecting on port 443 instead of 8883

docs/Docker.md

Bustel · 2021-05-19T19:31:24Z

Is the edenhaus/bumper image rpi compatible? The container instantly dies with standard_init_linux.go:219: exec user process caused: exec format error.

I guess the go code was not compiled for ARM.

edenhaus · 2021-05-19T19:46:25Z

Is the edenhaus/bumper image rpi compatible? The container instantly dies with standard_init_linux.go:219: exec user process caused: exec format error.

I guess the go code was not compiled for ARM.

I have tested it only on my linux x64 server.

I have no idea about the go program... But if you create the certs manually and map it as volume inside the container the go program will not be executed. I would recommend it to create it manually as a bug in the go program exists #117

Bustel · 2021-05-19T20:22:18Z

Does not seem to be an issue with the go code in bumper but rather docker itself. I tried to build the image on my rpi4 and the build failed with the same error:

pi@rpi4:~/smarthome/bumper-dev $ docker build -t foobar/bumper .
Sending build context to Docker daemon  41.99MB
Step 1/16 : ARG FROM_ARCH=amd64
Step 2/16 : FROM alpine as builder
 ---> b6342193e7c0
Step 3/16 : ADD https://github.com/balena-io/qemu/releases/download/v3.0.0%2Bresin/qemu-3.0.0+resin-arm.tar.gz .
Downloading [==================================================>]  1.719MB/1.719MB

 ---> Using cache
 ---> 4b90884cb9bd
Step 4/16 : RUN tar zxvf qemu-3.0.0+resin-arm.tar.gz --strip-components 1
 ---> Using cache
 ---> 8552add857b4
Step 5/16 : ADD https://github.com/balena-io/qemu/releases/download/v3.0.0%2Bresin/qemu-3.0.0+resin-aarch64.tar.gz .
Downloading [==================================================>]  1.857MB/1.857MB

 ---> Using cache
 ---> 56fb8e80c4f2
Step 6/16 : RUN tar zxvf qemu-3.0.0+resin-aarch64.tar.gz --strip-components 1
 ---> Using cache
 ---> e0a169240d89
Step 7/16 : FROM $FROM_ARCH/python:3.7-alpine as base
 ---> ec8ed031b5be
Step 8/16 : COPY --from=builder qemu-arm-static /usr/bin
 ---> Using cache
 ---> 1ce35e30773f
Step 9/16 : COPY --from=builder qemu-aarch64-static /usr/bin
 ---> Using cache
 ---> b9d7e4a15f11
Step 10/16 : FROM base as builderfinal
 ---> b9d7e4a15f11
Step 11/16 : RUN apk add build-base
 ---> [Warning] The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm/v7) and no specific platform was requested
 ---> Running in a64e938c918e
standard_init_linux.go:219: exec user process caused: exec format error
The command '/bin/sh -c apk add build-base' returned a non-zero code: 1

I tried different FROM_ARCH values and --platform options but I cannot build the image. Ill try again tomorrow on my laptop.

edenhaus · 2021-05-19T20:31:27Z

Does not seem to be an issue with the go code in bumper but rather docker itself. I tried to build the image on my rpi4 and the build failed with the same error:

pi@rpi4:~/smarthome/bumper-dev $ docker build -t foobar/bumper .
Sending build context to Docker daemon  41.99MB
Step 1/16 : ARG FROM_ARCH=amd64
Step 2/16 : FROM alpine as builder
 ---> b6342193e7c0
Step 3/16 : ADD https://github.com/balena-io/qemu/releases/download/v3.0.0%2Bresin/qemu-3.0.0+resin-arm.tar.gz .
Downloading [==================================================>]  1.719MB/1.719MB

 ---> Using cache
 ---> 4b90884cb9bd
Step 4/16 : RUN tar zxvf qemu-3.0.0+resin-arm.tar.gz --strip-components 1
 ---> Using cache
 ---> 8552add857b4
Step 5/16 : ADD https://github.com/balena-io/qemu/releases/download/v3.0.0%2Bresin/qemu-3.0.0+resin-aarch64.tar.gz .
Downloading [==================================================>]  1.857MB/1.857MB

 ---> Using cache
 ---> 56fb8e80c4f2
Step 6/16 : RUN tar zxvf qemu-3.0.0+resin-aarch64.tar.gz --strip-components 1
 ---> Using cache
 ---> e0a169240d89
Step 7/16 : FROM $FROM_ARCH/python:3.7-alpine as base
 ---> ec8ed031b5be
Step 8/16 : COPY --from=builder qemu-arm-static /usr/bin
 ---> Using cache
 ---> 1ce35e30773f
Step 9/16 : COPY --from=builder qemu-aarch64-static /usr/bin
 ---> Using cache
 ---> b9d7e4a15f11
Step 10/16 : FROM base as builderfinal
 ---> b9d7e4a15f11
Step 11/16 : RUN apk add build-base
 ---> [Warning] The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm/v7) and no specific platform was requested
 ---> Running in a64e938c918e
standard_init_linux.go:219: exec user process caused: exec format error
The command '/bin/sh -c apk add build-base' returned a non-zero code: 1

I tried different FROM_ARCH values and --platform options but I cannot build the image. Ill try again tomorrow on my laptop.

Please give a look at https://github.com/bmartin5692/bumper/blob/9b1ae262a807ad9fe09b33841ba7054b23c91453/hooks/build

I used the hook scripts to push the image for the same arch as the official one. So probably you should be able to use it on the rpi

Bustel · 2021-05-20T20:11:38Z

Thank you for the hint! I managed to get it running for the rpi4 by running this on my laptop:
docker build --build-arg FROM_ARCH=arm32v7 . --tag bumper-arm32

and the using docker save and docker image load to get the image onto the rpi.
However, after recreating the bumper.crt the app now shows an empty screen.

Do I need to update the CA cert as well? I had already reduced the validity before.

edenhaus · 2021-05-20T20:38:29Z

Do I need to update the CA cert as well? I had already reduced the validity before.

I don't think so but I have recreate it also as I was not sure what the go program does.

Also I have tested only the iOS App version. Maybe the android one uses different endpoints. As I don't have a rooted phone I couldn't test it

Bustel · 2021-05-21T06:43:16Z

I checked the logs via adb and the app throws RobotUI Errors for the following endpoints:

If I try them out manually they all return 404. Are those different in the iOS app?

edenhaus · 2021-05-21T08:37:16Z

I checked the logs via adb and the app throws RobotUI Errors for the following endpoints:
* https://api-app.dc-eu.ww.ecouser.net/api/rapp/sds/user/data/map/get?key=EcovacsHome&subKeys=name

* https://api-app.dc-eu.ww.ecouser.net/api/pim/product/software/config/batch

* https://api-app.dc-eu.ww.ecouser.net/api/ecms/app/ad/res?locations=robot_device_list_first
If I try them out manually they all return 404. Are those different in the iOS app?

The endpoints exists and return also some data when called correctly. Be aware that you need to call them correctly.
For example: https://api-app.dc-eu.ww.ecouser.net/api/pim/product/software/config/batch
must be called as post request with some pid in the body.

A GET request on https://api-app.dc-eu.ww.ecouser.net/api/rapp/sds/user/data/map/get?key=EcovacsHome&subKeys=name returns

{"code": 0, "data": {"data": {"name": "My Home"}, "tag": null}, "message": "success"}

Probably you don't have setup bumper correctly or you don't use the version with the code of this PR....

Bustel · 2021-05-21T09:25:50Z

Probably you don't have setup bumper correctly or you don't use the version with the code of this PR....

Just had the same realization and noticed that I build the main branch instead of the PR branch of your repo. Sorry for wasting your time on this 🙈

Also the manual build was not necessary. I just had to run docker pull with the correct platform on the rpi: docker pull edenhaus/bumper --platform linux/arm/v7. I naively assumed that docker is able to figure out which platform it is running on...

~~Now the Deebot is shown in the app but marked as offline. Bumper status shows that the client did not connect to MQTT, but I dont see any errors~~ After restarting the app a couple times I just had my first successful cleaning run 🎉

d4rkd3v1l · 2021-05-22T10:36:32Z

After pulling the docker image like @Bustel suggested, and after fixing linuxserver/docker-papermerge#4 (comment), I finally got the docker containers running on my rpi using docker-compose up.

ports unreachable

Unfortunately the services are not reachable on their ports (either not running, or not exposed, or otherwise "blocked"?).

While docker-compose ps shows the containers running and the ports:

docker-compose ps
     Name                    Command               State                       Ports
--------------------------------------------------------------------------------------------------------
bumper_bumper_1   python3 -m bumper                Up
bumper_nginx_1    /docker-entrypoint.sh ngin ...   Up      443/tcp, 5223/tcp, 80/tcp, 8007/tcp, 8883/tcp

They won't appear in netstat, e.g.:

sudo netstat -ntl | grep "8007"
sudo netstat -ntl | grep "443"

And of course everything else cannot reach them as well (e.g. curl):

curl -X GET https://api-app.dc-eu.ww.ecouser.net/api/rapp/sds/user/data/map/get?key=EcovacsHome&subKeys=name
curl: (7) Failed to connect to api-app.dc-eu.ww.ecouser.net port 443: Connection refused

The domain is resolved correctly by pi-hole:

host api-app.dc-eu.ww.ecouser.net
api-app.dc-eu.ww.ecouser.net has address 192.168.178.3

Stopping docker and starting up the old "bumper.service" works just fine.

sudo netstat -ntl | grep 8007
tcp        0      0 192.168.178.3:8007      0.0.0.0:*               LISTEN

Note: I don't want to keep using the "old service" approach, as I would have to setup nginx etc. manually and actually therefore prefer the docker-way.

iptables

Since docker seems to add a lot of iptables rules to the system and therefore broke my piVCCU installation, I had to modify the rules like here: alexreinert/piVCCU#264 (comment)
But I already reverted all those changes and still none of the ports is reachable for me :/

Did anyone else face a similar issue? Or does anyone have an idea what might go wrong here? Any help is highly appreciated!
Thx in advance.

edenhaus · 2021-05-26T20:36:38Z

@d4rkd3v1l I have updated the docker-compose example. The problem was that I only exposed the ports in my example as I use in my setup macvlan.

A working example can be found here

d4rkd3v1l · 2021-05-29T14:37:31Z

Thx @edenhaus
I got it working now as well, after using ports instead of expose. Also be aware that the example docker-compose.yaml uses the original docker image from bmartin5692, and not the one you provided on docker hub, so this must also be changed to edenhaus/bumper in order to make it work (for the moment, until everything got merged, I guess).

edenhaus · 2021-06-01T17:45:34Z

@d4rkd3v1l I know that, put I don't want to change it as my images are only test images, which can be updated and also be broken anytime

edenhaus · 2021-06-04T20:03:48Z

@bmartin5692 Have you time for a review? I think some people are waiting for this PR

bmartin5692 · 2021-06-05T11:55:44Z

I'll try to get this tested and reviewed this weekend.

slev1n · 2021-08-25T20:01:24Z

@edenhaus
system: rpi 4 @ARM32 with deebot 901 and adguard for dns rewrites

I had to add 'privileged: true' to start bumper from composer.
The iobroker plugin worked and i see the login in the logs.
But the IOS app didnt not work, timeout -> no login attempt in the logs.

Do u have any suggestions?

edenhaus · 2021-08-27T13:56:39Z

I use the docker-compose setup described in https://github.com/bmartin5692/bumper/tree/master/example/docker-compose

I have no idea what is wrong with your setup, but it can be a lot.... Wrong certs, wrong dns config, endpoints not reachable,....
Also I don't have your model and as I can remember it correctly your robot uses the protocol MQTT/xml and my 950 uses MQTT/json. So it can also be that there are missing endpoints...
Also you need to trust the certificate on your apple device.
As you can see there can be a lot and sorry but I currently don't have time to give you support.

edenhaus added 2 commits April 8, 2021 19:37

add some new endpoints for app v2

e8b61b1

add oauth

0b9f192

edenhaus mentioned this pull request Apr 11, 2021

Config entry and various changes And3rsL/Deebot-for-Home-Assistant#91

Merged

edenhaus added 2 commits April 15, 2021 18:15

update EcoVacsHomeProducts

03277b9

- new attribute model

add some missing endpoints for app v2

ca32e0e

edenhaus mentioned this pull request Apr 15, 2021

Add T8 series models #108

Open

edenhaus mentioned this pull request Apr 16, 2021

[Deebot950 ]No connection after update bot to 1.8.2 #103

Open

edenhaus added 5 commits April 19, 2021 14:03

- use only one level subdomain urls on service list

1342118

- add status

add new used subdomains (for europe)

dc49e96

improve docker images creation as requirements are change less often

f050213

reformat area list

c9257f3

add revoke expired oauths

a5359ac

edenhaus commented Apr 19, 2021

View reviewed changes

add docker-compose example

63330a6

edenhaus mentioned this pull request Apr 19, 2021

Ecovacs Home App not working on IOS since Version 2.0.0 #114

Closed

edenhaus added 2 commits May 1, 2021 17:47

wait for mqtt start fixes bmartin5692#107

6579e9a

make images even smaller by copying only required folders instead of …

4394e58

…all (.git,...)

edenhaus marked this pull request as ready for review May 10, 2021 20:58

edenhaus changed the title ~~[WIP] add new endpoints for app v2~~ add new endpoints for app v2 May 10, 2021

edenhaus added 2 commits May 19, 2021 17:52

fix exception during logging

4d0a428

return fail if we can't get clean logs

c726381

Bustel reviewed May 19, 2021

View reviewed changes

docs/Docker.md Outdated Show resolved Hide resolved

improve docker docs. Thanks for the hit @Bustel

607201f

edenhaus force-pushed the ecovacs-home branch from 8bda4ca to 607201f Compare May 19, 2021 20:29

fix body not json serializable

c5dfad4

add docker-compose example in separate folder

d61c8f5

edenhaus force-pushed the ecovacs-home branch from 76e7153 to d61c8f5 Compare May 26, 2021 20:33

bmartin5692 approved these changes Jun 5, 2021

View reviewed changes

bmartin5692 merged commit df36bdd into bmartin5692:master Jun 5, 2021

edenhaus deleted the ecovacs-home branch June 8, 2021 17:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add new endpoints for app v2 #116

add new endpoints for app v2 #116

edenhaus commented Apr 8, 2021 •

edited

Loading

edenhaus commented Apr 16, 2021

edenhaus Apr 19, 2021 •

edited

Loading

puddly May 26, 2021 •

edited

Loading

edenhaus May 27, 2021

puddly May 27, 2021

d4rkd3v1l commented May 16, 2021 •

edited

Loading

edenhaus commented May 16, 2021 •

edited

Loading

Bustel commented May 19, 2021 •

edited

Loading

edenhaus commented May 19, 2021

Bustel commented May 19, 2021

edenhaus commented May 19, 2021 •

edited

Loading

Bustel commented May 20, 2021

edenhaus commented May 20, 2021

Bustel commented May 21, 2021

edenhaus commented May 21, 2021

Bustel commented May 21, 2021

d4rkd3v1l commented May 22, 2021

edenhaus commented May 26, 2021

d4rkd3v1l commented May 29, 2021

edenhaus commented Jun 1, 2021 •

edited

Loading

edenhaus commented Jun 4, 2021

bmartin5692 commented Jun 5, 2021

slev1n commented Aug 25, 2021

edenhaus commented Aug 27, 2021

add new endpoints for app v2 #116

add new endpoints for app v2 #116

Conversation

edenhaus commented Apr 8, 2021 • edited Loading

edenhaus commented Apr 16, 2021

edenhaus Apr 19, 2021 • edited Loading

Choose a reason for hiding this comment

puddly May 26, 2021 • edited Loading

Choose a reason for hiding this comment

edenhaus May 27, 2021

Choose a reason for hiding this comment

puddly May 27, 2021

Choose a reason for hiding this comment

d4rkd3v1l commented May 16, 2021 • edited Loading

edenhaus commented May 16, 2021 • edited Loading

Bustel commented May 19, 2021 • edited Loading

edenhaus commented May 19, 2021

Bustel commented May 19, 2021

edenhaus commented May 19, 2021 • edited Loading

Bustel commented May 20, 2021

edenhaus commented May 20, 2021

Bustel commented May 21, 2021

edenhaus commented May 21, 2021

Bustel commented May 21, 2021

d4rkd3v1l commented May 22, 2021

ports unreachable

iptables

edenhaus commented May 26, 2021

d4rkd3v1l commented May 29, 2021

edenhaus commented Jun 1, 2021 • edited Loading

edenhaus commented Jun 4, 2021

bmartin5692 commented Jun 5, 2021

slev1n commented Aug 25, 2021

edenhaus commented Aug 27, 2021

edenhaus commented Apr 8, 2021 •

edited

Loading

edenhaus Apr 19, 2021 •

edited

Loading

puddly May 26, 2021 •

edited

Loading

d4rkd3v1l commented May 16, 2021 •

edited

Loading

edenhaus commented May 16, 2021 •

edited

Loading

Bustel commented May 19, 2021 •

edited

Loading

edenhaus commented May 19, 2021 •

edited

Loading

edenhaus commented Jun 1, 2021 •

edited

Loading