regression: calling builtin js libs fails with "Uncaught SyntaxError" since b8634fd8e0040c09946e5b6e0eb0bf4b6587acec

[ssievert42]

On both Bangle.js 2 and the Bangle.js 2 emulator, calling builtin js libs like Bangle.setUI() doesn't work anymore.

The following code:

Bangle.setUI();

only results in:

Uncaught SyntaxError: Got '*' expected EOF
 at line 1 col 1
*k(){1{Bangle.buzz(30)}catch(b){}},a={};#(object
^
in function called from line 1 col 14
Bangle.setUI();

but at least Bangle.loadWidgets()` is affected as well.

I bisected the issue, and it looks like b8634fd is the first bad commit.

No idea how to fix that though 😅

To reproduce, try a firmware / the emulator built from b8634fd that doesn't work, and compare it with the firmware / emulator from 617e1a2 which should be working.

[gfwilliams]

You're sure you've tested on an actual device? Because I've been using firmwares with this change in on a device on my wrist for ages now without issues.

Just tried again with Bangle.setUI on 2v24.183 (downloaded as a 'cutting edge' build from https://www.espruino.com/binaries/travis/master/) and there are no errors - it seems to 'just work'.

Are these builds you're doing yourself? What platform are you building on? The minified JS files in libs/js now include non-ASCII characters, so I wonder whether git has checked them out in the wrong format?

I've had of issues with Git changing line endings between windows/unix, but not other things... But it might not be a huge surprise if a platform is trying to interpret the binary file as UTF8 for some reason.

Maybe check the MD5sum of the file? this is what it is at current git head 1aed934

md5sum libs/js/banglejs/Bangle_setUI_Q3.min.js 
2be6d6d2bba8f9559327b212e7a47f14  libs/js/banglejs/Bangle_setUI_Q3.min.js

[ssievert42]

You're sure you've tested on an actual device? Because I've been using firmwares with this change in on a device on my wrist for ages now without issues.

Yup 👍 I've had to use dfu to reflash my watch after building from current master, because it would only show that error and I couldn't get into the recovery menu.
I even took a picture while bisecting (was currently at a99791f and it looks like it failed to load antonclk)

Are these builds you're doing yourself? What platform are you building on?

I'm doing these builds myself in a container based on debian:buster (to avoid messing up my host), with the following Dockerfile:

FROM debian:buster
RUN apt-get update && apt-get upgrade -y && apt-get install -y build-essential tar python3 python2 python-pip git curl unzip cmake && apt-get clean
RUN pip install --upgrade pip && python -m pip install --ignore-installed nrfutil protobuf==3.17.3
RUN git clone 'https://github.com/espruino/Espruino.git' /espruino
WORKDIR /espruino
RUN bash -c 'source scripts/provision.sh BANGLEJS2'
RUN bash -c 'source scripts/provision.sh EMSCRIPTEN2'

where I then run:

cd /espruino
source scripts/provision.sh BANGLEJS2
make clean && BOARD=BANGLEJS2 DFU_UPDATE_BUILD=1 RELEASE=1 make

with the only addition of an ssh server and rsync for using that container from CLion.
I've been producing a bunch of builds using that setup, and this is the first time that I notice any issues.

An emulator built from b8634fd using that container errors out on Bangle.setUI():

source scripts/provision.sh EMSCRIPTEN2
make clean && BOARD=EMSCRIPTEN2 RELEASE=1 make

And using an emulator built from current master, the Web IDE shows "Unable to retrieve board information. Connection Error?" and "Connected to Emulator (No response from board)".

The MD5sum of setUI matches, so ideally git is not to blame:

echo '2be6d6d2bba8f9559327b212e7a47f14  libs/js/banglejs/Bangle_setUI_Q3.min.js' | md5sum -c
libs/js/banglejs/Bangle_setUI_Q3.min.js: OK

I guess I must've messed something up with the build process then 🤷

Just tried the firmware from https://www.espruino.com/binaries/travis/master/espruino_2v24.196_banglejs2.zip and everything 'just works' for me now too.

Sorry for bothering you with this 😅

Still want to build my own firmwares though...

[ssievert42]

Switching my container image from debian:buster to debian:bookworm and not installing python2 gets rid of that "Uncaught SyntaxError" message! (At least when using the emulator, still have to wait for my watch to be fully flashed.)

FROM debian:bookworm
RUN apt-get update && apt-get upgrade -y && apt-get install -y build-essential tar python3 python-is-python3 python3-pip git curl unzip cmake && apt-get clean
RUN pip install --break-system-packages --upgrade pip && python -m pip install --break-system-packages --ignore-installed nrfutil protobuf==3.17.3
RUN git clone 'https://github.com/espruino/Espruino.git' /espruino
WORKDIR /espruino
RUN bash -c 'source scripts/provision.sh BANGLEJS2'
RUN bash -c 'source scripts/provision.sh EMSCRIPTEN2'

Not sure why I thought I needed python2 anyways.

But now the emulator doesn't work with the Web IDE since ad27875; the IDE only shows "Unable to retrieve board information. Connection Error?" and "Connected to Emulator (No response from board)".

Progress, I guess 🤷

[gfwilliams]

Thanks for the updates! That's good to know - so I guess it was installing Python2 that did it?

But now the emulator doesn't work with the Web IDE since ad27875

Thanks for letting me know and tracking it down! I imagine the thing that's causing the problem is the addition of EV_CUSTOM to the enum (it seems like the emulator hard-codes the value for EV_USB). I've just pushed a change which moved EV_CUSTOM to after the serial devices and that should fix it!

[ssievert42]

Thanks for the fix! Can confirm that as of ffd7270 the emulator works again :)

About building for the Bangle.js 2:
Turns out I was a little eager - creating a DFU ZIP requires nrfutil to work.
But the original python nrfutil by nordic is apparently abandoned and replaced by this shiny new version, where I could only find prebuilt binaries and no sources whatsoever.
In addition to being abandoned, the latest version of nrfutil from pypi (6.1.7) requires python >=3.7 and <3.11, which is problematic because debian bookworm comes with python 3.11.2.
So pip just installs nrfutil 5.2.0 instead, which was the last version without any python version requirements, but secretly depends on python 2 and doesn't run at all on python 3.
Aside from that, nrfutil 6.1.7 depends on some newer version of pc_ble_driver_py, but pip cannot see pc_ble_driver_py versions greater than 0.11.4.

I will probably never be allowed near pip again, but came up with this unholy thing of a Dockerfile to force pip to install nrfutil 6.1.7.

FROM debian:bookworm
RUN apt-get update && apt-get upgrade -y && apt-get install -y openssh-server build-essential rsync tar python3 python3-pip python-is-python3 git curl unzip cmake && apt-get clean
RUN pip install --break-system-packages protobuf==3.17.3
RUN pip install --break-system-packages --ignore-requires-python --no-deps nrfutil==6.1.7
RUN pip check | sed -e 's/^.*requires \(.*\),.*/\1/' > requirements.txt && pip install --break-system-packages -r requirements.txt && rm requirements.txt
RUN git clone 'https://github.com/espruino/Espruino.git' /espruino
WORKDIR /espruino
RUN bash -c 'source scripts/provision.sh BANGLEJS2'
RUN bash -c 'source scripts/provision.sh EMSCRIPTEN2'

The proper solution would probably be to switch to the non-abandoned new version of nrfutil though. (But I'm not such a huge fan of using prebuilt binaries when the sources are not available.)

I guess it was installing Python2 that did it?

So what fixed the builtin js libraries for me was switching from debian buster to debian bookworm, and only using python3.
Maybe if the system default python is python2 instead of python3, some script garbles that non-ASCII text 🤷

Now I can create working binaries :)

[gfwilliams]

Thanks for the update - yes, I hit the nrfutil issue too, but I was able to just install that version of Python on my system. That script to force the install is really handy though.

Personally, I wonder whether I shouldn't just include a working nrfutil in the Espruino repo (it's not like we need the bit of it that uses pc_ble_driver_py) so we can ensure it stays working? Because even if we manage to hack something up this time, it's only a matter of time before Python updates and breaks something else. I had to do that for intelhex already so one more wouldn't be a big deal.

... as an aside, I just don't know what language developers are thinking? C compilers will still compile code written 50 years ago but somehow Python seems to think everyone should rewrite their code every 6 months. Who introduces breaking changes between versions 3.10 and 3.11??

[ssievert42]

I wonder whether I shouldn't just include a working nrfutil in the Espruino repo

That'd be really neat!

I'm not a Python-person, so kind of totally out of the loop there, but maybe if you iterate fast (and don't want to end up like C++), you've just got to break backwards-compatibility every once in a while, to do away with the stuff that turned out to be suboptimal?
I don't get it either - at least put the breakage in a new major version or something...