Build spontaneously broken (involving vega-lite)

[benjimin]

The most recent commit that passed all tests (955fbe6) seems to no longer pass those same unit tests.

The problem involves test_s3_svg_chart_upload (in module test_s3) failing.

From the trace: the test causes altair.Chart.save to be invoked, which tries to run an external node.js script "vl2vg" (apparently part of vega-lite) that fails. (Note altair is basically a python API for vega-lite, a higher level wrapper of vega, which wraps the javascript data-visualisation library D3. The failure specifically presents as a SyntaxError seeming to involve a recent javascript language feature, the "nullish coalescing operator", which may require node v14.)

The dockerfile line RUN npm install vega-lite elicits a warning npm WARN notsup Unsupported engine for [email protected]: wanted: {"node":">=12"} (current: {"node":"10.19.0","npm":"6.14.4"}). Thanks @emmaai for reporting. The automatic build usually hides this because of docker cache.

[benjimin]

Apparently vega-lite 5.1.1 (released 9 days ago) raised their node engine requirement while updating to D3 v7. This conflicted with the engine distributed by the last ubuntu LTS, breaking our build. (Note vega-lite did declare this properly but, as one of the few non-python dependencies, the compatibility is not automated by pip.)

We should probably enforce --engine-strict to the node package manager, to prevent the default silent completion of broken builds. Probably also ought stay up to date with at least the node.js LTS, rather than installing ubuntu default.

If wps processes saved altair charts in html rather than svg format, then we wouldn't need node. That is, the javascript for translating the data into drawing directions could be executed in the browser rather than the WPS; I think svg format also forfeits user-interactivity.

[emmaai]

Agree with charts in html rather than svg

[benjimin]

I think this is resolved. Observed a remaining intermittent build test failure, where test_api.py::test_wit elicits a UnicodeDecodeError (invalid start bytes for utf8) from pyproj (and the trace involves witprocess cal_area, dask core, and datacube geometry __hash__ and to_wkt).