[Build] Upgrade Node.js from v14 to v16

[tmarkley]

We're currently running Node.js v10.24.1. This issue will track progress towards upgrading to a newer version.

v16 is the upcoming active LTS: https://nodejs.org/en/about/releases/

Tasks:

• v10 -> v14 Node.js v14 Upgrade #920
• v14 -> v16 [RFC] Node.js v16 Upgrade Plan #835

[seanneumann]

Let's do a patch release now and we can prioritize moving to 14 at a later point.

[tmarkley]

For 1.0.0 we're upgrading to 10.24.1 and will work on the major version upgrade post-GA.

[kavilla]

Created a sub-task to track a version upgrade to v10 to v12 here: #716.

That way if we have a dependency and that requires at least Node.js 12 then we can track it better. Also, the change from v10 to v16 might be large so it breaks it into chunks.

[hackacad]

Node 14+ will be a requirement for a FreeBSD port of OS Dashboards as well. (current major release is 16, node 14 backport would be possible)
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=257558#c4

[boktorbb]

Closed the sub-task issue for upgrading from v10 - v12 in favor of the plan to upgrade node straight to v16 from v10

[boktorbb]

Added new subtask: #835 that outlines a plan for moving towards upgrading to v16

[boktorbb]

Progress update:

7 packages, not including Node, have been updated to new versions that are supported by Node 16.9.1:

• lmdb-store
• node-sass
• sass-loader
• @types/node
• tslib
• @types/webpack-env
• globby (currently being upgraded)

The biggest type of error seen is type and interface changes across major versions. This manifested itself in the osd-optimizer cache implementation in the cache.ts file, which required time to investigate and refactor the implementation using lmdb-store.

Currently, yarn osd bootstrap is working and the project builds up to that point; however, there are still some runtime errors being investigated. Particularly in the cluster module of our cli source. Errors have cropped up in the worker wrapper around the Node built-in cluster module and within the osd-optimizer logic for parsing plugin paths and manifest paths.

[boktorbb]

Node upgrade progress update:

Overview

OpenSearch Dashboards needs to upgrade the Node.js version from the current version, 10.24.1, to mitigate high severity CVEs affecting the application. The upgrade process intended to move to Node.js v16.9.1. During the upgrade process, it became clear that the amount of investigation and work required to upgrade to v16 was quite large and the team wanted to prioritize mitigating CVEs for the next OpenSearch release. Since the Node v16 upgrade wouldn’t be finished in time for the next release, the team decided on a move to Node.js v14 instead.

Issues faced

The jump to v16 is several major versions higher than the current Dashboards version. Particularly, the jump from v14 to v16 is a much bigger jump in terms of breaking changes than even the jump to v10 to v14. Node v16 has many breaking changes and we encountered the following issues:

• The first major issue is dependency mismatches. Dependencies have silent errors and breaking changes when we update the mandatory dependencies that are required to match Node.js v16.
  • One of the major issues of these dependency updates is the changes in types and interfaces. When they change, it requires us to refactor code and modules in Dashboards. This is mitigated by moving back to Node v14 where these breaking interface changes are minimal
  • The other major issue faced was silent failures during runtime. Certain dependencies are mismatched with others but the mismatch doesn’t manifest in errors or log information but causes silent runtime crashes and failures. This is mitigated by moving to Node v14 where the package changes and updates are minimized
• The next major issue is code refactor in the Dashboards repo
  • The cluster module in the Dashboards core would require a refactor to mitigate the changes in the Node.js v16 built-in cluster module API changes.
    • A refactor of the way we determine worker types and scheduling would be required
    • A refactor of the osd-optimizer process that parses plugin paths and loads plugins would be required
• Another issue is the extra effort that is required from individual plugin owners to upgrade their plugins for compatibility with Node v16. The jump from v10 to v16 is big enough that the amount of work required from each plugin can be large. With 9 internal Dashboards plugins being affected, moving to Node v14 would reduce effort on this front significantly.

Possible paths forward

Moving to Node 14 (preferred solution)

Node.js v14 is currently in LTS and will continue to be in LTS until 2023. This gives us an easier and earlier target to hit since we’re prioritizing mitigating CVEs. It also gives us time to plan and prioritize work for a future upgrade to Node v16 where we aren’t crunched for time because of CVEs and meeting predetermined release schedules.

Continuing Node 16 and missing OpenSearch release

Another option would be bringing on more people to help with the Node 16 upgrade. This would take more effort and would require us to miss the OpenSearch 1.3 release and wait until the following release. Since mitigating the high severity CVEs in Dashboards motivated the whole Node upgrade project, pushing off the CVE fixes further out while Node v10 is EOL is not a good option

Continuing Node 16 and delaying OpenSearch release

This option is the same as the above option but instead of missing the release date and waiting until the next one, we have the OpenSearch 1.3 release delayed until Node.js v16 upgrade is complete.

[tmarkley]

Node.js v18 was just released, CHANGELOG, on April 19. v3.0 is targeted for January 2023 so it might be best to attempt to upgrade to v18 for that release.

[seanneumann]

Closing out this issue as we shipped Node 14 upgrade. Will open a new issue for the next version upgrade.