feat(build): uplift Lerna + replace insecure shortid with nanoid + uplift Yeoman-related packages + ESM-ize generator-superset #29419

hainenber · 2024-06-30T15:56:24Z

feat(build): uplift major version for CI-related packages

SUMMARY

Uplift major version for notable CI-related packages. Notable changes:

Uplift Lerna v6 to latest v8
Replace unsupported and insecure shortid with successor nanoid + corresponding refactors
Uplift yeoman-related packages to latest v7
ESM-ize the generator-superset package in order to work with yeoman-generator v7

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

Has associated issue:
Required feature flags:
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

Signed-off-by: hainenber <[email protected]>

…rform corresponding refactors Signed-off-by: hainenber <[email protected]>

Signed-off-by: hainenber <[email protected]>

… column char limit Signed-off-by: hainenber <[email protected]>

…aries + fix Jest tests + add specific ESLint config Signed-off-by: hainenber <[email protected]>

Signed-off-by: hainenber <[email protected]>

rusackas

This is looking great to me... I just need to find time to test it out!

mistercrunch

LGTM

mistercrunch · 2024-07-16T22:08:58Z

I really appreciate all your solid contributions! I'll take a look at all your open PRs today.

rusackas

Generator seems to work! I echo Max... thanks for all the contributions!

fisjac · 2024-07-19T20:45:43Z

After the Lerna uplift, I'm experiencing build errors in docker when running build-dev. It appears the optional dependency @nx isn't being installed, but is required by Lerna post update. Anyone experiencing this?

hainenber · 2024-07-21T14:10:58Z

Can you share the build failure's screenshot? I ran npm run build-dev locally and it finishes as usual. Maybe you can run npm ci to ensure all packages installed as specified in the lockfile? 👀

➜  superset-frontend git:(master) npm run build-dev

> [email protected] build-dev
> cross-env NODE_OPTIONS=--max_old_space_size=8192 NODE_ENV=development webpack --mode=development --color

[Superset Plugin] Use symlink source for @superset-ui/chart-controls @ ./packages/superset-ui-chart-controls
[Superset Plugin] Use symlink source for @superset-ui/core @ ./packages/superset-ui-core
[Superset Plugin] Use symlink source for @superset-ui/legacy-plugin-chart-calendar @ ./plugins/legacy-plugin-chart-calendar
[Superset Plugin] Use symlink source for @superset-ui/legacy-plugin-chart-chord @ ./plugins/legacy-plugin-chart-chord
[Superset Plugin] Use symlink source for @superset-ui/legacy-plugin-chart-country-map @ ./plugins/legacy-plugin-chart-country-map
[Superset Plugin] Use symlink source for @superset-ui/legacy-plugin-chart-event-flow @ ./plugins/legacy-plugin-chart-event-flow
[Superset Plugin] Use symlink source for @superset-ui/legacy-plugin-chart-heatmap @ ./plugins/legacy-plugin-chart-heatmap
[Superset Plugin] Use symlink source for @superset-ui/legacy-plugin-chart-histogram @ ./plugins/legacy-plugin-chart-histogram
[Superset Plugin] Use symlink source for @superset-ui/legacy-plugin-chart-horizon @ ./plugins/legacy-plugin-chart-horizon
[Superset Plugin] Use symlink source for @superset-ui/legacy-plugin-chart-map-box @ ./plugins/legacy-plugin-chart-map-box
[Superset Plugin] Use symlink source for @superset-ui/legacy-plugin-chart-paired-t-test @ ./plugins/legacy-plugin-chart-paired-t-test
[Superset Plugin] Use symlink source for @superset-ui/legacy-plugin-chart-parallel-coordinates @ ./plugins/legacy-plugin-chart-parallel-coordinates
[Superset Plugin] Use symlink source for @superset-ui/legacy-plugin-chart-partition @ ./plugins/legacy-plugin-chart-partition
[Superset Plugin] Use symlink source for @superset-ui/legacy-plugin-chart-rose @ ./plugins/legacy-plugin-chart-rose
[Superset Plugin] Use symlink source for @superset-ui/legacy-plugin-chart-sankey @ ./plugins/legacy-plugin-chart-sankey
[Superset Plugin] Use symlink source for @superset-ui/legacy-plugin-chart-sankey-loop @ ./plugins/legacy-plugin-chart-sankey-loop
[Superset Plugin] Use symlink source for @superset-ui/legacy-plugin-chart-world-map @ ./plugins/legacy-plugin-chart-world-map
[Superset Plugin] Use symlink source for @superset-ui/legacy-preset-chart-deckgl @ ./plugins/legacy-preset-chart-deckgl
[Superset Plugin] Use symlink source for @superset-ui/legacy-preset-chart-nvd3 @ ./plugins/legacy-preset-chart-nvd3
[Superset Plugin] Use symlink source for @superset-ui/plugin-chart-echarts @ ./plugins/plugin-chart-echarts
[Superset Plugin] Use symlink source for @superset-ui/plugin-chart-handlebars @ ./plugins/plugin-chart-handlebars
[Superset Plugin] Use symlink source for @superset-ui/plugin-chart-pivot-table @ ./plugins/plugin-chart-pivot-table
[Superset Plugin] Use symlink source for @superset-ui/plugin-chart-table @ ./plugins/plugin-chart-table
[Superset Plugin] Use symlink source for @superset-ui/plugin-chart-word-cloud @ ./plugins/plugin-chart-word-cloud
[Superset Plugin] Use symlink source for @superset-ui/switchboard @ ./packages/superset-ui-switchboard

28% building 4/12 entries 58/59 dependencies 25/41 modules`isModuleDeclaration` has been deprecated, please migrate to `isImportOrExportDeclaration`
    at isModuleDeclaration (/Users/dotronghai/Projects/github.com/apache/superset/superset-frontend/node_modules/@babel/types/lib/validators/generated/index.js:2748:35)
    at PluginPass.Program (/Users/dotronghai/Projects/github.com/apache/superset/superset-frontend/node_modules/babel-plugin-lodash/lib/index.js:102:44)
`isModuleDeclaration` has been deprecated, please migrate to `isImportOrExportDeclaration`
    at isModuleDeclaration (/Users/dotronghai/Projects/github.com/apache/superset/superset-frontend/node_modules/@babel/types/lib/validators/generated/index.js:2748:35)
    at PluginPass.Program (/Users/dotronghai/Projects/github.com/apache/superset/superset-frontend/node_modules/babel-plugin-lodash/lib/index.js:102:44)
`isModuleDeclaration` has been deprecated, please migrate to `isImportOrExportDeclaration`
    at isModuleDeclaration (/Users/dotronghai/Projects/github.com/apache/superset/superset-frontend/node_modules/@babel/types/lib/validators/generated/index.js:2748:35)
    at PluginPass.Program (/Users/dotronghai/Projects/github.com/apache/superset/superset-frontend/node_modules/babel-plugin-lodash/lib/index.js:102:44)
`isModuleDeclaration` has been deprecated, please migrate to `isImportOrExportDeclaration`
    at isModuleDeclaration (/Users/dotronghai/Projects/github.com/apache/superset/superset-frontend/node_modules/@babel/types/lib/validators/generated/index.js:2748:35)
    at PluginPass.Program (/Users/dotronghai/Projects/github.com/apache/superset/superset-frontend/node_modules/babel-plugin-lodash/lib/index.js:102:44)
`isModuleDeclaration` has been deprecated, please migrate to `isImportOrExportDeclaration`
    at isModuleDeclaration (/Users/dotronghai/Projects/github.com/apache/superset/superset-frontend/node_modules/@babel/types/lib/validators/generated/index.js:2748:35)
    at PluginPass.Program (/Users/dotronghai/Projects/github.com/apache/superset/superset-frontend/node_modules/babel-plugin-lodash/lib/index.js:102:44)
28% building 4/12 entries 563/594 dependencies 84/248 modules`isModuleDeclaration` has been deprecated, please migrate to `isImportOrExportDeclaration`
    at isModuleDeclaration (/Users/dotronghai/Projects/github.com/apache/superset/superset-frontend/node_modules/@babel/types/lib/validators/generated/index.js:2748:35)
    at PluginPass.Program (/Users/dotronghai/Projects/github.com/apache/superset/superset-frontend/node_modules/babel-plugin-lodash/lib/index.js:102:44)
`isModuleDeclaration` has been deprecated, please migrate to `isImportOrExportDeclaration`
    at isModuleDeclaration (/Users/dotronghai/Projects/github.com/apache/superset/superset-frontend/node_modules/@babel/types/lib/validators/generated/index.js:2748:35)
    at PluginPass.Program (/Users/dotronghai/Projects/github.com/apache/superset/superset-frontend/node_modules/babel-plugin-lodash/lib/index.js:102:44)
28% building 4/12 entries 851/1000 dependencies 177/299 modules`isModuleDeclaration` has been deprecated, please migrate to `isImportOrExportDeclaration`
    at isModuleDeclaration (/Users/dotronghai/Projects/github.com/apache/superset/superset-frontend/node_modules/@babel/types/lib/validators/generated/index.js:2748:35)
    at PluginPass.Program (/Users/dotronghai/Projects/github.com/apache/superset/superset-frontend/node_modules/babel-plugin-lodash/lib/index.js:102:44)
28% building 4/12 entries 1000/1071 dependencies 186/375 modules`isModuleDeclaration` has been deprecated, please migrate to `isImportOrExportDeclaration`
    at isModuleDeclaration (/Users/dotronghai/Projects/github.com/apache/superset/superset-frontend/node_modules/@babel/types/lib/validators/generated/index.js:2748:35)
    at PluginPass.Program (/Users/dotronghai/Projects/github.com/apache/superset/superset-frontend/node_modules/babel-plugin-lodash/lib/index.js:102:44)
`isModuleDeclaration` has been deprecated, please migrate to `isImportOrExportDeclaration`
    at isModuleDeclaration (/Users/dotronghai/Projects/github.com/apache/superset/superset-frontend/node_modules/@babel/types/lib/validators/generated/index.js:2748:35)
    at PluginPass.Program (/Users/dotronghai/Projects/github.com/apache/superset/superset-frontend/node_modules/babel-plugin-lodash/lib/index.js:102:44)
1059 assets
14017 modules
webpack 5.89.0 compiled successfully in 29595 ms

mistercrunch · 2024-07-22T17:32:37Z

Confirming that running npm i; npm run build-dev worked for me. @fisjac please open an issue with more details on your env and error messages if you can recreate.

fisjac · 2024-07-22T17:49:23Z

I believe this error is specific to building with docker. When I sh into the superset_app contianer, I see that the optional dependencies aren't being built within the node_modules directory. Building locally with npm i seems fine. Opening a separate issue ticket.

…lift Yeoman-related packages + ESM-ize generator-superset (#29419) Signed-off-by: hainenber <[email protected]>

eschutho · 2024-07-27T00:39:38Z

I'm hitting the same issue that @fisjac reported in our ci now. https://github.com/apache/superset/actions/runs/10119252433/job/27987414693?pr=29723

and also when running docker compose up superset
I believe it's related to nrwl/nx#20088

kgabryje · 2024-08-01T16:42:22Z

I'm also hitting it here #29788 🙁

eschutho · 2024-08-05T20:40:44Z

The side effects of not loading the dependencies were fixed with @mistercrunch's PR here. But I noted in @kgabryje's PR that the lock file can still get out of sync because it will only pull in the dependency required for the OS that built the lockfile. Although at last look the package-lock file has all the dependencies.

hainenber added 3 commits June 30, 2024 20:32

chore(build): migrate fontsource-fira-code to correct workspace

d0467da

Signed-off-by: hainenber <[email protected]>

feat(build): migrate Lerna to v8 and perform corresponding refactors

bf5d777

Signed-off-by: hainenber <[email protected]>

feat(build): replace non-secure shortid with modern nanoid and pe…

b447b0f

…rform corresponding refactors Signed-off-by: hainenber <[email protected]>

pull-request-size bot added the size/L label Jun 30, 2024

github-actions bot added the dependencies:npm label Jun 30, 2024

hainenber added 3 commits June 30, 2024 23:03

fix(ci): add EoF newline to pass pre-commit check

ffc2c0b

Signed-off-by: hainenber <[email protected]>

fix(ci): sync lockfile for cypress-base package

9c42ebd

Signed-off-by: hainenber <[email protected]>

fix(cd): restrict query's client_id to 11 char to avoid breaking DB's…

2ee2f81

… column char limit Signed-off-by: hainenber <[email protected]>

michael-s-molina added the review:draft label Jul 1, 2024

feat(pkg/generator): ESM-ize the package + uplift Yeoman-related libr…

d4cec32

…aries + fix Jest tests + add specific ESLint config Signed-off-by: hainenber <[email protected]>

github-actions bot added the packages label Jul 13, 2024

hainenber added 6 commits July 13, 2024 23:44

Merge branch 'master' into uplift-major-version-for-ci-related-packages

a63ae1d

Signed-off-by: hainenber <[email protected]>

chore(build/frontend): sync lockfile with latest master branch

777e5f5

Signed-off-by: hainenber <[email protected]>

fix(ci): add EoF newline to prevent failed pre-commit check

e94dffe

Signed-off-by: hainenber <[email protected]>

fix(ci): sync lockfile with latest master branch

918d423

Signed-off-by: hainenber <[email protected]>

chore: add LICENSE banner to newly added .eslintrc

91f7cdd

Signed-off-by: hainenber <[email protected]>

fix(ci): run Jest test with ESM support for generator-superset

604e100

Signed-off-by: hainenber <[email protected]>

github-actions bot added the github_actions Pull requests that update GitHub Actions code label Jul 13, 2024

hainenber marked this pull request as ready for review July 14, 2024 05:17

hainenber requested review from villebro, geido, eschutho, rusackas, betodealmeida, nytai, mistercrunch, craig-rueda, john-bodley, kgabryje and dpgaspar as code owners July 14, 2024 05:17

hainenber requested a review from jinghua-qa as a code owner July 14, 2024 05:18

hainenber changed the title ~~feat(build): uplift major version for CI-related packages~~ feat(build): uplift Lerna + replace insecure shortid with nanoid + uplift Yeoman-related packages + ESM-ize generator-superset Jul 14, 2024

rusackas reviewed Jul 16, 2024

View reviewed changes

mistercrunch approved these changes Jul 16, 2024

View reviewed changes

rusackas removed the review:draft label Jul 16, 2024

rusackas approved these changes Jul 16, 2024

View reviewed changes

rusackas merged commit c30ca53 into apache:master Jul 16, 2024
62 checks passed

hainenber deleted the uplift-major-version-for-ci-related-packages branch July 17, 2024 01:58

fisjac mentioned this pull request Jul 22, 2024

superset_node build failure @nx/nx-linux-x64-gnu missing module #29658

Closed

3 tasks

eschutho pushed a commit that referenced this pull request Jul 24, 2024

feat(build): uplift Lerna + replace insecure shortid with nanoid + up…

8fb6058

…lift Yeoman-related packages + ESM-ize generator-superset (#29419) Signed-off-by: hainenber <[email protected]>

github-actions bot added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 4.1.0 labels Nov 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(build): uplift Lerna + replace insecure shortid with nanoid + uplift Yeoman-related packages + ESM-ize generator-superset #29419

feat(build): uplift Lerna + replace insecure shortid with nanoid + uplift Yeoman-related packages + ESM-ize generator-superset #29419

hainenber commented Jun 30, 2024 •

edited

Loading

rusackas left a comment

mistercrunch left a comment

mistercrunch commented Jul 16, 2024

rusackas left a comment

fisjac commented Jul 19, 2024

hainenber commented Jul 21, 2024

mistercrunch commented Jul 22, 2024

fisjac commented Jul 22, 2024

eschutho commented Jul 27, 2024 •

edited

Loading

kgabryje commented Aug 1, 2024

eschutho commented Aug 5, 2024 •

edited

Loading

feat(build): uplift Lerna + replace insecure shortid with nanoid + uplift Yeoman-related packages + ESM-ize generator-superset #29419

feat(build): uplift Lerna + replace insecure shortid with nanoid + uplift Yeoman-related packages + ESM-ize generator-superset #29419

Conversation

hainenber commented Jun 30, 2024 • edited Loading

SUMMARY

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

rusackas left a comment

Choose a reason for hiding this comment

mistercrunch left a comment

Choose a reason for hiding this comment

mistercrunch commented Jul 16, 2024

rusackas left a comment

Choose a reason for hiding this comment

fisjac commented Jul 19, 2024

hainenber commented Jul 21, 2024

mistercrunch commented Jul 22, 2024

fisjac commented Jul 22, 2024

eschutho commented Jul 27, 2024 • edited Loading

kgabryje commented Aug 1, 2024

eschutho commented Aug 5, 2024 • edited Loading

hainenber commented Jun 30, 2024 •

edited

Loading

eschutho commented Jul 27, 2024 •

edited

Loading

eschutho commented Aug 5, 2024 •

edited

Loading