Package publishing workflow and Angular packaging update [Retroactive PR]

[rajsite]

Pull Request

🤨 Rationale

This PR covers the changes associated with TASK 1744736 and covers the commits pushed directly to main to address the task.

Note: This PR will not be merged, it is being used to gather feedback for a follow-up PR.

The goal of this change was to be able to customize the publish behavior for Beachball during the beachball publish action (which does versioning and publishing) and this was accomplished by completely decoupling package versioning from package publishing.

Before these changes the workflow was as follows:

• main workflow runs build and test
• main workflow runs publish
  • beachball assumes each package is a normal npm package (build contents are stored in package directory and npm publish can be called in a directory)
  • Angular does not create normal npm packages, instead it forms a dist folder outside of the package folder with a different custom generated package.json and folder structure
  • We had a workaround that ran during the npm pack phase to mutate the package on disk to copy in the contents of the external dist folder temporarily so beachball can publish it as if it was a normal npm package
    • This is a bit messy and error prone and an oversight in this process when switching to ViewEngine based libraries was one of the issues resulting in the sl-header build stack overflows
  • beachball publish -> creates a temporary branch, updates the package.jsons for new versions, calls npm publish in each package (which causes the Angular pack dance above), pushes the branch remote as a new commit with tags, reverts the temporary branch locally in the CI
• main workflow creates a lockfile by pulling in the remote branch, running npm install (which relies on the published packages existing in npmjs.com), and commits the new lockfile in a separate commit.

So at a high-level changes to the above workflow are the following:

• beachball was updated to not publish to npm during the beachball publish step
  • It still runs the branch, versioning, push with tags, delete branch steps.
    • the minimal change would have been skipping publishing of just the angular package, but that did not seem to work. From source it looks like beachball only checks the publish option once for all packages and the publish option not a condition tested per package.
  • The separate workaround was added so lockfile updates happen during this step (text editing of the package-lock.json)
• A separate workflow listens for the tags created by beachball and publishes the associated packages

The new workflow is as follows:

• main workflow runs build and test
• main workflow runs publish
  • beachball publish updates the package jsons, updates the lockfile with the new workaround, and creates tags and pushes them, but does not push to npm
• tag workflows run once for each tag (currently up to three; nimble-angular, nimble-components, nimble-tokens)
• tag workflows run build (but do not re-run tests, assumes tags came from main and are tested)
• tag workflows run specific publish action
  • which is npm publish inside the package for normal packages and custom behavior for npm (running npm publish in the dist folder)

Pros:

• Can publish out of the Angular dist folder which is Angular's expected workflow instead of the workaround needed to pretend to be a normal npm package.
• Can do more advanced publishing for a specific tag, ie next step is add the ViewEngine build to create a separate parallel package built with View Engine
• The lockfile update workaround does not pull from the registry. This was always a concern because it can pull in random updates and it does it at a dangerous stage after all testing has run. The lockfile update can pull in new packages that evade all testing and is immediately published.

Cons:

• Each tag is a separate git object and triggers a separate build. They all run in parallel so it's not too bad time wise but it is wasteful in resources.
• The publish is being done from a separate build that did not undergo testing. The goal of lockfiles etc is reproducible builds so if that is working correctly this is not a practical concern. Just feels a bit better knowing the exact file that was published is one that was tested. Probably wouldn't increase complexity to reproduce that though.
• The package-lock.json update is done from a brute force text search and replace rather than using tools that can safely manipulate a package-lock.json
  • Ideally beachball would handle this for us. Surprisingly it does not look like npm handles this correctly either 🤦‍♂️

👩‍💻 Implementation

See above.

🧪 Testing

Did it live, in production :)

✅ Checklist

• I have updated the project documentation to reflect my changes or determined no changes are needed. I don't think it impacts developers, just infrastructure changes.

[rajsite]

Closing in favor of #209 which addresses feedback in this PR.