[Meta] Monorepo and package-based publishing

[thompsongl]

EUI has had the long-term goal to increase modularity, and recent efforts related to styling (#3912; moving away from Sass) have opened the door to beginning the process in earnest.

A rough outline of phasing:

Validation

• Research monorepo infrastructure (e.g., yarn workspaces, lerna, turborepo)
• Establish future build system

Utilities

• Establish utilities and services that can exist in a standalone package
• Theming needs to exist in a standalone fashion

Core & Supplements

• Establish which components compose the EUI core package
• Split non-core components into relevant packages
• @elastic/eui should still exist as an all-inclusive package

[chandlerprall]

Three thoughts to help find potential boundaries for packages:

1. look at usage in downstream projects, are there common groupings?
2. do our EUI PRs create logical groups?
3. how about EUI imports of other components?

[thompsongl]

I did an exploratory spike to expand my understanding of monorepos generally and where the pain points will be for EUI specifically. We will pick up the work again soon, but the gist of the outcome is as follows.

Tooling

Dependency management: yarn workspaces
Build system: TBD; turborepo was positive
Release help: TBD; changesets potentially

Phasing

It is unfeasible to begin splitting up @elastic/eui at this moment. We will need to take a phased approach where in certain cases the lines between packages are blurred and/or crossed. For instance, the docs site (src-docs/) presents a unique challenge in that it relies on files generated specifically for its consumption as part of build scrips in @elastic/eui, as well as importing files not part of @elastic/eui's top-level API.

Proposed tentative migration

Phase 1 - Transition

• Keep src-docs/
• Move src/ to packages/eui/
• Split packages only where necessary
  • proptypes-from-ts-props
  • Considerations for packages related to shared test and build utils (tsconfig, eslintrc)
• Be ok with blurred line between packages/eui/ and src-docs/
  • Jest
  • Cypress
  • Sass compilation

Phase 2 - Illustration

• @elastic/eui-illustration
• apps/docs-eui-illustration
• apps/figma-eui-illustration

Phase 3 - Non-component packages

• eui-theme
• eui-utils
• eui-services
• Considerations for similar groups

Phase 4 - Component packages

• Scope TBD
• Progressively reduce packages/eui until it simply imports and re-exports other EUI packages

Other recommendations

• Move all EUI build output into dist/
  • dist/lib, dist/es, etc.
• Find a scripting method to git mv files to packages to ensure git history is maintained

Outstanding questions & further exploration

Testing

• Jest configuration: per package or run from root?
• How does this affect the eventual move to react-testing-library

Dependencies

• Version resolution between packages: manual or hoist to root?
  • babel, @types/*, typescript, moment presented version conflict problems
• Investigate nohoist for packages

Publishing releases

• Investigate changesets
• Further investigate turborepo and alternatives
• CI
• Build and release scripts
  • Does moving to changesets unblock the Jenkins release job??

[yakhinvadim]

Just wanted to mention that we at Search UI are facing the same questions as in "Publishing releases" section.

We also found https://github.com/elastic/apm-agent-rum-js that seemingly had the same challenges and added some scripts for publishing: https://github.com/elastic/apm-agent-rum-js/tree/main/scripts. We plan to chat with them soon, let me know if you'd like to participate.

[thompsongl]

@yakhinvadim Yes, I'd like be involved in that chat!
As we progress with this work I'd also like to talk to the Search UI team about what (if any) EUI packages would be of use. Any input on drawing lines in Phase 3/4 would be helpful.

[yakhinvadim]

@thompsongl Awesome, we'll invite you when we set it up (likely not in the next 2 weeks).

As for which EUI packages to release first: our priorities have shifted, and we paused the project that needed to use EUI. We plan to return to it in the future, but I don't think it'll happen in the next 6 months, maybe a year.

But if you have no other inputs, we were planning to build something like this (early design):

[thompsongl]

Sounds good, @yakhinvadim! Thank you

[PhilippeOberti]

@thompsongl I see that you've looked turborepo, I'm curious if you've also looked at Nx? I've used it in a pretty big team (75 developers) and decent codebase for the last 5 years. I'm also using it on very small side projects (2 people). It's extremely powerful and very simple to use.
I don't know how it compares with turborepo but I highly recommend taking a look. You can get up and running in 30 seconds.

I'm happy to hop on a call if you need any details or have any questions :)

[thompsongl]

Hey @PhilippeOberti!
Someone else recommended Nx a week or two ago. We'll give it a look for sure!

[PhilippeOberti]

Awesome! I was part of the frontend infrastructure team so I've used Nx extensively, I've helped set it up for all the devs, implemented custom generators and executors and performed Nx upgrades...
I love this stuff so don't hesitate to reach out if you have any questions!! 😄

[thompsongl]

For handoff purposes:

thompsongl/eui@main...thompsongl:eui:turborepo
WIP branch in a current state that builds a valid @elastic/eui package with tests passing. It is meant as a reference branch and I did not consider git history, only functionality. So use it as you methodically work through each conceptual step and make smaller commits along the way.

I haven't made significant progress beyond the previous comment, so the team will need to make evaluations between the spike I've worked on and other contending approaches like https://nx.dev/

Regardless, the procedure for moving to a monorepo is similar.

1. git mv or otherwise move src/ to packages/eui to ensure git history remains.
   Same goes for any config/script/documentation that becomes a package.
   This is critical. Maintaining git history for src/ should be a blocker for any platform changes.

2. Update import paths and path reference
   Files in scripts/, various configs.
   Maybe ignore files in src-docs/ for now. Updates there will be involved.

The following assumes an approach whereby base/default configs live at root and packages/apps inherit and extend.
Another approach would to make the configs packages themselves that each package and app has as a dependency. Inheritance and extension is the paradigm either way.

3. Babel configs
   More config files in more locations. Every package and app is likely to get its own config that inherits from the root config.
   Note that the config at the root is required to be named babel.config.js

4. Webpack configs
   Probably just path updates.

5. TypeScript configs
   More config files in more locations. Every package and app is likely to get its own config that inherits from the root config.

6. Lint configs
   More config files in more locations. Every package and app is likely to get its own config that inherits from the root config.
   .eslintignore must be in cwd, so it can't be used in a monorepo structure. Use ignorePatterns in local .eslintrc instead

7. scripts/
   Move scripts to relevant locations. That is, src-docs/ gets its own set of scripts. Some can remain in root if it makes sense.

8. Introduce yarn workspaces and turborepo
   This could be done as Step 0, also. Doesn't have bearing on other steps outside of making yarn install work correctly.

9. Triage the various package.json files
   Some dependencies should live at root so the correct versions are installed globally.
   Cannot have multiple copies of babel/babel/core
@types dependencies probably make sense at root.

10. Update component import statements in src-docs/ to use @elastic/eui
index file at root of eui package is necessary for yarn workspaces install, but the dtsgenerator messes up when looking to the baseDir
    Another approach would be to import from @elastic/eui/src/

11. Introduce other packages
    https://github.com/miukimiu/eui-illustration is a high priority
    In my branch I split ^ into a packages and two apps

---

Still unverified:

• Using changesets for changelog

[JasonStoltz]

With #7736, the EUI repository has now been moved to a monorepo.

While this original Meta issues was written to capture the work for both the move to a monorepo as well as splitting the library into discrete packages, I'm going to close this issue as completed with this PR.

The EUI team will likely create separate packages in the future, but I think this Meta issue has outlived itself. If we decide to make that move, we will create a new issue to capture that work.