Internal Dependency Management

[tylersmalley]

FYI: We're working on defining more about about how packages will work and will be passing around a project specification for input from primary stakeholders but in the meantime we think this effort will interact with a lot of existing issues for the Operations team so we wanted to put something on Github that we can point people to.

---

Kibana currently has essentially three types of "code":

1. plugins, which are automatically transpiled in the server and bundled using the optimizer for the front-end. When changes happen in plugins either the server is automatically restarted or the optimizer automatically rebuilds their bundle. Refreshes in the browser pause while changes in files are reflected on the file system or while the server is restarted and updated modules are loaded
2. packages, built using bazel to produce specific outputs and rebuilt during bootstrap or as files change when the yarn kbn watch command is running. The server will restart when the built output is updated, and the optimizer will rebundle when they change as well, but we don't currently integrate with this external build process for pausing network requests or anything. Packages can be customized to support arbitrary build tasks via bazel, but don't provide out of the box support for plugins at this time
3. legacy root level files not associated with any specific package or plugin which are automatically transpiled via babel/register and don't have any browser-side component.

This issue represents the Operations teams' intention to move Kibana to a more cohesive code-organization strategy where every piece of code in the Kibana repository is organized via packages. These packages will behave similarly to the packages we have today, they will have files like package.json, BUILD.bazel, expose a single interface via a global module ID like @kbn/foo, but will also have a few new features:

• can live anywhere in the repo, like src/observability/kbn-observability-package
• builds will be integrated with the dev CLI so that as builds occur network requests from the browser are paused
• packages will be able to optionally expose a plugin which will be used by the plugin system, causing a plugin bundle to be created by the build tasks for that package and that bundle will be loaded by the plugin system for use in the browser

Today we have a package generator for teams to start experimenting with breaking some shared logic out of their plugins and moving it into a package. The intention is to make this process easy so that we can do this for all of the things that plugins are currently sharing via static exports. This will enable much easier code sharing without the massive drawbacks static exports have on page load times and bundle sizes, not to mention the overhead of new plugins created to mitigate some of the existing drawbacks.

The goal is to be able to automatically convert plugins to packages at some point. This might be harder than we originally expected, but we're still hoping to provide something like this. Without it we'll likely need to have a transition period where plugins can exist in the current system and be migrated to the new package-based approach, but we haven't defined how that will work at this point.

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)