Support packages and plugins co-existing in the same folder

[stacey-gammon]

Problem statement

Our current directory structure, of keeping packages and plugins separate, causes some issues:

1. Increasing bundle size

A developer building a plugin will usually export stateless helper utilities from the index.ts file. This unnecessarily increases bundle size and slows down the page load for every application. We have plugins that don’t need to be plugins at all (kibana_react, kibana_utils).

2. Discoverability

If that developer does understand the benefits of putting stateless code in a package, we’ll end up with related code in two folders that are far away from each other. This leads to duplicate platform utilities, a larger public API footprint, and services that are hard to find.

Goals

1. Decrease page load times in Kibana
2. Improve discoverability of platform services
3. Reduce duplicate code, which will improve maintenance

Proposal

1. Support the ability for packages and plugins to co-exist in the same folder.

For example:

Would simply become:

2. All stateless index.ts exports would automatically part of the package. Relative plugin exports would be no more.

As far as I am aware, the support for this will actually fall out naturally with some of the work the Ops team is already doing.

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[stacey-gammon]

Should we consider following the same strategy for packages that we have for plugins in terms of using "scope" folders, common, public, server? I suspect it will be difficult to merge them without creating consistency.

[mattkime]

I've been thinking about tradeoffs between using packages, plugins, and how discoverability is affected. This seems to offer the best of both worlds. It would be nice to use this with the data and data_views plugins.

[stacey-gammon]

Note, this is currently blocked on #107648, as moving code into packages will increase build times compared to plugins.

[tylersmalley]

Closing in favor of #128012