Suggestion: merge integrations, package-spec, and elastic-package projects

[andrewstucki]

Thinking about the breakdown of repos that we currently have related to packages:

https://github.com/elastic/integrations
https://github.com/elastic/elastic-package
https://github.com/elastic/package-spec
https://github.com/elastic/package-storage
https://github.com/elastic/package-registry

The separation of package-storage/package-registry make sense to me -- we have a repo for releasable artifacts + the hosting code. What doesn't quite make as much sense to me is the distinction between integration, elastic-package, and package-spec.

When developing integration packages, at times I have to refer back to the spec, which is found in another project and dictates package formats which are found pretty much only in this repo. Likewise, much of the documentation around how to use the development and testing tooling for the packages in this repo are actually contained in elastic-package. In other words, to develop integrations you really need to know about all 3 repos.

I think that we'd likely have less repo confusion and cognitive overhead by merging the package-spec, elastic-package, and integrations repos as they're all related directly to package development.

cc: @elastic/security-external-integrations , @mtojek , @ruflin

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[mtojek]

Actually it was split on purpose to separate concerns. There are different responsibilities of projects:

1. Package spec defines the spec of a package and ultimately will be used by package-registry, Kibana, package-storage and Integrations as package validator.
2. Integrations contain only sources of package integrations and are not "polluted" with any builder tools as beats (and mage tools).
3. Elastic package is a builder tool, which has a totally different release cycle and board. Ultimately you don't need to use it against the Integrations repo, but in every place you like (your own package folders or single files).

I'm happy to discuss this topic, but actually the separation of concerns was the main goal. /cc @ycombinator

[ycombinator]

The only thing I'd add to @mtojek's excellent summary is to elaborate a bit on the first point, specifically why elastic-package and package-spec are in different repos.

The package-spec repo can house validator implementations in multiple languages. At the moment we only have a validator implement in Golang so I can see why it would seem tempting to combine the package-spec repo with the elastic-package repo and reduce the cognitive complexity. However, we plan to add a Typescript one as well soon (elastic/package-spec#58). Once that happens, combining elastic-package and package-spec would make even less sense IMHO.

Regarding your point about documentation, I agree, this is a major pain point and we need to solve it. It would absolutely be desirable to have a single place to go for all package-related documentation. I'm not sure where that is just yet but once we have something, all these repos' READMEs need to link to it so there's no room for ambiguity and it's convenient as well.