Support “input” package type

[mtojek]

This is the meta issue for all actions required to introduce the new package type.

Background:

The concept of the Elastic Package is GA for a while and helped us migrate Beats modules to Elastic Integrations and create new integrations from scratch. So far, we identified only one package type - the Elastic Integration, which meets most of the technical requirements for data streams, fields, and agent policies.
It has been proved that the format of the Elastic Integration, defined by the Package Spec, is flexible enough to wrap Beats inputs as separate integrations (for example Custom Logs or HTTP JSON), but not so flexible to implement custom Ingest configurations. On the other hand, the Integration type seems to be an overkill for the Input-only package type, as it doesn’t define multiple data streams or come with a specific ingest pipeline.

Design doc: link

Let's review the doc, comment, and create subtasks.

Dependency:

• [Fleet] Add support for input type packages kibana#133296

[mtojek]

As agreed with @jlind23, I'm moving on with this topic:

• Distinguish two different package types in spec (and validate).
  • Prepare for support of multiple, independent package types #323
  • Add sample "input" packages #325
  • Write input package spec #328
  • Add positive and negative tests.
• Package Registry to support "input packages"
  • Support input packages package-registry#809
• Package Storage to support "input packages"
  • Serve "input packages", make sure that pseudo-validation works as intended
• Package Storage v2 to support "input packages"
  • https://github.com/elastic/package-storage-infra/pull/127 Use latest package spec
  • Adjust existing indexing logic to cover input packages, regenerate indices after all. It will happen anyway.
• Modify Package Spec to support _dev extensions
  • Enable development resources for input packages #350
• Elastic Package to support "input packages"
  • Support "input packages" elastic-package#811 (build, validate, stack)
  • Enable developer extensions elastic-package#840 (validate non-data stream fields, import fields from ECS, render table with fields in docs)
• Elastic Package to facilitate testing methods (blocked by Fleet's) part

[mtojek]

Status update on input packages:

elastic/elastic-package#840 - is the last PR enabling support for input package in elastic-package. Post-MVP: what is left is figuring out and adapting system tests logic, but to do this we need API support on the Fleet side.

To close this chapter of works we have to release a new version of package-spec, elastic-package, and include them in Integrations.