Generate integrations from Beats modules

[exekias]

As Beats module development and packages registry will coexist for a while, it is in our interest to avoid duplicating the development process, to avoid human errors and inconsistencies. The aim of this issue is to automate as much as possible the generation of registry packages based on the Beats repo.

To scope this down we can initially focus on:

• Create an initial script that fills the directories an basic manifest.yml, reading from both Metricbeat and Filebeat modules (done)
• Keep iterating over the script to add support for importing as many aspects as possible from the original modules, including:
  • fields.yml
  • Dashboard screenshots
  • Dashboards and visualizations
  • Icons
  • Docs
  • Input configs (manifest.yml)
  • Streams (stream.yml)

We can initially focus on a small list of packages, and grow it later. In some cases we may need to update the Beats repo to include some metadata that is only useful for packages registry. For example, there are no icons in the beats repo, but we can add them for the shake of this process.

We may also find that automating the import process becomes too difficult for some pieces, we can leave some steps as manual, trying to keep these to a minimum.

There is an, now outdated, initial try at this: #137

[exekias]

cc @urso @ruflin feel free to comment / update

[mtojek]

Hi, I've started researching this area, browsing docs, etc., in general - preparing for development here.

Is there any model package that can be considered as template/archetype/skeleton?
Is the "nginx" package the best candidate? https://github.com/elastic/package-registry/tree/master/dev/package-examples/nginx-1.2.0

EDIT:

I apologize for multiple posts, Github is facing an outage.

[ruflin]

@mtojek Yes, the nginx package is the best skeleton at the moment to be used.

[mtojek]

I'm going with the initial step:

Create an initial script that fills the directories an basic manifest.yml, reading from both Metricbeat and Filebeat modules

I will keep you updated.

[mtojek]

@ruflin in https://github.com/elastic/package-registry/pull/137/files#r340031719

As discussed, we probably best start with 0.1.0?
As we keep doing conversions, I wonder if we should introduce on the beats side in a yaml file the target version for each module because over time we will generate multiple version.s

I'm not sure if I understand the concept of target versions, but I'm sure you have a vision, so I'd rather discuss it here instead of enforcing a different one.

(on the beats side)
Definitely it's easy to forget to manually bump up version/update the file. I was thinking about a merkle tree, a simple tree of hashes of all files inside the module. If anything changes, then hash will be different. We may have a semver-task per module to bump up a module's minor if the total hash has changed.
I don't see a problem to keep this file either on the beats or package-registry side.

/cc @exekias

[ruflin]

@mtojek I like the idea around having a hash that can note us about changes. Not sure if we need something like this from day one. For now it is ok even if we have different versions that all have the version 0.0.1 as it is not released yet.

[mtojek]

I'm going with fields.yml.

[mtojek]

I'm going with dashboards and visualizations.

[mtojek]

I'm going with icons.

[mtojek]

I'm going with docs.

[mtojek]

I'm going with input config.

[mtojek]

I'm going with streams.

[mtojek]

All items cleared. I will apply next changes/bugfixes/updates in the follow up issues.

Resolving.

[exekias]

Thank you for working on this! I think we need a plan about docs, right? Perhaps we can create an issue to discuss what we need to do and schedule it at some point

[mtojek]

Thank you for working on this! I think we need a plan about docs, right? Perhaps we can create an issue to discuss what we need to do and schedule it at some point

Issue has been created: #262