Deprecate generating custom Beats

[kvch]

What does this PR do?

This PR adds a warning to users who are generating their own Beats. From 8.0.0, it is not going to be supported.

Why is it important?

To manage users' expectations.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[mergify]

This pull request does not have a backport label. Could you fix it @kvch? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2021-11-04T16:51:54.353+0000

• Duration: 109 min 14 sec

• Commit: 600882a

Test stats 🧪

Test	Results
Failed	0
Passed	54122
Skipped	5354
Total	59476

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

[georgebridgeman]

I'd like to hear more about the reason for deprecating custom beat generation.

Is there an issue or discussion where I can find out more? I've only found issues mentioning the work to be done, such as #28427.

[nimarezainia]

@georgebridgeman the beat generator was experimental and not supported by Elastic. All that said we have been spending considerable amount of time debugging custom code to help the community. But due to the fact that it's best effort, it was creating delay in our responsiveness to our customers.
I'm reviewing this. We (I) should have been better at proactively providing notice and an alternative for the community who is building their own.

7.16 is log lived. the community still has access to the beat generator.
Would you be able to let us know how you were using this and whether there are alternatives we could provide?

[georgebridgeman]

@nimarezainia

the beat generator was experimental and not supported by Elastic.

I don't believe this. It was never stated in any of the Developer Guide documentation, even as of 7.15. If it was known internally, not putting a notice in the docs is very bad form as it gives the community more faith in the framework than it deserves.

All that said we have been spending considerable amount of time debugging custom code to help the community.

I can believe this, but mostly because the documentation is poor. I've contributed code and documentation to beats through work, and even a simple PR to fix documentation took a year to be reviewed. Improving the documentation, or supporting the community to do so, may have reduced the support required.

7.16 is log lived.

Not as long lived as code built using that generator. Pulling support for the generator shows Elastic don't want people to build custom Beats. It's a massive red flag and - going forward - nobody in their right mind is going to invest in building a production Beat for a new data project.

Would you be able to let us know how you were using this and whether there are alternatives we could provide?

I wrote a custom Beat to pull state from Cisco ACI into Elasticsearch (via Logstash). The alternative I'm going to fall back to (we've already committed to reimplement our Beat) is a Python script, with Celery (or similar) to manage the tasks. Any alternative like this requires a lot of functionality that a custom Beat gave for free, like interpreting response codes, backing off, coping with back-pressure, etc. Custom Beats also allow for great observability as you could use Metricbeat easily and see the state in Kibana.

There were plenty of signals that this change was coming. I half-jokingly suggested to colleagues a few months ago that Elastic will deprecate custom Beats before long. I'm just interested in the reasoning for it.

[ruflin]

Hi @georgebridgeman As the initial creator of the beat generator I would like to share a bit more background here.

A few years ago we create the beat generator to bring some standardisation on how beats are generated to the community. The list of community beats kept growing on a daily base which was amazing to see. One of the main problems we had, every beat had its own main files and imported X but not Y so when we upgrade libbeat, things didn't work anymore. At least with the beat generator we new more or less what most community beats were using including the folder structure. Having said that, we never did a great job especially across majors to provide enough guidance to the community beats on how to get to the next version.

The beat generator was mostly used by engineers and devops that know their way around Golang, Git etc. If the generator was broken or the docs, people like you @georgebridgeman contributed back to get it fixed (I'm sorry that we didn't get these contributions in quick enough). The goal of the beat generator was to get you off the ground. As soon as you have the beat, the generator becomes mostly useless. The part that changed over the last 2 years is that the expectations changed on what the beats generator does. Instead of focusing on fixing issues around the generator we had users that are missing git, no or old golang versions, go modules problems, platforms we never built it for and so on. Often it was not clear if the issue was the generator or the users system. That doesn't mean we shouldn't have better docs for it.

There are several ways on how to extend beats. One is building your own beat, alternatives are for example taking metricbeat as a shell like auditbeat does and create your own Beat for it with all the modules inside. This is all still possible and your existing community beats will still keep working. What we take away is the templating part. Also it is important to note, beat generator stays around in 7.16 and as you see here in beats 8.x use still 7.x libbeat as nothing has really changed on that front. So the community beats generated with this generator will keep working with 8.x.

One thing that was never great around the beat generator is that even though you got a lot of tooling out of the box with libbeat, it was always quite a bit of work to keep these beats up to date. One of the questions we ask ourself is how we can fix this?

The most fundamental question for me here is if Beats/Elastic needs and extension mechanism for data collection. The answer is simple from my perspective and it is a YES. One of the topics we keep discussing is around an EXEC input, see #18323. You would still get all the Beats magic but you could write your input in your preferred language. With Elastic Agent as supervisor there are now also new options. For example in theory Elastic Agent could run community beats and even manage them. Or Elastic Agent could run just the input piece mentioned above, it could even be delivered as part of an Elastic Package. Be aware, I'm in brainstorming mode here but I want to make it clear, removing beats generator does not mean removing custom beats, these keep working. And on the extension side we are working on ideas how to make it easier to plug in your collection code.

@georgebridgeman I hope the above shares a bit more background information, I really appreciate you chiming in on this issue.