Cherry-pick #19125 to 7.x: Add support for cloudfoundry logs collection using consumer v1 API

[jsoriano]

Cherry-pick of PR #19125 to 7.x branch. Original message:

What does this PR do?

Add support for collection of logs from cloud foundry using consumer v1 API. v1 API uses doppler/trafficcontrol to get events from the firehose, v2 API uses the reverse log proxy (RLP).

v2 API is still supported, and can be selected by adding version: v2 to the configuration, v1 is used by default as is in principle more reliable. version is ignored by now by Metricbeat, that continues using v2. Support for v1 in Metricbeat will be implemented in a future change.

This change adds some additional dependencies for the doppler consumer (noaa).

Why is it important?

We already support the v2 API, but we saw that this API is not reliable enough (see #18202).

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• Review FIXMEs in code.
• Test with PCF

How to test this PR locally

• Run filebeat with the cloudfoundry input and check that it collects events. Try these combinations of the version option:
  • No version option (what would use v1)
  • version: v1.
  • version: v2.
  • For other values of version it should fail to start.

Related issues

• Fixes Filebeat : Cloud foundry input dropping logs #18202
• Relates [Meta] Cloud Foundry Support #16520
• Includes integration tests from Add basic cloudfoundry integration tests #19018

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[elasticmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [Pull request #19212 opened]

• Start Time: 2020-06-16T08:09:53.674+0000

• Duration: 83 min 44 sec

Test stats 🧪

Test	Results
Failed	0
Passed	6694
Skipped	1069
Total	7763

Steps errors

Expand to view the steps failures

• Name: Report to Codecov

  • Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done

  • Duration: 2 min 22 sec

  • Start Time: 2020-06-16T08:45:25.718+0000

  • log

• Name: Report to Codecov

  • Description: curl -sSLo codecov https://codecov.io/bash for i in auditbeat filebeat heartbeat libbeat metricbeat packetbeat winlogbeat journalbeat do FILE="${i}/build/coverage/full.cov" if [ -f "${FILE}" ]; then bash codecov -f "${FILE}" fi done

  • Duration: 2 min 22 sec

  • Start Time: 2020-06-16T09:29:42.790+0000

  • log

[ChrsMark]

@jsoriano how about fixing #19213 here to avoid having an extra backport PR?
nah, this PR seems to not have any lint issues? 🤔

[jsoriano]

@jsoriano how about fixing #19213 here to avoid having an extra backport PR?
nah, this PR seems to not have any lint issues? 🤔

#19213 seems to be needed in combination with #19135.