[GCP] add compute datastream tests #2849

endorama · 2022-03-18T16:43:49Z

This PR is part of the effort to migrate Beats metricsets to data stream and adding system tests for them.

What does this PR do?

add GCP envs to Jenkins pipelines
add gcp.compute datastream tests

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Migrate some Metricbeat modules to GCP package #490

Screenshots

elasticmachine · 2022-03-18T18:23:24Z

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-04-05T15:11:25.141+0000
Duration: 92 min 4 sec

Test stats 🧪

Test	Results
Failed	1
Passed	4306
Skipped	7
Total	4314

Test errors

Expand to view the tests failures

`Check integrations / kubernetes / kubernetes: check / system test: default – kubernetes.pod`

Expand to view the error details

 null

Expand to view the stacktrace

 one or more errors found in documents stored in metrics-kubernetes.pod-ep data stream: [0] field "container.network.egress.bytes" is undefined
[1] field "container.network.ingress.bytes" is undefined

Steps errors

Expand to view the steps failures

`Test integration: docker`

Took 3 min 31 sec . View more details here
Description: eval "$(../../build/elastic-package stack shellinit)" ../../build/elastic-package test -v --report-format xUnit --report-output file --test-coverage

`Take down the Elastic stack`

Took 0 min 16 sec . View more details here
Description: build/elastic-package stack down -v

`Check integration: gcp`

Took 0 min 3 sec . View more details here
Description: ../../build/elastic-package check -v

`Test integration: kubernetes`

Took 30 min 0 sec . View more details here
Description: eval "$(../../build/elastic-package stack shellinit)" ../../build/elastic-package test -v --report-format xUnit --report-output file --test-coverage

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

endorama · 2022-03-21T09:30:10Z

CI failed with a permission error on the GCP Service Account:

Error: Error loading zone 'us-central1-a': googleapi: Error 403: Required 'compute.zones.get' permission for 'projects/elastic-obs-integrations-dev/zones/us-central1-a', forbidden

@v1v I'm using the same SA we setup in this PR elastic/elastic-package#701 (see https://github.com/elastic/elastic-package/pull/701/files#diff-ef8b2f5e4d21d81c0eba4754dea3a3dfc2c67d26284f195165b650cf4aff7beaR23). Could it be the SA should be enabled for this repository?

v1v · 2022-03-21T09:33:43Z

Could it be the SA should be enabled for this repository?

IIRC, @mtojek created the service account, though I'm inclined to suggest to use another service account with the right permissions instead, to avoid messing up the other pipelines/consumers.

I can provide you some details about how to update the secret in vault if needed

endorama · 2022-03-21T10:14:50Z

I'm inclined to suggest to use another service account with the right permissions instead, to avoid messing up the other pipelines/consumers.

I think we want this to be the same SA for this reason.

But there was a bug in the code that's why the permissions error showed up, sorry for involving you

mtojek · 2022-03-21T14:31:30Z

IIRC, @mtojek created the service account, though I'm inclined to suggest to use another service account with the right permissions instead, to avoid messing up the other pipelines/consumers.

I suggest using the same GCP service account as elastic-package uses for GCP testing (elastic-package-gcp).

elastic-obs-integrations-dev

Shouldn't it be elastic-observability?

@endorama Please try not using the force-push mode as you lose the commit history and we don't know what have you tried and what haven't.

kaiyan-sheng · 2022-03-21T18:50:42Z

packages/gcp/manifest.yml

@@ -50,64 +50,6 @@ vars:
    required: false
    show_user: true
 policy_templates:
-  - name: audit


Are we moving these into a separate package?

No, but tests are failing with some gcp-pubsub related failures and I wanted to remove that to observe if the issue was being introduced by these changeset.

Failure:

[2022-03-21T17:00:29.474Z] Error: error running package system tests: could not complete test run: could not add data stream config to policy: could not add package to policy; API status code = 500; response body = {"statusCode":500,"error":"Internal Server Error","message":"Input template not found, unable to find input type gcp-pubsub"}

endorama · 2022-03-22T08:28:33Z

@mtojek

I suggest using the same GCP service account as elastic-package uses for GCP testing (elastic-package-gcp).

That's the plan

Shouldn't it be elastic-observability?

yes, there was a bug and the code didn't use the proper project, fixed in 9b6f981

@endorama Please try not using the force-push mode as you lose the commit history and we don't know what have you tried and what haven't.

You're right, sorry!

endorama · 2022-03-22T10:46:02Z

Due to the data streams test errors I see on this PR that are unrelated to the changes in this PR I'm pausing the work and ensure the CI is green in gcp-metrics branch, the branch this PR is based on.

aspacca · 2022-03-22T03:10:12Z

.ci/Jenkinsfile

@@ -257,6 +259,11 @@ def withCloudTestEnv(Closure body) {
    [var: "AWS_ACCESS_KEY_ID", password: aws.access_key],
    [var: "AWS_SECRET_ACCESS_KEY", password: aws.secret_key],
  ])
+  // GCP
+  withGCPEnv(secret: env.ELASTIC_PACKAGE_GCP_SECRET) {


where is withGCPEnv defined?

Here: https://github.com/elastic/apm-pipeline-library/blob/main/vars/withGCPEnv.groovy :)

This commit has been changed to remove changes to README included in #2141 to allow documentation changes from #2842 Changes to packages/gcp/data_stream/audit/manifest.yml has been removed too.

This commit has been changed to remove changes to README included in #2141 to allow documentation changes from #2842

This reverts commit fe54af3. `alternative_host` was a input level variable that is not shared by all inputs but used only in log related inputs. It was moved to data stream level but was not implemented for all data streams.

This commit has been changed to remove changes to README included in #2141 to allow documentation changes from #2842 Changes to packages/gcp/data_stream/audit/manifest.yml has been removed too.

This reverts commit afe2beef112fa288e8aafdc862988ee83a79391c.

This reverts commit 3f41f4dc5135db53ba97c9cf9e6d43e02f403a5d.

endorama requested review from a team as code owners March 18, 2022 16:43

endorama self-assigned this Mar 18, 2022

endorama added Integration:gcp Google Cloud Platform Team:Cloud-Monitoring Label for the Cloud Monitoring team labels Mar 18, 2022

endorama mentioned this pull request Mar 18, 2022

Migrate some Metricbeat modules to GCP package #490

Closed

11 tasks

kaiyan-sheng reviewed Mar 21, 2022

View reviewed changes

endorama force-pushed the gcp-metrics branch from 7ed83f9 to f248b18 Compare March 22, 2022 09:27

endorama force-pushed the gcp-metrics branch from 8cf80ec to fe54af3 Compare March 22, 2022 15:14

aspacca reviewed Mar 28, 2022

View reviewed changes

legoguy1000 and others added 7 commits April 5, 2022 17:07

[GCP] Add GCP Billing Metricset (#2141)

9718549

This commit has been changed to remove changes to README included in #2141 to allow documentation changes from #2842 Changes to packages/gcp/data_stream/audit/manifest.yml has been removed too.

[GCP] Add GCP Compute Metricset (#2301)

330797b

This commit has been changed to remove changes to README included in #2141 to allow documentation changes from #2842

remove edits to firewall and vpcflow data streams

5da526a

Revert "remove edits to firewall and vpcflow data streams"

64c30dd

This reverts commit fe54af3. `alternative_host` was a input level variable that is not shared by all inputs but used only in log related inputs. It was moved to data stream level but was not implemented for all data streams.

gcp: add alternative_host to audit and dns

49d037f

gcp: define alternative_host at data stream level

6a159de

CHANGELOG

cc98bd1

endorama force-pushed the gcp-metrics branch from aaab6bd to cc98bd1 Compare April 5, 2022 15:08

legoguy1000 and others added 4 commits April 5, 2022 17:10

[GCP] Add GCP Billing Metricset (#2141)

a695a27

This commit has been changed to remove changes to README included in #2141 to allow documentation changes from #2842 Changes to packages/gcp/data_stream/audit/manifest.yml has been removed too.

add GCP envs to Jenkins pipelines

1f5cef1

add gcp.compute datastream tests

7d7800f

fix package linting

fec888b

endorama added 9 commits April 5, 2022 17:10

fix project_id var name

a494a03

move compute definition on top

c4c1560

test: removing all other templates to focus on compute

b2dfa49

format

0ddfd3c

Revert "test: removing all other templates to focus on compute"

5b3f938

This reverts commit afe2beef112fa288e8aafdc862988ee83a79391c.

Revert "move compute definition on top"

bbfbc7d

This reverts commit 3f41f4dc5135db53ba97c9cf9e6d43e02f403a5d.

resolve testing instance conflict names

b339cc3

bump to 2.0.0

40fbb66

tag cloud resources

459646d

endorama force-pushed the gcp-metrics branch from 471f0fc to fa4dc99 Compare June 9, 2022 13:07

endorama force-pushed the gcp-metrics branch 2 times, most recently from edc3905 to 826b985 Compare July 13, 2022 07:54

endorama deleted the branch elastic:gcp-metrics July 14, 2022 08:49

endorama closed this Jul 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[GCP] add compute datastream tests #2849

[GCP] add compute datastream tests #2849

endorama commented Mar 18, 2022

elasticmachine commented Mar 18, 2022 •

edited

Loading

Build stats

Test stats 🧪

`Check integrations / kubernetes / kubernetes: check / system test: default – kubernetes.pod`

`Test integration: docker`

`Take down the Elastic stack`

`Check integration: gcp`

`Test integration: kubernetes`

endorama commented Mar 21, 2022

v1v commented Mar 21, 2022

endorama commented Mar 21, 2022 •

edited

Loading

mtojek commented Mar 21, 2022 •

edited

Loading

kaiyan-sheng Mar 21, 2022

endorama Mar 22, 2022

endorama commented Mar 22, 2022

endorama commented Mar 22, 2022

aspacca Mar 22, 2022

mtojek Mar 28, 2022

[GCP] add compute datastream tests #2849

[GCP] add compute datastream tests #2849

Conversation

endorama commented Mar 18, 2022

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Mar 18, 2022 • edited Loading

💔 Tests Failed

Build stats

Test stats 🧪

Test errors

Check integrations / kubernetes / kubernetes: check / system test: default – kubernetes.pod

Steps errors

Test integration: docker

Take down the Elastic stack

Check integration: gcp

Test integration: kubernetes

🤖 GitHub comments

endorama commented Mar 21, 2022

v1v commented Mar 21, 2022

endorama commented Mar 21, 2022 • edited Loading

mtojek commented Mar 21, 2022 • edited Loading

kaiyan-sheng Mar 21, 2022

Choose a reason for hiding this comment

endorama Mar 22, 2022

Choose a reason for hiding this comment

endorama commented Mar 22, 2022

endorama commented Mar 22, 2022

aspacca Mar 22, 2022

Choose a reason for hiding this comment

mtojek Mar 28, 2022

Choose a reason for hiding this comment

elasticmachine commented Mar 18, 2022 •

edited

Loading

`Check integrations / kubernetes / kubernetes: check / system test: default – kubernetes.pod`

`Test integration: docker`

`Take down the Elastic stack`

`Check integration: gcp`

`Test integration: kubernetes`

endorama commented Mar 21, 2022 •

edited

Loading

mtojek commented Mar 21, 2022 •

edited

Loading