-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add_cloud_metadata - Retrieve AKS cluster name and id #37685
Merged
MichaelKatsoulis
merged 29 commits into
elastic:main
from
MichaelKatsoulis:add-aks-cluster-name-id
Mar 11, 2024
Merged
Changes from 20 commits
Commits
Show all changes
29 commits
Select commit
Hold shift + click to select a range
7e74460
Retrieve aks cluster name and id
MichaelKatsoulis 7906f04
Add tests
MichaelKatsoulis eada5b2
Update docs
MichaelKatsoulis c8d065a
Lint errors and notice txt
MichaelKatsoulis 2c1095d
make check updates
MichaelKatsoulis 47b4fea
Update tests
MichaelKatsoulis d6da3c5
Merge remote-tracking branch 'upstream/main' into add-aks-cluster-nam…
MichaelKatsoulis 775afc9
Use fake credentials
MichaelKatsoulis d778f99
Unused logger
MichaelKatsoulis 2f67b55
Review comments and declare new cloud.resourceGroup.name field
MichaelKatsoulis f728115
Rename var cloud.resourceGroup.name to cloud.resource_group.name
MichaelKatsoulis df2096b
Merge remote-tracking branch 'upstream/main' into add-aks-cluster-nam…
MichaelKatsoulis 406863c
Update CHANGELOG.next.asciidoc
MichaelKatsoulis c73cba1
Fix asciidoc file
MichaelKatsoulis 9c7cd50
Merge remote-tracking branch 'upstream/main' into add-aks-cluster-nam…
MichaelKatsoulis 0e08561
Rename new field to azure.resourcegroup.name to align with otel
MichaelKatsoulis ef3e8da
Update unit test
MichaelKatsoulis 70865f3
Update changelog
MichaelKatsoulis d57fa5f
Add azure.resourcegroup.name field under cloud key in fields.yml
MichaelKatsoulis 757fc01
Add fmt.Errorf in error returns
MichaelKatsoulis 884af33
Remove azure.resourcegroup.name
MichaelKatsoulis e8c5400
Update libbeat/processors/add_cloud_metadata/docs/add_cloud_metadata.…
MichaelKatsoulis 2ce7d88
Update libbeat/processors/add_cloud_metadata/docs/add_cloud_metadata.…
MichaelKatsoulis 17a1286
Update libbeat/processors/add_cloud_metadata/provider_azure_vm.go
MichaelKatsoulis 99db284
Update libbeat/processors/add_cloud_metadata/provider_azure_vm.go
MichaelKatsoulis df6f6c5
Update libbeat/processors/add_cloud_metadata/provider_azure_vm.go
MichaelKatsoulis c74720c
Update Id to ID
MichaelKatsoulis 665f9d5
Update libbeat/processors/add_cloud_metadata/provider_azure_vm.go
MichaelKatsoulis 785135f
Merge branch 'main' into add-aks-cluster-name-id
MichaelKatsoulis File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1773,11 +1773,11 @@ SOFTWARE | |
|
|
||
| -------------------------------------------------------------------------------- | ||
| Dependency : github.com/Azure/azure-sdk-for-go/sdk/azidentity | ||
| Version: v1.2.2 | ||
| Version: v1.4.0 | ||
| Licence type (autodetected): MIT | ||
| -------------------------------------------------------------------------------- | ||
|
|
||
| Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/azidentity@v1.2.2/LICENSE.txt: | ||
| Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/azidentity@v1.4.0/LICENSE.txt: | ||
|
|
||
| MIT License | ||
|
|
||
|
|
@@ -1832,6 +1832,36 @@ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, | |
| OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE | ||
| SOFTWARE. | ||
|
|
||
| -------------------------------------------------------------------------------- | ||
| Dependency : github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/v4 | ||
| Version: v4.6.0 | ||
| Licence type (autodetected): MIT | ||
| -------------------------------------------------------------------------------- | ||
|
|
||
| Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/resourcemanager/containerservice/armcontainerservice/[email protected]/LICENSE.txt: | ||
|
|
||
| MIT License | ||
|
|
||
| Copyright (c) Microsoft Corporation. All rights reserved. | ||
|
|
||
| Permission is hereby granted, free of charge, to any person obtaining a copy | ||
| of this software and associated documentation files (the "Software"), to deal | ||
| in the Software without restriction, including without limitation the rights | ||
| to use, copy, modify, merge, publish, distribute, sublicense, and/or sell | ||
| copies of the Software, and to permit persons to whom the Software is | ||
| furnished to do so, subject to the following conditions: | ||
|
|
||
| The above copyright notice and this permission notice shall be included in all | ||
| copies or substantial portions of the Software. | ||
|
|
||
| THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | ||
| IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | ||
| FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE | ||
| AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER | ||
| LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, | ||
| OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE | ||
| SOFTWARE. | ||
|
|
||
| -------------------------------------------------------------------------------- | ||
| Dependency : github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/costmanagement/armcostmanagement | ||
| Version: v1.1.0 | ||
|
|
@@ -1894,11 +1924,11 @@ SOFTWARE. | |
|
|
||
| -------------------------------------------------------------------------------- | ||
| Dependency : github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources | ||
| Version: v1.0.0 | ||
| Version: v1.1.1 | ||
| Licence type (autodetected): MIT | ||
| -------------------------------------------------------------------------------- | ||
|
|
||
| Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources@v1.0.0/LICENSE.txt: | ||
| Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/resourcemanager/resources/armresources@v1.1.1/LICENSE.txt: | ||
|
|
||
| MIT License | ||
|
|
||
|
|
@@ -29549,6 +29579,98 @@ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE | |
| SOFTWARE | ||
|
|
||
|
|
||
| -------------------------------------------------------------------------------- | ||
| Dependency : github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/internal | ||
| Version: v1.1.2 | ||
| Licence type (autodetected): MIT | ||
| -------------------------------------------------------------------------------- | ||
|
|
||
| Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/resourcemanager/[email protected]/LICENSE.txt: | ||
|
|
||
| MIT License | ||
|
|
||
| Copyright (c) Microsoft Corporation. | ||
|
|
||
| Permission is hereby granted, free of charge, to any person obtaining a copy | ||
| of this software and associated documentation files (the "Software"), to deal | ||
| in the Software without restriction, including without limitation the rights | ||
| to use, copy, modify, merge, publish, distribute, sublicense, and/or sell | ||
| copies of the Software, and to permit persons to whom the Software is | ||
| furnished to do so, subject to the following conditions: | ||
|
|
||
| The above copyright notice and this permission notice shall be included in all | ||
| copies or substantial portions of the Software. | ||
|
|
||
| THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | ||
| IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | ||
| FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE | ||
| AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER | ||
| LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, | ||
| OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE | ||
| SOFTWARE | ||
|
|
||
|
|
||
| -------------------------------------------------------------------------------- | ||
| Dependency : github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/internal/v2 | ||
| Version: v2.0.0 | ||
| Licence type (autodetected): MIT | ||
| -------------------------------------------------------------------------------- | ||
|
|
||
| Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/resourcemanager/internal/[email protected]/LICENSE.txt: | ||
|
|
||
| MIT License | ||
|
|
||
| Copyright (c) Microsoft Corporation. | ||
|
|
||
| Permission is hereby granted, free of charge, to any person obtaining a copy | ||
| of this software and associated documentation files (the "Software"), to deal | ||
| in the Software without restriction, including without limitation the rights | ||
| to use, copy, modify, merge, publish, distribute, sublicense, and/or sell | ||
| copies of the Software, and to permit persons to whom the Software is | ||
| furnished to do so, subject to the following conditions: | ||
|
|
||
| The above copyright notice and this permission notice shall be included in all | ||
| copies or substantial portions of the Software. | ||
|
|
||
| THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | ||
| IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | ||
| FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE | ||
| AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER | ||
| LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, | ||
| OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE | ||
| SOFTWARE | ||
|
|
||
|
|
||
| -------------------------------------------------------------------------------- | ||
| Dependency : github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/managementgroups/armmanagementgroups | ||
| Version: v1.0.0 | ||
| Licence type (autodetected): MIT | ||
| -------------------------------------------------------------------------------- | ||
|
|
||
| Contents of probable licence file $GOMODCACHE/github.com/!azure/azure-sdk-for-go/sdk/resourcemanager/managementgroups/[email protected]/LICENSE.txt: | ||
|
|
||
| MIT License | ||
|
|
||
| Copyright (c) Microsoft Corporation. All rights reserved. | ||
|
|
||
| Permission is hereby granted, free of charge, to any person obtaining a copy | ||
| of this software and associated documentation files (the "Software"), to deal | ||
| in the Software without restriction, including without limitation the rights | ||
| to use, copy, modify, merge, publish, distribute, sublicense, and/or sell | ||
| copies of the Software, and to permit persons to whom the Software is | ||
| furnished to do so, subject to the following conditions: | ||
|
|
||
| The above copyright notice and this permission notice shall be included in all | ||
| copies or substantial portions of the Software. | ||
|
|
||
| THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | ||
| IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | ||
| FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE | ||
| AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER | ||
| LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, | ||
| OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE | ||
| SOFTWARE. | ||
|
|
||
| -------------------------------------------------------------------------------- | ||
| Dependency : github.com/Azure/go-amqp | ||
| Version: v0.16.0 | ||
|
|
@@ -31253,11 +31375,11 @@ SOFTWARE. | |
|
|
||
| -------------------------------------------------------------------------------- | ||
| Dependency : github.com/AzureAD/microsoft-authentication-library-for-go | ||
| Version: v0.9.0 | ||
| Version: v1.1.1 | ||
| Licence type (autodetected): MIT | ||
| -------------------------------------------------------------------------------- | ||
|
|
||
| Contents of probable licence file $GOMODCACHE/github.com/!azure!a!d/microsoft-authentication-library-for-go@v0.9.0/LICENSE: | ||
| Contents of probable licence file $GOMODCACHE/github.com/!azure!a!d/microsoft-authentication-library-for-go@v1.1.1/LICENSE: | ||
|
|
||
| MIT License | ||
|
|
||
|
|
@@ -39522,12 +39644,12 @@ Contents of probable licence file $GOMODCACHE/github.com/godror/[email protected]/L | |
|
|
||
|
|
||
| -------------------------------------------------------------------------------- | ||
| Dependency : github.com/golang-jwt/jwt/v4 | ||
| Version: v4.5.0 | ||
| Dependency : github.com/golang-jwt/jwt/v5 | ||
| Version: v5.0.0 | ||
| Licence type (autodetected): MIT | ||
| -------------------------------------------------------------------------------- | ||
|
|
||
| Contents of probable licence file $GOMODCACHE/github.com/golang-jwt/jwt/[email protected].0/LICENSE: | ||
| Contents of probable licence file $GOMODCACHE/github.com/golang-jwt/jwt/[email protected].0/LICENSE: | ||
|
|
||
| Copyright (c) 2012 Dave Grijalva | ||
| Copyright (c) 2021 golang-jwt maintainers | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this consistent with what we have been using in Fleet integrations? I think I recall
azure.resource.groupbeing used over there.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am also a little concerned about this processor beginning to produce fields outside of the ECS
cloudandorchestratornamespaces.Downstream consumers (e.g Fleet integrations) should ideally be updated to have this field included in their mappings since it's not part of ECS. This affects almost all integrations since Agent includes
add_cloud_metadatawith most input types from Beats. (Imagine if we ran our CI on Azure for Fleet integration system testing, after this merges every integration's system tests would begin failing because this field was not defined in its mappings.)The other concern is whether or not existing pipelines will safely handle the addition of this field. For example if an ingest pipeline is already using this field then does it safely handle the case where the field already exists.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The
azure.resource.group, created by a pipeline? as I noticed is the resource group stripped. It only includes the group's name. The addition of theazure.resourcegroup.namefield I did was to me a nice to have and definitely not the reason of this PR.I understand that it then needs to be added in all the integrations because it comes from the processor.
Also it is confusing having two similar named fields. I chose the naming to be compliant with Otel.
I hear your concerns and I will remove this field as it beats the purpose of this PR which was to add the orchestrator ecs fields.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not familiar with these integrations, but, yes, I see it being created by a grok in a few pipeline.
https://github.com/search?q=repo%3Aelastic%2Fintegrations+path%3A**%2F*.yml+%2Fazure%5C.resource%5C.%3Fgroup%2F&type=code
This is a really general problem we have with Fleet integrations in that there is no easy or centralized way to account for the fields being added by the processors.
👍