Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Metricbeat][Kubernetes Volume] Add pvc reference to distinguish ephemeral from persistent volumes #38839

Merged
merged 19 commits into from
May 16, 2024
Merged

[Metricbeat][Kubernetes Volume] Add pvc reference to distinguish ephemeral from persistent volumes #38839

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
5b4a487
Kubernetes add pvc reference in the kubernetes.volume dataset if present
herrBez Apr 9, 2024
ede64ae
Reuse field kubernetes.persistentvolumeclaim instead of creating a ne…
herrBez Apr 10, 2024
5026f50
Add only persistentvolumeclaim name and no namespace (that is redunda…
herrBez Apr 10, 2024
375e742
Merge branch 'elastic:main' into kubernetes-pvc-volume
herrBez Apr 10, 2024
b0dd8a4
Add test to see if persistentvolumeclaim name is extracted
herrBez Apr 10, 2024
d5cfb1f
Add a separate test for pvc
herrBez May 2, 2024
e1ba07b
Merge branch 'main' into kubernetes-pvc-volume
herrBez May 2, 2024
0f78c41
Add changelog entry for the PR
herrBez May 2, 2024
9a0cf91
Add the persistentvolume claim field
herrBez May 2, 2024
698eb17
Run mage update to update the fields
herrBez May 2, 2024
39d255e
Remove the duplicate definition of the field kubernetes.persistentvol…
herrBez May 8, 2024
8a6eeff
Remove newline
herrBez May 8, 2024
a8b3092
Fix key from kubernetes.volume.persistentvolumeclaim.name to kubernet…
herrBez May 10, 2024
d206ac5
Merge branch 'main' into kubernetes-pvc-volume
herrBez May 10, 2024
9e9a600
Merge branch 'main' into kubernetes-pvc-volume
herrBez May 13, 2024
cec10b2
Fix unit test after changing the populated field name
herrBez May 14, 2024
14cff40
Merge branch 'kubernetes-pvc-volume' of github.com:herrBez/beats into…
herrBez May 14, 2024
ef19ff0
Removed printf function
herrBez May 14, 2024
835d991
Remove empty lines added by mistake
herrBez May 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -192,6 +192,7 @@ https://github.com/elastic/beats/compare/v8.8.1\...main[Check the HEAD diff]
- Enable early event encoding in the Elasticsearch output, improving cpu and memory use {pull}38572[38572]
- The environment variable `BEATS_ADD_CLOUD_METADATA_PROVIDERS` overrides configured/default `add_cloud_metadata` providers {pull}38669[38669]
- Introduce log message for not supported annotations for Hints based autodiscover {pull}38213[38213]
- Add persistent volume claim name to volume if available {pull}38839[38839]


*Auditbeat*
Expand Down
16 changes: 15 additions & 1 deletion metricbeat/module/kubernetes/_meta/test/stats_summary.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,21 @@
"inodes": 473560,
"inodesUsed": 9,
"name": "default-token-sg8x5"
}
},
{
"time": "2024-04-09T17:34:17Z",
"availableBytes": 31509590016,
"capacityBytes": 31526391808,
"usedBytes": 24576,
"inodesFree": 1966069,
"inodes": 1966080,
"inodesUsed": 11,
"name": "pvc-demo-vol",
"pvcRef": {
"name": "pvc-demo",
"namespace": "default"
}
}
]
}
]
Expand Down
4 changes: 4 additions & 0 deletions metricbeat/module/kubernetes/types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,10 @@ type Summary struct {
InodesUsed uint64 `json:"inodesUsed"`
Name string `json:"name"`
UsedBytes uint64 `json:"usedBytes"`
PvcRef struct {
Name string `json:"name"`
Namespace string `json:"namespace"`
} `json:"pvcRef"`
} `json:"volume"`
} `json:"pods"`
}
3 changes: 3 additions & 0 deletions metricbeat/module/kubernetes/volume/data.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,9 @@ func eventMapping(content []byte, logger *logp.Logger) ([]mapstr.M, error) {
if volume.Inodes > 0 {
kubernetes2.ShouldPut(volumeEvent, "fs.inodes.pct", float64(volume.InodesUsed)/float64(volume.Inodes), logger)
}
if volume.PvcRef.Name != "" {
kubernetes2.ShouldPut(volumeEvent, mb.ModuleDataKey+".persistentvolumeclaim.name", volume.PvcRef.Name, logger)
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have the feeling that this way the field that will be generated will be kubernetes.volume.persistentvolumeclaim.name . Is that what you want?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow, great static analysis :D.. Indeed, its creating the field you mention. I actually want to reuse the field kubernetes.persistentvolumeclaim.name to allow correlation, but I am honestly a little bit lost.

After some thinking I tried with:

kubernetes2.ShouldPut(volumeEvent, mb.ModuleDataKey+".persistentvolumeclaim.name", volume.PvcRef.Name, logger)

But it does not seem to work 🤔

Copy link
Contributor Author

@herrBez herrBez May 10, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, the line I wrote in the comment is correct. For some reasons I were still using the old binary 🤦

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had the feeling 😂

events = append(events, volumeEvent)
}

Expand Down
38 changes: 24 additions & 14 deletions metricbeat/module/kubernetes/volume/volume_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ import (

"github.com/stretchr/testify/assert"

"github.com/elastic/beats/v7/metricbeat/mb"
"github.com/elastic/elastic-agent-libs/logp"
"github.com/elastic/elastic-agent-libs/mapstr"
)
Expand All @@ -44,23 +45,32 @@ func TestEventMapping(t *testing.T) {
events, err := eventMapping(body, logger)
assert.NoError(t, err, "error mapping "+testFile)

assert.Len(t, events, 1, "got wrong number of events")
assert.Len(t, events, 2, "got wrong number of events")

testCases := map[string]interface{}{
"name": "default-token-sg8x5",

"fs.available.bytes": 1939689472,
"fs.capacity.bytes": 1939701760,
"fs.used.bytes": 12288,
"fs.used.pct": float64(12288) / float64(1939701760),
"fs.inodes.used": 9,
"fs.inodes.free": 473551,
"fs.inodes.count": 473560,
"fs.inodes.pct": float64(9) / float64(473560),
testCases := []map[string]interface{}{
// Test for ephemeral volume
{
"name": "default-token-sg8x5",
"fs.available.bytes": 1939689472,
"fs.capacity.bytes": 1939701760,
"fs.used.bytes": 12288,
"fs.used.pct": float64(12288) / float64(1939701760),
"fs.inodes.used": 9,
"fs.inodes.free": 473551,
"fs.inodes.count": 473560,
"fs.inodes.pct": float64(9) / float64(473560),
},
// Test for the persistent volume claim
{
mb.ModuleDataKey + ".persistentvolumeclaim.name": "pvc-demo",
"name": "pvc-demo-vol",
},
}

for k, v := range testCases {
testValue(t, events[0], k, v)
for i := range testCases {
for k, v := range testCases[i] {
testValue(t, events[i], k, v)
}
}
}

Expand Down
Loading