[vSphere][datastore] Add additional metrics #40441

niraj-elastic · 2024-08-06T11:59:39Z

Description

Added following new metrics for the datastore metricset in the vSphere metricbeat module. added new performance API to get more detailed info from the vSphere.

Metrics Type	Metrics	Field Mappings
Datastore performance
	datastore.read.average (KBps)	datastore.read.bytes
	datastore.write.average (KBps)	datastore.write.bytes
	datastore.datastoreIops.average (num)	datastore.iops
	datastore.totalReadLatency.average (ms)	datastore.read.latency.total.ms
	datastore.totalWriteLatency.average (ms)	datastore.write.latency.total.ms
Datastore Summry
	Datastore.OverallStatus	datastore.status
	Datastore.Host	datastore.host.count
	Datastore.Host	datastore.host.names
	Datastore.Vm	datastore.vm.count
	Datastore.Vm	datastore.vm.names

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Relates https://github.com/elastic/obs-integration-team/issues/19

mergify · 2024-08-06T12:00:22Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @niraj-elastic? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

elasticmachine · 2024-08-08T05:15:47Z

Pinging @elastic/obs-dc (Team:Obs-DC)

CHANGELOG.next.asciidoc

metricbeat/module/vsphere/datastore/_meta/fields.yml

metricbeat/module/vsphere/datastore/data.go

metricbeat/module/vsphere/datastore/_meta/fields.yml

CHANGELOG.next.asciidoc

metricbeat/module/vsphere/datastore/datastore_test.go

metricbeat/module/vsphere/datastore/data.go

shmsr · 2024-08-22T06:29:52Z

metricbeat/module/vsphere/datastore/datastore_test.go

@@ -27,7 +27,8 @@ import (
 )

 func TestFetchEventContents(t *testing.T) {
-	model := simulator.ESX()
+	// Creating a new simulator model with VPX server to collect broad range of data.
+	model := simulator.VPX()
 	if err := model.Create(); err != nil {
 		t.Fatal(err)
 	}


Suggested change

}

}

defer model.Remove()

nitpick

metricbeat/module/vsphere/datastore/data.go

metricbeat/module/vsphere/datastore/data_test.go

metricbeat/module/vsphere/datastore/datastore.go

…j-elastic/beats into vsphere_datastore_metricset

mergify · 2024-08-22T09:52:53Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b vsphere_datastore_metricset upstream/vsphere_datastore_metricset
git merge upstream/main
git push upstream vsphere_datastore_metricset

ishleenk17

Changes look good, other than one comment that I have left.
Lets merge the conflicts and move ahead.

ishleenk17 · 2024-08-22T09:56:34Z

metricbeat/module/vsphere/datastore/datastore.go

+	outputVmNames := make([]string, 0, len(ds.Vm))
+	for _, ob := range objects {
+		if ob.Reference().Type == "VirtualMachine" {
+			name := strings.ReplaceAll(ob.Name, ".", "_")


IMO, we should not change the names from "." to "_".
But as discussed we can take care of this as part of this issue as this could be a breaking change for existing users and we will have to log that.

…phere_datastore_metricset Conflicts: metricbeat/module/vsphere/fields.go

metricbeat/module/vsphere/datastore/datastore.go

metricbeat/module/vsphere/datastore/datastore_test.go

metricbeat/module/vsphere/datastore/datastore.go

* add data collection * update changelog * update datacollection * update fields.yml * update test files * update field discriptions * address review comments * address review comments * address review commetns * update fields * update usint test * address review comments * update data_test file * update unit test * address review comments * update fields.asci * address review comments * update data_test file * update go.mod file * update NOTICE.txt * address review comment * address review comments * mage update * address review comments * Address review comments --------- Co-authored-by: Niraj Rathod <[email protected]> (cherry picked from commit 94ca509) # Conflicts: # metricbeat/module/vsphere/fields.go

…40782) * [vSphere][datastore] Add additional metrics (#40441) * add data collection * update changelog * update datacollection * update fields.yml * update test files * update field discriptions * address review comments * address review comments * address review commetns * update fields * update usint test * address review comments * update data_test file * update unit test * address review comments * update fields.asci * address review comments * update data_test file * update go.mod file * update NOTICE.txt * address review comment * address review comments * mage update * address review comments * Address review comments --------- Co-authored-by: Niraj Rathod <[email protected]> (cherry picked from commit 94ca509) # Conflicts: # metricbeat/module/vsphere/fields.go * Update fields.go * update fields.go --------- Co-authored-by: niraj-elastic <[email protected]> Co-authored-by: Ishleen Kaur <[email protected]> Co-authored-by: ishleenk17 <[email protected]>

add data collection

ea05a06

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Aug 6, 2024

mergify bot assigned niraj-elastic Aug 6, 2024

niraj-elastic added 4 commits August 6, 2024 17:36

update changelog

decfc87

update datacollection

de913ec

update fields.yml

0b49482

update test files

88387bf

niraj-elastic marked this pull request as ready for review August 7, 2024 13:08

niraj-elastic requested a review from a team as a code owner August 7, 2024 13:08

niraj-elastic requested review from ishleenk17, harnish-elastic and lalit-satapathy August 7, 2024 13:08

niraj-elastic added enhancement Team:Obs-DC Label for the Data Collection team labels Aug 8, 2024

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Aug 8, 2024

niraj-elastic added the Metricbeat Metricbeat label Aug 8, 2024

update field discriptions

42bb440

kush-elastic requested changes Aug 8, 2024

View reviewed changes

harnish-elastic reviewed Aug 9, 2024

View reviewed changes

metricbeat/module/vsphere/datastore/_meta/fields.yml Show resolved Hide resolved

address review comments

e5bb66c

ishleenk17 reviewed Aug 12, 2024

View reviewed changes

CHANGELOG.next.asciidoc Outdated Show resolved Hide resolved

ishleenk17 reviewed Aug 12, 2024

View reviewed changes

metricbeat/module/vsphere/datastore/datastore_test.go Show resolved Hide resolved

ishleenk17 reviewed Aug 12, 2024

View reviewed changes

metricbeat/module/vsphere/datastore/data.go Outdated Show resolved Hide resolved

address review comments

b2197e4

niraj-elastic requested a review from kush-elastic August 13, 2024 05:36

address review commetns

7616495

niraj-elastic requested a review from ishleenk17 August 13, 2024 05:40

update fields

f6e481e

niraj-crest and others added 2 commits August 21, 2024 17:59

address review comment

a875047

Merge branch 'elastic:main' into vsphere_datastore_metricset

2e35a11

niraj-elastic requested a review from kush-elastic August 21, 2024 15:43

kush-elastic approved these changes Aug 22, 2024

View reviewed changes

shmsr reviewed Aug 22, 2024

View reviewed changes

devamanv requested changes Aug 22, 2024

View reviewed changes

niraj-crest added 2 commits August 22, 2024 14:52

address review comments

eaa3429

Merge branch 'vsphere_datastore_metricset' of https://github.com/nira…

e38f7e2

…j-elastic/beats into vsphere_datastore_metricset

niraj-elastic requested review from devamanv and shmsr August 22, 2024 09:23

shmsr approved these changes Aug 22, 2024

View reviewed changes

ishleenk17 approved these changes Aug 22, 2024

View reviewed changes

niraj-crest added 2 commits August 22, 2024 16:06

Merge branch 'main' of https://github.com/niraj-elastic/beats into vs…

4931d09

…phere_datastore_metricset Conflicts: metricbeat/module/vsphere/fields.go

mage update

3479446

devamanv requested changes Aug 22, 2024

View reviewed changes

metricbeat/module/vsphere/datastore/datastore.go Outdated Show resolved Hide resolved

metricbeat/module/vsphere/datastore/datastore.go Outdated Show resolved Hide resolved

metricbeat/module/vsphere/datastore/datastore_test.go Outdated Show resolved Hide resolved

address review comments

47893ac

niraj-elastic requested a review from devamanv August 23, 2024 06:41

devamanv requested changes Aug 23, 2024

View reviewed changes

metricbeat/module/vsphere/datastore/datastore.go Outdated Show resolved Hide resolved

metricbeat/module/vsphere/datastore/datastore.go Outdated Show resolved Hide resolved

Address review comments

bc8f19b

devamanv approved these changes Aug 23, 2024

View reviewed changes

niraj-elastic merged commit 94ca509 into elastic:main Aug 23, 2024
29 checks passed

niraj-elastic mentioned this pull request Aug 27, 2024

[vSphere][virtualmachine] Add metrics to virtualmachine metricset #40485

Merged

6 tasks

lucian-ioan mentioned this pull request Aug 28, 2024

[vSphere][datastore] Add support for additional metrics elastic/integrations#10911

Merged

11 tasks

niraj-elastic added backport-8.x Automated backport to the 8.x branch with mergify and removed backport-8.x Automated backport to the 8.x branch with mergify labels Sep 10, 2024

ishleenk17 added the backport-8.15 Automated backport to the 8.15 branch with mergify label Sep 12, 2024

mergify bot mentioned this pull request Sep 12, 2024

[8.15](backport #40441) [vSphere][datastore] Add additional metrics #40782

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[vSphere][datastore] Add additional metrics #40441

[vSphere][datastore] Add additional metrics #40441

niraj-elastic commented Aug 6, 2024 •

edited by devamanv

Loading

mergify bot commented Aug 6, 2024

elasticmachine commented Aug 8, 2024

shmsr Aug 22, 2024

mergify bot commented Aug 22, 2024

ishleenk17 left a comment

ishleenk17 Aug 22, 2024

[vSphere][datastore] Add additional metrics #40441

[vSphere][datastore] Add additional metrics #40441

Conversation

niraj-elastic commented Aug 6, 2024 • edited by devamanv Loading

Description

Checklist

Related issues

mergify bot commented Aug 6, 2024

elasticmachine commented Aug 8, 2024

shmsr Aug 22, 2024

Choose a reason for hiding this comment

mergify bot commented Aug 22, 2024

ishleenk17 left a comment

Choose a reason for hiding this comment

ishleenk17 Aug 22, 2024

Choose a reason for hiding this comment

niraj-elastic commented Aug 6, 2024 •

edited by devamanv

Loading