Refactor and enrich cloudbeat events #69

uri-weisman · 2022-05-01T10:19:03Z

Refactor and add a name, sub-type, and type to the resource struct.

revert changes to justfile

…sman/cloudbeat into refactor_resource_struct

resources/fetchers/file_system_fetcher.go

eyalkraft · 2022-05-01T11:51:47Z

resources/fetchers/ecr_fetcher.go

 func (res ECRResource) GetData() interface{} {
 	return res
 }
+
+func (res ECRResource) GetMetadata() fetching.ResourceMetadata {
+	//TODO implement me


please open an issue for that

A task for implementing the missing GetMetadata method - #71

eyalkraft · 2022-05-01T11:52:07Z

resources/fetchers/eks_fetcher.go

+
+func (r EKSResource) GetMetadata() fetching.ResourceMetadata {
+	//TODO implement me
+	return fetching.ResourceMetadata{


in the issue you open include a list of fetchers to fix

eyalkraft · 2022-05-01T11:55:49Z

resources/fetchers/file_system_fetcher.go

+	}
+}
+
+func getSubType(isDir bool) string {


that's a bit weird.
what about passing info?

eyalkraft · 2022-05-01T11:56:04Z

resources/fetchers/file_system_fetcher.go

@@ -31,13 +31,20 @@ import (
 	"github.com/elastic/cloudbeat/resources/fetching"
 )

+const (
+	FSResourceType = "file"


see elastic/csp-security-policies#84 (review)

eyalkraft · 2022-05-01T11:58:03Z

resources/fetchers/file_system_fetcher.go

+		ID:      r.Inode,
+		Type:    FSResourceType,
+		SubType: r.SubType,
+		Name:    r.Path,


leave a note about this being the path from the container and not from the host

eyalkraft · 2022-05-01T12:07:06Z

resources/fetchers/kube_provider.go

-func (r K8sResource) GetData() interface{} {
-	return r.Data
+func (r K8sResource) GetSubType() string {
+	k8sObj := reflect.Indirect(reflect.ValueOf(r.Data))


try and redesign to avoid calling reflect.Indirect(reflect.ValueOf(r.Data)) twice

eyalkraft · 2022-05-01T12:11:44Z

transformer/events_creator.go

 		}
 	}
 }

-func (c *Transformer) createBeatEvents(fetchedResource fetching.Resource, metadata ResourceMetadata) error {
+func (c *Transformer) createBeatEvents(fetchedResource fetching.Resource, metadata fetching.ResourceMetadata, cycleMetadata CycleMetadata) error {


now since fetchedResource itself has .GetMetadata() why do you pass both?

eyalkraft · 2022-05-01T12:14:32Z

transformer/events_creator.go

-		Raw:  fetcherResult.Resource,
+	resource := fetching.ResourceFields{
+		ResourceMetadata: metadata,
+		Raw:              fetcherResult.Resource,


type and subtype?
or are they waiting for another PR?

No, they are part of the ResourceMetadata type.
plz see fetcher.go

jeniawhite · 2022-05-01T12:24:29Z

resources/fetchers/iam_fetcher.go

@@ -60,3 +60,13 @@ func (r IAMResource) GetID() (string, error) {
 func (r IAMResource) GetData() interface{} {
 	return r.Data
 }
+


You did not remove GetID() in iam_fetcher.go

Thanks 💪

jeniawhite · 2022-05-01T12:32:07Z

resources/fetchers/kube_provider.go

 }

-func (r K8sResource) GetData() interface{} {
-	return r.Data
+func (r K8sResource) GetSubType() string {


Is there a reason not to fail and continue with an empty string value instead of the kind?
Just wondering if we are interested in sending data without kind that might create problems down the line of the pipeline that expects to groupBy or do any logic on these values?

good question👆🏼
I think i'd prefer events with subtype missing than no events at all,
but the possibility for problems in Kibana resulting from this field missing is interesting...

I think sending with empty subtype matches the current design pattern and is a good start (best effort).
for example currently if a fetcher registration fails cloudbeat will log an error and continue.
anyways it's a good idea to ask @kfirpeled / @ari-aviran or someone else from the FE, or even comment here.

Additionally, except for bugs I don't see any reason for this method ever failing so I wouldn't waste too much time thinking/debating it.

Thats a good point,
In case of an issue - we can define it to be empty. And we would know to treat empty values as error.

What happens when there's no subtype? For example, in process.
Just to make things easier, I think it is better to have value we would like to see. And not solve this in the UI only or assume if it is process there's no error and it is empty. Just so we can allow our users to built their own dashboard over the data without having some sort of logic.

…sman/cloudbeat into refactor_resource_struct

github-actions · 2022-05-02T05:40:40Z

Cloudbeat CI 🤖

Integration tests status: success
Tests Summary:

=========================== short test summary info ============================
PASSED integration/tests/test_output_to_elasticsearch.py::test_elastic_index_exists[file]
PASSED integration/tests/test_output_to_elasticsearch.py::test_elastic_index_exists[process]
PASSED integration/tests/test_output_to_elasticsearch.py::test_elastic_index_exists[k8s_object]
PASSED product/tests/test_cloudbeat.py::test_cloudbeat_pod_exist
PASSED product/tests/test_cloudbeat.py::test_cloudbeat_pods_running
================= 5 passed, 4 deselected, 6 warnings in 0.90s ==================

Link to detailed report: https://elastic.github.io/cloudbeat/428

kfirpeled · 2022-05-02T08:04:26Z

resources/fetchers/process_fetcher.go

-	CMDArgumentMatcher = "\\b%s[\\s=]\\/?(\\S+)"
+	CMDArgumentMatcher  = "\\b%s[\\s=]\\/?(\\S+)"
+	ProcessResourceType = "process"
+	ProcessSubType      = "process"


amitkanfer · 2022-05-02T08:47:22Z

resources/fetchers/kube_provider.go

-	return r.Data
+func getK8sSubType(k8sObj reflect.Value) string {
+	typeMeta, ok := k8sObj.FieldByName(k8sTypeMetadataField).Interface().(metav1.TypeMeta)
+	if !ok {


now i see what you meant...

uri-weisman and others added 10 commits April 26, 2022 14:29

add type and subtypes fields

b68dceb

commit before pr splits

4812158

add name field

e105383

Merge branch 'main' into refactor_resource_struct

77a443a

unit tests + csp policies repo tag change

73adf59

update go.mod

6cf20f5

Update JUSTFILE

322e71f

revert changes to justfile

Update JUSTFILE

a7b24d1

revert justfile changes

5e80a03

Merge branch 'refactor_resource_struct' of https://github.com/uri-wei…

f4b13e5

…sman/cloudbeat into refactor_resource_struct

uri-weisman commented May 1, 2022

View reviewed changes

resources/fetchers/file_system_fetcher.go Show resolved Hide resolved

uri-weisman added 2 commits May 1, 2022 13:44

prepare pr for review

b35a8fd

update integration tests with new types

4b2373a

uri-weisman requested a review from eyalkraft May 1, 2022 11:50

uri-weisman marked this pull request as ready for review May 1, 2022 11:50

uri-weisman requested a review from a team as a code owner May 1, 2022 11:50

Merge branch 'main' into refactor_resource_struct

43baff4

uri-weisman mentioned this pull request May 1, 2022

Update data adapter with new FS type elastic/csp-security-policies#84

Merged

eyalkraft requested changes May 1, 2022

View reviewed changes

jeniawhite reviewed May 1, 2022

View reviewed changes

uri-weisman added 2 commits May 1, 2022 15:54

CR changes

d1dbb9c

Merge branch 'refactor_resource_struct' of https://github.com/uri-wei…

ebdb3f1

…sman/cloudbeat into refactor_resource_struct

uri-weisman requested a review from eyalkraft May 1, 2022 13:01

uri-weisman and others added 2 commits May 1, 2022 17:14

Merge branch 'main' into refactor_resource_struct

366d31c

Merge branch 'main' into refactor_resource_struct

2905f65

uri-weisman requested a review from a team as a code owner May 2, 2022 05:24

kfirpeled reviewed May 2, 2022

View reviewed changes

amitkanfer reviewed May 2, 2022

View reviewed changes

eyalkraft approved these changes May 2, 2022

View reviewed changes

uri-weisman merged commit 15a339b into elastic:main May 2, 2022

uri-weisman deleted the refactor_resource_struct branch May 2, 2022 09:27

uri-weisman mentioned this pull request May 8, 2022

[Cloud Posture] Update latest findings index mapping elastic/kibana#131504

Merged

9 tasks

orouz pushed a commit to orouz/cloudbeat that referenced this pull request Sep 6, 2023

remove cis 1.2.2 (elastic#69)

3f77666

orestisfl pushed a commit to orestisfl/cloudbeat that referenced this pull request Oct 11, 2023

[Security Policies] remove cis 1.2.2 (elastic#69)

020bcb3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor and enrich cloudbeat events #69

Refactor and enrich cloudbeat events #69

uri-weisman commented May 1, 2022

eyalkraft May 1, 2022

uri-weisman May 2, 2022

eyalkraft May 1, 2022

eyalkraft May 1, 2022

eyalkraft May 1, 2022

eyalkraft May 1, 2022

eyalkraft May 1, 2022

eyalkraft May 1, 2022

eyalkraft May 1, 2022

uri-weisman May 1, 2022 •

edited

Loading

jeniawhite May 1, 2022

uri-weisman May 1, 2022

jeniawhite May 1, 2022

eyalkraft May 1, 2022

eyalkraft May 1, 2022

eyalkraft May 1, 2022

kfirpeled May 1, 2022

github-actions bot commented May 2, 2022

kfirpeled May 2, 2022

amitkanfer May 2, 2022

Refactor and enrich cloudbeat events #69

Refactor and enrich cloudbeat events #69

Conversation

uri-weisman commented May 1, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

uri-weisman May 1, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented May 2, 2022

Cloudbeat CI 🤖

Choose a reason for hiding this comment

Choose a reason for hiding this comment

uri-weisman May 1, 2022 •

edited

Loading