migrate fleet integration_policy to framework, fix secret churn #797

daemitus · 2024-09-26T20:52:19Z

Dependent on #785 (will fix this up after merge to clean up the diff)
Solves #689

Migrate fleet_integration_policy to framework
Migrate fleet_output to framework
Fix [Bug] elasticstack_fleet_integration_policy render consistent diff for policy secret #689, churn on integration policy secrets by storing a map of secretID -> actual in the private resource state.

Actually testing the secrets required implementing an actual fleet server. So theres a bunch of changes in the makefile and github actions setting that up.

Change from previously: for integration policies, sortInputs would leave new in the original order if old was not present, e.g. during import. Since only the TCP integration was ever tested, the go maps are unordered/randomized issue never cropped up. Now new is sorted by inputID if old is not present. If both are present, existing functionality is preserved.

breml · 2024-09-27T06:04:59Z

@daemitus Thanks for the work you have put into this.

if I understand correctly, this PR is based on #785. In order to make the diff in this PR only containing the changes necessary to fix the issue with the secret ref (#689), I suggest to use daemitus:feat as merge target instead of main, at least for the time until daemitus:feat is merged. What do you think?

daemitus · 2024-09-27T10:41:40Z

Oh sure

…

On Fri, Sep 27, 2024, 2:05 AM Lucas Bremgartner ***@***.***> wrote: @daemitus <https://github.com/daemitus> Thanks for the work you have put into this. if I understand correctly, this PR is based on #785 <#785>. In order to make the diff in this PR only containing the changes necessary to fix the issue with the secret ref (#689 <#689>), I suggest to use daemitus:feat as merge target instead of main, at least for the time until daemitus:feat is merged. What do you think? — Reply to this email directly, view it on GitHub <#797 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFMTBHWGTGU4MCCRXXWQSLZYTYSDAVCNFSM6AAAAABO52NGMWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNZYGQ3DOMZVHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

daemitus · 2024-09-29T10:41:34Z

.github/workflows/test.yml

@@ -63,7 +63,6 @@ jobs:
          xpack.security.enabled: true
          xpack.security.authc.api_key.enabled: true
          xpack.security.authc.token.enabled: true
-          xpack.security.http.ssl.enabled: false


defaults to false, not included in the makefile

daemitus · 2024-09-29T10:42:27Z

.github/workflows/test.yml

+          KIBANA_FLEET_PASSWORD: ${{ env.ELASTIC_PASSWORD }}
+        ports:
+          - 8220:8220
+        options: --restart="unless-stopped"


To test for secret storage, an actual fleet server is necessary. It'll fail until the kibana password is set, so keep restarting it.

daemitus · 2024-09-29T10:42:42Z

.github/workflows/test.yml

@@ -123,8 +140,6 @@ jobs:
      - name: Setup Kibana user
        run: make set-kibana-password
        env:
-          ELASTICSEARCH_ENDPOINTS: "http://localhost:9200"
-          ELASTICSEARCH_USERNAME: "elastic"


unused arguments in this makefile directive

daemitus · 2024-09-29T10:42:50Z

.github/workflows/test.yml

@@ -134,10 +149,17 @@ jobs:
        run: |-
          echo "apikey=$(make create-es-api-key | jq -r .encoded)" >> "$GITHUB_OUTPUT"
        env:
-          ELASTICSEARCH_ENDPOINTS: "http://localhost:9200"
-          ELASTICSEARCH_USERNAME: "elastic"


unused arguments in this makefile directive

daemitus · 2024-09-29T10:43:46Z

Makefile

 SOURCE_LOCATION ?= $(shell pwd)
+, := ,


escaping JSON commas in the setup fleet payloads. This makes me sad, but not much of a better way to do it when wrapping in the retry func

daemitus · 2024-09-29T10:46:55Z

internal/fleet/integration_policy/secrets.go

+		}
+	}
+
+	handleVars(utils.Deref(resp.Vars))


fwiw, without using the x-go-remove-unnecessary-pointers flag, you have to sprinkle these deref methods everywhere. Being able to range a nil map is nice.

daemitus · 2024-09-29T10:50:48Z

internal/utils/tfsdk.go

+	Key   string
+	Path  path.Path
+	Diags diag.Diagnostics
+}


The idea with using these, is to make the iter functions in each of these helpers a little less wordy. If you don't inline the function, you'll still have access to the parent metadata.

daemitus · 2024-09-29T10:52:54Z

internal/fleet/integration_policy/models.go

+// 'existing' will be placed at the end of the list. Inputs are identified by
+// their ID ('input_id'). The 'incoming' slice will be sorted in-place.
+func sortInputs(incoming []integrationPolicyInputModel, existing []integrationPolicyInputModel) {
+	if len(existing) == 0 {


This is a change from previous that I think addresses a bug that was not noticed before. When importing, existing will be empty/null. And if you have more than 1 import, the order will get randomized (thanks go-maps), so the import test will randomly fail.

tobio

Can we split this PR into one for the integration_policy resource and one for the output resource please?

tobio · 2024-09-29T21:43:48Z

generated/fleet/getschema.go

Can we just use a newer version of the spec?

most of the fleet changes in getschema (including secret_references) are still missing on the main branch. i was playing around with it yesterday, seeing what would need doing.

I can look into bumping the spec once this is done. I wanted to incorporate some of the transforms upstream anyways. Based on what I've seen, its likely going to require some extra work on getschema anyways.

internal/fleet/integration_policy/models.go

internal/fleet/integration_policy/secrets.go

internal/fleet/output/models.go

tobio

This looks great, thanks for making this change!

* origin/main: fix package policy secrets (#821) chore(deps): update codecov/codecov-action digest to b9fd7d1 (#815) Bump release memory Switch to golang 1.23.2 in release Bump release memory Prepare release v0.11.8 (#810) Add key_id as an explicit attribute (#789) standalone-output resource (#811) Add URL support to data_view field_formats (#812) migrate fleet integration_policy to framework, fix secret churn (#797) Allow mappings to be unknown to support mappings defined in index templates (#803)

daemitus marked this pull request as draft September 26, 2024 20:52

daemitus force-pushed the newfeat branch 2 times, most recently from 26d5955 to 9310f58 Compare September 26, 2024 22:35

kpollich mentioned this pull request Sep 27, 2024

[Bug] elasticstack_fleet_integration_policy render consistent diff for policy secret #689

Closed

daemitus force-pushed the newfeat branch 2 times, most recently from 8726779 to 1dd7646 Compare September 29, 2024 10:40

daemitus commented Sep 29, 2024

View reviewed changes

daemitus marked this pull request as ready for review September 29, 2024 10:53

tobio reviewed Sep 30, 2024

View reviewed changes

daemitus added 6 commits September 30, 2024 08:16

integration policy

870b590

extract fn

7709be0

init empty map

87ba881

secrets change

992fb83

fmt

54e4db8

tweak

6cf9b4c

daemitus force-pushed the newfeat branch from 3b13e5b to 6cf9b4c Compare September 30, 2024 13:03

daemitus changed the title ~~migrate fleet integration_policy and output to framework, fix secret churn~~ migrate fleet integration_policy to framework, fix secret churn Sep 30, 2024

tobio approved these changes Oct 1, 2024

View reviewed changes

tobio merged commit ba9abc9 into elastic:main Oct 1, 2024
20 checks passed

daemitus deleted the newfeat branch October 1, 2024 13:19

breml mentioned this pull request Oct 18, 2024

[Bug] Semantic Equality Check Error in elasticstack_fleet_integration_policy #820

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

migrate fleet integration_policy to framework, fix secret churn #797

migrate fleet integration_policy to framework, fix secret churn #797

daemitus commented Sep 26, 2024 •

edited

Loading

breml commented Sep 27, 2024

daemitus commented Sep 27, 2024 via email

daemitus Sep 29, 2024

daemitus Sep 29, 2024

daemitus Sep 29, 2024

daemitus Sep 29, 2024

daemitus Sep 29, 2024

daemitus Sep 29, 2024

daemitus Sep 29, 2024

daemitus Sep 29, 2024

tobio left a comment

tobio Sep 29, 2024

daemitus Sep 30, 2024

daemitus Sep 30, 2024

tobio left a comment

migrate fleet integration_policy to framework, fix secret churn #797

migrate fleet integration_policy to framework, fix secret churn #797

Conversation

daemitus commented Sep 26, 2024 • edited Loading

breml commented Sep 27, 2024

daemitus commented Sep 27, 2024 via email

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tobio left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tobio left a comment

Choose a reason for hiding this comment

daemitus commented Sep 26, 2024 •

edited

Loading