Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore: Improve config builders #3207

Merged
merged 82 commits into from
Nov 29, 2024
Merged

chore: Improve config builders #3207

merged 82 commits into from
Nov 29, 2024

Conversation

sfc-gh-asawicki
Copy link
Collaborator

@sfc-gh-asawicki sfc-gh-asawicki commented Nov 19, 2024 •
edited
Loading

Improve how we build config models:

  • explore HCL v1 lib
  • generate provider config builder
  • generate datasource config builder
  • generate config using hcl v1 lib
  • support objects, lists, and sets
  • use provider config in provider setup tests
  • use datasource generated config in one test for databases datasource
  • use support for objects and lists in one view resource acceptance test
  • extract nullVariable to own file
  • cleanup in bettertestspoc/config directory
  • add all datasources to provider/datasources
  • deprecate old config builder and config variables from config methods - undeprecated for now because of linter (and I could not quickly find how to disable it; will do in a follow-up)
  • multiline values (using <<EOT...EOT) are handled far from ideally but work for now

References:

TODOs (next PRs - soon):

  • rename package model to resourcemodel
  • add generating MarshalJSON() function
  • migrate resources to new config generation method (above needed first)
  • improve unquoteDependsOnReferences method

TODOs (next PRs - later, left as next steps in readme):

  • add config builders for other block types (Variable, Output, Localsl, Module, Terraform)
  • add provider to resource/datasource models (use in the grant_ownership_acceptance_test)
  • explore HCL v2 in more detail (especially struct tags generation; probably with migration to plugin framework because of schema models)
  • introduce some common interface for all three existing models (ResourceModel, DatasourceModel, and ProviderModel)
  • rename ResourceSchemaDetails (because it is used for the datasources and provider too)
  • consider duplicating the builders template from resource (currently same template used for datasources and provider)
  • consider merging ResourceModel with DatasourceModel (currently the implementation is really similar)

@sfc-gh-asawicki
Test the HCL v1 printing
4f9d1e3
@sfc-gh-asawicki
Experiment with V2 unsuccessfully, reverting to V1
f0c5ba8
@sfc-gh-asawicki
Replace the test data
5379e27
@sfc-gh-asawicki
Split conversion and formatting
5fc63e6
@sfc-gh-asawicki
Add more complex test case
abc5928
@sfc-gh-asawicki
PoC generating configs with lists and sets
04f8fbc
@sfc-gh-asawicki
Reorganized
5abe92e
@sfc-gh-asawicki
Extract logical parts
add6635
@sfc-gh-asawicki
Rename and add defaults
17dde40
@sfc-gh-asawicki
Rename json provider method
90f59d3
@sfc-gh-asawicki
List missing block types
d15f882
@sfc-gh-asawicki
Prepare provider block config
927037b
@sfc-gh-asawicki
Prepare provider config generator
b4e1ee8
@sfc-gh-asawicki
Generate provider config
c28b622
@sfc-gh-asawicki
Test provider model generation
bd0832c
@sfc-gh-asawicki
Extract to separate package
740ea6f
@sfc-gh-asawicki
Reorganize files in model, add datasource gen
928718b
@sfc-gh-asawicki
Generate databases config builder
7f94097
@sfc-gh-asawicki
Add datasource generation
3d15762
@sfc-gh-asawicki
Fix block quoting
204a69d
@sfc-gh-asawicki
Fix depends_on for datasources
9549b0e
@sfc-gh-asawicki
Unquote arguments
f8054f7
@sfc-gh-asawicki
Use provider config builder instead of string ones (WIP)
8d18777
@sfc-gh-asawicki
Add model joining
a8ae363
@sfc-gh-asawicki
Use multi model config builder
bb69b5c
@sfc-gh-asawicki
Inline provider building functions
9cd1d5a
@sfc-gh-asawicki
Try to use new config generation for view acceptance test (WIP)
42e83b2
@sfc-gh-asawicki
Add fix for block types equal sign
67994b9
@sfc-gh-asawicki
Add test for single object
97b2444
@sfc-gh-asawicki
Use datasource model builder in existing test
db3d412
@github-actions GitHub Actions
Copy link

Integration tests cancelled for c12c11f3d95b13ebc00bc750ec69b81aec6de20c

@github-actions GitHub Actions
Copy link

Integration tests failure for 7d49a98fac6a394432556c3d9c6046f16991f793

sfc-gh-jmichalak
sfc-gh-jmichalak approved these changes Nov 29, 2024
View reviewed changes
pkg/acceptance/bettertestspoc/config/providermodel/snowflake_model_ext.go
}

func (m *SnowflakeModel) WithClientStoreTemporaryCredentialBool(clientStoreTemporaryCredential bool) *SnowflakeModel {
m.ClientStoreTemporaryCredential = tfconfig.BoolVariable(clientStoreTemporaryCredential)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: why not With... here?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will change with some other change

pkg/acceptance/bettertestspoc/config/hcl_config_provider.go
@@ -79,6 +79,20 @@ func fixBlockArguments(s string) (string, error) {
return argumentRegex.ReplaceAllString(s, `$1$2$3$4`), nil
}

// TODO [SNOW-1501905]: fix new lines replacement totally in this method, consider better placeholders (now there can't be multilinekeys for private key)
func fixMultilinePrivateKey(s string) (string, error) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So it's not clear whether we should use this function at all.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will change the comment to a more readable one in the next PRs

@sfc-gh-asawicki sfc-gh-asawicki merged commit 425787c into main Nov 29, 2024
9 of 10 checks passed
@sfc-gh-asawicki sfc-gh-asawicki deleted the improve-config-builders branch November 29, 2024 09:51
@snowflake-release-please snowflake-release-please bot mentioned this pull request Nov 29, 2024
Merged
@sfc-gh-asawicki sfc-gh-asawicki mentioned this pull request Dec 4, 2024
Merged
sfc-gh-jcieslak pushed a commit that referenced this pull request Dec 12, 2024
@snowflake-release-please
chore(main): release 0.100.0 (#3237)
5139b1a 
🤖 I have created a release *beep* *boop*
---


##
[0.100.0](v0.99.0...v0.100.0)
(2024-12-12)


### 🎉 **What's new:**

* Account v1 readiness
([#3236](#3236))
([5df33a8](5df33a8))
* Account v1 readiness generated files
([#3242](#3242))
([3df59dd](3df59dd))
* Account v1 readiness resource
([#3252](#3252))
([8f5698d](8f5698d))
* Add a new account roles data source
([#3257](#3257))
([b3d6b9e](b3d6b9e))
* Add account data source
([#3261](#3261))
([6087fc9](6087fc9))
* Add all other functions and procedures implementations
([#3275](#3275))
([7a6f68d](7a6f68d))
* Basic functions implementation
([#3269](#3269))
([6d4a103](6d4a103))
* Basic procedures implementation
([#3271](#3271))
([933335f](933335f))
* Docs, test, and missing parameter
([#3280](#3280))
([10517f3](10517f3))
* Functions and procedures schemas and generated sources
([#3262](#3262))
([9b70f87](9b70f87))
* Functions sdk update
([#3254](#3254))
([fc1eace](fc1eace))
* Handle missing fields in function and procedure
([#3273](#3273))
([53e7a0a](53e7a0a))
* Procedures schemas and generated sources
([#3263](#3263))
([211ad46](211ad46))
* Procedures sdk update
([#3255](#3255))
([682606a](682606a))
* Rework account parameter resource
([#3264](#3264))
([15aa9c2](15aa9c2))
* Rework data types
([#3244](#3244))
([05ada91](05ada91))
* support table data type
([#3274](#3274))
([13401d5](13401d5))
* Tag association v1 readiness
([#3210](#3210))
([04f6d54](04f6d54))
* Test imports and small fixes
([#3276](#3276))
([a712195](a712195))
* Unsafe execute v1 readiness
([#3266](#3266))
([c4f1e8f](c4f1e8f))
* Use new data types in sql builder for functions and procedures
([#3247](#3247))
([69f677a](69f677a))


### 🔧 **Misc**

* Add ShowByID filtering generation
([#3227](#3227))
([548ec42](548ec42))
* Adress small task-related todos
([#3243](#3243))
([40de9ae](40de9ae))
* Apply masking
([#3234](#3234))
([c209a8a](c209a8a))
* fix missing references in toOpts and changes with newlines
([#3240](#3240))
([246547f](246547f))
* function tests
([#3279](#3279))
([5af6efb](5af6efb))
* Improve config builders
([#3207](#3207))
([425787c](425787c))
* Revert to proper env
([#3238](#3238))
([5d4ed3b](5d4ed3b))
* Use service user for ci
([#3228](#3228))
([2fb50d7](2fb50d7))


### 🐛 **Bug fixes:**

* Make blocked_roles_field optional in OAuth security integrations
([#3267](#3267))
([7197b57](7197b57))
* Minor fixes
([#3231](#3231))
([1863bf6](1863bf6))
* Minor fixes 2
([#3230](#3230))
([73b7e74](73b7e74))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: snowflake-release-please[bot] <105954990+snowflake-release-please[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sfc-gh-jmichalak sfc-gh-jmichalak sfc-gh-jmichalak approved these changes

@sfc-gh-jcieslak sfc-gh-jcieslak Awaiting requested review from sfc-gh-jcieslak

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@sfc-gh-asawicki @sfc-gh-jcieslak @sfc-gh-jmichalak