Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add remote state deployment + use by default #97

Merged
merged 52 commits into from
Sep 26, 2023
Merged

Add remote state deployment + use by default #97

merged 52 commits into from
Sep 26, 2023

Conversation

strickvl
Copy link
Contributor

@strickvl strickvl commented Sep 18, 2023 •
edited
Loading

Added default deployment of remote state when using mlstacks.

  • includes choice of random bucket name for state unless connecting to pre-existing bucket.
  • also includes some docs updates on how to use this / connect to remote state.
  • improves error raising / messaging also.

This change is non-breaking for ZenML core.

Implemented for GCP and AWS modular stacks.

@strickvl
add AWS remote state bucket config
36d593c
@strickvl
alphabetical sort
5de0d0c
@strickvl
manually merge develop
b1e3b4b
@strickvl
add gcp remote state files
cb3c6e1
@strickvl
add lockfile to gitignore
eb1eb96
@strickvl
more gitignore files
930fd11
@strickvl
add commented out backend hint
07934a7
@strickvl
update tfvars files
7d3fbab
@strickvl
update output names for gcp
87c331a
@strickvl
add bucket_url output for s3 remote state
e686705
@strickvl
Merge remote-tracking branch 'origin/develop' into feature/OSS-2328-r…
ac1f618 
…emote-state-first
@strickvl
update gcp definition
0341107
@strickvl
add remote state deployment
2ede8f6
@strickvl
fix remote state deployment bugs
ee30820
@strickvl
add mlstacks destroy including remote state
07bbbf4
@strickvl
add documentation for remote state
0cc1509
@strickvl
fix long line
6b1bb7c
@strickvl
update docs for remote state deployment
3573eb3
@strickvl
update CLI to accommodate remote state deployment
abb7360
@strickvl
add tests for remote state existence check
1c7609e
@strickvl
implement remote state bucket existence check
e7a526a
@strickvl
add remote state bucket validation check to init
4a97a0e
@strickvl
implement random bucket name
e1b488f
@strickvl strickvl added enhancement New feature or request internal labels Sep 18, 2023
avishniakov
avishniakov approved these changes Sep 18, 2023
View reviewed changes
Copy link

@avishniakov avishniakov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few nitpicks

src/mlstacks/terraform/aws-remote-state/main.tf Show resolved Hide resolved
src/mlstacks/terraform/aws-remote-state/main.tf Show resolved Hide resolved
fa9r
fa9r approved these changes Sep 21, 2023
View reviewed changes
Copy link
Contributor

@fa9r fa9r left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a bunch of smaller comments but overall this looks good to go already 👍

docs/book/getting-started/core-concepts.md Outdated Show resolved Hide resolved
docs/book/getting-started/gcp.md Outdated Show resolved Hide resolved
src/mlstacks/cli/cli.py Outdated Show resolved Hide resolved
src/mlstacks/cli/cli.py Outdated Show resolved Hide resolved
src/mlstacks/terraform/aws-remote-state/variables.tf Show resolved Hide resolved
src/mlstacks/utils/terraform_utils.py Show resolved Hide resolved
src/mlstacks/utils/terraform_utils.py Show resolved Hide resolved
src/mlstacks/utils/terraform_utils.py Show resolved Hide resolved
src/mlstacks/utils/terraform_utils.py Show resolved Hide resolved
src/mlstacks/utils/terraform_utils.py Outdated Show resolved Hide resolved
strickvl and others added 21 commits September 21, 2023 11:02
@strickvl @fa9r
Update docs/book/getting-started/gcp.md
3ba9f9d 
Co-authored-by: Felix Altenberger <[email protected]>
@strickvl @fa9r
Update src/mlstacks/terraform/aws-remote-state/variables.tf
4737766 
Co-authored-by: Felix Altenberger <[email protected]>
@strickvl
add link to new documentation section
e24de3f
@strickvl
lengthen variable name as better description
8fd757c
@strickvl @fa9r
Update src/mlstacks/terraform/gcp-remote-state/variables.tf
479c4fb 
Co-authored-by: Felix Altenberger <[email protected]>
@strickvl
Merge branch 'feature/OSS-2328-remote-state-first' of github.com:zenm…
22e965d 
…l-io/mlops-stacks into feature/OSS-2328-remote-state-first
@strickvl
PR suggestions
91d7a0d
@strickvl
refactor backend config assembly
6aafd39
@strickvl @fa9r
Update src/mlstacks/cli/cli.py
4362a7d 
Co-authored-by: Felix Altenberger <[email protected]>
@strickvl
formatting changes
96823a4
@strickvl
add missing docstrings
68a758a
@strickvl
add defaults for variables
b1bc965
@strickvl
docstring fix
6a0c43e
@strickvl
fix output property
4d295a5
@strickvl
remote state bucket check for s3 needs region
0aad950
@strickvl
update tests with new param
39add07
@strickvl
aws fixes
426507b
@strickvl
private functions and add region param
a102e23
@strickvl
raise more & better errors
5a0f83c
@strickvl
mypy and docstring fixes
3ac10bd
@strickvl
fix tests
1113e7e
@strickvl strickvl merged commit 64ce5c9 into develop Sep 26, 2023
@strickvl strickvl deleted the feature/OSS-2328-remote-state-first branch September 26, 2023 09:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@fa9r fa9r fa9r approved these changes

@avishniakov avishniakov avishniakov approved these changes

Assignees
No one assigned
Labels
enhancement New feature or request internal
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@strickvl @fa9r @avishniakov