Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Integration tests executed on a real deployment as part of the CICD - Redshift Connections #1628

Merged
merged 8 commits into from
Oct 15, 2024
Merged

Integration tests executed on a real deployment as part of the CICD - Redshift Connections #1628

merged 8 commits into from
Oct 15, 2024

Conversation

dlpzx
Copy link
Contributor

@dlpzx dlpzx commented Oct 10, 2024 •
edited
Loading

⚠️ NEEDS SOME CHANGES AFTER #1638 is merged

Feature or Bugfix

  • Feature: Testing

Detail

Add integration tests for Redshift connections
It also includes the global conftest fixtures that will be used for redshift-datasets and redshift dataset sharing

At the moment it assumes Redshift infrastructure is already provided; we might want to implement the deployment of this pre-required infra, but that can be done on a separate PR.

+++ Additional fixes found during testing and fixed in this PR

  • Fixed Pivot role permissions in CDK when no workgroups are defined (it handled None wrong)
  • Add update environment stack on Connection delete to ensure permissions to pivot role are removed

Testing

  • Fixes deployed in CICD pipeline in AWS
  • Tests executed locally (in this case we can be sure it will work as in AWS as there are no AWS Clients, just API calls)

image

Relates

Security

Please answer the questions below briefly where applicable, or write N/A. Based on
OWASP 10.

  • Does this PR introduce or modify any input fields or queries - this includes
    fetching data from storage outside the application (e.g. a database, an S3 bucket)?
    • Is the input sanitized?
    • What precautions are you taking before deserializing the data you consume?
    • Is injection prevented by parametrizing queries?
    • Have you ensured no eval or similar functions are used?
  • Does this PR introduce any functionality or component that requires authorization?
    • How have you ensured it respects the existing AuthN/AuthZ mechanisms?
    • Are you logging failed auth attempts?
  • Are you using or adding any cryptographic features?
    • Do you use a standard proven implementations?
    • Are the used keys controlled by the customer? Where are they stored?
  • Are you introducing any new policies/roles/users?
    • Have you used the least-privilege principle? How?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@dlpzx dlpzx force-pushed the feat/redshift-dataset-integration-test branch from b21c8da to 57d4e2b Compare October 10, 2024 12:45
@dlpzx dlpzx force-pushed the feat/redshift-dataset-integration-test branch from 57d4e2b to d658890 Compare October 10, 2024 15:06
@dlpzx dlpzx mentioned this pull request Oct 11, 2024
Merged
@dlpzx dlpzx marked this pull request as ready for review October 11, 2024 11:26
@dlpzx dlpzx force-pushed the feat/redshift-dataset-integration-test branch from 2a3108e to eaa2d53 Compare October 11, 2024 11:28
noah-paige
noah-paige reviewed Oct 11, 2024
View reviewed changes
Copy link
Contributor

@noah-paige noah-paige left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still need to test - setting that up now

But left some comments on code review in the meantime

tests_new/integration_tests/README.md Show resolved Hide resolved
tests_new/integration_tests/modules/redshift_datasets/test_redshift_connection.py Outdated Show resolved Hide resolved
tests_new/integration_tests/modules/redshift_datasets/test_redshift_connection.py Outdated Show resolved Hide resolved
tests_new/integration_tests/modules/redshift_datasets/test_redshift_connection.py Outdated Show resolved Hide resolved
tests_new/integration_tests/modules/redshift_datasets/test_redshift_connection.py Outdated Show resolved Hide resolved
tests_new/integration_tests/modules/redshift_datasets/test_redshift_connection.py Show resolved Hide resolved
@noah-paige
Copy link
Contributor

got all 30 tests passed after ensuring testdata.json configured appropriately. PR looks good pending above comments

@dlpzx dlpzx mentioned this pull request Oct 14, 2024
Merged
dlpzx added a commit that referenced this pull request Oct 14, 2024
@dlpzx
Restrict invitation to Redshift Connections and edit permission name (#…
0eed4cd 
…1638)

### Feature or Bugfix
- Bugfix
- Refactoring

### Detail
- Rename MANAGE_REDSHIFT_CONNECTION_PERMISSIONS as
EDIT_REDSHIFT_CONNECTION_PERMISSIONS to avoid confusion with Tenant
permissions that are always defined as MANAGE....
- Limit the groups that can be granted connection permissions to those
groups in the same Environment
- Adjust unit tests accordingly

### Relates
- #1628 

### Security
Please answer the questions below briefly where applicable, or write
`N/A`. Based on
[OWASP 10](https://owasp.org/Top10/en/).

- Does this PR introduce or modify any input fields or queries - this
includes
fetching data from storage outside the application (e.g. a database, an
S3 bucket)?
  - Is the input sanitized?
- What precautions are you taking before deserializing the data you
consume?
  - Is injection prevented by parametrizing queries?
  - Have you ensured no `eval` or similar functions are used?
- Does this PR introduce any functionality or component that requires
authorization?
- How have you ensured it respects the existing AuthN/AuthZ mechanisms?
  - Are you logging failed auth attempts?
- Are you using or adding any cryptographic features?
  - Do you use a standard proven implementations?
  - Are the used keys controlled by the customer? Where are they stored?
- Are you introducing any new policies/roles/users?
  - Have you used the least-privilege principle? How?


By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
@dlpzx
Copy link
Contributor Author

dlpzx commented Oct 15, 2024

Hi @noah-paige I added a new test and tested all of them again locally:
image

Because I am playing a bit with the invitation to environments/organizations I will run the tests on a real AWS CICD pipeline

noah-paige
noah-paige approved these changes Oct 15, 2024
View reviewed changes
Copy link
Contributor

@noah-paige noah-paige left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tested on my end as well and looks good

@dlpzx dlpzx merged commit 8dfd9ba into main Oct 15, 2024
9 checks passed
dlpzx added a commit that referenced this pull request Oct 16, 2024
@dlpzx
Integration tests executed on a real deployment as part of the CICD -…
85dbfdc 
… Redshift Datasets (#1636)

⚠️ MERGE AFTER #1628
### Feature or Bugfix
- Feature: Testing

### Detail
Add integration tests for Redshift datasets
It also includes the global conftest fixtures that will be used for
redshift dataset sharing

At the moment it assumes Redshift infrastructure is already provided; we
might want to implement the deployment of this pre-required infra, but
that can be done on a separate PR.

## Testing
Tested locally:

![image](https://github.com/user-attachments/assets/564ff84f-61ac-4055-9551-e7186cacdd69)


### Relates
- #1619 
- #1220 

### Security
Please answer the questions below briefly where applicable, or write
`N/A`. Based on
[OWASP 10](https://owasp.org/Top10/en/).

- Does this PR introduce or modify any input fields or queries - this
includes
fetching data from storage outside the application (e.g. a database, an
S3 bucket)?
  - Is the input sanitized?
- What precautions are you taking before deserializing the data you
consume?
  - Is injection prevented by parametrizing queries?
  - Have you ensured no `eval` or similar functions are used?
- Does this PR introduce any functionality or component that requires
authorization?
- How have you ensured it respects the existing AuthN/AuthZ mechanisms?
  - Are you logging failed auth attempts?
- Are you using or adding any cryptographic features?
  - Do you use a standard proven implementations?
  - Are the used keys controlled by the customer? Where are they stored?
- Are you introducing any new policies/roles/users?
  - Have you used the least-privilege principle? How?


By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
@dlpzx dlpzx deleted the feat/redshift-dataset-integration-test branch October 25, 2024 12:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@noah-paige noah-paige noah-paige approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@dlpzx @noah-paige