Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Integration tests executed on a real deployment as part of the CICD - Redshift Datasets #1636

Merged
merged 15 commits into from
Oct 16, 2024
Merged

Integration tests executed on a real deployment as part of the CICD - Redshift Datasets #1636

merged 15 commits into from
Oct 16, 2024

Conversation

dlpzx
Copy link
Contributor

@dlpzx dlpzx commented Oct 11, 2024 •
edited
Loading

⚠️ MERGE AFTER #1628

Feature or Bugfix

  • Feature: Testing

Detail

Add integration tests for Redshift datasets
It also includes the global conftest fixtures that will be used for redshift dataset sharing

At the moment it assumes Redshift infrastructure is already provided; we might want to implement the deployment of this pre-required infra, but that can be done on a separate PR.

Testing

Tested locally:
image

Relates

Security

Please answer the questions below briefly where applicable, or write N/A. Based on
OWASP 10.

  • Does this PR introduce or modify any input fields or queries - this includes
    fetching data from storage outside the application (e.g. a database, an S3 bucket)?
    • Is the input sanitized?
    • What precautions are you taking before deserializing the data you consume?
    • Is injection prevented by parametrizing queries?
    • Have you ensured no eval or similar functions are used?
  • Does this PR introduce any functionality or component that requires authorization?
    • How have you ensured it respects the existing AuthN/AuthZ mechanisms?
    • Are you logging failed auth attempts?
  • Are you using or adding any cryptographic features?
    • Do you use a standard proven implementations?
    • Are the used keys controlled by the customer? Where are they stored?
  • Are you introducing any new policies/roles/users?
    • Have you used the least-privilege principle? How?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@dlpzx dlpzx linked an issue Oct 11, 2024 that may be closed by this pull request
Integration tests executed on a real deployment as part of the CICD - Redshift Datasets #1619
Closed
dlpzx added 7 commits October 11, 2024 13:28
@dlpzx
Fix assertion in test_redshift_schema_tables + clean PR
eaa2d53
@dlpzx
Merge remote-tracking branch 'origin/feat/redshift-dataset-integratio…
293fc30 
…n-test' into feat/redshift-dataset-integration-test-2

# Conflicts:
#	tests_new/integration_tests/modules/redshift_datasets/global_conftest.py
@dlpzx
Redshift-dataset tests
1e853c4
@dlpzx
PR review comments
e879794
@dlpzx
Merge remote-tracking branch 'origin/main' into feat/redshift-dataset…
821baea 
…-integration-test
@dlpzx
PR review comments
b2a2c76
@dlpzx
Merge remote-tracking branch 'origin/feat/redshift-dataset-integratio…
693a67c 
…n-test' into feat/redshift-dataset-integration-test-2

# Conflicts:
#	tests_new/integration_tests/modules/redshift_datasets/global_conftest.py
@dlpzx
Copy link
Contributor Author

dlpzx commented Oct 15, 2024

Re-tested with changes from PR of Redshift Connections #1628

@dlpzx dlpzx marked this pull request as ready for review October 15, 2024 10:03
@dlpzx
Merge remote-tracking branch 'origin/main' into feat/redshift-dataset…
4885590 
…-integration-test-2

# Conflicts:
#	tests_new/integration_tests/core/environment/global_conftest.py
#	tests_new/integration_tests/modules/redshift_datasets/dataset_queries.py
#	tests_new/integration_tests/modules/redshift_datasets/global_conftest.py
#	tests_new/integration_tests/modules/redshift_datasets/test_redshift_connection.py
noah-paige
noah-paige reviewed Oct 15, 2024
View reviewed changes
Copy link
Contributor

@noah-paige noah-paige left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very minor comments - to test now and will comment on any additional findings

tests_new/integration_tests/modules/redshift_datasets/global_conftest.py Outdated Show resolved Hide resolved
tests_new/integration_tests/modules/redshift_datasets/global_conftest.py Outdated Show resolved Hide resolved
tests_new/integration_tests/modules/redshift_datasets/test_redshift_dataset.py Outdated Show resolved Hide resolved
tests_new/integration_tests/modules/redshift_datasets/test_redshift_dataset.py Outdated Show resolved Hide resolved
tests_new/integration_tests/modules/redshift_datasets/test_redshift_dataset.py Outdated Show resolved Hide resolved
@noah-paige
Copy link
Contributor

All 54 test passed between redshift connection and redshift datasets against AWS deployment - will approve pending responses to small nit comments above

@dlpzx dlpzx requested a review from noah-paige October 16, 2024 06:28
@dlpzx dlpzx mentioned this pull request Oct 16, 2024
Merged
@dlpzx dlpzx merged commit 85dbfdc into main Oct 16, 2024
9 checks passed
dlpzx added a commit that referenced this pull request Oct 22, 2024
@dlpzx
Integration tests executed on a real deployment as part of the CICD -…
3747c9d 
… Redshift Shares (#1643)

⚠️ MERGE AFTER #1636 
### Feature or Bugfix
- Feature: Testing

### Detail
Add integration tests for Redshift shares. Implements #1620 
- Implemented inside the shares modules in a subdirectory so that each
share type can have its own conftest but still re-use common methods
from shares queries
- This PR is focused on testing the Redshift shares functionality, it
does not include all tests that test the workflow of the share (e.g.
submit, reject...)
- It does not validate if after a share the user has access to data. We
could implement it using the Redshift Data API, but I left it as
optional for a separate PR

### Tested
Locally:

![image](https://github.com/user-attachments/assets/3a2acc79-d025-483f-949b-23e31b23d26e)


### Relates
- #1620
- #1619
- #1220

### Security
Please answer the questions below briefly where applicable, or write
`N/A`. Based on
[OWASP 10](https://owasp.org/Top10/en/).

- Does this PR introduce or modify any input fields or queries - this
includes
fetching data from storage outside the application (e.g. a database, an
S3 bucket)?
  - Is the input sanitized?
- What precautions are you taking before deserializing the data you
consume?
  - Is injection prevented by parametrizing queries?
  - Have you ensured no `eval` or similar functions are used?
- Does this PR introduce any functionality or component that requires
authorization?
- How have you ensured it respects the existing AuthN/AuthZ mechanisms?
  - Are you logging failed auth attempts?
- Are you using or adding any cryptographic features?
  - Do you use a standard proven implementations?
  - Are the used keys controlled by the customer? Where are they stored?
- Are you introducing any new policies/roles/users?
  - Have you used the least-privilege principle? How?


By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
@dlpzx dlpzx deleted the feat/redshift-dataset-integration-test-2 branch October 25, 2024 12:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@noah-paige noah-paige noah-paige approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Integration tests executed on a real deployment as part of the CICD - Redshift Datasets
2 participants
@dlpzx @noah-paige