Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Retrieve child results during fides connector execution #1967

Merged
merged 4 commits into from
Dec 1, 2022
Merged

Retrieve child results during fides connector execution #1967

merged 4 commits into from
Dec 1, 2022

Conversation

adamsachs
Copy link
Contributor

@adamsachs adamsachs commented Dec 1, 2022 •
edited
Loading

Closes #1966

Code Changes

  • Reach out to child's privacy_request_data_transfer endpoint immediately after polling in the FidesConnector's main retrieve_data method
  • Make one retrieval per rule in the provided policy, and store each result as an entry in a dict keyed by the rule.key
  • Unpack this dict with some special-cased post-processing for Fides connectors at the framework level, depending on the rule whose results we are processing/persisting to storage
  • Update tests as needed

Steps to Confirm

Pre-Merge Checklist

  • All CI Pipelines Succeeded
  • Documentation Updated:
    • documentation complete, or draft/outline provided (tag docs-team to complete/review on this branch)
    • documentation issue created (tag docs-team to complete issue separately)
  • Issue Requirements are Met
  • Relevant Follow-Up Issues Created
  • Update CHANGELOG.md

Description Of Changes

Write some things here about the changes and any potential caveats

@adamsachs adamsachs linked an issue Dec 1, 2022 that may be closed by this pull request
distributed fides - retrieve child results within fides connector execution #1966
Closed
@adamsachs adamsachs force-pushed the 1966-distributed-fides-retrieve-child-results-within-fides-connector-execution branch from 0da5a32 to d489a5c Compare December 1, 2022 19:52
sanders41
sanders41 reviewed Dec 1, 2022
View reviewed changes
Copy link
Contributor

@sanders41 sanders41 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like this update! Just a couple small comments.

src/fides/api/ops/service/connectors/fides_connector.py Show resolved Hide resolved
src/fides/api/ops/service/connectors/fides_connector.py Outdated Show resolved Hide resolved
src/fides/api/ops/task/filter_results.py Outdated Show resolved Hide resolved
@adamsachs adamsachs marked this pull request as ready for review December 1, 2022 20:24
adamsachs and others added 2 commits December 1, 2022 15:47
@adamsachs
Update docstring
814624a 
Co-authored-by: Paul Sanders <[email protected]>
@adamsachs
update type annotation
3133cc5 
Co-authored-by: Paul Sanders <[email protected]>
@sanders41 sanders41 merged commit 557a656 into main Dec 1, 2022
@sanders41 sanders41 deleted the 1966-distributed-fides-retrieve-child-results-within-fides-connector-execution branch December 1, 2022 21:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sanders41 sanders41 sanders41 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

distributed fides - retrieve child results within fides connector execution
2 participants
@adamsachs @sanders41