Improve DynamoDB Field Generation and Error Handling

[SteveDMurphy]

Closes FIDES-635

Description Of Changes

• Scans Dynamo (ignoring values) to generate a more accurate field description
• Handles errors generating from incomplete permissions (either to view or operate on resources)
• Optionally Generate DynamoDB datasets individually instead of nested into a single Dataset

Code Changes

• Added CLI option to have separate datasets
• Handle exceptions from Boto3 around permissions errors
• Scan the DynamoDB table for Fields
• Ensure pagination is handled correctly

Steps to Confirm

• credentials_id shared in 1Password to use for testing with

Pre-Merge Checklist

• All CI Pipelines Succeeded
• Documentation:
  • documentation complete, PR opened in fidesdocs
  • documentation issue created in fidesdocs
  • if there are any new client scopes created as part of the pull request, remember to update public-facing documentation that references our scope registry
• Issue Requirements are Met
• Relevant Follow-Up Issues Created
• Update CHANGELOG.md
• For API changes, the Postman collection has been updated
• If there are any database migrations:
  • Ensure that your downrev is up to date with the latest revision on main
  • Ensure that your downgrade() migration is correct and works
    • If a downgrade migration is not possible for this change, please call this out in the PR description!

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
fides-plus-nightly	⬜️ Ignored (Inspect)	Visit Preview		Jun 4, 2024 6:40pm

[cypress]

Passing run #8106 ↗︎

0

4

0

0

0

Details:

Merge c769649 into 1b0acd8...
Project: fides	Commit: 850cb60000 ℹ️
Status: Passed	Duration: 00:34 💡
Started: Jun 4, 2024 6:51 PM	Ended: Jun 4, 2024 6:51 PM

Review all test suite changes for PR #4943 ↗︎

[codecov]

Codecov Report

Attention: Patch coverage is 17.07317% with 34 lines in your changes missing coverage. Please review.

Project coverage is 86.86%. Comparing base (d50f5b3) to head (c769649).
Report is 2 commits behind head on main.

Files	Patch %	Lines
src/fides/connectors/aws.py	13.88%	31 Missing ⚠️
src/fides/api/api/v1/endpoints/generate.py	0.00%	1 Missing ⚠️
src/fides/cli/commands/generate.py	50.00%	1 Missing ⚠️
src/fides/core/dataset.py	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4943      +/-   ##
==========================================
- Coverage   86.92%   86.86%   -0.06%     
==========================================
  Files         346      346              
  Lines       21209    21253      +44     
  Branches     2785     2793       +8     
==========================================
+ Hits        18435    18461      +26     
- Misses       2285     2304      +19     
+ Partials      489      488       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[adamsachs]

@SteveDMurphy nice job cleaning this up! i made one minor change on my end that allows us to fully utilize these underlying functions in the new monitor code over in plus, hope that's alrigh.t

generally all these changes seem sound to me, and i've tested them with an eye to how they integrate with new dynamo monitor code i'm working on in plus (will put the PR link once i've got the PR up).

i haven't tested or looked super closely into the changes in relation to our CLI workflows - but i trust that you've done due diligence there (and i know you've been testing it!). coupled with the fact that CI is passing, i feel comfortable merging this 👍