Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding retries to new database task sessions #5448

Merged
merged 12 commits into from
Nov 19, 2024

Conversation

galvana
Copy link
Contributor

@galvana galvana commented Nov 1, 2024

Closes LA-89

Description Of Changes

This change is intended to resolve an intermittent error we are seeing in production

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SSL connection has been closed unexpectedly

We haven't been able to reproduce it locally but given the intermittent, and short-lived nature of the issue, we believe a retry should account for these disconnections. We also disabled connection pooling for task workers since we believe this is causing issues https://docs.sqlalchemy.org/en/14/core/pooling.html#using-connection-pools-with-multiprocessing-or-os-fork

Code Changes

  • Adding a retry to DatabaseTask.get_new_session()
  • Disabled connection pooling for task workers

Steps to Confirm

  • Unable to verify locally, we just need to make sure all the existing tests pass

Pre-Merge Checklist

  • All CI Pipelines Succeeded
  • Issue Requirements are Met
  • Update CHANGELOG.md

Copy link

vercel bot commented Nov 1, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment
Name Status Preview Comments Updated (UTC)
fides-plus-nightly ⬜️ Ignored (Inspect) Visit Preview Nov 19, 2024 6:24pm

Copy link

cypress bot commented Nov 1, 2024

fides    Run #11085

Run Properties:  status check passed Passed #11085  •  git commit b3aaa34a26 ℹ️: Merge 3f9d563b2908258b696107a66256224981d6f895 into 8c59e5d3961d9e23206566f12192...
Project fides
Branch Review refs/pull/5448/merge
Run status status check passed Passed #11085
Run duration 00m 37s
Commit git commit b3aaa34a26 ℹ️: Merge 3f9d563b2908258b696107a66256224981d6f895 into 8c59e5d3961d9e23206566f12192...
Committer Adrian Galvan
View all properties for this run ↗︎

Test results
Tests that failed  Failures 0
Tests that were flaky  Flaky 0
Tests that did not run due to a developer annotating a test with .skip  Pending 0
Tests that did not run due to a failure in a mocha hook  Skipped 0
Tests that passed  Passing 4
⚠️ You've recorded test results over your free plan limit.
Upgrade your plan to view test results.
View all changes introduced in this branch ↗︎

Copy link

codecov bot commented Nov 1, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 85.47%. Comparing base (e5f240d) to head (e8219f2).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5448      +/-   ##
==========================================
+ Coverage   77.00%   85.47%   +8.46%     
==========================================
  Files         384      384              
  Lines       24116    24122       +6     
  Branches     2624     2624              
==========================================
+ Hits        18571    20618    +2047     
+ Misses       4971     2950    -2021     
+ Partials      574      554      -20     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@galvana galvana requested review from tvandort and NevilleS November 1, 2024 23:55
Copy link
Contributor

@tvandort tvandort left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's try it!

@galvana galvana added the run unsafe ci checks Runs fides-related CI checks that require sensitive credentials label Nov 6, 2024
@galvana galvana requested a review from tvandort November 7, 2024 21:34
Copy link
Contributor

@eastandwestwind eastandwestwind left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome work on this @galvana ! I'm excited to have this in prod, as it adds extra polish and resilience to our product 🌟

Just a couple notes / asks, mainly for ease of understanding for future devs.

src/fides/api/tasks/__init__.py Show resolved Hide resolved
tests/ops/tasks/test_database_task.py Show resolved Hide resolved
tests/ops/tasks/test_database_task.py Show resolved Hide resolved
src/fides/config/database_settings.py Show resolved Hide resolved
src/fides/api/tasks/__init__.py Show resolved Hide resolved
@galvana galvana merged commit c7e645f into main Nov 19, 2024
17 checks passed
@galvana galvana deleted the LA-89-add-connection-retries-to-database-tasks branch November 19, 2024 18:30
Copy link

cypress bot commented Nov 19, 2024

fides    Run #11086

Run Properties:  status check passed Passed #11086  •  git commit c7e645fd2d: Adding retries to new database task sessions (#5448)
Project fides
Branch Review main
Run status status check passed Passed #11086
Run duration 00m 39s
Commit git commit c7e645fd2d: Adding retries to new database task sessions (#5448)
Committer Adrian Galvan
View all properties for this run ↗︎

Test results
Tests that failed  Failures 0
Tests that were flaky  Flaky 0
Tests that did not run due to a developer annotating a test with .skip  Pending 0
Tests that did not run due to a failure in a mocha hook  Skipped 0
Tests that passed  Passing 4
⚠️ You've recorded test results over your free plan limit.
Upgrade your plan to view test results.
View all changes introduced in this branch ↗︎

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
run unsafe ci checks Runs fides-related CI checks that require sensitive credentials
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants