configure logger on worker initialization

[adamsachs]

Description Of Changes

Without this change, we don't configure the loguru logger on the worker app/process. This means that even though we may have set e.g. logging.serialization = json on the worker container, it wouldn't actually impact the log behavior on the worker in any way.

With this change, before booting up the celery worker app, we now invoke the same setup() logging function that's invoked when creating our fides fast API app (i.e. within create_fides_app). This ensures consistent logging configuration across both our server and worker apps 👍

Code Changes

• invoke fides.api.util.logger.setup within the _create_celery helper function that's invoked as part of the worker runtime bootstrap, i.e. within fides.api.worker.start_worker

Steps to Confirm

• set logging.serialization = "json" and celery.task_always_eager = false in your app config (i.e. in fides.toml)
• run nox -s dev -- worker in fides (OSS)
• create/enable a bogus system with a bogus integration (connector) that will be run on an access request (i used "braze")
• run privacy center and admin UI by running turbo run dev in the clients dir
• submit an access request for an arbitrary email, wait to see logs come through on the worker container (you can tail logs with docker logs -f fides-worker-1
• ensure logs are serialized as JSON and have the additional (structured) metadata they should have (added in Structured logging for DSR requests #4594)
  • example:

{"text": "2024-02-20 20:58:04.084 | ERROR    |  - Connector request failed. | {'action_type': 'access', 'system_key': 'asdf', 'connection_key': 'asdf_braze_api', 'collection': 'subscription_groups', 'privacy_request_id': 'pri_b0a1b442-4a6a-4589-8a12-f9a991ef9b2f', 'error_group': 'NetworkError', 'error_details': 'Timeout occurred connecting to https://asdf.com.', 'status_code': None}\n", "record": {"elapsed": {"repr": "0:03:29.848315", "seconds": 209.848315}, "exception": null, "extra": {"action_type": "access", "system_key": "asdf", "connection_key": "asdf_braze_api", "collection": "subscription_groups", "privacy_request_id": "pri_b0a1b442-4a6a-4589-8a12-f9a991ef9b2f", "error_group": "NetworkError", "error_details": "Timeout occurred connecting to https://asdf.com.", "status_code": null}, "file": {"name": "authenticated_client.py", "path": "/fides/src/fides/api/service/connectors/saas/authenticated_client.py"}, "function": "result", "level": {"icon": "❌", "name": "ERROR", "no": 40}, "line": 138, "message": "Connector request failed.", "module": "authenticated_client", "name": "fides.api.service.connectors.saas.authenticated_client", "process": {"id": 22, "name": "ForkPoolWorker-2"}, "thread": {"id": 281473068298720, "name": "ThreadPoolExecutor-0_0"}, "time": {"repr": "2024-02-20 20:58:04.084453+00:00", "timestamp": 1708462684.084453}}}

Pre-Merge Checklist

• All CI Pipelines Succeeded
• Documentation:
  • documentation complete, PR opened in fidesdocs
  • documentation issue created in fidesdocs
• Issue Requirements are Met
• Relevant Follow-Up Issues Created
• Update CHANGELOG.md
• For API changes, the Postman collection has been updated

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
fides-plus-nightly	⬜️ Ignored (Inspect)	Visit Preview		Feb 21, 2024 2:51am

[cypress]

Passing run #6310 ↗︎

0	4	0	0	0
⚠️ You've recorded test results over your free plan limit. Upgrade your plan to view test results.

Details:

Merge 585a3e3 into 68c204a...
Project: fides	Commit: a95b007783 ℹ️
Status: Passed	Duration: 00:35 💡
Started: Feb 21, 2024 3:01 AM	Ended: Feb 21, 2024 3:01 AM

Review all test suite changes for PR #4624 ↗︎

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (68c204a) 86.83% compared to head (585a3e3) 86.83%.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4624   +/-   ##
=======================================
  Coverage   86.83%   86.83%           
=======================================
  Files         332      332           
  Lines       19975    19979    +4     
  Branches     2569     2569           
=======================================
+ Hits        17345    17349    +4     
  Misses       2157     2157           
  Partials      473      473           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pattisdr]

this makes sense Adam 👍