Catch startup error and log to console #1239
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Update config.py * Update crypto imports to fideslib and remove legacy file and tests * Update Identity salt * checkpoint, lowercasing all of the things * update more config values to lowercase * fix linting errors, fix config validators * update the configs to use lowercase keys * lowercase the allowed keys * Update create_test_data.py * bump fideslib version * Lowercase config variables * Fix linting issues * Fix some test failures * Fix application fixtures * Remove old celery config options * Sort script imports * Update changelog Co-authored-by: Thomas <[email protected]>
* Replace config/gh secrets with secrets from vault * sorting and update changelog * fix path for importing test helpers * more sorting * run ci again with empty commit * update unsafe_pr_checks * allow for no vault vars for unit tests, adds back back for saas config toml vars * check for client * sort * init client to None * fix imports * fix pytest markers to better indicate which tests rely on actual secrets
Bumps [pandas](https://github.com/pandas-dev/pandas) from 1.3.3 to 1.4.3. - [Release notes](https://github.com/pandas-dev/pandas/releases) - [Changelog](https://github.com/pandas-dev/pandas/blob/main/RELEASE.md) - [Commits](pandas-dev/pandas@v1.3.3...v1.4.3) --- updated-dependencies: - dependency-name: pandas dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Paul Sanders <[email protected]>
* initial sendgrid saas connector integration. access only, contacts only * erasure (update) support for sendgrid * remove unused imports * update sendgrid test fixture to expect 404 response status code because of ignore_errors enhancement * Fixing import order and cleaning up the retry logic for consistency * add sendgrid env var support to makefile and unsafe_pr_checks config. remove DELETE endpoint per PR comments * Added delete endpoint for contacts * Fixing data_path for contacts endpoint * Reverting search query to improve performance and avoid server timeouts * Updated delete endpoint request, used request instead of SaaSRequest in tests * updated imports after check suggestion * Updated code after review * Removed unused variables, imports * Restoring Makefile * Fixed import cryptographic_util error * Misc fixes * Updated Changelog file * Updated Changelog for unreleased section and pulled main * Updated Changelog and added Sendgrid in unreleased section * Updated Changelog and added Sendgrid in unreleased section with link * Updated Changelog and added Sendgrid in added section after Adam's suggestion Co-authored-by: Adam Sachs <[email protected]> Co-authored-by: Adam Sachs <[email protected]> Co-authored-by: Adrian Galvan <[email protected]> Co-authored-by: Hamza W <[email protected]> Co-authored-by: Adrian Galvan <[email protected]>
…ype filter to open up the dropdown (#903) * 747 - Users should be able to click on the full field of a dropdown-type filter to open up the dropdown * Disabled eslint import/extensions rule for certain files with an alias path in the import statement * Updated CHANGELOG.md file * Resolved jest unit tests from failing * Removed eslint-disable import/extensions declarations
Bumps [types-redis](https://github.com/python/typeshed) from 4.3.2 to 4.3.4. - [Release notes](https://github.com/python/typeshed/releases) - [Commits](https://github.com/python/typeshed/commits) --- updated-dependencies: - dependency-name: types-redis dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* adds saas config base info to connection config responses * update changelog * add properties to base class * use diff naming to avoid recursion * update saas_config prop for unit tests * another occurance of saas config in unit test * use pydantic alias for type field * sort * revert change to saas type naming * unused import * init parent class so that props are avail on self * revert to using props
Bumps [mypy](https://github.com/python/mypy) from 0.961 to 0.971. - [Release notes](https://github.com/python/mypy/releases) - [Commits](python/mypy@v0.961...v0.971) --- updated-dependencies: - dependency-name: mypy dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [pydash](https://github.com/dgilland/pydash) from 5.0.2 to 5.1.0. - [Release notes](https://github.com/dgilland/pydash/releases) - [Changelog](https://github.com/dgilland/pydash/blob/develop/CHANGELOG.rst) - [Commits](dgilland/pydash@v5.0.2...v5.1.0) --- updated-dependencies: - dependency-name: pydash dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Updates the requirements on [boto3](https://github.com/boto/boto3) to permit the latest version. - [Release notes](https://github.com/boto/boto3/releases) - [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst) - [Commits](boto/boto3@1.18.14...1.24.34) --- updated-dependencies: - dependency-name: boto3 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [dask](https://github.com/dask/dask) from 2022.6.1 to 2022.7.0. - [Release notes](https://github.com/dask/dask/releases) - [Changelog](https://github.com/dask/dask/blob/main/docs/release-procedure.md) - [Commits](dask/dask@2022.6.1...2022.7.0) --- updated-dependencies: - dependency-name: dask dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…changed for config variables. (#921)
* Correct build arg variable name * Update CHANGELOG Co-authored-by: Paul Sanders <[email protected]>
* Update python docker base image from slim-buster to slim-bullseye * Update CHANGELOG * Remove ipython from dev-requirements.txt Co-authored-by: Paul Sanders <[email protected]> Co-authored-by: Sean Preston <[email protected]>
Updates the requirements on [boto3](https://github.com/boto/boto3) to permit the latest version. - [Release notes](https://github.com/boto/boto3/releases) - [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst) - [Commits](boto/boto3@1.24.34...1.24.36) --- updated-dependencies: - dependency-name: boto3 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [faker](https://github.com/joke2k/faker) from 13.15.0 to 13.15.1. - [Release notes](https://github.com/joke2k/faker/releases) - [Changelog](https://github.com/joke2k/faker/blob/master/CHANGELOG.md) - [Commits](joke2k/faker@v13.15.0...v13.15.1) --- updated-dependencies: - dependency-name: faker dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [types-ujson](https://github.com/python/typeshed) from 5.2.0 to 5.4.0. - [Release notes](https://github.com/python/typeshed/releases) - [Commits](https://github.com/python/typeshed/commits) --- updated-dependencies: - dependency-name: types-ujson dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* add db vs saas to connection type api * adds changelog line * mypy * fix test * format * more tests * formatting * adds system type query param * format * adjust test saas search * correct query param
* Create noxfile.py * update the dockerfile with more stages * add GitPython as a dev requirement (used by nox builds) * add the noxfiles, all docker builds work * fix the "make compose-build" command * remove the worker docker stage due to it being redundant with prod * create a unified compose file for integrations * update the password prompt to be accurate * copy/pasta run_infrastructure into the noxfiles dir so it can be called directly via python * add create_user and seed_test_data to the nox utils * docs commands work * get the generic dev command working * add db commands to utils * clean up mypy configuration in pyproject.toml and remove config from setup.cfg * simplify some of the CI targets and start updating the pytest targets * cleanup run_infrastructure * update ci_suite and other CI nox targets * add nox as a dev-requirement * get dev commands working, sans quickstart * tweak to the compose_down constant * get the new pylint target passing * remove analytics_id * updated the changelog * Apply suggestions from code review Co-authored-by: Paul Sanders <[email protected]> * fix the xenon command * add a few tweaks to the worker dev command * fix an import issue * update dockerignore * move mssql to the bottom of the datastore list Co-authored-by: Paul Sanders <[email protected]> Co-authored-by: Paul Sanders <[email protected]>
* Changed Debian base image * Update the python version to be the same as used in the fidesops app image Co-authored-by: Paul Sanders <[email protected]> * Updated changelog Co-authored-by: Dave Quinlan <[email protected]> Co-authored-by: Paul Sanders <[email protected]>
* Change docker password environment variable to MSSQL_SA_PASSWORD * Revert password environment variable name * Set MSSQL user to root in docker-compose * Revert setting root user in docker-compose * Change environment variable name from REQUIRE_MSSQL to SKIP_MSSQL_INSTALLATION * Add healthcheck to mssql compose file * Modify healthcheck command * Revert healthcheck * Try for more loging information * Another try for logging * Another try for logging * Try running only mssql tests to avoid timeout * Revert mssql only flag * Extend time out to try to get logs * Revert extra logging and extended timeout * Set network mode to host * Make mssql run on its own * Remove network from docker-compose and only run mssql in integration * Increase integration test logging * Revert mssql only * Use cache for docker * Fix workflow error * Fix workflow error * Fix target * Fix make traget * Fix make traget * Revert cache * Verify that mssql is running from pytest fixture * Add restart to mssql * Revert rester in docker-compose * Revert wait for mssql in pytest fixture * Wait for mssql to be ready before adding test data * Add count of retries * Update CHANGELOG Co-authored-by: Paul Sanders <[email protected]>
We've defined generic issue templates for all Ethyca repos here: https://github.com/ethyca/.github/tree/main/.github/ISSUE_TEMPLATE Removing the templates from this repo allows it to automatically pull in the organization templates for consistency.
* Make nox test name sessions more logical * consolidate the nox test sessions
* Resolve conflicts and upgrade next * Resolve merge conflicts * Fix missing required prop * Update postman collection /policy --> /dsr/policy
* Instead of rebuilding node_modules, just make docker-compose not overwrite them during volume mounting * Increase count for starting up fides, likely needed for M1 macs
Co-authored-by: Adam Sachs <[email protected]>
…d into create_or_update_email_config. (#1146)
…go (#1158) * 1149 - Update all fidesops/fidesctl logos to use the default Fides logo Updated logo.svg file to the new Fides logo * Updated CHANGELOG.md file
* Correctly `isort` `versioneer` in files * Make `optional-requirements.txt` a source of truth * Update `CHANGELOG.md` * Include `optional_requirements.txt` in the package * Specify an encoding when opening requirements files Quiets the pylint warning * Remove superfluous variables * Match dependency names more strictly, add docstring
…res (#1178) * Delete fidesctl user routes in favor of fidesops user routes. They are practically copies of each other except fidesops user logout route has improved logic such as allowing the user to logout with a malformed or expired token. The fidesctl route was taking priority. * Use new CONFIG variable in test_saas_queryconfig * Adjust fidesops health endpoint test. We're using the fidesctl health endpoint now and they surface a version number while fidesops did not. * Add missing CONFIG variable in existing execution tests. Fix incorrect timescale test name. * Add missing timescale secrets to integration_test_config.toml.
…st mode" is True. (#1162)
* 1191-Duplicate key in fides admin ui package.json file * Updated CHANGELOG.md file
* Stop creating new engines as part of the process of running health checks which cause us to have too many idle connections opened against the application database. Use the same engine that is being shared across the "ops" API endpoints. - Remove unused get_db_for_health_check * Remove the ctl get_db used in the endpoints in favor of the ops get_db. * Mock get_db_health needs two parameters.
* Re-add SaaS template update script to startup hook Also update dependency getter that returns the shared db session directly to be importable from outside classes. * Use a context manager for db session Co-authored-by: Adam Sachs <[email protected]>
* Bump pre-commit from 2.9.3 to 2.20.0 (#853) (#1373) Bumps [pre-commit](https://github.com/pre-commit/pre-commit) from 2.9.3 to 2.20.0. - [Release notes](https://github.com/pre-commit/pre-commit/releases) - [Changelog](https://github.com/pre-commit/pre-commit/blob/main/CHANGELOG.md) - [Commits](pre-commit/pre-commit@v2.9.3...v2.20.0) --- updated-dependencies: - dependency-name: pre-commit dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * 1016 frontend ability for users to manually enter pii to an in progress subject request (#1377) * On successful connector creation the empty yaml text input displays a validation error * Updated caniuse-lite NPM dependency * Added min/max character limitation validation for Manual Webhook DSR Customization * Prevented Chakra Divider component opacity from being overridden at runtime. User could not see visible horizontal line on UI before. * Updated the Manual Processing detail form to not be submitted until the form is dirty. * Removed form dirty check on ManualProcessingDetail component * Update URL when user is navigating to DSR Customization screen when creating a Manual Webhook * Enable retries on saas connectors for failures at the http request level (#1376) * Add decorator to send method that retries throttles * Fix black/mypy * Fix pylint * Add tests for authenticated client * Small fixes and typos * Update CHANGELOG.md * Small changes. Update retry logic to not retry general exceptions Co-authored-by: Eduardo Armendariz <[email protected]> * Add Consent Request API (#1387) Co-authored-by: Paul Sanders <[email protected]> * Add new template for consent requets emails (#1405) * Add new template for consent requets emails * Remove analytics id * Update subject Co-authored-by: Paul Sanders <[email protected]> * Add authenticated route to get consent preferences (#1402) Co-authored-by: Paul Sanders <[email protected]> * Backend: Updating or Deleting Access Webhooks [#1388][#1389] (#1394) * If a manual webhook is deleted or disabled, check if there are any remaining active manual webhooks configured. If not, queue any Privacy Requests stuck in "requires_input" for processing. * In the "view_uploaded_manual_webhook_data", load cached webhook data for a privacy request in strict mode. If it fails (no data saved, extra field saved, field missing), return checked=True, so the user knows they need to reupload data for this webhook before it can be submitted. Return the data in non-strict mode, so we just show the overlap between the data saved and the fields defined. * Update changelog. * Move queue_requires_input_requests to the connection_endpoints where this is the only module it's being called - both where you update and delete a connection config. * Clarify docstring. * Braze Connector: Access Endpoints (#1248) * [#1393] Update Fidesops config with sane defaults where necessary (#1395) * add sane defaults * make subsections of config with complete defaults optional * lowercase database.enabled, set defaults for optional configs * update return type * updates changelog * make PORT an env var * cast env var to int * remove unnecessary unpinned dependency * bump fideslib version * bump fideslib to 3.1.4 * add defaults for the non optional config subclasses * set empty dict to default for config subclasses that require some fields * use .get() in assemble URL for correct error message, correct comment * update jwt_key type annotation * Explain Privacy Request Execution [#1397] (#1396) Co-authored-by: Cole <[email protected]> * Update docker command on privacy center step 4 (#1410) removing the typo ` . at the end of the command so that the command works. * 1319 consent UI api integration (#1407) * Add consent UI back in * Finish initial integration with consent api * WIP consent page * Get initial consent updating working * Improve button look and feel * Add untracked files * Format VerificationForm.tsx * Remove comments * Rename Privacy modal variables * Rename variable * 1401 admin UI persist redux store to localstorage (#1409) * 1401 - Admin UI: Persist Redux store to localStorage * Resolved React memory leak when user attempts to logout via the Subject Requests landing page * Resolved UI unit test failure * Rollback previous change * Removed blacklist property from Redux store config * Refactored due to recommended code review feedback * update footer links (#1406) * update footer links * changelog * Update CHANGELOG.md Co-authored-by: Paul Sanders <[email protected]> Co-authored-by: Sean Preston <[email protected]> Co-authored-by: Paul Sanders <[email protected]> * firebase auth integration (#1399) * Initial implementation of firebase auth connector Introduces firebase_admin sdk as a fidesops dependency Leverages request overrides to define custom functionality * Update changelog * Minimize unused client config in firebase auth config * Add user.photo_url as a supported Firebase Auth field Improve readability in access implementation * Fix field name to be more consistent with python SDK properties * Add clarifying comment to unused function docstring * Add photo_url field to firebase auth dataset * Properly test provider data in firebase auth integration. Also include test coverage for delete function, even thought it's not being invoked by default saas config. * Only add fields to result if popluated * Fix empty attribute logic within provider data block * Ignore pylint error for too many variables * fix up provider data check Co-authored-by: Adam Sachs <[email protected]> * Add Braze connector registry entry (#1418) * Braze SVG added * Fixing change log message Co-authored-by: Adrian Galvan <[email protected]> Co-authored-by: Adam Sachs <[email protected]> * Removing PII fields from conversations collection (#1386) * Fidesops v1.8.1 Release Checklist (#1420) * updates changelog * add UI testing steps to release checklist * DX Improvements from `v1.8.1` release (#1421) * stop swallowing integrity errors on dataset update * dont attach logging middleware if analytics opt_out is true * handle diff correctly * disable consider-using-f-string as we need this for logging statements to work effectively in fideslog * Updated CHANGELOG.md file from prior merge * get the server up and running * fix test collection * fix isort and black * update dataset annotations * Merge latest fidesops UI into unified ui * Fixing post UI merge issues * Fix formatting and import issues * get all unit tests passing * Fix cypress failure 🤞 * Remove auth.slice.test.ts Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Eduardo Armendariz <[email protected]> Co-authored-by: Eduardo Armendariz <[email protected]> Co-authored-by: Paul Sanders <[email protected]> Co-authored-by: Paul Sanders <[email protected]> Co-authored-by: Dawn Pattison <[email protected]> Co-authored-by: Noonari <[email protected]> Co-authored-by: Sean Preston <[email protected]> Co-authored-by: Cole <[email protected]> Co-authored-by: shawnplusplus <[email protected]> Co-authored-by: Andrew Jackson <[email protected]> Co-authored-by: Cole Isaac <[email protected]> Co-authored-by: Adam Sachs <[email protected]> Co-authored-by: Adam Sachs <[email protected]> Co-authored-by: Adrian Galvan <[email protected]> Co-authored-by: Thomas <[email protected]>
* Remove SaaS type enum and dynamically reference registered types Update tests to dynamically compare results rather than looking for static values. * Remove references to specific custom SaaSType Since we no longer have an enum for SaaSType, we don't need to have a specific 'custom' type - instead, users can simply create their own type dynamically. There is also now no more invalid 'type' value for saas configs, so we remove the test for that functionality. * Remove unused import * update changelog Co-authored-by: Adam Sachs <[email protected]>
* Rate limiter implementation copied from fidesops repo * Change enum values to not be plural * Add test using rate limiter from multiple threads * Update function names and test parameters * Update error messages to include better details * Update changelog * Add an additional comment to test * fix typo in test name * Fix typo in test comment * Fix one more typo * Run black one more time Co-authored-by: Eduardo Armendariz <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #1159
gracefully.fail.and.log.error.mov
Code Changes
Steps to Confirm
Pre-Merge Checklist
CHANGELOG.mdDescription Of Changes
Update the config api call to log error to console instead of throwing runtime error.