chore(datasets): Refactor DatasetDAO update to leverage bulk logic for create, update, and delete operations

[john-bodley]

SUMMARY

Somewhat akin to #22413, this PR optimizes the PUT operation for the RESTful API /api/v1/dataset/{pk} endpoint.

Though updates are sparse in nature, i.e., per here where only the specified properties are updated, from a dataset perspective all the columns and metrics must be specified—even if one is only updating a subset—otherwise the undefined (unseen) columns and metrics will be deleted per here and here respectively.

For large datasets (like Airbnb's uber virtual dataset—comprising of thousands of columns and metrics) these updates are rather expensive (and thus slow) given that the create, update, and delete operations work—courtesy of a for loop—on a single entity (which could be sparse in nature) at a time. In our production environment this request times out as NGINX is configured with a 300 second (5 minute) timeout.

The proposed fix is to use bulk logic (courtesy of SQLAlchemy) for the create, update, and delete operations which significantly improves the performance, i.e., for the dataset in question (with sparsely defined columns/metrics containing only the minimum attributes, i.e., name, ID, etc.) the time for the PUT operation for the RESTful API /api/v1/dataset/{pk} endpoint reduced from ~ 590 seconds to ~ 45 seconds—a ~ 1,200% increase in performance.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

CI. Yay for existing unit/integration tests with good coverage. Makes these types of refactors a joy!

ADDITIONAL INFORMATION

• Has associated issue:
• Required feature flags:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[john-bodley]

metrics (plural) and not metric (singular).

[john-bodley]

Using the same synchronize_session as the bulk_delete method for deleting multiple datasets.

[codecov]

Codecov Report

Merging #22957 (0021148) into master (e13ebb6) will increase coverage by 0.82%.
The diff coverage is 0.00%.

❗ Current head 0021148 differs from pull request most recent head a1d4d74. Consider uploading reports for the commit a1d4d74 to get more accurate results

@@            Coverage Diff             @@
##           master   #22957      +/-   ##
==========================================
+ Coverage   65.81%   66.64%   +0.82%     
==========================================
  Files        1876     1876              
  Lines       72084    72072      -12     
  Branches     7857     7857              
==========================================
+ Hits        47444    48031     +587     
+ Misses      22626    22027     -599     
  Partials     2014     2014              

Flag	Coverage Δ
hive	52.75% <0.00%> (+0.01%)	⬆️
mysql	?
postgres	?
presto	52.64% <0.00%> (+0.01%)	⬆️
python	80.52% <0.00%> (+1.71%)	⬆️
sqlite	76.66% <0.00%> (+0.02%)	⬆️
unit	52.44% <0.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
superset/datasets/dao.py	48.14% <0.00%> (-46.41%)	⬇️
superset/datasets/commands/create.py	30.61% <0.00%> (-69.39%)	⬇️
superset/datasets/commands/update.py	25.00% <0.00%> (-69.05%)	⬇️
superset/datasets/commands/bulk_delete.py	33.33% <0.00%> (-53.34%)	⬇️
superset/datasets/columns/commands/delete.py	44.11% <0.00%> (-52.95%)	⬇️
superset/datasets/metrics/commands/delete.py	44.11% <0.00%> (-52.95%)	⬇️
superset/sql_validators/postgres.py	50.00% <0.00%> (-50.00%)	⬇️
superset/datasets/commands/delete.py	48.57% <0.00%> (-48.58%)	⬇️
superset/datasets/commands/refresh.py	43.75% <0.00%> (-43.75%)	⬇️
superset/datasets/commands/duplicate.py	53.24% <0.00%> (-37.67%)	⬇️
... and 106 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more