Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

💣 "Delete means delete" when deleting a table #2500

Closed
MatMoore opened this issue Nov 24, 2023 · 0 comments
Closed

💣 "Delete means delete" when deleting a table #2500

MatMoore opened this issue Nov 24, 2023 · 0 comments

Comments

@MatMoore
Copy link
Contributor

MatMoore commented Nov 24, 2023 •
edited by tom-webber
Loading

User Story

As a data owner
I want a way to completely remove data from all versions of a data product
So that I can correct mistakes

Value / Purpose

We decided to leave this out of the original delete table implementation (#2496) due to confusion about the use case.

Some questions to ask when refining this:

  1. Should the user be free to delete their data at any point? Or should this depend on the status of the data product, i.e. I can perform this action while my data product is in a draft state, but not when it is "live" and being consumed from
  2. Should this create a new version of the data product? (If we are modifying all previous versions, then the new version seems redundant: we will have two subsequent versions with the exact same metadata and data)
@MatMoore MatMoore added enhancement enhancing an existing feature data-platform-labs labels Nov 24, 2023
@MatMoore MatMoore mentioned this issue Nov 24, 2023
Merged
@github-actions github-actions bot mentioned this issue Dec 1, 2023
Closed
jacobwoffenden pushed a commit that referenced this issue Dec 14, 2023
@MatMoore @murdo-moj
Revise delete table endpoint to take into account database versioning
0a776e9 
* Allow specifying db_meta when creating a db

This allows us to create databases based on other databases.

* Expand glue utilities

Ensure we have create/get/list/delete operations for tables and
databases.

Additionally, add a clone_database operation for us during major version
updates. In this scenario we need to create a copy of a database with
a new version number.

* Add method to copy a database between versions

* Do not delete table from existing version

* When deleting a table, only delete data from the new version

For now, we will just delete the table & data from the new version. This
deviates from our original plan of deleting from all historical versions
"delete means delete", but we had some doubts about implementing
contradictory behaviour (i.e. creating a new major version while at the
same time making breaking changes to existing versions doesn't make
sense).

This behaviour will be revisited in #2500

* Bump version

* Update delete table to work with the new base image

---------

Co-authored-by: Murdo <[email protected]>
@seanprivett seanprivett removed enhancement enhancing an existing feature data-platform-labs labels Dec 21, 2023
@tom-webber tom-webber moved this to Todo in Data Catalogue Apr 11, 2024
@seanprivett seanprivett closed this as not planned Won't fix, can't repro, duplicate, stale Apr 12, 2024
@github-project-automation github-project-automation bot moved this from Todo to Done in Data Catalogue Apr 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
Data Catalogue
Status: Done ✅
Milestone
No milestone
Development

No branches or pull requests
3 participants
@MatMoore @jacobwoffenden @seanprivett