Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Metadata Detection] Only make primary/foreign keys sdtype id (leave others as unknown) #1598

Closed
npatki opened this issue Sep 20, 2023 · 0 comments · Fixed by #1601 or #1610
Closed
Assignees
Labels
feature request Request for a new feature
Milestone

Comments

@npatki
Copy link
Contributor

npatki commented Sep 20, 2023

This issue is for the in-progress metadata-improvment branch, which aims to improve the accuracy of metadata detection from datasets.

Problem

The new metadata detection logic is described in #1515. This is implemented according to spec but we've found that it's not intuitive.

When the values in a column are unique, the script is very liberal in calling it an 'id' column. In reality, many PII columns will also have unique values (eg. SSNs, phone numbers, email addresses, etc.).

Expected Behavior

To not automatically assume that all unique values are sdtype 'id'. The end result should be the following:

  • Primary keys should be sdtype 'id'
  • Foreign keys should be sdtype 'id'
  • Other columns with unique values should be sdtype 'unkown' (so they are treated as PII)

Details

This may be accomplished by a few changes to the logic in #1515:

  • 3 (iv): Elif the values are all unique, sdtype is id. unknown. Add it to a list of potential primary keys.
  • 5 (iv): Elif the values are all unique, sdtype is id unknown. Add it to a list of potential primary keys.

Then when detecting relationships:

@npatki npatki added the feature request Request for a new feature label Sep 20, 2023
@npatki npatki changed the title [Metadata Auto-Detection] [Metadata Detection] Only make primary/foreign keys sdtype id (leave others as unknown) Sep 20, 2023
@amontanez24 amontanez24 self-assigned this Oct 11, 2023
@amontanez24 amontanez24 added this to the 1.5.0 milestone Oct 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request Request for a new feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants