You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This issue is for the in-progress metadata-improvment branch, which aims to improve the accuracy of metadata detection from datasets.
Problem
The new metadata detection logic is described in #1515. This is implemented according to spec but we've found that it's not intuitive.
When the values in a column are unique, the script is very liberal in calling it an 'id' column. In reality, many PII columns will also have unique values (eg. SSNs, phone numbers, email addresses, etc.).
Expected Behavior
To not automatically assume that all unique values are sdtype 'id'. The end result should be the following:
Primary keys should be sdtype 'id'
Foreign keys should be sdtype 'id'
Other columns with unique values should be sdtype 'unkown' (so they are treated as PII)
Details
This may be accomplished by a few changes to the logic in #1515:
3 (iv): Elif the values are all unique, sdtype is id.unknown. Add it to a list of potential primary keys.
5 (iv): Elif the values are all unique, sdtype is idunknown. Add it to a list of potential primary keys.
Then when detecting relationships:
Primary key: We can continue to choose the leftmost column of the potential keys we've identified. Only that column should have sdtype updated to 'id'. (See Detect primary keys in metadata #1521 for initial specs)
npatki
changed the title
[Metadata Auto-Detection]
[Metadata Detection] Only make primary/foreign keys sdtype id (leave others as unknown)
Sep 20, 2023
This issue is for the in-progress
metadata-improvment
branch, which aims to improve the accuracy of metadata detection from datasets.Problem
The new metadata detection logic is described in #1515. This is implemented according to spec but we've found that it's not intuitive.
When the values in a column are unique, the script is very liberal in calling it an
'id'
column. In reality, many PII columns will also have unique values (eg. SSNs, phone numbers, email addresses, etc.).Expected Behavior
To not automatically assume that all unique values are sdtype
'id'
. The end result should be the following:'id'
'id'
'unkown'
(so they are treated as PII)Details
This may be accomplished by a few changes to the logic in #1515:
id.unknown. Add it to a list of potential primary keys.idunknown. Add it to a list of potential primary keys.Then when detecting relationships:
'id'
. (See Detect primary keys in metadata #1521 for initial specs)The text was updated successfully, but these errors were encountered: