[Metadata detection] Add a cardinality cap when choosing between categorical vs. numerical

[npatki]

This issue is for the in-progress metadata-improvment branch, which aims to improve the accuracy of metadata detection from datasets.

Background

Sometimes a column may be represented using int/float values but it is actually encoding distinct categories. For example, an HTTP error code such as 404 or 200. To distinguish between numerical vs. categorical, the metadata detection looks at the cardinality, or the # of unique values in a column.

Current logic: If cardinality < (length of data)/10, then mark it as "categorical"

Problem

The current logic is giving too much tolerance for cardinality for large datasets. Eg. with 500K rows, we allow a cardinality of up to 50K.

Expected behavior

We propose capping the cardinality at 10. Which means using the following logic:

# pseudocode
if cardinality(column) < min(len(column)/10, 10):
    sdtype = "categorical"