Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inequality fails with None and datetime #1471

Closed
amontanez24 opened this issue Jun 21, 2023 · 1 comment · Fixed by #1481
Closed

Inequality fails with None and datetime #1471

amontanez24 opened this issue Jun 21, 2023 · 1 comment · Fixed by #1481
Assignees
Labels
bug Something isn't working
Milestone

Comments

@amontanez24
Copy link
Contributor

Environment Details

Please indicate the following details about the environment in which you found the bug:

  • SDV version: 1.2
  • Python version: Any
  • Operating System: Any

Error Description

Inequality doesn't work with datetime that has None values

Note that this works if (a) I don't add the constraint or (b) I keep the constraint and I convert the None to pd.NaT

InvalidDataError: The provided data does not match the metadata:
'NoneType' object has no attribute 'to_datetime64'

Steps to reproduce

import pandas as pd
from sdv.metadata import SingleTableMetadata
from sdv.single_table import GaussianCopulaSynthesizer

data = pd.DataFrame(data={
    'A': [None, None, '2020-01-02', '2020-03-04']*2,
    'B': [None, '2021-03-04', '2021-12-31', None]*2
})

metadata = SingleTableMetadata.load_from_dict({
    'columns': {
        'A': { 'sdtype': 'datetime', 'datetime_format': '%Y-%m-%d' },
        'B': { 'sdtype': 'datetime', 'datetime_format': '%Y-%m-%d' }
    }
})

metadata.validate()
synth = GaussianCopulaSynthesizer(metadata)
synth.validate(data)

synth.add_constraints([{
    'constraint_class': 'Inequality',
    'constraint_parameters': {
        'high_column_name': 'A', 
        'low_column_name': 'B'
    }
}])

synth.fit(data)

InvalidDataError: The provided data does not match the metadata:
'NoneType' object has no attribute 'to_datetime64'
@amontanez24 amontanez24 added the bug Something isn't working label Jun 21, 2023
@amontanez24
Copy link
Contributor Author

This can probably be resolved by updating this function in the same way as SDV-Enterprise

def cast_to_datetime64(value):
"""Cast a given value to a ``numpy.datetime64`` format.
Args:
value (pandas.Series, np.ndarray, list, or str):
Input data to convert to ``numpy.datetime64``.
Return:
``numpy.datetime64`` value or values.
"""
if isinstance(value, str):
value = pd.to_datetime(value).to_datetime64()
elif isinstance(value, pd.Series):
value.apply(lambda x: pd.to_datetime(x).to_datetime64())
value = value.astype('datetime64[ns]')
elif isinstance(value, (np.ndarray, list)):
value = np.array([
pd.to_datetime(item).to_datetime64()
for item in value
])
return value

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants