Transformer should not be fit on test data

[dionman]

Environment Details

• SDMetrics version: v0.8.1
• Python version: 3.9
• Operating System: Ubuntu

Error Description

At current implementation of MLEfficacyMetric base class, the transformer is fit on the test data

SDMetrics/sdmetrics/single_table/efficacy/base.py

Line 50 in 6983bc0

test_data = ht.fit_transform(test_data)

This undesirably leaks information from the (unseen) test dataset, biasing the reported metrics. Instead fit_transform should be applied on the train data and transform on the test data.

[npatki]

Thanks for filing this @dionman. I'll check back in with the team in case there's any reason why it was implemented the way it is. Will keep this issue open to report any progress or updates.

[dionman]

thanks - are there any updates on this?

[npatki]

Hi @dionman, this bug has been added to our queue. We'll use this issue to provide any updates.