HyperTransformer: Memory usage increase when reverse_transform is called

[pvk-developer]

Environment Details

• RDT version: 0.4.0
• Python version: Python 3.8
• Operating System: PopOS! 20.04

Error Description

When trying to reverse_transform categorical data using rdt.transformers.OneHotEncoder having more than 500 categories the memory usage is being increased drastically.

Steps to reproduce

In [1]: import pandas as pd
In [2]: data = pd.DataFrame({
   ...:     'categorical': [str(x) for x in range(600)] * 10
   ...: })

In [3]: from rdt.hyper_transformer import HyperTransformer 
In [4]: from rdt.transformers import OneHotEncodingTransformer
In [5]: ht = HyperTransformer(transformers={'categorical': OneHotEncodingTransformer()})
In [6]: ht.fit(data)
In [7]: transform = ht.transform(data)
In [8]: reverse = ht.reverse_transform(transform)

On the image can be observed the memory consumption which increases when reverse_transform is run.

This issue was originally raised on sdv-dev/SDV#304

Cause

The main cause of this memory increase is that using pandas.DataFrame.pop duplicates the data frame view and increases the memory used by the application instead of reducing it. When we try to pop more than 500 columns we end up creating 500+ views of the data frame which ends up increasing the memory drastically.