data-duper is a tool to replicate the structure of private or protected data for testing.
When testing the data handling of software, it is best to use data as similar to the real data as possible - without revealing sensitive information to the test environment. This is where data-duper comes into play. It allows you to create an authentic replicate of your private or protected data.
The source code is currently hosted on GitHub at: https://github.com/kjanker/data-duper.
Binary installers for the latest released version are available at the Python Package Index (PyPI).
data-duper works like a learning model. You train the duper on your real data and, afterwards, generate a new data set of arbitrary size. The new data set - or dupe - has the same structure as the real data, i.e., columns, dtypes, as well as string composition and distribution of numerical values. Occurrences of NA values are ignored by default but can optionally be included as well.
- numerical values (float, int, datetime) are drawn from an interpolated empirical distribution
- identifier strings of fixed length and structure are replicated with regular expressions
- features with only few values (category, bool) are redrawn according to their occurrence
- value distributions are replicated as draw probability. Thus, for small dupe sets the realized distribution may differ slightly
- correlations between columns are not replicated (this ensures real data is better obscured)
- descriptive strings like notes, names, etc are not obscured but reshuffled
You simply initialize a new Duper instance, fit it on your real data df_real, and make a data dupe df_dupe of desired size n.
from duper import Duper
duper = Duper()
duper.fit(df=df_real)
df_dupe = duper.make(size=10000)- capture index-like data, e.g., counting integers, time series
- handle numerical data with pre- or subseeding characters, e.g., '$100', '100€', '50.25 mm'
- include optional correlations between selected rows
- improve algorithm of regex duper
The official documentation is hosted on readthedocs.org: https://data-duper.readthedocs.io/en/latest/?badge=latest
Don't hesitate to contact me if you like the idea and want to get in touch.