Sequence index values should be strictly increasing in the synthetic data

[silvaac]

Environment Details

Please indicate the following details about the environment in which you found the bug:

• SDV version: 0.10.0
• Python version: 3.8.10
• Operating System: windows

Error Description

Synthetic time-series dates are out of sequence, repeated and/or with large gaps between dates as well. I think that one wants to keep the same time step or at least the time sequence when generating Dates of synthetic time-series.

Steps to reproduce

In fact the basic example in the web-page shows it. This is a scene shot:

This is an alternative code that presents the issues:

import pandas as pd
import numpy as np
from pandas_datareader import data
from sdv.timeseries import PAR
if __name__ == '__main__':

    P  = data.DataReader('SPY',data_source='yahoo',start='1990-01-01')
    x0 = P.assign(logP = np.log(P["Adj Close"]))
    x  = x0.assign(r = x0.logP.diff())
    x = x.dropna()
    R = x["r"]
    # Simulator
    sequence_index = 'Date'
    model = PAR(
        sequence_index=sequence_index,
        epochs=3000,
        sample_size=10,
        verbose=True,
    )
    model.fit(R.reset_index())
    sample = model.sample(num_sequences=1)

Ability to contribute

<If you are available to contribute the necessary changes to the project to solve this issue please indicate so here, so the maintainers can assign the issue to you and provide the necessary support. Otherwise, please remove this section.>

[npatki]

Thanks for filing this issue @silvaac.

There is a related feature request in #678 about ensuring that sequence index rows are all unique. I will turn this one into a complementary feature request for ensuring the values in sequence index are strictly increasing.

You touched upon another issue, which is that there are sometimes large gaps between sequence indices. It would be great if you could file a new feature request for this, as I have some follow-up questions about what is in the input data vs. expected as output.

[npatki]

Update on this: We have identified a root cause, which is now linked in #1760. You can follow that issue for an update.

Please note that this may not be the only root cause so until we resolve #1760, we will keep this one open.

[npatki]

Hi @silvaac,

Good news -- this issue should be fixed in the latest SDV release (1.10.0). Please upgrade for the latest results.

On the NASDAQ demo dataset, I've confirmed that the values are strictly increasing within a sequence.

Do note that if your real data contains decreasing values, then your synthetic data would as well -- as the synthesizer learns what is possible based on your real data. Please feel fee to reach out if you have any questions. Thanks.

P.S. Since the original issue was filed, we've significantly updated the SDV API as well as the docs. Here is the new reference for PARSynthesizer.