Problems encountered in synthetic data

[PuJeff]

Environment details

• SDV version:1.26
• Python version: 3.12
• Operating System:Linux

Problem description

I want to synthesize time series data using the CPAR model provided by SDV. However, my real data has a large long-tail distribution, with a large proportion of 0 and sparse non-zero data. I can now learn the distribution of 0 very well, but the non-zero portion is always much smaller than the real data. How can I generate data similar to my real data?

What I already tried

My real data contains 198,110 values, of which 176,200 are 0s, and the rest range from 0.0001 to 35.9779. The range of my synthesized data is 0.01 to 4.61. How can I get the model to generate data within this range?

If possible, also add below the exact code that you are running.>

Paste the command(s) you ran and the output.
If there was a crash, please include the traceback here.

[PuJeff]

I looked up questions other users have asked My question is more like this: PAR seems to create a very narrow distribution around the mean

[npatki]

Hi @PuJeff, nice to meet you.

I tried this out with a fake dataset myself, consisting of ~90% values being 0 and the remaining values ranging up to 35, and I can see what you mean. In comparison to the real data, the synthetic data is a bit more smoothened out.

Root Cause

I've seen this problem before for both 0-heavy and long-tailed distributions. The root cause is that most of core generative AI models are designed to work under the assumption that your data is normally distributed.

This can explain what you're seeing. For example, the mean of your data may be ~1.5 (due to the data being so 0-heavy) and its standard deviation may be ~6-7 at best (again due to so many data points being 0). As a result, synthesizing a value as high as 35 will be rare, because this represents a data point that is roughly 5 standard deviations from the mean. It will be rare, but not impossible. So I think if you were to generate a lot more data points, eventually you will find that it covers the range. (I recognize that may not be the most useful solution though).

Solutions

The solution is to generally preprocess your data to get it into a state that the AI model is designed to handle. You could try some preprocessing to change the distribution shape to normal (the RDT library may help).

Though for your particular case, I think there could be a simpler solution. In your data, 0 is a very special, static value that you may want SDV to learn separately -- especially if it's supposed to indicate an absence of something (eg. a missing value). Converting all the 0s in your dataset explicitly to np.nan values will make it easier for the model to understand that something special is going on, and that the 0 values should not really be considered when learning the overall distribution shape.

import numpy as np

# make a copy of the data and replace all the 0 values with NaN
data_copy = data.copy()
data_copy['COLUMN_NAME'] = data_copy['COLUMN_NAME'].replace(0, np.nan)

# now run the PARSynthesizer
synthesizer = PARSynthesizer(metadata)
synthesizer.fit(data_copy)
synthetic_data = synthesizer.sample(num_sequences=100)

# at the end, make sure to turn the NaN values back to zero
synthetic_data['COLUMN_NAME'] = synthetic_data['COLUMN_NAME'].replace(np.nan, 0)

This gets me a much broader range -- from 0 to about 25.

Hope that helps!

[PuJeff]

I tried it and it did improve. Thank you very much for your answer.