Imbalanced data and conditional sampling about sdv

[npatki]

Hi @gnomesanta, when you use conditional sampling with a neural network-based synthesizer such as TVAE, it will require many attempts to match the conditions. It appears that you are running out of attempts because a Regex (in your metadata) is limiting the possibilities.

But to confirm this, would you be able to share your final metadata? You can do this by printing the metadata or by saving it as a JSON file and attaching it here (recommended approach).

metadata.save_to_json(filepath='my_metadata.json')

Once we confirm this, I may be able to suggest a few workarounds.

[npatki]

Hi @gnomesanta, are you still running into this issue?

I'm not sure what your metadata looks like so I don't know if this will help: If you have a Regex that is limiting the # of possibilities, you may want to update your metadata to provide a larger Regex.

For eg. the following regex will allow for 1,000 possibilities (id_000 to id_999): id_[0-9]{3}. To allow for more possibilities you can either update the allowed characters (id_[0-9a-z]) or increase the number of digits (id_[0-9]{10}). Do note that we plan to update the defaults in the future so the synthesizer will not crash if it runs out of Regexes (see RDT issue #749).

[npatki]

Hi @gnomesanta, are you still working on this project? I'm closing off this issue since it has been inactive for some time and we've discussed one possible cause and suggestion for conditional sampling.

If there is more to discuss, please feel free to reply to this issue and we can always reopen for further investigation. Thanks.