TabDDPM generating NaNs for large datasets about synthcity

[robsdavis]

Hi @HLasse,

I have not seen this issue before. Have you tried experimenting with batch size to see if that get round the issue?

One other thing to check would be numerical instability. There are a few divisions and logarithms in the code here do any the denominators/log arguments of them approach zero for your dataset?

[muellermarkus]

I actually encountered the same problem on larger datasets. For me, it is not just that NaNs are sampled but the training loss also becomes NaN after a couple of iterations.

I agree with @robsdavis that this is related to numerical instability. In particular, I traced the error (in my case) down to

synthcity/src/synthcity/plugins/core/models/tabular_ddpm/utils.py

Lines 151 to 154 in 943fa28

	@torch.jit.script
	def log_sub_exp(a: Tensor, b: Tensor) -> Tensor:
	m = torch.maximum(a, b)
	return torch.log(torch.exp(a - m) - torch.exp(b - m)) + m

I managed to stabilize the function and can create a related pull request if you want. However, I cannot guarantee that this gives the same results on the datasets for which the non-adjusted variant works without issues. I only tried this for a couple of datasets and the results were close enough.