ValueError: Cannot convert float NaN to integer about sdv

[npatki]

Hi @ThomasK1018, thanks for the detailed description. We were able to replicate your error. For future reference, I'm attaching the stack trace below.

stack_trace.txt

Root Cause

We will investigate more but in all likelihood, this is probably related to #1691. The known bug is that HMA sometimes creates null values even when there aren't any in the real data. Usually this happens for float64 or string types in the child table. In this particular case, you have int64 columns (that are categorical). The int64 storage type does not support null values, hence the crash.

As a next step, the team can investigate more to confirm that the root cause is, indeed, #1691. If so, we will dedupe the issue. We can also do a better job at preventing the crash and instead just returning the null value (even though this is not quite correct). I've filed RDT issue 747 to keep track of this.

Workarounds

Changing the default distribution for each table to 'norm' seems to solve this issue because it makes the error less likely. Unfortunately, this may impact the data quality (but at least it won't crash!)

synthesizer = HMASynthesizer(metadata)

for table_name in ['unique_id', 'ads_dataset', 'feeds_dataset']:
  synthesizer.set_table_parameters(
      table_name=table_name,
      table_parameters={
          'enforce_min_max_values': True,
          'default_distribution': 'norm'
      }
  )

synthesizer.fit(datasets)
synthesizer.sample()

Do note that the HMA is only meant for smaller datasets though. So even with this workaround in place, you'll still see the performance alert:

PerformanceAlert: Using the HMASynthesizer on this metadata schema is not recommended. To model this data, HMA will generate a large number of columns. (1189 columns)


      Table Name  # Columns in Metadata  Est # Columns
0      unique_id                      1           1129
1  feeds_dataset                     26             26
2    ads_dataset                     34             34

We recommend simplifying your metadata schema by dropping columns that are not necessary. If this is not possible, contact us at [email protected] for enterprise solutions.

Ultimately, I'd recommend using the HSASynthesizer as it's designed to handle larger datasets more robustly. I've confirmed that it works without issue on this dataset (fitting in a few seconds).

[ThomasK1018]

Hi Neha,

Thanks for your reply. That change on default distribution has certainly helped. May I also ask what are the available choices to be filled in for this section? I have also tried inputting truncnorm and the output is different from leaving in blank. So I would like to know if there are more choices for the distribution. Thanks.

Best Regards,
Thomas

[npatki]

Hi @ThomasK1018, no problem. For more info, I'd recommend checking our docs here. Note that the default_distributions sets the shape for all columns but you can override individual columns by using the numerical_distributions parameter too.

[npatki]

Hi @ThomasK1018, I'm closing off this issue as a duplicate since we do have now have more evidence that it is caused by #1691. We are actively looking into this root cause and hope to have fix up in an upcoming release.

Please feel free to reply directly to #1691 if there is anything more to discuss and we can always continue the conversation there.