Check on missing_percent assumes 0.999984 == 1 about soda-core

[tools-soda]

SAS-3526

[benjamin-pirotte]

Hi,

Looking at your check, I am wondering if it is the expected behavior
missing_percent(my_column_name) < 100%, this means you are expecting all values to be null. In this case, it is not so it fails.

Aren't you looking for the following check: missing_percent(my_column_name) < 0% ?

[lewis-anderson53]

missing_percent(my_column_name) < 0% would ensure that I had 0 missing data and that single row was not NULL.
That's not my intention with this check, I want to make sure that at least some rows are non-NULL.

I lowered my threshold and looked at another column, to demonstrate. Here I will say that less than 65% NULL is acceptable, any higher and that's bad.

checks for my_table_name [daily]:
  - missing_percent(my_other_column_name) < 65%:
      name: Feature column my_other_column_name contains values that are not NULL

Which shows the following results:

AssertionError: Check results failed: 
[missing_percent(my_other_column_name) < 65%] FAIL (check_value: 73.92, row_count: 58335240, missing_count: 43121299)

My check_value here is rounded to 2 decimal places, further demonstrating potentially what my problem is.