nz-social-investment-agency / social_investment_data_foundation Goto Github PK

There is no rollup parameter available in the si_control.sas file for rolling up CYF related tables. This needs to be added.

si_create_rollup_vars.sas : parameter si_aggs_cols needs to = department. This doesnt work if there is no roll up chosen. i.e if we want to just roll up by snz_uid

for example if we choose si_aggs_cols=%str(snz_uid) , this would fail.

Might need to re code this?

The SI data foundation takes too long to roll up events that have high dates, because it essentially breaks up events into periods specified by period duration. This poses a problem for cases where the end date is 31-12-9999. Need to default to current date or date of IDI refresh in such cases.

Setting si_rollup_duration equal to false does not stop the creation of duration columns in the output.

Around 20 files in ./sasautos/ and other folders of this repo have the SIA's IDI_Sandpit schema (DL-MAA2016-15) hard coded into it and it will need to be changed to the schema available to other projects that obviously can't write to that schema. In the long term a solution would be to parameterise this, in the short term perhaps just a note in the readme alerting people to the need to do these changes.

Initially reported by users from PwC.

Instead of having equally sized periods for all roll-up variables, it would be better to have variable-sized periods closer to the "as-at" date. For example, having variables 3 months before & after the as-at date, 6 months before & after, 12 months before & after, etc.

The data foundation needs a library declaration file to be run to define the sand library (and other IDI libraries) that are used in the code. This has to be either added as a dependency, or explicitly invoked from the si_main.sas script.

The macro call to generate highest qualification does not have the same parameters as the recently updated highest qualification macro

https://github.com/nz-social-investment-agency/social_investment_data_foundation/blob/master/sasautos/si_get_outcomes_ext.sas

vs

https://github.com/nz-social-investment-agency/social_investment_data_foundation/blob/master/sasautos/si_get_highest_qualifications.sas

Code is mismatching the keys and conesequently returning values that belong to another record

The SASAUTOS macro called si_subset_idi_dataset macro has used hard-coded values for the schema name. It uses DL-MAA2016-15 as the hard-coded value, which tends to succeed when users within the SIA project execute the code for creating qualifications data. However, this code will fail when anybody external to that project uses it.