open-aims / adria-synthetic-data Goto Github PK

Current ReadMe refers to example files, but snippets of these should be added to the ReadMe file for clarity.

• Currently dhw, connectivity and site data is created separately.
• ADRIA requires input data in a particular datapackage format
• create function which automatically organises input data files in this structure, given the locations of created synthetic data files as input

Currently using first nearest neighbour for spatial interpolation of connectivity and dhw data, but could use nearest k median for better interpolation.

• synthetic data models require data in a certain form.
• currently transformations are done in data fitting files, but should be separated into their own functions for re-use

Compare current PAR model for environmental data with timeseries surrogate model. Note any time improvements to consider changing methods.

As site data was originally as a csv, a conversion function to csv was added as a temporary fix. This should be changed so that gpkg files are instead used by each of the models.

Currently the datapackage.json file which accompanies each data package is made by altering a copy of the original data package. On discussion it was agreed it is better to create one from scratch to avoid back tracking.

Currently synthetic site data positions are translated in a randomised distance within a 2000km by 2000km box. Increase this size for better anonymity.

Currently, the GAN model for connectivity has a reasonably long run time and is prone to mode collapse. Compare this with simple probabilistic model by fitting Gaussians over several layers of connectivity data, run for multiple iterations and take median of outcomes.

• currently models have been created for synthetic site data, connectivity and dhw data.
• wave data is also a necessary input layer for the ADRIA model
• first try out temporal model from SDV package, then try out models from Deep Echo package if no success.

Currently, synthetic site polygons (which are created as circles around site centroids satisfying the synthetically generated areas) may overlap. This should be fixed to avoid double counting when it comes to coral cover and site areas.

Currently functions which save data for plotting comparisons (original, sampled and real as csvs) save the non-anonymised lats and longs. Plotting should be done with anonymised lats and longs for the synthetic data.

• currently no function wrappers for visualisations comparing synthetic and original datasets
• create functions which alllow easy comparisons between synthetic and original data, both for tabular, spatial and temporal datasets.

README should have quick set up and examples added.

• site data in ADRIA is now in a geopackage format
• synthetic site data cannot be created with geometries easily (synthetic data packages cannot easily create the geometry column).
• resolve this maybe with randomised placeholder geometry

Currently the environments used for the SDV models and GAN connectivity models have insoluble conflicts (NetCDF4 conflicts with geopandas). To resolve this, see if later or earlier netCDF versions do not cause conflict.