The dfm_tools from deltares

Decide on curvigrid interpolation method in dfmt.interp_regularnc_to_plipoints()

For CMCC, HIRLAM, Delft3D4, WAQUA and other models, there are 2D lat/lon variables in the dataset (curvilinear grids, sometimes polar). Decide on interpolation method used in dfmt.interp_regularnc_to_plipoints(). ds.interp() does not work with 2D lat/lon vars (pydata/xarray#2281), so currently using KDTree for CMCC.

conversion to xugrid UgridDataset is also possible: #396
this also works for Delft3D4/WAQUA datasets, but the function should be made less hard-coded as described in #798 (comment)
applied for plotting in example script
conversion method still requires improvements
in case of CMCC, uo and vo variables have different grid, so combining them is inconvenient.
replace nearest/triangular interpolation with uds.ugrid.sel_points()
how is performance when concatenating many datasets in time?
investigate potential use of xcdat, useful for spherical grids?

Improve map-polyline-slice dataset structure and cleanliness

Very maybe uds.ugrid.sel(x/y) could help out here (Deltares/xugrid#26). Or for instance implement uds.ugrid.isel(faces=idx_faces), where also edges/nodes are subsetted (Deltares/xugrid#32) >> fixed

If it would be possible subset an entire dataset (dims: faces, edges, nodes) based on faceidx, it would also be easier to remove ghostcells as in dfmt.open_partitioned_dataset() (#207)

Discuss role of z-values in mapfile

variables mesh2d_flowelem_zcc/mesh2d_flowelem_zw (=fullgrid output) are set as coords in dataset (geen data_vars)
mesh2d_layer_sigma/mesh2d_interface_sigma zijn geen coords maar data_vars
mesh2d_layer_z/mesh2d_interface_z zijn geen coords maar data_vars

This should probably be aligned, all z-variables as coords or all as data_vars.

First check if this is also different in models that are run with new dflowfm kernel. If so, report as UNST issue.

Deprecate and eventually remove old code

Tasks:

raise DeprecationWarning for all functions in dfm_tools.io, with new hydrolib-core+dfm_tools method in warning message.
Raise deprecation warnings for other code
Eventually remove old code

Functions that currently raise a DeprecationWarning:

get_ugrid_verts
write_bcfile/read_bcfile
write_timfile/read_timfile, was deprecated after Deltares/HYDROLIB-core#348 and #301 were solved
Polygon and Polygon.fromfile
scatter_to_regulargrid
get_varnamefromattrs() (including unittest): #310

Functions that currently print a DeprecationWarning:

corner2center/corner2center

Add rename for fourier varnames based on attributes

would consist of:

set up translation dictionary with help of fourier_analysis.f90.
get quantity and maybe analysistype from long_name (e.g. "temperature, average value"). Issue for m
maybe get analysistype from varname (e.g. "mesh2d_fourier001_mean") instead (standard_name is not always present), this prevents unclarities with underscores/spaces. However, is there always an underscore present there?
get tstart/tstop datestrings from numstart+numstop+reftime. However, numstart/numstop are defined twice and would have to be merged: starttime_fourier_analysis_in_minutes_since_reference_date and starttime_min_max_analysis_in_minutes_since_reference_date
rename variable to something like f"{quantity}_{analysistype}_{tstart}_{tstop}" names would then probably make most sense. However, some quantities have spaces/underscores in them, so it might be better to have shorter names like ux/uy/wl etc.
possible to include tidal constituent translations? Only numcycles/numstart/numstop is available there, so computed freq would have to be matched to closest value in some online list? (see below) This is source for errors, so warn user. Also, the columns knfac+v0 from the inputfile are not available in the output, also add warning.
select all tidal variables by filtering vars that have numcyc attr: xarray.Dataset.filter_by_attrs
in case of a 3D model with sigma/z-sigma layers, which z-values are written to the file?

Getting Foreman tidal frequencies:

import pandas as pd
file_freqs = 'https://raw.githubusercontent.com/Deltares/hatyan/main/hatyan/data/data_foreman_frequencies.txt'
freqs = pd.read_csv(file_freqs,names=['freq','dependents'],delim_whitespace=True,comment='#')
print(freqs.loc['M2','freq'])

Used example files:

p:\archivedprojects\11203379-005-mwra-updated-bem\03_model\02_final\A72_ntsu0_kzlb2\DFM_OUTPUT_MB_02_fou\MB_02_0000_fou.nc
p:\1230882-emodnet_hrsm\GTSMv3.0EMODnet\EMOD_MichaelTUM_yearcomponents\GTSMv4.1_yeartide_2014_2.20.06\output\gtsm_model_0000_fou.nc

kml export of grids and other features

Add kml export support, for instance like:

Improve dfmt.open_partitioned_dataset()

Improve dfmt.open_partitioned_dataset():

support for edge selection (maybe xugrid: Deltares/xugrid#32)
support for arbitrary grid names (dependent on Deltares/xugrid#25), that makes it possible to remove some parts of the code.
support for multiple grids (1D2D3D model)
make renaming more efficient
TODO in function
documentation

This might fix the issue of reprojecting a uds as in the header of https://github.com/Deltares/dfm_tools/blob/main/tests/examples_workinprogress/workinprogress_grid_convert_coordinates.py

This should fix improper edges plotting in https://github.com/Deltares/dfm_tools/blob/main/tests/examples_workinprogress/workinprogress_plot_edges.py

This might fix #203

Profile xr.open_dataset() for large mapfiles and hisfiles

Opening large mapfiles takes quite some time, this might for instance be because of the decoding of time etc. This could also be done only once, after merging of the mapfile. However, the stuff that takes time is cached so the second opening is more than 10 times faster. Beware the performance for the second opening does not get less.

Some timings:
- DCSM 3D 20 partitions 367 timesteps: 219.0 sec
- RMM 2D 8 partitions 421 timesteps: 60.6 sec
- GTSM 2D 8 partitions 746 timesteps: 73.8 sec
- RMM 3D 40 partitions 146 timesteps: 166.0 sec
- MWRA 3D 20 partitions 2551 timesteps: 826.2 sec

Fixes related to meshkernelpy issues

Next release (4.2.0 or 5.0.0):

Test GTSM-specific new features:

Deltares/MeshKernelPy#97 >> https://github.com/Deltares/MeshKernelPy/blob/feature/GRIDEDIT-768_global_grid_example/docs/examples/10_mesh2d_global_grid.ipynb >> issue for xugrid that straigt line is on the right instead of left?
test ridge refinement (implemented in GRIDEDIT-502)
the mesh2d_refine_based_on_gridded_samples() API now supports multiple dtypes (Deltares/MeshKernelPy#146). Check if gtsm refinement with meshkernelpy is possible with coarsefac 4 >> short/float distinction >> test gtsm memory consumtion for two dtypes (with release)

Deltares/MeshKernelPy#31
Deltares/MeshKernelPy#91
Deltares/MeshKernelPy#72 >> Deltares/xugrid#148
Deltares/MeshKernelPy#74
refining/z-interpolation with gridded samples with nans in it (also UNST-5600)
Deltares/MeshKernelPy#35 (currently a workaround exists in dfm_tools, also auto-closing all polygons)

dfm_tools/dfm_tools/meshkernel_helpers.py

Line 19 in 14fc1ae

def meshkernel_delete_withpol(mk, file_ldb, minpoints=None):
Deltares/MeshKernelPy#70

Related issues:

#217
meshkernel(py) issues in JIRA
possible to check orthogonality in mk? More grid validators/cleanups (modelbuilder example script)
possible to use connect_cells separately (without refining)?
polygon generation/editing, maybe already possible?
check TODO in meshkernel example script and modelbuilder example script
minimal interacter network is 1D: p:\dflowfm\maintenance\JIRA\06000-06999\06548\meshkernel_interp.py, can be used to interpolate bathy to?
do we need sphericalaccurate grids for dcsm/gtsm? These are not implemented yet (only spherical)
overview of RGFGRID functions

Workflows:

GTSM: global base grid, bathy+gradient refinement, cut landpart with ldb (incl drypoints), example script with some issues
1D2D connected like GTSM+rivers
DCSM: bathy+polygonen refinement, cut landpart with ldb+depth (drypoints with matlab, details at JG?)
RMM: multiblock rivers, curvigrid generation based on splines, triangles in estuaries, squares in seapart. For the second, we need a to divide a polygon in equal length parts, part of meshkernel?
D-HYDRO course materials workflow
tutorial materials from manual
RGFGRID functions overview, manual edits like in edit>reg/irreg
create unittests of all workflows to make sure behaviour is tested and future changes do not result in undesired behaviour

DCSM steps:

Release 3.0.0:

mk 4.1.0 (released 2024-02-15):

Remove test_import_libraries()

Shapely minimal version is tested (shapely>=1.7.0), put this in requirements instead

Update dependencies after hydrolib-core and meshkernel release

When hydrolib-core 0.5.0 and meshkernel 2.0.3 is available, update deps to:

hydrolib-core>=0.5.0 (contains TimModel, pytest and generate-documentation fail until then)
meshkernel>=2.0.3 (working is_geometric etc)
check TODO in example scripts

Test with new env from yml.

Recompute scaling/offset for dtype(int) mfdataset ds

Recompute like suggested in ArcticSnow/TopoPyScale#60 (comment)

Implementation: https://github.com/ArcticSnow/TopoPyScale/blob/494f4e7ea17830ba3d23627bf22ee200a6c4f082/TopoPyScale/topo_export.py#L21

cleanup deprecated code (`polyline_mapslice` and `polygon_intersect`)

Release numpy dependency restriction when possible

Currently installing an older numpy version to avoid "SystemError: initialization of _internal failed without raising an exception"

This happens in the generate-documentation actions 57 to 61 and in binder, but in general upon creation of dfm_tools_env from environment.yml.

This post suggests to restrict the numpy version and using "numpy<1.24" indeed resolves the issue.
https://stackoverflow.com/questions/74947992/how-to-remove-the-error-systemerror-initialization-of-internal-failed-without

Release this restriction when the dependency conflict is resolved by numba: numba/numba#8464

drop Python 3.8 support

xarray dropped py3.8 support in version 2023.02.0 (07-02-2023)
scipy dropped py3.8 support in version 1.11.0 (25-06-2023)
numpy dropped py3.8 support in version 1.25.0 (17-06-2023)
pandas dropped py3.8 support in version 2.1.0 (30-08-2023)
dask dropped py3.8 support in version 2023.5.1 (26-05-2023)
matplotlib dropped py3.8 support in version 3.8 (15-09-2023)
xugrid dropped py3.8 support in version 0.7.1 (17-11-2023)
copernicus-marine-client also does not support py38
python 3.8 is EOL in October 2024: https://devguide.python.org/versions/
py39 suggestion in installation/contributing guide was introduced in June 2023, wait at least a few months before dropping py38 support. Also check py38 percentage at https://pypistats.org/packages/dfm-tools
py3.11 suggestion in installation/contributing since 30 Sept 2023, just to be safe for the near future. Py 3.11 is faster than py 3.10

Also:

update supported python versions in pyproject.toml (classifiers and requires-python)
remove pandas version restriction for py38 from pyproject.toml (and simplify xarray/dask requirement)
remove pytest-py38.yml workflow and badge in readme
#576

xarray writing mfdataset results in incorrect data when not using manual encoding

xarray.to_netcdf() of opened mfdataset results in incorrect data when not using manual encoding

import os
import xarray as xr
import matplotlib.pyplot as plt
plt.close('all')

#open data
dir_data = r'p:\11207892-pez-metoceanmc\3D-DCSM-FM\workflow_manual\01_scripts\04_meteo\era5_temp'
file_nc = os.path.join(dir_data,'era5_mslp_*.nc')
data_xr = xr.open_mfdataset(file_nc)

#optional encoding
#data_xr.msl.encoding['dtype'] = 'float32' #TODO: updating dtype in encoding solves the issue. Source data is int, opened data is float, but encoding is still int.
#data_xr.msl.encoding['_FillValue'] = float(data_xr.msl.encoding['_FillValue'])
#data_xr.msl.encoding['missing_value'] = float(data_xr.msl.encoding['missing_value'])
#data_xr.msl.encoding['zlib'] = True #no effect
#data_xr.msl.encoding['scale_factor'] = 0.01
#data_xr.msl.encoding['add_offset'] = 0

#write to netcdf file
file_out = os.path.join('era5_mslp_out.nc')
data_xr.to_netcdf(file_out)

fig,(ax1,ax2) = plt.subplots(1,2,figsize=(11,5))
data_xr.msl.sel(time='2023-01-24 02:00:00').plot(ax=ax1,cmap='jet') #original dataset
with xr.open_dataset(file_out) as data_xr_check:
    data_xr_check.msl.sel(time='2023-01-24 02:00:00').plot(ax=ax2,cmap='jet') #written dataset
fig.tight_layout()

This results in incorrect data in the written file (right):

When updating the dtype (from int to float) in the variable encoding, this issues is solved:

The encoding in the source dataset:

data_xr.msl.encoding
Out[28]: 
{'source': 'p:\\11207892-pez-metoceanmc\\3D-DCSM-FM\\workflow_manual\\01_scripts\\04_meteo\\era5_temp\\era5_mslp_2022-11.nc',
 'original_shape': (720, 93, 121),
 'dtype': dtype('int16'),
 'missing_value': -32767,
 '_FillValue': -32767,
 'scale_factor': 0.11615998809759968,
 'add_offset': 99924.34817000595}

Possible issue: source data is integers, but opening files with different scaling_factors (from different files) converts it to floats (or maybe this always happens). The dtype in the encoding is still int, so this is how the netcdf is written, but probably something does not fit within the int-bounds.

Design simple FM/FM nesting workflow

Nesthd1:
- input: rooster grof (evt ook epsg code, of uit rooster lezen)
- input: pli-line met bnd langs rooster-fijn (incl epsg code, default is WGS84 maar check dan: coords.min()>=-180 en coords.max()<=180)
- action: kdtree van cellcenters grove rooster
- action: find 3 nearest neighbors (cartesian) or find cell numbers and connected cells via meshgrid oid
- action: generate list of cellscenters and connected cellcenters
- action: drop duplicate coordinates, drop coordinates with distance larger than np.sqrt(cellarea.mean() of nestcells)
- returns: obsfile met coordinaten van deze cellcenter values
Run grof model met obsfile
Nesthd2:
- input: pli-line
- input: hisfile grove model
- (input: obsfile, to check if all obspoints are present in ncfile (geen adminfile nodig))
- action: make kdtree van obspoints, query per pli punt (incl distance, cartesian), invdist avg of neighbors, results in timeseries
  - alternative: hisfile.interp() met alle pli punten (zoals interpolate_nc_to_bc() function) (lijkt niet zomaar mogelijk, maar kan ongetwijfeld)
- action: put timeseries in forcingmodel and write to bc file
- eerste opzet in "workinprogress_interpolate_his.py" example script
er is een idee om interpolatie in te bouwen in de FM kernel, dan zou het nog veel makkelijker worden (plifile omzetten naar xyn, grof model draaien met xyn, history inlezen en omzetten naar nc/bc

First setup: https://github.com/Deltares/dfm_tools/blob/main/tests/examples_workinprogress/workinprogress_nestingFMtoFM.py

Also request from RS/PK to add Neuman boundary, where inflowing direction of grid should be known (possible?). Or alternatively, combination of waterlevel/ux/uy boundary, which is simpler to implement.

Related, nesting SFINCS in FM is done in GC with:

Generated FM obsfile with: #578
Nest SFINCS from FM hisfile: https://github.com/Deltares/gc-japan-cyclone/blob/issue_22/src/sfincs/sfincs_update_forcing.py
Uses functions from: https://github.com/Deltares/gc-japan-cyclone/blob/issue_22/src/sfincs/sfincs_utils.py

Also request from WO to add Riemann boundary, so combination of waterlevel and discharge (latter from cross-sections). For morphology we also require nesting bed level changes (form obsfiles) and sediment transport (form cross-sections).

Also align with nesting in ocean models like CMEMS, maybe also consider nesting in other existing models (shyfem, schism, delft3d4, etc).

add support for mapformat=1

Removing deprecated code makes https://github.com/Deltares/dfm_tools/blob/main/tests/examples/preprocess_ini_rst_nc_to_xyz.py fail, since it reads rst files which are in the unsupported mapformat=1. Add preprocess function to generate mesh2d variable with pointers to relevant topology variable.

Improve dataset consistency of dfmt.open_partitioned_dataset()

The internal structure of datasets opened with dfmt.open_partitioned_dataset() is not consistent. This might be because edges/nodes are not ghostcell-filtered or reindexed.

import xugrid as xu
import dfm_tools as dfmt

file_nc = r'c:\DATA\dfm_tools_testdata\DFM_3D_z_Grevelingen\computations\run01\DFM_OUTPUT_Grevelingen-FM\Grevelingen-FM_0000_map.nc'

data_frommap = xu.open_dataset(file_nc)
data_frommap.ugrid.to_netcdf('test.nc') #this works
data_frommap2 = dfmt.open_partitioned_dataset('test.nc') #this also works

data_frommap = dfmt.open_partitioned_dataset(file_nc)
data_frommap.ugrid.to_netcdf('test.nc') #TODO: "ValueError: cannot reindex or align along dimension 'mesh2d_nEdges' because of conflicting dimension sizes: {77761, 67906}"
#data_frommap2 = dfmt.open_partitioned_dataset('test.nc') #not tested yet

Add ds rename funtion for waq variables

Add model builder example notebook

Tasks for first implementation:

build on examples for grid creation and model builder
also add to tutorials in docs
check TODO in notebook
last block: status = os.system('dir')
how to handle apikeys?
also point to updated meshkernelpy notebooks
commit+push, run binder a few times so it loads quickly (after that no pushes to main anymore)
improve grid/bathy plot

Improve reading/writing of FM bathymetry files

For FM, there are several file types for bathymetry and serveral packages that could read/write them:

xyz:
- pandas.read_csv()
- hydrolib XYZModel (but blocked by Deltares/HYDROLIB-core#415)
asc:
- dfm_tools writer: https://github.com/Deltares/dfm_tools/blob/main/dfm_tools/bathymetry.py
- geotiff (library gdal not found): https://gis.stackexchange.com/questions/425152/how-to-write-a-georeferenced-geotiff-from-known-coordinates-with-python-rasterio
- gdal: import gdal or from osgeo import gdal (does not work in dfm_tools_env for some reason)
- we will keep on needing this since FM does not support netcdf data
netcdf:
- maybe this is a convenient replacement for asc. It is not supported by the FM kernel, but it will be supported by meshkernelpy (for grid generation/refinement and bathy interpolation)

Fixes related to hydrolib-core issues

Checks for next release (may 2024):

mdu settings (more strict)
true/false booleans instead of 1/0 in mdu

From hydromt_delft3dfm:

Other:

Can be closed? (dfm_tools/xugrid alternatives):

Deltares/HYDROLIB-core#485
Deltares/HYDROLIB-core#268
- select via station name now possible with xarray+dfmt.preprocess_hisnc(): ds.sel(station='stationname')
- selecting nearest station not directly possible, also not with multiindex of name/x/y >> ValueError: multi-index does not support 'method' and 'tolerance'. Alternative method is via KDTree
- selecting stations within bounding box: possible with bool as in example script, but not with multiindex and ds.sel(x=slice(),y=slice() >> TypeError: float() argument must be a string or a number, not 'slice'. select with polygon would also result in bool, so might not matter
Deltares/HYDROLIB-core#400
Deltares/HYDROLIB-core#430
Deltares/HYDROLIB-core#322

hydrolib-core issues, to be included in 0.5.0:

Replacement for dfmt.scatter_to_regulargrid() (rasterize)

dfmt.scatter_to_regulargrid() was replaced by dfmt.rasterize_ugrid(), which is a working implementation for a new regridder based on the 500x500 regridder in xugrid for uda.ugrid.plot.imshow() (Deltares/xugrid#31). The rasterization is used in e.g. workinprogress_mapfile_to_regulargrid.py and postprocess_mapnc_ugrid.py

Left tasks:

#271
deprecate dfmt.scatter_to_regulargrid()

Improve release workflow

moved to #215

Deprecate dfmt.get_ncmodeldata() and related functions

Deprecate dfmt.get_ncmodeldata() and related functions, these have "DeprecationWarning" in these scripts:

https://github.com/Deltares/dfm_tools/blob/main/dfm_tools/get_nc.py
https://github.com/Deltares/dfm_tools/blob/main/dfm_tools/get_nc_helpers.py
removal of dfmt.plot_netmapdata() is blocked by Deltares/xugrid#49

`ds.to_crs()` implementation for hisfiles

Currently results in an error: AttributeError: 'Dataset' object has no attribute 'set_crs'

It was added to xr.UgridDataset via the ugrid accessor. Consider doing something similar for hisfiles.

remove additional xugrid uds.grid.plot() method from dfm_tools

Currently added a xugrid uds.grid.plot() method via the init.py file of dfm_tools. This is not necessary anymore, since Deltares/xugrid#28 and Deltares/xugrid#54 are closed.

remove non unicode bytes from HISTORY.rst and README.md

dfm_tools version: 0.7.23
Python version: 3.8
Operating System: Win 10

Description

Install toolbox from conda .yml file

What I Did

did: conda env update -f newenvironment.yml
with following lines in yml that give error
dependencies:
  - pip:
    - git+https://github.com/openearth/dfm_tools.git@6ac91e323ad9228cd10903a3e6af4ac15ad72b20

error:

File "\dfm_tools\setup.py", line 8, in <module>
        readme = readme_file.read()
      File "\lib\codecs.py", line 322, in decode
        (result, consumed) = self._buffer_decode(data, self.errors, final)
    UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa0 in position 3617: invalid start byte

similar error will be found later in the history file

What I expected:

seamless installation ;)

cleanup dfmt.regulargrid.rasterize_ugrid() input arguments

unsure if tests are possible with a branch clone

This is just a test issue to see if it works.
First branch has been created and there is a howto_git.txt explaining the theoretical way to work with it. Please check if it works

in tests/examples/CMEMS_interpolate_example.py: incorrect quantity for .ext

Input quantity for Boundary should be the bcvarname as is correctly specified in get_conversion_dict in interpolate_grid2bnd.py. E.g. salinity should become salinitybnd.

boundary_object = Boundary(quantity=quantity, #TODO: nodeId / bndWidth1D / bndBlDepth are written as empty values, but they should not be written if not supplied. https://github.com/Deltares/HYDROLIB-core/issues/319
                                   locationfile=Path(dir_out,file_pli.name),
                                   forcingfile=ForcingModel_object,
                                   )

Decide where to land grid-related operations

Could land in xugrid, dfm_tools, meshkernel or hydrolib-core.

Actions like >> current package:

grid generation/refinement >> meshkernel
grid writing >> xugrid (implemented, including meshkernel>xugrid conversion)
cross-section trough map data (side view) so dfmt.polyline_mapslice() >> xugrid+dfm_tools combination (#206)
slicing at depths is probably not grid-specific (#208)

feat: remove_periodic_cells()

Temporary fix for cells that go "around the back" of global models. Later properly fix in xugrid: Deltares/xugrid#63

mapfile/hisfile subselect in polygon

only return depth-sliced variables in get_Dataset_atdepths

temporarily drop edge/node/interface dims to avoid extra dimensions in returned dataset

Fixes related to xugrid issues

0.8.1:

Soon:

Then:

Deltares/xugrid#180
Deltares/xugrid#181
Deltares/xugrid#98
Deltares/xugrid#100
Deltares/xugrid#127
Deltares/xugrid#119 >> also add test for gridwriting to xugrid or dfm_tools (check on min/max fnc, start_index and _FillValue), make sure the grid has both triangles as squares
Deltares/xugrid#59 (although it would be nice to have it in xugrid instead, discuss this and #206)
Deltares/xugrid#135

Later:

Deltares/xugrid#80
Deltares/xugrid#58
Deltares/xugrid#56
Deltares/xugrid#172
Deltares/xugrid#68 (for westernscheldt model)
Deltares/xugrid#88
hanging edges/nodes are allowed in ugrid convention according to AvD

xugrid 0.7.0 or older:

get_Dataset_atdepths() requires bedlevel even if it is not used

Remove deprecated testcases

Add slev obsdata download function

Todo:

Follow-up: #712

proper cleanup of extra dims in atdepths return dataset

replace extra-dim-removal code by looping over relevant variables myself instead of applying bool to entire uds. >> variables = [var for var in ds.data_vars if set(["layer", "time", grid.face_dimension]).issubset(ds[var].dims)]
Atdepth bool hoeft alleen maar toegepast te worden op vars met evenveel en dezelfde dimensies
remove edge and node dim from uds, but before .where(). Might solve extra-dim issue already? Add note that edges/nodes are removed since zcoords-filter is currently only on faces, maybe once add edge-support if needed?
clean up unneccesary vars in returned dataset

Fix failing testbank cases

Fix failing testbank cases (24-01-2023).

_____________________________ test_run_examples[preprocess_meteo_mergenetCDFtime_xarray] ______________________________
C:\DATA\dfm_tools\tests\test_dfm_tools.py:43: in test_run_examples
------------------------------------------------ Captured stdout call -------------------------------------------------
opening multifile dataset of 180 files matching "era5_.*(chnk|mslp|u10n|v10n)_.*\.nc" (can take a while with lots of files)
------------------------------------------------ Captured stderr call -------------------------------------------------
Traceback (most recent call last):
[...]
OSError: [Errno -51] NetCDF: Unknown file format: b'p:\\metocean-data\\open\\ERA5\\data\\Irish_North_Baltic_Sea\\u10n\\era5_u10n_1992.nc'

______________________________________ test_zlayermodel_correct_layers_THISFAILS ______________________________________
C:\DATA\dfm_tools\tests\test_dfm_tools.py:165: in test_zlayermodel_correct_layers_THISFAILS
    assert (np.abs(vals_zcc_top-vals_wl)<1e-6).all() #this should pass
E   AssertionError: assert False
E    +  where False = <built-in method all of numpy.ndarray object at 0x00000220BFA8E570>()
E    +    where <built-in method all of numpy.ndarray object at 0x00000220BFA8E570> = array([2.5124 , 2.2804 , 2.2921 , ..., 2.284  , 2.25245, 2.25245]) < 1e-06.all
E    +      where array([2.5124 , 2.2804 , 2.2921 , ..., 2.284  , 2.25245, 2.25245]) = <ufunc 'absolute'>((array([-0.875, -0.875, -0.875, ..., -0.875, -0.875, -0.875]) - array([1.6374 , 1.4054 , 1.4171 , ..., 1.409  , 1.37745, 1.37745])))
E    +        where <ufunc 'absolute'> = np.abs

Deprecate dfmt.get_ugrid_verts()

Deprecate dfmt.get_ugrid_verts(). Since Deltares/xugrid#48 was solved, this can be replaced with uds.grid.face_node_coordinates everywhere. Maybe first add Deprecationwarning, and/or put new code in function for now.

Convert FM input/output to shapefiles/kml

Convert FM data (map/grid/obspoints/thd/pli/etc) to kml/shp, including coordinate conversion

Shapefile example (kml was also added for grid): https://github.com/Deltares/dfm_tools/blob/main/tests/examples_workinprogress/workinprogress_exporttoshapefile.py

KML example for obspoints: #808

Also convert obs/crs/etc to shape/kml

Maybe geowomat: https://geowombat.readthedocs.io/en/latest/api/geowombat.core.geoxarray.GeoWombatAccessor.html

maybe this is workable? https://deltares.github.io/xugrid/api/xugrid.Ugrid2d.intersect_edges.html
in the meantime, replace timestep argument with uds.isel(time=timestep), or make it optional?

Github related improvements

Documentation/license:

Code style/quality:

replace readme badge with overall (?): https://sonarcloud.io/summary/overall?id=Deltares_dfm_tools
style guide: https://www.python.org/dev/peps/pep-0008/
isort/black: https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html
argument type checking (pydantic?)
improve sonarcloud code quality and fix code issues: https://sonarcloud.io/project/issues?resolved=false&id=Deltares_dfm_tools
solve flake8 messages in pytest workflows
introduce types for essential variables: https://docs.python.org/3/library/typing.html (and update docstring)
properly deprecate deprecated functions
pre-commit-config: https://github.com/savente93/automation-demo

Create issues:

convert TODO from dfm_tools functions and example scripts into Github issues: https://sonarcloud.io/project/issues?resolved=false&severities=INFO&id=Deltares_dfm_tools (216 on 15-5-2024)
convert example scripts into Github issues, unittests and notebooks
create issues from bullets in #852

Testbank:

enable pytest via github actions
fix pytest by temporary ignoring testcases marked with requiresdata (reduces codecov but that is ok)
add codecov badge (fix percentage)
#336
#683
add windows and macos testbanks
phase out requireslocaldata tests by making testdata available (and avoiding p-drive links). Now we have opendap download. Maybe use HYDROLIB-data repos instead (or xugrid cached downloads)? Maybe use dsctestbench data on repos, how to authenticate on github? At least make cds/cmems testcases work on github by setting env vars
increase test coverage: https://app.codecov.io/gh/deltares/dfm_tools?displayType=list
pytest ignore UserWarning via pyproject.toml

deltares / dfm_tools Goto Github PK

dfm_tools's People

Contributors

Stargazers

Watchers

Forkers

dfm_tools's Issues

Description

What I Did

What I expected:

Recommend Projects

Recommend Topics

Recommend Org