Ideas on generalization of spatial package backends and file sources using GDAL (terra, sf, stars, etc.) about geotargets

[Aariq]

These are great ideas and I totally agree that the ideal situation would be one in which users can provide format = tar_raster or format = tar_vector and geotargets takes care of figuring out if the data is coming from terra, sf, stars, etc. and has some defaults for how targets are stored. I think the tricky part, which I don't quite understand if you have a plan for, is how those targets should be read back in and "unmarshalled". How can we know if the current target assumes an upstream target is a terra SpatVector or a sf MULTIPOLYGON? I would think geotargets would have to be as specific as providing tar_<vector/raster>_<package name> (e.g. tar_vector_terra, tar_vector_sf) as formats.

[mdsumner]

excellent descript @brownag

note there is also {gdalraster} now with already support for SOZip creation and file management, and nascent 'gdalvector' support:

https://usdaforestservice.github.io/gdalraster/reference/addFilesInZip.html

https://usdaforestservice.github.io/gdalraster/articles/gdalvector-draft.html

gdalraster has become very richly featured very quickly, and could be the GDAL API that's otherwise entirely missing from R atm.

[brownag]

Regarding Item 2 (/vsizip/), it should be possible to drop the .zip extension from file and target name. I was not aware of the alternate syntax!

From https://gdal.org/user/virtual_file_systems.html#vsizip-zip-archives:

Starting with GDAL 2.2, an alternate syntax is available so as to enable chaining and not being dependent on .zip extension, e.g.: /vsizip/{/path/to/the/archive}/path/inside/the/zip/file. Note that /path/to/the/archive may also itself use this alternate syntax.

[brownag]

I think the tricky part, which I don't quite understand if you have a plan for, is how those targets should be read back in and "unmarshalled". How can we know if the current target assumes an upstream target is a terra SpatVector or a sf MULTIPOLYGON? I would think geotargets would have to be as specific as providing tar_<vector/raster>_ (e.g. tar_vector_terra, tar_vector_sf) as formats.

I lean towards this conclusion also-- so that the choice is explicit and not changeable through magical options or settings. How these operations would be handled on the backend could/should be more generic, but for reproducibility and clarity it is probably best to give users options that require explicit choices. This might mean many combinations of thin wrapper methods around the core functions, but I don't think that is inherently bad as long as there is some overall order to how they are named.

excellent descript @brownag

note there is also {gdalraster} now with already support for SOZip creation and file management, and nascent 'gdalvector' support:

https://usdaforestservice.github.io/gdalraster/reference/addFilesInZip.html

https://usdaforestservice.github.io/gdalraster/articles/gdalvector-draft.html

gdalraster has become very richly featured very quickly, and could be the GDAL API that's otherwise entirely missing from R atm.

Sweet! I have seen {gdalraster} and watched some of the (rapid) progress on that with interest... but I don't think I was aware of the plans to provide bindings for the OGR vector API! I have used your vapour package for some of my generic/vector GDAL needs that go beyond terra/sf

Something truly generic, mirroring the GDAL API/"close to the GDAL metal" would allow for all sorts of capabilities and customization--perhaps {gdalraster} would be a good choice for an imported package doing the core backend work for plumbing to the various user-facing types/formats.

[njtierney]

Thanks for this @brownag !

Following on from @Aariq 's #7 - I quite like the idea of tar_{pkg}_<filetype> convention. I think that a lot of/most users would know what package they are reading/creating things with, so it should hopefully facilitate discovery in that way?

I'm still learning about a lot of spatial things, so there is a bit of this I don't quite understand, but I think that this issue could be split out into multiple components, eventually, as there are a few threads in here.

Overall my preference for syntax would be something like:

tar_terra_raster(
  new_raster
  raster_creation_function(args)
)

But overall this would work the same as:

tar_target(
  new_raster
  raster_creation_function(args),
  format = "format_terra_raster"
)

Or something?

with tar_{pkg}_<filetype> we can expose a filetype argument to give users control, e.g.,:

tar_terra_shapefile(
  my_shape,
  create_shapefile(args),
  filetype = "parquet"
)

But then I wonder if

tar_parquet_shapefile(
  my_shape,
  create_shapefile(args)
)

Would be better?

Naming things is hard. But I think that it is worthwhile thinking about the API design - once we have ideas on how we want the user to interact with the package my experience is that it is usually easier to write the code.

[Aariq]

Naming things is hard. But I think that it is worthwhile thinking about the API design - once we have ideas on how we want the user to interact with the package my experience is that it is usually easier to write the code.

I agree we should try to think through this and make some decisions before getting too far. Super helpful discussion here.

[Aariq]

I think everything in here is either in the package already, a PR, or a separate issue. Thanks for the contributions @brownag!