emodnet / emodnetwfs Goto Github PK
View Code? Open in Web Editor NEWAccess EMODnet Web Feature Service data through R
Home Page: https://emodnet.github.io/emodnet.wfs/
License: Other
Access EMODnet Web Feature Service data through R
Home Page: https://emodnet.github.io/emodnet.wfs/
License: Other
ows4r allows to pass CQL and OGC filters to the WFS requests. We should add this option as some layers are too large to be queried. For example, the EMODnet Biology occurrences (+30M at the moment!).
The CQL and OGC filters are passed to getFeatures()
with the arguments cql_filter
and filter
. The filter has to be URL encoded. This has been fixed in the development version but it is not yet on CRAN.
These filters can be added in ews_get_layer
(@annakrystalli please correct me if I'm wrong or if you have a better idea!):
https://github.com/EMODnet/EMODnetWFS/blob/5ac0e2fcc0d4b5869cbf78c8dce7212673a4687c/R/layers.R#L79
This test works:
version
#> _
#> platform x86_64-w64-mingw32
#> arch x86_64
#> os mingw32
#> system x86_64, mingw32
#> status
#> major 4
#> minor 0.2
#> year 2020
#> month 06
#> day 22
#> svn rev 78730
#> language R
#> version.string R version 4.0.2 (2020-06-22)
#> nickname Taking Off Again
library(EMODnetWFS)
library(ows4R)
library(leaflet)
library(tibble)
# Start client
wfs_bio <- emodnet_init_wfs_client("biology_occurrence_data")
#> v WFS client created succesfully
#> i Service: 'http://geo.vliz.be/geoserver/Dataportal/wfs'
#> i Version: '2.0.0'
# Get capabilities
wfs_bio_caps <- wfs_bio$getCapabilities()
# Find basic occurrence data
wfs_bio_caps_feat <- wfs_bio_caps$findFeatureTypeByName("Dataportal:eurobis-obisenv_basic")
# Get data
cql_filter = "scientificnameaccepted='Posidonia oceanica'"
ly_qry <- wfs_bio_caps_feat$getFeatures(cql_filter = URLencode(cql_filter))
ly_qry
# A tibble: 509 x 10
fid id datasetid datecollected decimallongitude decimallatitude scientificname aphiaid scientificnamea~ coordinateuncer~
<chr> <int> <chr> <dttm> <dbl> <dbl> <chr> <chr> <chr> <dbl>
1 eurobis~ 5805417 http://www.e~ 2006-09-05 00:00:00 13.7 45.5 Posidonia oce~ http://m~ Posidonia ocean~ NA
2 eurobis~ 5805426 http://www.e~ 2006-09-05 00:00:00 13.7 45.5 Posidonia oce~ http://m~ Posidonia ocean~ NA
3 eurobis~ 5805435 http://www.e~ 2006-09-05 00:00:00 13.7 45.5 Posidonia oce~ http://m~ Posidonia ocean~ NA
4 eurobis~ 5805444 http://www.e~ 2006-09-05 00:00:00 13.7 45.5 Posidonia oce~ http://m~ Posidonia ocean~ NA
5 eurobis~ 5805453 http://www.e~ 2006-09-05 00:00:00 13.7 45.5 Posidonia oce~ http://m~ Posidonia ocean~ NA
6 eurobis~ 5805462 http://www.e~ 2006-09-05 00:00:00 13.7 45.5 Posidonia oce~ http://m~ Posidonia ocean~ NA
7 eurobis~ 5805471 http://www.e~ 2006-09-05 00:00:00 13.7 45.5 Posidonia oce~ http://m~ Posidonia ocean~ NA
8 eurobis~ 5869917 http://www.e~ NA 38.6 41.6 Posidonia oce~ http://m~ Posidonia ocean~ 849349
9 eurobis~ 5869918 http://www.e~ NA 38.6 41.6 Posidonia oce~ http://m~ Posidonia ocean~ 849349
10 eurobis~ 5869919 http://www.e~ NA 6.78 42.5 Posidonia oce~ http://m~ Posidonia ocean~ NA
# ... with 499 more rows
Created on 2020-12-02 by the reprex package (v0.3.0)
Not in all obviously, but this might make things easier for communication given that @salvafern is an author of this package?
To be checked:
Might affect this test:
library(EMODnetWFS)
emodnet_get_all_wfs_info()
#> Warning in CPL_crs_from_input(x): GDAL Message 1: +init=epsg:XXXX syntax is
#> deprecated. It might return a CRS with a non-EPSG compliant axis order.
#> Error in curl::curl_fetch_memory(url, handle = handle): Timeout was reached: [emodnet02.cineca.it] Connection timed out after 10002 milliseconds
sessionInfo()
#> R version 4.0.2 (2020-06-22)
#> Platform: x86_64-w64-mingw32/x64 (64-bit)
#> Running under: Windows 10 x64 (build 18363)
#>
#> Matrix products: default
#>
#> locale:
#> [1] LC_COLLATE=English_United States.1252
#> [2] LC_CTYPE=English_United States.1252
#> [3] LC_MONETARY=English_United States.1252
#> [4] LC_NUMERIC=C
#> [5] LC_TIME=English_United States.1252
#>
#> attached base packages:
#> [1] stats graphics grDevices utils datasets methods base
#>
#> other attached packages:
#> [1] EMODnetWFS_0.0.2
#>
#> loaded via a namespace (and not attached):
#> [1] Rcpp_1.0.5 pillar_1.4.6 ows4R_0.1-5 compiler_4.0.2
#> [5] highr_0.8 class_7.3-17 tools_4.0.2 digest_0.6.25
#> [9] checkmate_2.0.0 jsonlite_1.7.0 evaluate_0.14 lifecycle_0.2.0
#> [13] tibble_3.0.3 lattice_0.20-41 pkgconfig_2.0.3 rlang_0.4.7
#> [17] cli_2.0.2 DBI_1.1.0 curl_4.3 yaml_2.2.1
#> [21] rgdal_1.5-16 xfun_0.16 e1071_1.7-3 geometa_0.6-3
#> [25] dplyr_1.0.2 stringr_1.4.0 httr_1.4.2 knitr_1.29
#> [29] fs_1.5.0 generics_0.0.2 vctrs_0.3.2 askpass_1.1
#> [33] classInt_0.4-3 grid_4.0.2 tidyselect_1.1.0 glue_1.4.1
#> [37] sf_0.9-6 R6_2.4.1 fansi_0.4.1 XML_3.99-0.5
#> [41] rmarkdown_2.5 sp_1.4-4 tidyr_1.1.1 purrr_0.3.4
#> [45] magrittr_1.5 usethis_1.6.3 backports_1.1.7 units_0.6-7
#> [49] ellipsis_0.3.1 htmltools_0.5.0 assertthat_0.2.1 KernSmooth_2.23-17
#> [53] stringi_1.4.6 openssl_1.4.2 crayon_1.3.4
Created on 2021-01-20 by the reprex package (v0.3.0)
Currently, the default service if no service is specified in many of the functions is seabed_habitats_individual_habitat_map_and_model_datasets
. This is somewhat arbitrary and the result of that being the service of interest when I first started working on the package.
Let me know your thoughts @salvafern & @LennertSchepers !
When downloading seabed substrate data from geology_seabed_substrate_maps , the CRS is not provided by the server. The CRS is then assumed to be 4326. This is incorrect.
wfs <- emodnet_init_wfs_client(service="geology_seabed_substrate_maps")
test <- emodnet_get_layers(wfs = wfs, layers = "seabed_substrate_1m", reduce_layers = T)
gives: Warning: crs missing. Set to default 4326
Working from the assumption that it is 3857 also yields incorrect maps, so I am a bit at a loss at what the correct CRS is.
version 1.0.0 in R3.6.1
Now with #56 layers (features) can be selected by exact name again. It's good as it avoids getting "abiotic" stuff when querying "biotic" for instance.
Now, exact means "with the namespace" e.g. ms:sl_fishing
.
How to we expect users to feed layer names? I'd say
exact
that we'd pass to caps$findFeatureTypeByName()
.Should exact
be TRUE or FALSE by default? I'd say TRUE?
Get a DOI form Zenodo (as ows4r does). Include funding details
@salvafern I have noticed that sf objects returned for the servers often encode CRSs in the deprecated PROJ4 string format (eg +init=epsg:3034
returned by emodnet_get_layers(service="geology_seabed_substrate_maps", layers = "seabed_substrate_1m", reduce_layers = T)
).
I know there's been big changes in the geospatial software landscape in terms of expressing CRSs as a result of changes in versions of the PROJ library > 5 & GDAL > 3 (see for example discussions of handling changes in package rgdal
and here in sf
. So my questions are:
I.e. a searchable table of all services & layers, e.g. as a regularly updated article in the pkgdown website.
# Start client
View(emodnet_wfs)
emodnet_client <- emodnet_init_wfs_client("seabed_habitats_general_datasets_and_products")
# See list of layers available and get Sandbanks of Europe
View(emodnet_get_wfs_info(emodnet_client))
subs <- emodnet_get_layers(emodnet_client, layers = "eusm2019_subs_full", reduce_layers = TRUE)
returns error
Warning: Download of layer 'eusm2019_subs_full' failed: Error in readBin(content, character()): R character strings are limited to 2^31-1 bytes
Is there a way to import such large layers through this package?
I feel we need good communication channels set up with service maintainers to be able to quickly figure out whether problems are from bugs in the package or service issues.
We should discuss at the next meeting.
I think at the very least a list of contact details for each service is necessary but would be interesting to explore whether there are better options.
And if not possible open a FR with ows4R
E.g. could emodnet_get_layer_info
be emod_layer_info()
or so?
Geology servers cannot be supported on Linux yet.
In the meantime the solutions are
We don't recommend changing the security levels system wide.
Thanks for your patience.
It feels like a lot of the failing tests recently relate more to failing services. While I'm glad we are testing them (and look forward to implementing during runtime too as discussed in #24 ), I feel it's not appropriate for our build to be showing as failing when it's services that are the issue.
The best way to implement it might well be separating service testing into a different GitHub Action. What do you think @salvafern ?
library("EMODnetWFS")
wfs <- emodnet_init_wfs_client(service = "biology")
layer_attribute_descriptions(wfs, layer = "Species_gridded_abundances_2year")
emodnet_get_layer_info()
has that info so what kind of call could help with that?Hi,
I wonder if it is possible to crop data when requesting a layer from a service.
Looking at the documentation there is a bbox(expr, num, num, num, num) that can be used in cql_filter, but I wonder what expr to provide if I want to crop the entire layer. Is this even possible? If so I think it would be helpful to add to the vignette about cql_filter().
Thanks once more for your work,
Jan Jaap
As per https://devguide.ropensci.org/collaboration.html#contributing-guide
Here for instance there's usage of testthis, and of httptest, for which one sentence or two could be added.
Following the tutorial, the "getbathymetry" function results in a wrong raster, which provides the altitudinal values, instead of the bathymethrycal. In fact, it seems that the bathymetry is masked with values of 0s. I think that the error is given by the "name" parameter. If it is changed to "emodnet:mean_2016", the resulting raster is correct, but then I cannot plot it.
emodnet_get_wfs_info(wfs_cml)
layers <- c("mediseh_zostera_m_pnt","mediseh_posidonia_nodata")
emodnet_get_layer_info(wfs = wfs_cml, layers = layers)
All our GAs are failing atm so I'm taking a closer look at them today.
macOS-latest (devel): Looks like this emanates from package sf
. Likely an issue with using binaries vs source.
ERROR: dependencies ‘ows4R’, ‘sf’ are not available for package ‘EMODnetWFS’
* removing
macOS-latest (release): change in epsg code (no longer returning +init
). fixed in af9717b
windows-latest:
Error: Error: processing vignette 'emodnetwfs.Rmd' failed with diagnostics:
package or namespace load failed for 'mapview' in loadNamespace(i, c(lib.loc, .libPaths()), versionCheck = vI[[i]]):
namespace 'terra' 1.4-22 is being loaded, but >= 1.5.12 is required
This appears to be related to the fact that The update for {terra} to 1.5.12 from 1.4-22 was released on January 13 and it has been slow to appear as compiled binaries, which means that if the tools to compile R packages (e.g., the Rtools4 software for Windows) is not installed, it will not be possible to install 1.5.12 yet. We can either
ubuntu-16.04 (release): Annoyingly it just says the job has failed with no further info. oldrel and 3.6 R pass though so it might be one of the issues described above.
This uses ubuntu-20.04
and seems to be installing package dependencies twice, once here with remotes::install_deps
: https://github.com/EMODnet/EMODnetWFS/blob/b4f2f9b258fdd12f6999033365b2d81dad3c6027/.github/workflows/Test-Services.yaml#L20-L23 and then again here with install.packages(".", repos = NULL, type="source")
: https://github.com/EMODnet/EMODnetWFS/blob/b4f2f9b258fdd12f6999033365b2d81dad3c6027/.github/workflows/Test-Services.yaml#L27-L32. I'll do some more investigations on this.
Overall I think most problems here are an issue with using binaries vs source. I found this discussion on the topic r-lib/actions#37 and while I generally prefer binaries, not least for their speed, at least when testing r devel we should likely be using source. I am happy to hear thoughts on whether we should also do this for windows @salvafern / @maelle.
In any case, I wills set it up for r-devel and dig a little deeper into the Test-Services GA issue and report back.
Hey @salvafern ,
I was just wondering, are metadata on the attributes of each layer available through the server? For example, when I download windfarm data:
library(EMODnetWFS)
emodnet_get_layers(service = "human_activities", layers = "windfarms")
#> Loading ISO 19139 XML schemas...
#> Loading ISO 19115 codelists...
#> ✓ WFS client created succesfully
#> ℹ Service: 'https://ows.emodnet-humanactivities.eu/wfs'
#> ℹ Version: '2.0.0'
#> Warning: crs missing. Set to default 4326
#> $windfarms
#> Simple feature collection with 219 features and 9 fields
#> Geometry type: POINT
#> Dimension: XY
#> Bounding box: xmin: -16.4 ymin: 27.757 xmax: 25.97 ymax: 65.74
#> Geodetic CRS: WGS 84
#> First 10 features:
#> gml_id country n_turbines power_mw status start dist_coast notes
#> 1 windfarms.1 Spain 1 5.0 Production 2018 3211.17407 <NA>
#> 2 windfarms.24 France NA 8.0 Production 2015 10400.68077 <NA>
#> 3 windfarms.190 Greece NA 24.0 Planned <NA> 1711.23910 <NA>
#> 4 windfarms.4 Spain 1 4.5 Production 2013 22.36593 <NA>
#> 5 windfarms.58 Denmark NA 240.0 Planned <NA> 6518.04614 <NA>
#> 6 windfarms.6 Spain 5 50.0 Production 2017 3211.17407 <NA>
#> 7 windfarms.59 Denmark NA 160.0 Planned <NA> 8396.57314 <NA>
#> 8 windfarms.60 Denmark NA 120.0 Planned <NA> 7725.66890 <NA>
#> 9 windfarms.63 Denmark NA 320.0 Planned <NA> 9328.48812 <NA>
#> 10 windfarms.75 Norway NA 10.0 Planned <NA> 9225.51976 <NA>
#> ljstatus the_geom
#> 1 production POINT (-15.36 28.04)
#> 2 production POINT (-2.778 47.24)
#> 3 planned POINT (25.85 40.83)
#> 4 production POINT (-15.4 27.85)
#> 5 planned POINT (11.01 55.59)
#> 6 production POINT (-15.36 28.04)
#> 7 planned POINT (9.82 55.15)
#> 8 planned POINT (10.41 56.03)
#> 9 planned POINT (11.11 55.06)
#> 10 planned POINT (5.01 59.15)
Created on 2021-08-20 by the reprex package (v2.0.1)
How can I get information on for example, what the units dist_coast
is in?
Hi all,
I noticed that there is a section for an example in the readme but the example itself is missing https://github.com/EMODnet/EMODnetWFS#example
Just installed EMODnetWFS and getting the seabed_substrate_1m layer from geology_seabed_substrate_maps now fails. I think that it also failed for 1.0.0, but I am not sure. This is on windows, R3.6.1.
wfs <- emodnet_init_wfs_client(service="geology_seabed_substrate_maps")
Loading ISO 19139 XML schemas...
Loading ISO 19115 codelists...
√ WFS client created succesfully
i Service: 'https://drive.emodnet-geology.eu/geoserver/gtk/wfs'
i Version: '2.0.0'
test <- emodnet_get_layers(wfs = wfs, layers = "seabed_substrate_1m", reduce_layers = T)
Space required after the Public Identifier
SystemLiteral " or ' expected
SYSTEM or PUBLIC, the URI is missing
Entity 'nbsp' not defined
Warning: Download of layer 'seabed_substrate_1m' failed: Error: 1: Space required after the Public Identifier
2: SystemLiteral " or ' expected
3: SYSTEM or PUBLIC, the URI is missing
4: Entity 'nbsp' not defined
The error is thus "Space required after the Public Identifier". Some digging in emodnet_get_layers() reveals that this comes from a call to purrr::map2
debug: out <- purrr::map2(.x = layers, .y = cql_filter, ~ews_get_layer(.x,
wfs, cql_filter = .y), wfs, suppress_warnings) %>% stats::setNames(layers)
Space required after the Public Identifier
SystemLiteral " or ' expected
SYSTEM or PUBLIC, the URI is missing
Entity 'nbsp' not defined
Warning: Download of layer 'seabed_substrate_1m' failed: Error: 1: Space required after the Public Identifier
2: SystemLiteral " or ' expected
3: SYSTEM or PUBLIC, the URI is missing
4: Entity 'nbsp' not defined
at this time layers is a single string "seabed_substrate_1m" and cql_filter is NA. The version of purrr is:
packageDescription("purrr")
Package: purrr
Title: Functional Programming Tools
Version: 0.3.4
Any pointers?
To add: getting layers from chemistry_marine_litter as per the example on the github main page works fine.
When I try to get info about the layer "biotic_occurrences" I also get the ones about "abiotic_occurrences" as if there were some sort of grepl()
rather than ==
somewhere.
library("EMODnetWFS")
wfs <- emodnet_init_wfs_client(service = "biology_occurrence_data")
#> Loading ISO 19139 XML schemas...
#> Loading ISO 19115 codelists...
#> Loading IANA mime types...
#> No encoding supplied: defaulting to UTF-8.
#> ✓ WFS client created succesfully
#> ℹ Service: 'http://geo.vliz.be/geoserver/Dataportal/wfs'
#> ℹ Version: '2.0.0'
emodnet_get_wfs_info(wfs)
#> # A tibble: 27 × 9
#> data_source service_name service_url layer_namespace layer_name title
#> <chr> <chr> <chr> <chr> <chr> <chr>
#> 1 emodnet_wfs biology_occurrence_… http://geo… Dataportal abiotic_o… Data…
#> 2 emodnet_wfs biology_occurrence_… http://geo… Dataportal biotic_ob… Data…
#> 3 emodnet_wfs biology_occurrence_… http://geo… Dataportal eurobis-o… EMOD…
#> 4 emodnet_wfs biology_occurrence_… http://geo… Dataportal eurobis-o… EMOD…
#> 5 emodnet_wfs biology_occurrence_… http://geo… Dataportal eurobis-o… EMOD…
#> 6 emodnet_wfs biology_occurrence_… http://geo… Dataportal eurobis-o… EMOD…
#> 7 emodnet_wfs biology_occurrence_… http://geo… Dataportal eurobis_g… EMOD…
#> 8 emodnet_wfs biology_occurrence_… http://geo… Dataportal eurobis_g… EMOD…
#> 9 emodnet_wfs biology_occurrence_… http://geo… Dataportal eurobis_g… EMOD…
#> 10 emodnet_wfs biology_occurrence_… http://geo… Dataportal eurobis_g… EMOD…
#> # … with 17 more rows, and 3 more variables: abstract <chr>, class <chr>,
#> # format <chr>
emodnet_get_layer_info(wfs, c("biotic_observations", "abiotic_observations"))
#> Error in .f(.x[[i]], ...): tentative d'appliquer un objet qui n'est pas une fonction
Created on 2022-03-02 by the reprex package (v2.0.1)
The solution that I'll implement in #46 is to unlist what I get from capabilities$findFeatureTypeByName and return unique rows (I checked that I also get this problem on the default branch).
library("EMODnetWFS")
wfs <- emodnet_init_wfs_client(service = "biology_occurrence_data")
#> Loading ISO 19139 XML schemas...
#> Loading ISO 19115 codelists...
#> Loading IANA mime types...
#> No encoding supplied: defaulting to UTF-8.
#> ✓ WFS client created succesfully
#> ℹ Service: 'http://geo.vliz.be/geoserver/Dataportal/wfs'
#> ℹ Version: '2.0.0'
simple_filter_sf <- emodnet_get_layers(
wfs = wfs,
layers = "taxa",
cql_filter = "scientificname='Abra'",
reduce_layers = TRUE
)
#> Warning: Download of layer 'taxa' failed: Error in UseMethod("st_crs<-"): pas de méthode pour 'st_crs<-' applicable pour un objet de classe "data.frame"
Created on 2022-03-02 by the reprex package (v2.0.1)
I tried to install the package in a old ubuntu distribution and failed due to a downstream dependency on the gert package, through the usethis package. Note usethis it is only used at the moment for stylish UI messages, if the problem persisted we could remove the dependency on usethis.
devtools::install_github("r-lib/gert")
# Downloading GitHub repo r-lib/gert@HEAD
# ✓ checking for file ‘/tmp/RtmpFV0XvN/remotes69bf6e4495fb/r-lib-gert-6585a65/DESCRIPTION’ (342ms)
# ─ preparing ‘gert’:
# ✓ checking DESCRIPTION meta-information ...
# ─ cleaning src
# ─ running ‘cleanup’
# ─ checking for LF line-endings in source and make files and shell scripts
# ─ checking for empty or unneeded directories
# ─ building ‘gert_1.1.0.tar.gz’
#
# Installing package into ‘/data/home/salvadorf/R/x86_64-pc-linux-gnu-library/3.6’
# (as ‘lib’ is unspecified)
# * installing *source* package ‘gert’ ...
# ** using staged installation
# > curl::curl_download("http://r-lib.github.io/gert/get-libgit2-linux.sh","get-libgit2-linux.sh")
# >
# >
# > curl::curl_download('https://r-lib.github.io/gert/libgit2-1.1.0.x86_64_linux.tar.gz','bundle.tar.gz')
# >
# >
# Using static libgit2-1.1.0 for Linux x86_64
# Using PKG_CFLAGS=-DSTATIC_LIBGIT2 -I/tmp/Rtmp8ir5fb/R.INSTALL6a474e0e2c00/gert/libgit2/include
# Using PKG_LIBS=-L/tmp/Rtmp8ir5fb/R.INSTALL6a474e0e2c00/gert/libgit2/lib -lgit2 -lrt -lpthread -lssh2 -lssl -lcrypto -ldl -lpcre -lz
# Configuration OK!
# ** libs
# [...]
# /usr/bin/ld: /tmp/Rtmp8ir5fb/R.INSTALL6a474e0e2c00/gert/libgit2/lib/libgit2.a(annotated_commit.c.o): unrecognized relocation (0x2a) in section `.text'
# /usr/bin/ld: final link failed: Bad value
# collect2: error: ld returned 1 exit status
# make: *** [gert.so] Error 1
# ERROR: compilation failed for package ‘gert’
# * removing ‘/data/home/salvadorf/R/x86_64-pc-linux-gnu-library/3.6/gert’
# Error: Failed to install 'gert' from GitHub:
# (converted from warning) installation of package ‘/tmp/RtmpFV0XvN/file69bf33029d44/gert_1.1.0.tar.gz’ had non-zero exit status
sessionInfo()
#> R version 3.6.0 (2019-04-26)
#> Platform: x86_64-pc-linux-gnu (64-bit)
#> Running under: Ubuntu 14.04.6 LTS
#>
#> Matrix products: default
#> BLAS: /usr/lib/libblas/libblas.so.3.0
#> LAPACK: /usr/lib/lapack/liblapack.so.3.0
#>
#> locale:
#> [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
#> [3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
#> [5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
#> [7] LC_PAPER=en_US.UTF-8 LC_NAME=C
#> [9] LC_ADDRESS=C LC_TELEPHONE=C
#> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
#>
#> attached base packages:
#> [1] stats graphics grDevices utils datasets methods base
#>
#> loaded via a namespace (and not attached):
#> [1] Rcpp_1.0.5 fansi_0.4.2 digest_0.6.25 crayon_1.3.4
#> [5] assertthat_0.2.1 magrittr_2.0.1 reprex_1.0.0 evaluate_0.14
#> [9] highr_0.8 stringi_1.5.3 rlang_0.4.10 cli_2.2.0
#> [13] rstudioapi_0.13 fs_1.3.1 rmarkdown_2.6 tools_3.6.0
#> [17] stringr_1.4.0 glue_1.4.2 xfun_0.20 yaml_2.2.1
#> [21] compiler_3.6.0 htmltools_0.5.1.1 knitr_1.30
Created on 2021-01-28 by the reprex package (v1.0.0)
I.e. as a temporary workaround to #39
Hey Anna, I'm trying to access some of the big seabed map layers, am hitting an error when running:
wfs_seabedgeneral <- emodnet_init_wfs_client(service = "seabed_habitats_general_datasets_and_products")
hab_dat <- emodnet_get_layers(wfs = wfs_seabedgeneral, layers = "eusm2019_subs_full")
Error message is:
Warning: Download of layer 'eusm2019_subs_full' failed: Error in readBin(content, character()): R character strings are limited to 2^31-1 bytes
Seems like some problem parsing the layer?
Hi @annakrystalli !
I have written a vignette on how to use this package with some more examples. The idea is to use it as a product story, but I figured it would be good to include it in the package itself.
Could you please add a short introduction paragraph?
https://github.com/EMODnet/EMODnetWFS/blob/f9774fa2639be5661e24ac4810a569e747bf92e9/vignettes/emodnetwfs.Rmd#L17
And of course let me know if there is anything you want to change/improve/suggestions etc 😊
Example in the ECQL filtering vignette are failing:
library(EMODnetWFS)
wfs_ha <- emodnet_init_wfs_client(service = "human_activities")
#> Loading ISO 19139 XML schemas...
#> Loading ISO 19115 codelists...
#> Loading IANA mime types...
#> No encoding supplied: defaulting to UTF-8.
#> ✓ WFS client created succesfully
#> ℹ Service: 'https://ows.emodnet-humanactivities.eu/wfs'
#> ℹ Version: '2.0.0'
layer_attributes_summarise(wfs_ha, layer = "maritimebnds")
#> Error in sf::st_drop_geometry(.): st_drop_geometry only works with objects of class sf
layer_attributes_tbl(wfs_ha, layer = "maritimebnds")
#> Error in sf::st_drop_geometry(.): st_drop_geometry only works with objects of class sf
layer_attribute_inspect(wfs_ha, layer = "maritimebnds", attribute = "sitename")
Created on 2022-03-11 by the reprex package (v2.0.1)
Running functions with debugging shows that it is the filtering with PROPERTIES
filter that seems to not be working:
Related to #64
It'd be good to have, for developers, some sort of table of how requests are made when a function makes requests. I.e. is it using httr or something else.
library("EMODnetWFS")
wfs <- emodnet_init_wfs_client(service = "human_activities")
#> Loading ISO 19139 XML schemas...
#> Loading ISO 19115 codelists...
#> Loading IANA mime types...
#> No encoding supplied: defaulting to UTF-8.
#> ✓ WFS client created succesfully
#> ℹ Service: 'https://ows.emodnet-humanactivities.eu/wfs'
#> ℹ Version: '2.0.0'
or_filter_sf <- emodnet_get_layers(
wfs = wfs,
layers = "majorcatches",
cql_filter = "country='Baltic Sea' OR country='Bulgaria'",
reduce_layers = TRUE
)
#> Warning: Download of layer 'majorcatches' failed: Error: Cannot open "/tmp/RtmpZxxHz3/file630649c69027.gml"; The source could be corrupt or not supported. See `st_drivers()` for a list of supported formats.
Created on 2022-03-02 by the reprex package (v2.0.1)
It seems that some of the services mentioned in the documentation are not available anymore. I cannot get any data from
seabed_habitats_general_datasets_and_products or seabed_habitats_individual_habitat_map_and_model_datasets. This means that the default in emodnet_init_wfs_client() also does not work.
Using v1.0.0 in R3.6.1.
From Britt:
As far as I can tell, there is only one (relatively recent) web service missing, namely that of the EMODnet Chemistry contaminants (URL: https://nodc.inogs.it/geoserver/Contaminants/wfs). It would be nice if you could also include this in the package J
It should error
library("EMODnetWFS")
emodnet_init_wfs_client(service = "chemistry_marine_litter")
#> Loading ISO 19139 XML schemas...
#> Loading ISO 19115 codelists...
#> Loading IANA mime types...
#> No encoding supplied: defaulting to UTF-8.
#> x WFS client creation failed.
#> ℹ Service: 'https://www.ifremer.fr/services/wfs/emodnet_chemistry2'
#> Error in curl::curl_fetch_memory(url, handle = handle): OpenSSL SSL_connect: Connexion ré-initialisée par le correspondant in connection to www.ifremer.fr:443
Created on 2022-03-02 by the reprex package (v2.0.1)
I've been trying to include more detailed demos of using spatial ECQL predicates, following the following geoserver documentation: https://docs.geoserver.org/latest/en/user/filter/ecql_reference.html#ecql-expr
However I am hitting a lot of problems with getting filters to work and am also getting confusing results when they do work. I've isolated a couple of examples below to illustrate some of my confusion:
library(EMODnetWFS)
wfs_gs <- emodnet_init_wfs_client(service = "geology_seabed_substrate_maps")
#> Loading ISO 19139 XML schemas...
#> Loading ISO 19115 codelists...
#> Loading IANA mime types...
#> No encoding supplied: defaulting to UTF-8.
#> ✓ WFS client created succesfully
#> ℹ Service: 'https://drive.emodnet-geology.eu/geoserver/gtk/wfs'
#> ℹ Version: '2.0.0'
# start with a reference geometry. NOte call returns EWKT geometry MULTISURFACE
# which is not part of OCG standard
ref_sf <- emodnet_get_layers(wfs = wfs_gs, layers = "seabed_substrate_1m",
cql_filter ="id=10848", reduce_layers = TRUE)
ref_sf
#> Simple feature collection with 1 feature and 30 fields
#> Geometry type: MULTISURFACE
#> Dimension: XY
#> Bounding box: xmin: 3183730 ymin: 2544099 xmax: 3184146 ymax: 2546943
#> CRS: +init=epsg:3034
#> gml_id id objectid code country data_holder
#> 1 seabed_substrate_1m.10848 10848 10848 FR001 France SHOM
#> contact scale original_scale original_grain_size
#> 1 [email protected] 1000000 0 SHOM
#> references
#> 1 Garlan T. et Marches E. (2010) - DCSMM, Carte de nature des fonds de la sous-région Manche-Mer du Nord à 1/500 000
#> comments
#> 1 Based on a compilation of maps from differents institutions between 1970 and 2010
#> reclassification method sample_number original_substrate relation folk_16cl
#> 1 2 4 0 103 4 6
#> folk_16cl_txt folk_7cl folk_7cl_txt folk_5cl folk_5cl_txt
#> 1 6. No data at this level 2 2. Sand 2 2. Sand
#> surface_feature conf_rs conf_s conf_d conf_tot shape_length shape_area
#> 1 NA NA NA NA 0.05998287 8.306917e-05
#> geom
#> 1 MULTISURFACE (POLYGON ((318...
# create reference map
ref_map <- ref_sf %>%
sf::st_cast(to = "GEOMETRYCOLLECTION") %>% # mapview doesn't seem to work with
# EWKT so MULTIURFACE needs casting to a geometry collection
mapview::mapview(zcol = "folk_7cl_txt", burst = TRUE)
ref_map
#extract geometry as WKT
ref_geom_multisurface <- ref_sf %>%
sf::st_geometry() %>% sf::st_as_text()
ref_geom_multisurface
#> [1] "MULTISURFACE (POLYGON ((3184146 2544100, 3184141 2544099, 3184108 2544110, 3184055 2544148, 3184014 2544192, 3183974 2544250, 3183940 2544317, 3183914 2544387, 3183899 2544453, 3183890 2544510, 3183864 2544808, 3183730 2546928, 3183967 2546943, 3184058 2545528, 3184057 2545520, 3184054 2545456, 3184060 2545400, 3184057 2545336, 3184054 2545271, 3184058 2545205, 3184058 2545118, 3184090 2545026, 3184103 2544826, 3184102 2544806, 3184107 2544751, 3184098 2544557, 3184103 2544360, 3184103 2544317, 3184139 2544137, 3184146 2544100)))"
# Use WKT to construct DWITHIN query - FAILS with EWKT geometry MULTISURFACE event though that's the geometry type the data is returned as.
emodnet_get_layers(wfs = wfs_gs, layers = "seabed_substrate_1m",
cql_filter = paste0("DWITHIN(geom,", ref_geom_multisurface, ", 0.5, kilometers)"),
reduce_layers = TRUE )
#> Warning: Download of layer 'seabed_substrate_1m' failed: Error: Cannot open "/private/var/folders/8p/87cqdx2s34vfvcgh04l6z72w0000gn/T/Rtmpmzke5V/file1e416b4c119.gml"; The source could be corrupt or not supported. See `st_drivers()` for a list of supported formats.
#> NULL
# Transform to GEOMETRYCOLLECTION.
ref_geom_colllection <- ref_sf %>%
sf::st_cast(to = "GEOMETRYCOLLECTION") %>%
sf::st_geometry() %>% sf::st_as_text()
# Use WKT to construct DWITHIN query -
geom_colllection_res <- emodnet_get_layers(wfs = wfs_gs, layers = "seabed_substrate_1m",
cql_filter = paste0("DWITHIN(geom,", ref_geom_colllection, ", 0.5, kilometers)"),
reduce_layers = TRUE )
# WORKS
geom_colllection_res
#> Simple feature collection with 16 features and 30 fields
#> Geometry type: MULTISURFACE
#> Dimension: XY
#> Bounding box: xmin: 2333204 ymin: 2842851 xmax: 2751690 ymax: 3525843
#> CRS: +init=epsg:3034
#> First 10 features:
#> gml_id id objectid code country data_holder
#> 1 seabed_substrate_1m.17209 17209 17209 IE-016 Ireland GSI
#> 2 seabed_substrate_1m.17210 17210 17210 IE-016 Ireland GSI
#> 3 seabed_substrate_1m.17231 17231 17231 IE-016 Ireland GSI
#> 4 seabed_substrate_1m.17232 17232 17232 IE-016 Ireland GSI
#> 5 seabed_substrate_1m.17225 17225 17225 IE-016 Ireland GSI
#> 6 seabed_substrate_1m.17226 17226 17226 IE-016 Ireland GSI
#> 7 seabed_substrate_1m.17227 17227 17227 IE-016 Ireland GSI
#> 8 seabed_substrate_1m.17229 17229 17229 IE-016 Ireland GSI
#> 9 seabed_substrate_1m.17230 17230 17230 IE-016 Ireland GSI
#> 10 seabed_substrate_1m.17233 17233 17233 IE-016 Ireland GSI
#> contact scale original_scale original_grain_size references
#> 1 [email protected] 1000000 0 Folk
#> 2 [email protected] 1000000 0 Folk
#> 3 [email protected] 1000000 0 Folk
#> 4 [email protected] 1000000 0 Folk
#> 5 [email protected] 1000000 0 Folk
#> 6 [email protected] 1000000 0 Folk
#> 7 [email protected] 1000000 0 Folk
#> 8 [email protected] 1000000 0 Folk
#> 9 [email protected] 1000000 0 Folk
#> 10 [email protected] 1000000 0 Folk
#> comments
#> 1 Generalized from 250k scale to 1 M by GTK June 2016/EMODnet Geology.
#> 2 Generalized from 250k scale to 1 M by GTK June 2016/EMODnet Geology.
#> 3 Generalized from 250k scale to 1 M by GTK June 2016/EMODnet Geology.
#> 4 Generalized from 250k scale to 1 M by GTK June 2016/EMODnet Geology.
#> 5 Generalized from 250k scale to 1 M by GTK June 2016/EMODnet Geology.
#> 6 Generalized from 250k scale to 1 M by GTK June 2016/EMODnet Geology.
#> 7 Generalized from 250k scale to 1 M by GTK June 2016/EMODnet Geology.
#> 8 Generalized from 250k scale to 1 M by GTK June 2016/EMODnet Geology.
#> 9 Generalized from 250k scale to 1 M by GTK June 2016/EMODnet Geology.
#> 10 Generalized from 250k scale to 1 M by GTK June 2016/EMODnet Geology.
#> reclassification method sample_number original_substrate relation folk_16cl
#> 1 2 2 0 0 6
#> 2 2 2 0 0 6
#> 3 2 2 0 0 6
#> 4 2 2 0 0 6
#> 5 2 2 0 0 6
#> 6 2 2 0 0 6
#> 7 2 2 0 0 6
#> 8 2 2 0 0 6
#> 9 2 2 0 0 6
#> 10 2 2 0 0 6
#> folk_16cl_txt folk_7cl folk_7cl_txt folk_5cl
#> 1 6. No data at this level 3 3. Coarse-grained sediment 3
#> 2 6. No data at this level 3 3. Coarse-grained sediment 3
#> 3 6. No data at this level 4 4. Mixed sediment 4
#> 4 6. No data at this level 4 4. Mixed sediment 4
#> 5 6. No data at this level 4 4. Mixed sediment 4
#> 6 6. No data at this level 4 4. Mixed sediment 4
#> 7 6. No data at this level 4 4. Mixed sediment 4
#> 8 6. No data at this level 4 4. Mixed sediment 4
#> 9 6. No data at this level 4 4. Mixed sediment 4
#> 10 6. No data at this level 4 4. Mixed sediment 4
#> folk_5cl_txt surface_feature conf_rs conf_s conf_d conf_tot
#> 1 3. Coarse-grained sediment NA NA NA NA
#> 2 3. Coarse-grained sediment NA NA NA NA
#> 3 4. Mixed sediment NA NA NA NA
#> 4 4. Mixed sediment NA NA NA NA
#> 5 4. Mixed sediment NA NA NA NA
#> 6 4. Mixed sediment NA NA NA NA
#> 7 4. Mixed sediment NA NA NA NA
#> 8 4. Mixed sediment NA NA NA NA
#> 9 4. Mixed sediment NA NA NA NA
#> 10 4. Mixed sediment NA NA NA NA
#> shape_length shape_area geom
#> 1 0.01020992 5.222959e-06 MULTISURFACE (POLYGON ((252...
#> 2 1.99591150 1.361911e-01 MULTISURFACE (POLYGON ((252...
#> 3 8.59975782 2.658686e-01 MULTISURFACE (POLYGON ((256...
#> 4 0.14909377 8.219173e-04 MULTISURFACE (POLYGON ((254...
#> 5 0.12131420 9.190709e-04 MULTISURFACE (POLYGON ((251...
#> 6 0.06035866 2.032417e-04 MULTISURFACE (POLYGON ((251...
#> 7 0.21341242 2.291320e-03 MULTISURFACE (POLYGON ((251...
#> 8 0.18912688 1.402354e-03 MULTISURFACE (POLYGON ((253...
#> 9 0.46555359 7.035125e-03 MULTISURFACE (POLYGON ((252...
#> 10 0.19845728 1.973919e-03 MULTISURFACE (POLYGON ((255...
# However returns very strange results!
ref_map + geom_colllection_res %>%
sf::st_cast(to = "GEOMETRYCOLLECTION") %>%
mapview::mapview(zcol = "folk_7cl_txt", burst = TRUE)
I also tried to test out predicate TOUCHES
. Seems to fail with some geometry types:
# try to use TOUCHES predicate with GEOMETRYCOLLECTION - FAILS!
emodnet_get_layers(wfs = wfs_gs, layers = "seabed_substrate_1m",
cql_filter = paste0("TOUCHES(geom,", ref_geom_colllection,")"),
reduce_layers = TRUE )
#> Warning: Download of layer 'seabed_substrate_1m' failed: Error: Cannot open "/private/var/folders/8p/87cqdx2s34vfvcgh04l6z72w0000gn/T/Rtmpmzke5V/file1e413fb143f5.gml"; The source could be corrupt or not supported. See `st_drivers()` for a list of supported formats.
#> NULL
# Transform to single POLYGON and try again
ref_geom <- ref_sf %>%
sf::st_cast(to = "GEOMETRYCOLLECTION") %>%
sf::st_collection_extract(type = "POLYGON") %>%
sf::st_geometry() %>% sf::st_as_text()
# WORKS - although no features match the predicate. I guess polygons boundary do
# not share a single point?
emodnet_get_layers(wfs = wfs_gs, layers = "seabed_substrate_1m",
cql_filter = paste0("TOUCHES(geom,", ref_geom,")"),
reduce_layers = TRUE )
#> Simple feature collection with 0 features and 30 fields
#> Bounding box: xmin: NA ymin: NA xmax: NA ymax: NA
#> CRS: +init=epsg:3034
#> [1] gml_id id objectid
#> [4] code country data_holder
#> [7] contact scale original_scale
#> [10] original_grain_size references comments
#> [13] reclassification method sample_number
#> [16] original_substrate relation folk_16cl
#> [19] folk_16cl_txt folk_7cl folk_7cl_txt
#> [22] folk_5cl folk_5cl_txt surface_feature
#> [25] conf_rs conf_s conf_d
#> [28] conf_tot shape_length shape_area
#> [31] geom
#> <0 rows> (or 0-length row.names)
Created on 2021-11-26 by the reprex package (v2.0.1)
So it's really not clear from the ECQL literal docs docs what types of geometries should work for different predicates. The docs relating to a geometry expression state: All standard geometry types are supported: POINT, LINESTRING, POLYGON, MULTIPOINT, MULTILINESTRING, MULTIPOLYGON, GEOMETRYCOLLECTION.
So I'm finding it hard to understand why TOUCHES doesn't work with a GEOMETRYCOLLECTION
(when it does for DWITHIN yet POLYGON works for TOUCHES
It's taken a lot of time digging and I'm still left in a lot of confusion as to how to implement spatial filtering in particular and digging through docs on these topics online felt like a massive rabbit hole.
Given how much effort it took and how much confusion it generated in me, it might be safe to assume that users will have similar issues. I feel it would be good for firstly myself to understand better and inverst in more detailed documentation so that are users can successfully make use of filtering.
So my questions are:
For now, the only example I've got to work correctly is the BBOX
predicate so it is the only one I've included as an example in the filtering vignette so far (#29 ).
I've also been going back to basics and trying to collate useful information to help me understand what's going on here: https://annakrystalli.me/spatial-notes/ (in this repo: https://github.com/annakrystalli/spatial-notes)
But any ideas on how we can get a better understanding on these topics would be greatly appreciated @salvafern
from #34
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.