jjrom / resto

A metadata catalog and search engine for geospatialized data

About

resto is a metadata catalog and a search engine dedicated to geospatialized data. Originally, it’s main purpose it to handle Earth Observation satellite imagery but it can be used to store any kind of metadata localized in time and space.

resto search API conforms to the SpatioTemporal Asset Catalog (STAC) specification v1.0.0 and to the CEOS OpenSearch Best Practice Document.

It is mentioned in ESA's "Exploitation Platform Common Core Components" as the closest implementation of a catalogue component according to the requirements specified in ESA's "Exploitation Platform Open Architecture"

Demo

The [https://tamn.snapplanet.io] resto server provides up to date access to Landsat-8 and Sentinel-2 images.

You can browse it with the rocket web client

Or test the API :

• Get STAC root endpoint - https://tamn.snapplanet.io/?_pretty=1
• Get all collections - https://tamn.snapplanet.io/collections?_pretty=1
• Search Sentinel-2 products acquired on June 1st, 2021 - https://tamn.snapplanet.io/collections/S2/items?datetime=2021-05-06T00:00:00Z/2021-06-01T23:59:59Z&_pretty=1
• Get catalogs of products classified by continents - https://tamn.snapplanet.io/catalogs/classifications/geographical/continent?_pretty=1
• Get catalogs of products classified by european countries - https://tamn.snapplanet.io/catalogs/classifications/geographical/continent/continent:Europe:6255148?_pretty=1

Installation

TL;DR

The INSTALLATION.md file provides additional information on the installation process.

Deploy the service

To launch a default pre-configured resto instance, just type :

./deploy

This will build locally the jjrom/resto image and launch a resto container exposing the resto API service at http://localhost:5252

[IMPORTANT] Docker on Mac M1

If you're using docker on Mac with apple Silicon M1 chip, be sure to turn off "Use Rosetta for x86/amd64 emulation on Apple Silicon" in Docker Desktop > Settings > General.

When this option is turned on, every calls to PHP preg_match function (which is used by resto) leads to a segmentation fault within php fpm and an HTTP 502 Bad Gateway error in nginx. Why ? I just don't know !

Users, groups and rights

See ./USERS.md

Collection and catalogs

See .COLLECTIONS_AND_CATALOGS.md

References

Here are some projects that use resto.

• SnapPlanet
• European Digital Twin of the Ocean
• CREODIAS
• Rocket - The Earth in your pocket
• The French Sentinel Data Processing center
• The French Space Agency, THEIA land data center
• The Polish EO Data finder
• Remote Sensor Technology Center of Japan, EPIC project
• Sentinel Australia Regional Access
• ESA's Food Security Thematic Exploitation Platform
• ESA's Forestry Thematic Exploitation Platform
• CNES Kalideos platform
• CEOS Recovery Observatory
• EO4SD Lab - Earth observation for sustainable development
• IRSTEA Thisme project - THeia and Irstea Soil MoisturE catalog

If you plan to use resto and would like to have your project added to this list, feel free to contact support

Support

resto is developped and maintained by jeobrowser.

For questions, support or anything related to resto feel free to contact jerome[dot]gasperi[at]gmail[dot]com

resto's Issues

Security : Third party php-jwt (version < 2.0) is vulnerable

Critical vulnerabilities was found in several JSON Web Token libraries (see https://auth0.com/blog/2015/03/31/critical-vulnerabilities-in-json-web-token-libraries/).
The current version of php-jwt library is compromised. Upgrade to v2.0 is mandatory

Incorrect operator for dc:date operator in RestoModel

Operator for dc:date (i.e. updated) filter is '='. It should be '>=' since a request on the updated date should return all products that have been inserted after the updated date

Query Analyzer : "last month" incorrectly processed if found after a "in location" block

The query "in Toulouse last month" is incorrectly handled because "in ... last month" is detected as a whole by the WhenProcessor. Correct handling should be "in Toulouse" discarded by WhenProcessor then "last month" detected by WhenProcessor, then "in Toulouse" detected by WhereProcessor.
For instance the query "last month in Toulouse" is correctly analyzed

Unwanted PHP warning on unsuccessful login

Notice: Undefined index: userid in /Users/jrom/Devel/resto2/include/resto/RestoUser.php

on line 72



Notice: Undefined index: email in /Users/jrom/Devel/resto2/include/resto/RestoUser.php</b
on line 76



Notice: Undefined index: groupname in /Users/jrom/Devel/resto2/include/resto/RestoUser.php
on line 76



Notice: Undefined index: email in /Users/jrom/Devel/resto2/include/resto/RestoCart.php</b
on line 50

Can't create two admin users with different login but the same password.

$RESTO_HOME/_install/createAdminUser.sh -u admin -p mypass
INSERT 0 1

$RESTO_HOME/_install/createAdminUser.sh -u admin_resto -p mypass
ERROR: duplicate key value violates unique constraint "users_activationcode_key"
DETAIL: Key (activationcode)=(41b95ca5************b1fa5823a529c74a) already exists.

Missing normalize() function in PostgreSQL

With the following query : http://localhost/resto2/api/collections/RO/search.json?q=haiti

I get the following answer (only partial answer is quoted here) :

query: {
    searchFilters: {
        searchTerms: ""
    },
    analysis: {
        query: "haiti",
        language: "en",
    analyze: {
        What: [ ],
        When: [ ],
        Where: [ ],
        Errors: [
            {
            error: "LOCATION_NOT_FOUND",
            message: "haiti"
            }
        ]
        },
        processingTime: 0.0031042098999
    },
    processingTime: 0.00775694847107
},

I see this in the logs

[09-Apr-2015 15:40:34 Europe/Paris] PHP Warning:  pg_query(): Query failed: ERROR:  function normalize(character varying) does not exist
LINE 1: SELECT name,countryname,normalize(countryname) as countrynor...
                                ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts. in /Users/laurent.baresse/Documents/workspaces/workspaces/workspace_RO/LocalDeploy/resto2/include/resto/Modules/Gazetteer.php on line 296
[09-Apr-2015 15:40:34 Europe/Paris] PHP Warning:  pg_fetch_assoc() expects parameter 1 to be resource, boolean given in /Users/laurent.baresse/Documents/workspaces/workspaces/workspace_RO/LocalDeploy/resto2/include/resto/Modules/Gazetteer.php on line 297
[09-Apr-2015 15:40:34 Europe/Paris] PHP Warning:  pg_query(): Query failed: ERROR:  function normalize(character varying) does not exist
LINE 1: SELECT name,countryname,normalize(countryname) as countrynor...
                                ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts. in /Users/laurent.baresse/Documents/workspaces/workspaces/workspace_RO/LocalDeploy/resto2/include/resto/Modules/Gazetteer.php on line 296
[09-Apr-2015 15:40:34 Europe/Paris] PHP Warning:  pg_fetch_assoc() expects parameter 1 to be resource, boolean given in /Users/laurent.baresse/Documents/workspaces/workspaces/workspace_RO/LocalDeploy/resto2/include/resto/Modules/Gazetteer.php on line 297
[09-Apr-2015 15:40:34 Europe/Paris] PHP Warning:  pg_query(): Query failed: ERROR:  function normalize(character varying) does not exist
LINE 1: SELECT admin, normalize(admin) as normalized, continent, ST_...
                      ^
HINT:  No function matches the given name and argument types. You might need to add explicit type casts. in /Users/laurent.baresse/Documents/workspaces/workspaces/workspace_RO/LocalDeploy/resto2/include/resto/Modules/Gazetteer.php on line 489
[09-Apr-2015 15:40:34 Europe/Paris] PHP Warning:  pg_fetch_assoc() expects parameter 1 to be resource, boolean given in /Users/laurent.baresse/Documents/workspaces/workspaces/workspace_RO/LocalDeploy/resto2/include/resto/Modules/Gazetteer.php on line 490

Query Analyzer : add "on" as a synonym of "for" to quantityModifier

E.g. "on coastal area" - by the way add "area" to noisy words

Typo in french keywords data

Some french words are noted 'en' instead of 'fr' within generalKeywords.sql file

Administration module requires that getRights(...) function returns null instead of array of null values

To ease administration module, Function_rights/getFullRights and Function_rights/getRights should return null instead of array with null values

Move from license per collection to license per products

Currently license are per collection (i.e. each product within a collection share the same license).
However, in the frame of Recovery Observatory project, one collection should contain products with different licences.
This update implies :

• to add a new "licences" table that contains licence id, licence urls (json), licence signing type (i.e. once, always, never)
• to add a licence column within resto.features referencing licence id
• to apply collection licence to product when no licence is defined within a product

Replace all 'resto2' occurence by 'resto'

QueryAnalyzer : support for "vers" location keyword (french)

Sentences like "vers Toulouse" are note understood. "vers" should be added to the french locationModifiers keywords as a synonym of "in"

Query Analyzer : WhenProcessor clean seasonToInterval() function

This function contains duplicated code

deleteCollection.sh : typo in usage text

The usage text is ambiguous

Query Analyzer : "to" should be a noisy word

Add "to" to noise words otherwise it is detected as a toponym in last pass

QueryAnalyzer : "before next year" and "after next year" are incorrectly handled

Not easy to solve...

Query Analyzer : do not search for location if only a stop word remains

Assume the query "In the town of Lota". Analyzer detects two locations "in the town" and "of Lota".
The first search for "the-town" with no result and then "the" which appears to be a toponym in Pakistan.
General handling should be : never search location if remaining toponym name is a stop word

Query Analyzer : year:XXX to be replaced by time:start/time:end if no season

For consistency, year:XXXX should be replaced by time:start/time:end if no season:YYYY keyword is present

Add support to new iTag 'keywords' property

The keywords property in iTag v3.0 indicates :

• the season of a product (from timestamp)
• the relative location on earth i.e. northern, southern, tropical, equatorial, coastal

Query Analyzer : WhenProcessor to return an array of array to support multiple time block

Currently WhenProcessor returns an array of properties - thus it cannot support multiple times query like "between may and june 2014 and between september and november 2015".
When should return an array of array in this case

Query Analyzer : "of something" not detected

Add 'of' as a synonym of 'with' in quantityModifier

Query Analyzer : add support to "austral summer/winter/autumn/spring"

Incorrect error message when deleting non existing collection

When DELETE a non existing collection, resto returns a {"ErrorMessage":"instantiate - Class name is not set","ErrorCode":500} instead of a 404 Not Found

JWT - replace SignatureInvalidException, BeforeValidException and ExpiredException by UnexpectedValueException

Third party library JWT uses non standard exception - to be replaced by UnexpectedValueException instead of importing these non standard exception

Upgrade to iTag v3.0

Service users/{userid}/cart/{itemid} always returns HTTP 404

Typo in https://github.com/jjrom/resto2/blob/master/include/resto/Routes/RestoRoutePUT.php#L137 : assertion is always false which to a 404 error everytime !

Query Analyzer : "de Paris" does not detect location

Location is not detected with "de" ('in' equivalent) modifier. Apparently this is a regression introduced by previous commit

Remove old code from deploy.sh

The file deploy.sh contains references to old code which result in (non blockin) errors message

QueryAnalyzer : discard first stop word in location detection

During location detection process, if first word is a stop word discard it unless it is a part of the location name (e.g. "The Hague"). For instance in french "near Toulouse" would be "près (de) Toulouse". In this case "de" should be discarded (i.e. stop word). However in "near the Hague", "the" should not be discarded since it is a part of the city name.

Missing '/' in download link

The download link is missing a '/' between the collection name and the feature identifier (same problem for feature links)

PostgreSQLDatabaseDriver - fetch() function to return empty array instead of false when no result

Query Analyzer : "with something" should be discarded

When searching for "with something" and "something" is not a valid keyword, then words should be marked as processed and NOT_UNDERSTOOD.
This is not the case currently - as a consequence "something" is processed afterward by the WhereProcessor leading to incorrect results

Upgrade to upcoming iTag v3.0

Modify iTag module to handle iTag v3.0RC1+ new interface

Add detection of productType in QueryAnalyzer

Only instrument and platform are supported by QueryAnalyzer - should add productType

Endpoint /collections/:collection/:something.json returns /collections/:collection.json

The endpoint /collections/:collection/:something.json returns the collection description. It should returns an HTTP 404 error instead

Query Analyzer : add support to northern, southern, tropical and equatorial keywords

With the upgrade to iTag 3.0 relative location are stored in the database

Query Analyzer : "last year" not detected if followed by a non modifier word

"last year" is detected unless a non modifier word directly follows the sentence (e.g. "last year and next month")

Replace sha1(...) calls by a dedicated hash function

The sha1(...) functions is called in several part within the code. Proposal to move this function to RestoUtil so it could be easily upgraded to other hashing algorithms

Query Analyzer : "all" to be added to list of stop words

For instance "Search for all imagery" is incorrectly processed because "for all imagery" is understood as "for all" (location) and "imagery" (keyword).

Feature 'links' properties contains consecutive / characters

Consecutive '/' character is found between resto endpoint and 'collections'. For instance http://localhost/resto//collections/...

Query Analyzer : (French) need to update dictionary

The sentence "Images de cyclone durant l'été austral sur la zone côtière près de Sydney" is not completely understood (missing "cyclone", "austral", "zone côtière")

Support of JSON Web Tokens encrypted with other methods than SH256

Currently only SH256 encrypted JSON Web Tokens are supported (this is hardcoded in the call to JWT::decode within RestoContext decode function). It would be better to support a list of encryption methods stored within the config.php file instead.

Gazetteer - set simplification precision in constant

Set 3 different precisions for polygon simplification(i.e. for continents, countries and states)

Keywords composed by multiple words are not detected

Refactor of QueryAnalyzer in 2.0RC1 broke the detection of multiple word keywords. For instance, in the sentence "Volcano eruption in Villarrica last month" - "Volcano eruption" is not detected. As a consequence, "Volcano" is the location and Villarica is discared without being detected

resto needs at least PostGIS 2.0

Update README to tell so

Gazetteer - search for states when country is set

Search on "texas" would propose both toponyms and state. Search on "texas, usa" only returns toponyms and no more state.

Query Analyzer : location not detected a duration is following

QueryAnalyzer : (french) "avec tremblement de terre" vs "tremblement de terre"

Both queries are detected as "earthquake" but the latter indicated "de terre" as NOT_UNDERSTOOD which is obviously not the case

Query Analyzer : "since year" incorrectly handled

Problem in WhenExtractor / extractDuration() function - it should return null when duration is not valid (currently it returns an incomplete duration)

Query Analyzer : (French) tune quantityModifier

Add 'dans' as a quantity modifier