This repository is used for maintaining the SDMX RESTful web services specification.
This repository is used to maintain the SDMX REST API.
The API allows implementers to offer programmatic access to statistical data and metadata over HTTP.
This repository contains:
From the list of features expected for SDMX 3.0: Deals with web server clients requesting for specific language match in the RESTful API.
Note: Is this more than simply documentation how Content-Negotiation can be used for language resources?
Handling of the start and end periods might depend on the values of the collection attribute (such as beginning, middle, average or end of period).
This is not reflected in the documentation at the moment, but it would be valuable information for implementers.
Please add clear specifications for usage of HTTP verbs POST, PUT, DELETE, PATCH with restful SDMX API to allow managing artefacts and data in a coherent manner. An interesting article (in French) is here: http://blog.xebia.fr/2014/03/17/post-vs-put-la-confusion/
The examples at https://sdw-wsrest.ecb.europa.eu/ use this updated endpoint.
I know the a-sdw-wsrest.ecb.int end-point was intended as a test one.
However, as per the end of this thread: amattioc/SDMX#30
... it can lead to some confusion, as the test end-point has some bogus data to illustrate the updatedAfter and includeHistory parametres.
Proposed changes (in bold):
Additional parameter used for identifying a resource, for item scheme types:
SDMX uses the item scheme pattern to model SDMX collections of items. These are:
For these collections, it is possible to use a 4th parameter for identifying resources. The rules for the 3 other parameters, as defined in the section above, remain valid.
Parameter Type Description
itemList A concatenation of strings compliant with the SDMX common:NestedNCNameIDType for conceptscheme and agencyscheme, or with the SDMX common:NestedIDType in all other cases. The OR operator is supported using the + character. For example, the following string can be used to retrieve the codes for Germany, United States and Japan from REF_AREA codelist: DE+US.JP. The ids of the items to be returned.
This 4th parameter is used as follows:
protocol://ws-entry-point/resource/agencyID/resourceID/version/itemID+itemID
SDMX web services sometimes need to throw an exception because a syntactically valid request is not allowed within a particular context. For example, a registry client could send a request to update a final artefact, which is an operation that is not allowed by the standard.
The best HTTP status code in this case seems to be 403. As defined in the HTTP specification, 403 means that “the server understood the request, but is refusing to fulfill it. Authorization will not help and the request SHOULD NOT be repeated. If the request method was not HEAD and the server wishes to make public why the request has not been fulfilled, it SHOULD describe the reason for the refusal in the entity”.
If it is agreed to support this, there are 2 options:
When querying for data, a client has the possibility to indicate that it wants to retrieve what has changed since a certain point in time.
It would be nice to add a similar feature to metadata queries.
The fact that metadata messages do not have the equivalent of a dataset action might be an issue, though.
The RESTful API offers the possibility to supply default values for some of the query fields, such as ‘latest’ (to get the latest version currently in production) and ‘all’. Initially, these keywords were in uppercase but, following the feedback received, it was decided to opt for lowercase instead. Unfortunately, some occurrences of the uppercase variants remain on pages 19, 20 and 23 of the SDMX web services guidelines.
Please provide approach/solution for huge data selections when code selection exceeds maximum URL length.
Following several requests, it has been proposed to add a section about web services best practices. The section should contain useful tips and tricks for both data providers and data consumers.
Hi,
I was going through the SDMX RESTfull error handling wiki page and stumbled upon this line
Semantic error (403): If your request is syntactically correct but fails a semantic validation, a 403 code will be returned.
which is confusing taking into account what www.w3.org says about the HTTP 403 code here and what SDMX 2.1 Web Services Guidelines state in Paragraph 5.8 SDMX to HTTP Error Mapping
So, my questions are:
Semantic error (403) just a misprint here: https://github.com/sdmx-twg/sdmx-rest/wiki/Error-handling? I guess that it should be Semantic error (400).Thank you,
Denis
The regular expression used for KeyType (SDMXRestTypes.xsd) marks some valid queries as invalid (such as this one: M+D..EUR.SP00.A) while also failing to catch some invalid queries (such as: B.U2.EUR.RT.MM.EURIBOR1MD_.HST&startPeriod=2011-03-21).
Replace
([A-z0-9_@$-]+)(.(([A-z0-9_@$-]+)(+([A-z0-9_@$-]+)))?)
with
([A-Za-z0-9_@$-]+([+][A-Za-z0-9_@$-]+)_)?(.?)*
in SDMXRestTypes.xsd
It is proposed to group all structural metadata queries under a structure resource to ease the understanding of the API. After this has been done, the API would offer only 4 main resources:
data: to retrieve statistical datametadata: to retrieve reference metadataschema: to retrieve schemas that can be used to validate datastructure: to retrieve structural metadataThe new structure query would be as follow:
resource/type/agency/id/version (and of course resource/type/agency/id/version/item, for item scheme queries), where:
structure)agency/id/version) would not change compared to the current versionall keyword (in addition, version also accept latest)So, for example, to get the latest version of the SDMX CL_FREQ codelist, one would do:
https://entry-point/structure/codelist/SDMX/CL_FREQ/latest (latest could be omitted as it is the default value if not supplied)
To get the latest version of all structures maintained by SDMX, use the following:
https://entry-point/structure/all/SDMX/all/latest or simply https://entry-point/structure/all/SDMX, as all and latest are the default values for the id and version path parameters respectively.
Could an example of the Accept header for SDMX-JSON format (application/vnd.sdmx.data+json) please be added to the 4_6_conneg.md?
Resolution of the constraints on the server side for big datasets that a client would not be able to process, for which he provides a restriction for one or more dimension (CubeRegion) and the service would then return the valid set of codes (partial codelist) from the other dimensions.
While there are many examples of how to use the /data resource, I could not find any about the /metadata. The SDMX Web Services Guidelines has only information and examples for /data (see section 4.4.3).
Please redirect me to any resources that would explain how to retrieve reference metadata using REST and especially how to use the ‘key’ parameter as defined by the wadl.
Thanks.
In order to allow easily retrieving referential metadata attached to any structural artefact or to data, it is proposed to extend the Rest API in the following way (this extension was already implemented by MT):
Prefixe any data and structure query with the path parameter 'metadata', e.g.:
protocol://ws-entry-point/codelist/IMF/CL_FREQ - returns codelist Frequency
protocol://ws-entry-point/metadata/codelist/IMF/CL_FREQ - returns any reference metadata for the codelist Frequency (or any codes contained in the Codelist)
protocol://ws-entry-point/data/BIS,EXR,1.0/N.AR. - returns exchange rate dataset
protocol://ws-entry-point/metadata/data/BIS,EXR,1.0/N..AR. - returns the referential metadata for the dataset and for any series or observations contained in the dataset
This new syntax should replace the current more restrictive syntax:
protocol://ws-entry-point/metadata/flowRef/key/providerRef
VTL defines a few additional MaintainableArtefacts (e.g. TransformationScheme, RulesetScheme, etc.). They should be added as new resource types to the documentation and the OpenAPI definition.
The current documentation for data query parameter updateAfter could improved to specify how the response message should be. And maybe an example.
Some example documentation (if it is correct...)
Some sample example (provided the above is correct):
<url>rest/data/ESTAT,SSTSCONS_PROD_A,1.0/....NS0040../ALL/?detail=full&dimensionAtObservation=TIME_PERIOD&updatedAfter=2015-04-01T14%3A15%3A30
<?xml version="1.0" encoding="utf-8"?>
<message:StructureSpecificData xmlns:ss="http://www.sdmx.org/resources/sdmxml/schemas/v2_1/data/structurespecific" xmlns:footer="http://www.sdmx.org/resources/sdmxml/schemas/v2_1/message/footer" xmlns:ns1="urn:sdmx:org.sdmx.infomodel.datastructure.DataStructure=ESTAT:STS(2.0):ObsLevelDim:TIME_PERIOD" xmlns:message="http://www.sdmx.org/resources/sdmxml/schemas/v2_1/message" xmlns:common="http://www.sdmx.org/resources/sdmxml/schemas/v2_1/common" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xml="http://www.w3.org/XML/1998/namespace">
<message:Header>
<message:ID>IDTEST</message:ID>
<message:Test>false</message:Test>
<message:Prepared>2015-12-17T14:11:48</message:Prepared>
<message:Sender id="ZZ9" />
<message:Structure structureID="ESTAT_STS_2_0" namespace="urn:sdmx:org.sdmx.infomodel.datastructure.DataStructure=ESTAT:STS(2.0):ObsLevelDim:TIME_PERIOD" dimensionAtObservation="TIME_PERIOD">
<common:Structure>
<Ref agencyID="ESTAT" id="STS" version="2.0" />
</common:Structure>
</message:Structure>
</message:Header>
<message:DataSet action="Replace" ss:dataScope="DataStructure" xsi:type="ns1:DataSetType" ss:structureRef="ESTAT_STS_2_0">
<Series FREQ="M" REF_AREA="DE" ADJUSTMENT="N" STS_INDICATOR="PROD" STS_ACTIVITY="NS0040" STS_INSTITUTION="1" STS_BASE_YEAR="2000" TIME_FORMAT="P1M">
<Obs TIME_PERIOD="2012-02" OBS_VALUE="0" OBS_STATUS="A" />
<Obs TIME_PERIOD="2012-03" OBS_VALUE="1" OBS_STATUS="A" />
<Obs TIME_PERIOD="2012-04" OBS_VALUE="2" OBS_STATUS="A" />
<Obs TIME_PERIOD="2012-05" OBS_VALUE="3" OBS_STATUS="A" />
<Obs TIME_PERIOD="2012-06" OBS_VALUE="4" OBS_STATUS="A" />
</Series>
</message:DataSet>
<message:DataSet action="Delete" ss:dataScope="DataStructure" xsi:type="ns1:DataSetType" ss:structureRef="ESTAT_STS_2_0">
<Series FREQ="M" REF_AREA="DE" ADJUSTMENT="N" STS_INDICATOR="PROD" STS_ACTIVITY="NS0040" STS_INSTITUTION="1" STS_BASE_YEAR="2000">
<Obs TIME_PERIOD="2006-02" OBS_VALUE="NaN" />
<Obs TIME_PERIOD="2006-03" OBS_VALUE="NaN" />
<Obs TIME_PERIOD="2006-04" OBS_VALUE="NaN" />
<Obs TIME_PERIOD="2006-05" OBS_VALUE="NaN" />
<Obs TIME_PERIOD="2006-06" OBS_VALUE="NaN" />
</Series>
</message:DataSet>
</message:StructureSpecificData>The schema queries return XML schemas (xsd) as output.
However, now that SDMX offers support for new formats such as JSON and CSV, we could also return schemas for these formats, i.e. JSON Schema or W3C Metadata Vocabulary for Tabular Data.
In principle, the service could also support returning schemas for non-SDMX formats such as AVRO schemas for example.
There is a documentation issue with the section related to the applicability and meaning of the reference attribute (for metadata queries). Some of the artefacts to be returned are missing from the table. For example, in case the matching artefact is a codelist, the table only mentions hierarchical codelists in the column for references, but other types would be valid as well (such as Categorisation, Process, ConceptScheme, DataStructureDefinition, MetadataStructureDefinition and StructureSet).
Furthemore, it would be nice to distinguish between the artefacts referenced by the matching artefact and the artefacts that reference the matching artefact.
The RESTful API offers the possibility to clients to ask the service to package the observations in 3 different ways: as time series, as cross-section or as a flat list of observations. This is achieved using the dimensionAtObservation parameter. However, the documentation does not precise what the default should be if the client does not use this parameter.
It is proposed to :
Should we propose a way to version the API? It's considered good practice, and it brings additional guarantees to clients.
Considering SDMX adoption of semantic versioning, we could limit the versioning to the MAJOR segment of the version (e.g. v1, v2, v3, etc.) as MINOR and PATCH changes should not have an impact on clients.
Review the information returned with detail parameter (full, stubs, ) for both Soap and Rest which are today different (full, stubs, completestubs, referencestubs, allstubs). There is a general need to return more information with stubs than currently specified (e.g. isFinal, structureURL, urn, isExternalReference and some times also annotations (with CategoryScheme_node_order)). A mean to get descriptions, isFinal and annotations in Rest without the Items of the ItemScheme is needed.
The SDMX RESTful API currently features a detail parameter. When references are requested for a structural metadata query, all available information for these references will be returned, unless the value referencestubs is used.
SDMX item schemes (codelists, concept schemes, category schemes, etc.) feature a isPartial attribute, but this functionality is currently not used in the RESTful API. It would be nice to add a referencepartial (or something along these lines) to the detail parameter, so that only items relevant in a certain context get returned.
Let's take for example a query for a DSD and its children (i.e. codelists and concept schemes). When using full (the default), all concepts contained in the concept schemes will be returned, including those not used in the DSD. When using referencestubs, no concept will be returned. If we add referencepartial, we could retrieve the concepts used in the DSD and only those, i.e. only the ones relevant to the context of the query. It would be the same for other item schemes, such as when retrieving the codelists referenced by hierarchical codelists or content constraints.
Some short introduction to the issue. We (SDMX RI ) got a request for specifying the format/version from the URL and at the same time we see different solutions/approaches from various organisations e.g. from ISTAT (some kind of wrapper Service to StructureSpecificData) and Global Registry (version parameter).
Based on the above I assume there are use cases where setting the HTTP header Accept is not easy or not possible. Example on browsers without a REST plugin or using software that doesn't support this and cannot be easily modified.
In order to avoid having different custom solutions for specifying format/version without a HTTP Accept header, it might be better to have a common approach, i.e. one or two new parameters for data and structure requests in SDMX REST.
For example:
| Parameter | Description |
|---|---|
| format | Whatever [format].[xml/json/whatever] is supported via Accept. (without the mime parts and version) e.g. format=structurespecificdata.xml |
| version | SDMX version, whatever is supported via Accept [version] |
The error page contains information that might be more relevant for the SOAP API. So, there might be some opportunity to simplify that part of the spec.
The SDMX WADL file makes use of the WADL resource_type construct. This is very useful in order to define common behaviour that can then be inherited by various resources. This is used in the SDMX WADL file in order to define the parameters common to all types of metadata resources.
Unfortunately, various testing tools don't support resource_type.
In the WADL file the type of the agencyID parameter is NCNameIDType. This does not allow dots in the agencyID. However agency ids may optionally contain dots (e.g. SDMX.SECRETARIAT.TWG is a valid agency id).
Change the type of agencyID parameter to NestedNCNameIDType.
Currently the only SDMX Artifact that can not be retrieved via REST is the Data Registration. This will be useful for applicaitons that wish to have knowledge of where the data is located, enabling applications to browse the metadata, and then get related datasets via the Registration message.
A proposed solution is to be able to retrieve registraitons by
Agency/Id
Dataflow / Provision Agreement
Data Provider
For input to this discussion, we have added support into the Fusion Registry, please refer to section 3.6 in the following document
@egreising proposed to check whether referencing of structural metadata could be harmonized.
At the moment, we have the following:
There are reasons why these are handled differently but it can nevertheless be confusing.
Please add a mechanism to allow for filtering of data by attribute values.
The definition of the WS API should be updated into a newer format.
Potential candidates are:
A potential outcome or distributed would be the equivalent set of Postman collections that could be used to test the Web Services. Another would be a Swagger or RAML documentation web site.
In 4_7_errors.md, shouldn't the HTTP status code for the SDMX error 150 "Semantic error" not be "422 Unprocessable Entity" (rather than currently "403 Forbidden")?
The SDMX description is:
Semantic error - 150
A web service should return this error when a request is
syntactically correct but fails a semantic validation or
violates agreed business rules.
See https://tools.ietf.org/html/rfc4918#section-11.2 :
The 422 (Unprocessable Entity) status code means the server
understands the content type of the request entity (hence a
415(Unsupported Media Type) status code is inappropriate), and the
syntax of the request entity is correct (thus a 400 (Bad Request)
status code is inappropriate) but was unable to process the contained
instructions. For example, this error condition may occur if an XML
request body contains well-formed (i.e., syntactically correct), but
semantically erroneous, XML instructions."
A sub-question:
How should the server behave if a data query contains valid and invalid codes in the key parameter? Should the service return the available data only for the valid codes and ignore the others, or refuse the execution and return an SDMX 150 error? This would be worthwhile to be clarified in the standard so that client applications can expect a consistent behaviour.
The Wiki includes a list of providers of SDMX 2.1 RESTful web services: https://github.com/sdmx-twg/sdmx-rest/wiki/Where-is-it-used
This list could be improved by adding a table listing the RESTful API features as well as the SDMX formats supported by each service.
Many regular expressions in the SDMX-ML schemas use ‘A-z0-9’, as shortcut for lowercase and uppercase letters, and numbers. However, this is incorrect and will in fact allow invalid characters, such as ‘[‘. For example, according to the schemas, ‘A[‘ is a valid IDType, but it should not (the only allowed characters are ‘A-Z, a-z, @, 0-9, _, -, $’). The correct regular expression is: a-zA-Z0-9.
Hi There.
I'm trying to get data from the Rest API, with your example as it exists in your docs.
The call is GET https://sdw-wsrest.ecb.europa.eu/service/data/EXR/D.USD.EUR.SP00.A?startPeriod=2009-05-01&endPeriod=2009-05-31
The Rest API return internal error (status code 500) without any error description.
Please fix it.
Thanks!
The SDMX 2.1 RESTful web services currently allows to:
HTTP If-Modified-Since header);updatedAfter parameter);includeHistory parameter).However, it does not offer a direct way to get the data as they were at a certain point in time (there is an indirect way, using the includeHistory parameter but the work is then left to the client).
A new parameter (e.g. snapshot) could be added, where a timestamp can be supplied and the web service would then return the data matching the query as they were at that point in time (for example, as they were when a report was published).
A SDMX Content Constraint can have 2 distinct flavours – actual and allowable. Each flavour can fall into one of two categories (Keyset of Cube Region).
In a way a Content Constraint should be at least 2 different maintainable objects, which can be independently retrieved, but if they remain as one object type in the next version of the standard, it would be useful at the API level to be able to distinguish between them.
In the same way the REST API allows the user to query for organisationscheme (which brings back agency, data provider, and data consumer schemes). It would be nice to add two new resource parameters to the API:
allowableconstraint
actualconstraint
It is interesting to note that the actualconstraint may even be a bit redundant considering other requests (e.g. #40). The actual constraint says what data actually exists, this could be a keyset (list of series) or cube region. However for the former (keyset) it is an incredibly verbose response, a much less verbose (and more flexible) client to server query is for the all data with detail=seriesKeysOnly. The second query, cube region, may lose importance when bug #40 is resolved. So for this purpose the Fusion Registry has already re-purposed the actualconstraint as a way to create sub-cube (pre-defined query) definitions - a use case we have been asked to support on many occasions. The actual constraint is a useful mechanism for the client to build pre-selections for large dataflows, and hence the driver for us, to be able to distinguish between constraint used to restrict content for submission, to those that are used for dissemination.
How should the SDMX server respond when the request contains both valid and invalid SDMX (identifiable) artefact IDs?
Examples ('A' is a valid ID, 'AA' is not a valid ID):
Should the server ignore the semantically invalid IDs and return only the resources for the valid IDs, or should the server refuse to serve the whole request by returning an 422 error?
The SDMX-JSON structure message will soon be released for public comments. Once it is approved, we will need to add a media type for it.
Decision has been taken to migrate the documentation to readthedocs.org.
We should take this opportunity to consolidate and improve the current documentation.
Hi!
I have written some sdmx visualisation software to put SDMX data on a map, and one of the things I do is instead of using a drop down menu to select the time, I walk the tree of the data set, and pick out the latest time value, it seems a shame that i have to specify a time period range (for example 2008-2014) and then for each full key, I take only the latest observation for that country and display that on the map... and then discard the rest of the observations...
its not important that I get the latest observation for each full key, I just thought that it would be more efficient to be able to query for the latest observation rather than discarding the previous observations....
this is a minor issue =D
Cheers
JG
Currently SOAP supports Categorisation query parameters on the Categorisation TargetCategory via the CategorisationWhere type but REST does not. This feature would allow a REST API consumer to benefit from the power of the Categorisation in SDMX. Many times the CategoryScheme is used as a base discovery structure and from a chosen member category the navigation will branch to a search. In REST this is not possible. This is a major limitation, as it impacts the clients willing to browse data, using a category scheme.
A path parameter could be added to the metadata queries of the RESTful API. This parameter would accept the id of the item, in an item scheme, to be returned.
For example, currently, the following syntax can be used to retrieve a category scheme:
https://ws-entry-point/categoryscheme/AGENCY_ID/SCHEME_ID/VERSION
In order to retrieve only a particular category (e.g.: ID = XYZ), the following syntax could be used:
https://ws-entry-point/categoryscheme/AGENCY_ID/SCHEME_ID/VERSION/XYZ
Nested ids should be supported:
https://ws-entry-point/categoryscheme/AGENCY_ID/SCHEME_ID/VERSION/XYZ.A.B
When the references parameter is not used, the returned item scheme should only contain the matching item (including the parents in case a nested id is used). The isPartial attribute should be used and its value should be true.
When the references parameter is used, artefacts using/used by the matching item should be returned.
For example, the following would retrieve, along with the matching categoryscheme/category, all the categorisations categorising the matching category:
https://ws-entry-point/categoryscheme/AGENCY_ID/SCHEME_ID/VERSION/XYZ?references=categorisation
As another example, the following would allow to retrieve all the DSDs referencing the supplied concept:
https://ws-entry-point/conceptscheme/SDMX/CROSS_DOMAIN_CONCEPTS/1.0/FREQ?references=datastructure
In many use cases, knowing in advance the number of observations to be delivered by a data query may be very useful as it allow you to warn the user about "no data available" or excessive information (e.g. greater than a certain threshold) situations without having to wait for the actual data query to be completed.
It worth noting that in most databases, the query for "count" returns the amount of tuples in an invariant time, regardless of the amount of data.
The proposal is to add a new possible value "count" for the "detail=" parameter in the data query. When "detail=count" is specified, the number of observations to be returned by the query is returned.
For the returned message, the simplest way would be to return just the number of observations, regardless of the message format in effect. This has the advantage of being format independent and, as a matter of fact, if somebody is querying for the count, nothing else really matters. Besides, it will be faster, as the writer has no need to format a message. Especially in JSON, there's no need to query for the structural metadata to include.
Otherwise, if the message structure will be kept for the information returned, the count would require the definition of new optional elements in the message header of XML and in the meta of JSON formats to place the count, and some way should be agreed for the CSV data message to return the count.
TF3 (Message formats) should advise on these points.
Since version 1.1.0 of the SDMX RESTful API, it is possible to retrieve individual items within item schemes (such as a particular concept within a concept scheme). This becomes particularly powerful when used in combination with the references parameter, for example to see where a particular concept or code is used.
The functionality described above applies only to item schemes, but there are other types of collections in SDMX, such as hierarchical codelists and structure sets. It would be nice to extend the feature to these types of collections as well, so as to be able to retrieve, for example, a particular hierarchy within an hierarchical codelist.
It can be a requirement for some SDMX web services to return previous versions of the data, as they were disseminated in the past (“history” or “timeline” functionality). Although it is possible to represent the result set in one SDMX-ML data message, the current version of the SDMX RESTful API does not offer a parameter to activate this functionality.
It is proposed to add a query string parameter, named includeHistory, which accept boolean values and defaults to false.
The returned SDMX-ML data message should contain one or two datasets per data dissemination, depending on whether a dissemination also deleted observations from the data warehouse.
For example, let’s say that, for a particular series, there were, so far, 3 disseminations:
In this scenario, it is proposed that the web service returns one data message containing 4 datasets:
This proposal affects section 07 of the SDMX 2.1 Documentation package and the accompanying wadl file.
Section 4.8 of the SDMX RESTful specification documents the various error codes that implementers can return and maps them to their respective HTTP status codes.
Informational status codes (such as 200 -OK- or 304 -Not modified-) could also be added, for documentation purposes, as clients are expected to be able to handle these as well.
The Content-Negotiation section of the documentation contains information about the HTTP Accept and Accept-Encoding headers. However, there are other useful HTTP headers (such as If-Modified-Since, Accept-Language, Vary, etc.,) and these are not documented. These headers are not SDMX-specific of course, but it could nevertheless be useful for implementers if we at least mention these in the documentation.
Today, the SDMX Rest ws standard foresees one way for restricting number of observations through the firstNObservations and lastNObservations URL parameters. But those apply individually to each "series" and not to the whole data message, and are thus not useful when the client needs to restrict the message size, or for chunking the request into smaller pieces that are better manageable by the client.
A nice solution would be provided by the HTTP protocol that includes a way to request only a specific range of data. The related RFC 7233 standard says:
The "Range" header field on a GET request modifies the method semantics to request transfer of only one or more subranges of the selected representation data, rather than the entire selected representation data.
It works in the following way:
The server indicates with the HTTP header field "Accept-Ranges" that it supports range requests for the target resource, and which types of range units, e.g. bytes.
Accept-Ranges: bytes
The client uses the "Range" header field in the GET request to indicate the requested subrange, e.g. for the first 1000 bytes (with 0-based indexes):
Range: bytes=0-999
And for the next 1000 bytes:
Range: bytes=1000-1999
The server, if the range request is admissible, returns the 206 (Partial Content) status code that indicates that the server is successfully fulfilling a range request for the target resource by transferring one or more parts of the selected representation that correspond to the satisfiable ranges found in the request's "Range" header field. It also returns a "Content-Range" header field, describing what range of the selected representation is enclosed, e.g.
Content-Range: bytes 0-999/1234
Note: 1234 is the total number of bytes for the complete resource, or
Content-Range: bytes 0-999/*
Note: * is indicating that the total number of bytes for the complete resource is unknown by the server, or
Content-Range: bytes 0-566/567
Alternatively, the server generates a 416 (Range Not Satisfiable) response and a "Content-Range" header field with an unsatisfied-range value "*", e.g.
Content-Range: bytes */1234
Also see here and here, for examples of requests for multiple sub-ranges.
Since the bytes unit is inappropriate for our purposes, in .Stat Suite we have tested the HTTP Range header approach with another range unit: values. While this works fine in most environments, some users have reported issues in AWS hosting scenarios. Indeed, some cloud hosting services do not allow using range units that have not been registered with IANA,
We therefore decided to use a non-standard/proprietary X-Range HTTP header instead of Range. This worked in all environments. The specification we have used is as follow:
The server indicates with the HTTP header field "Accept-Ranges" that it supports range requests for the target resource and the range unit values.:
Accept-Ranges: values
The client uses the "X-Range" header field in the GET request to indicate the subrange of the selected representation data, e.g. for the first 1000 SDMX observation values:
X-Range: values=0-999
And for the next 1000 values:
X-Range: values=1000-1999
The server, if the range request is admissible, returns the 206 (Partial Content) status code that indicates that the server is successfully fulfilling a range request for the target resource by transferring one or more parts of the selected representation that correspond to the satisfiable ranges found in the request's "X-Range" header field. It also returns a "Content-Range" header field, describing what range of the selected representation is enclosed, e.g.
Content-Range: values 0-999/1234
Note: 1234 is the total number of values for the complete resource, or
Content-Range: values 0-999/*
Note: * is indicating that the total number of values for the complete resource is unknown by the server, or
Content-Range: values 0-566/567
Alternatively, the server generates a 416 (Range Not Satisfiable) response and a "Content-Range" header field with an unsatisfied-range value "*", e.g.
Content-Range: values */1234
This ticket should be address together with ticket Allow ordering the data query results by dimensions, since slicing/pagination makes only really sense if the order of the observations is deterministic.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.