Comments (7)
I fully agree. This makes things more complicated than they should be and leaves too much room for errors or different interpretations. There should be one obvious way to do things and not multiple competing ones. The order of the dimensions defined in the DSD (as represented in the structure part of the message) should be sufficient.
from sdmx-json.
Dear Leo, we are sorry for the late follow-up. The keyPosition indicates the position of the dimension in the SDMX "key" that is used in the rest URL syntax. This SDMX "key"does by definition never include the TIME_PERIOD dimension. Therefore the keyPosition wasn't given for that dimension in the examples.
I also agree with the unnecessary ambiguity.
Xavier, was your conclusion that the keyPosition should be completely removed from the standard or be
made obligatory for dimensions and forbidden for attributes? The trouble of removing would be that the order (position) information is lost when dimensions are split over dataset, series and observation nodes.
from sdmx-json.
I have mixed feelings about this.
On the one hand, I understand the use case, i.e. dimensions can be split and so you have no way to know the order unless you have the DSD.
On the other hand:
- You get the SDMX-JSON file via a RESTful data query, which means you most likely know the right dimension order already, else you would not have been able to retrieve the data in the first place.
- In addition, in a streaming scenario, it's unlikely that the web service provider will be able to make this kind of clever optimisation, such as, for example, attaching some dimensions to the dataset level. That means that, in practice, I would expect most implementers to return all the dimensions at the series level (except the special dimensionAtObservation of course), unless the client request a flat representation of the data, in which case this issue of splitting dimensions would not exist anyway, as they would all be attached to the observations.
So, in my opinion, we could live without a keyPosition, at least in the first release of the spec (i.e. until it has been proven that enough implementers can't live without), but I don't have a strong preference.
However, in any case, indicating the position of attributes makes no sense, as there is no guarantee of their order in any SDMX spec, AFAIK.
from sdmx-json.
from sdmx-json.
Hi Leo,
Thanks for your very useful and clear comments, as always!
I disagree with a few statements but, overall, let me reiterate that I'm fine with your proposal re. the keyPosition for dimensions, even more so when providers of great libraries like you have a clear preference :).
The few statements with which I disagree are:
- The standard does not enforce dimension order at any level. I would say that this is then a documentation issue that we need to address because series keys, heavily used by many SDMX implementers, wouldn't work if the order was random.
- Regarding the obvious optimization that you mentioned, I will confess it was not implemented in the ECB web service, as far as I can remember. And, to be fair, I'm not convinced it would have made much of a difference, especially with compressed data messages, at least for service providers. For service consumers, I suspect the opinions might be split, with some in favour (as you mentioned, it's a nice implementation) and some against (they would prefer not to bother and simply read sequentially).
But as you can see from the comments above, these are minor indeed and I'm fine with your proposal, as stated at the beginning.
from sdmx-json.
Dear Jens and Xavier,
let me clarify that contrary to Jens's initial comment TimeDimension does have a
key position attribute. TimeDimension inherits from Dimension the only specialty being that its role is always None.
This confirms that in the json format key position should be mandated for every Dimension including TimeDimension.
The following code example shows the key positions of all dimensions of an arbitrary
DSD taken from the ECB:
from pandasdmx import *
resp = Request('ecb').datastructure('ECB_EXR1')
dsd = resp.datastructure.ECB_EXR1
{id : dim._position for id, dim in dsd.dimensions.items()}
{'CURRENCY': 2,
'CURRENCY_DENOM': 3,
'EXR_SUFFIX': 5,
'EXR_TYPE': 4,
'FREQ': 1,
'TIME_PERIOD': 6}
I stay away from any politics, such as whether the TWG is dominated by representatives of data providers or any other constituency
who may be biased
by how they have prototyped their sdmx-json API. From a client implementor's perspective I am naturally interested in
the most rigorous standard so as to avoid ambiguities and divergent implementations across agencies.
Leo
from sdmx-json.
The SDMX-TWG decided: keyPosition becomes mandatory for all dimensions including special dimensions such as the Time dimension. It returns the information on the order of dimensions in the Data Structure Definition. It is not to be used for attributes.
Documentation, examples and schema have been updated.
from sdmx-json.
Related Issues (20)
- Proposal to reference DSD from data message via `agencyID` and `version` HOT 4
- Coded TIME_PERIOD?
- `contentLanguages` disallows most non-lowercase language identifiers HOT 1
- `timeDimension.localRepresentation` should be optional
- `name` and `description` on `$.data.structures[*]`
- `valueArray` schema requires `oneOf` `integer` xor `number`
- Bad schema for `dimValues` HOT 1
- Invalid regex pattern for `StandardTimePeriodType`
- Use `$defs` instead of `$.definitions` in JSON Schemas.
- Unclear representation of metadata in data messages
- Attribute relationship vs attachment level
- Dimension group relationships no longer resolved
- Confusing ID inheritance remark
- Missing `validFrom`/`validTo` properties on `TimeRangeValue`
- How are `s:GenericDataflow` elements represented?
- Deprecate using artefact types as link relations
- Time and time zone requirements vs SDMX-ML
- `meta` and `errors` should be annotable
- If `data` and `errors` are mutually exclusive, then where do warnings go?
- Possible inconsistency regarding coded attributes/measures with array values HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sdmx-json.