How to determine the dimension order when keyPosition is missing for a particular dimension? about sdmx-json

[sosna]

I fully agree. This makes things more complicated than they should be and leaves too much room for errors or different interpretations. There should be one obvious way to do things and not multiple competing ones. The order of the dimensions defined in the DSD (as represented in the structure part of the message) should be sufficient.

[dosse]

Dear Leo, we are sorry for the late follow-up. The keyPosition indicates the position of the dimension in the SDMX "key" that is used in the rest URL syntax. This SDMX "key"does by definition never include the TIME_PERIOD dimension. Therefore the keyPosition wasn't given for that dimension in the examples.
I also agree with the unnecessary ambiguity.

Xavier, was your conclusion that the keyPosition should be completely removed from the standard or be
made obligatory for dimensions and forbidden for attributes? The trouble of removing would be that the order (position) information is lost when dimensions are split over dataset, series and observation nodes.

[sosna]

I have mixed feelings about this.

On the one hand, I understand the use case, i.e. dimensions can be split and so you have no way to know the order unless you have the DSD.

On the other hand:

• You get the SDMX-JSON file via a RESTful data query, which means you most likely know the right dimension order already, else you would not have been able to retrieve the data in the first place.
• In addition, in a streaming scenario, it's unlikely that the web service provider will be able to make this kind of clever optimisation, such as, for example, attaching some dimensions to the dataset level. That means that, in practice, I would expect most implementers to return all the dimensions at the series level (except the special dimensionAtObservation of course), unless the client request a flat representation of the data, in which case this issue of splitting dimensions would not exist anyway, as they would all be attached to the observations.

So, in my opinion, we could live without a keyPosition, at least in the first release of the spec (i.e. until it has been proven that enough implementers can't live without), but I don't have a strong preference.

However, in any case, indicating the position of attributes makes no sense, as there is no guarantee of their order in any SDMX spec, AFAIK.

[dr-leo]

Dear Xavier, as a first comment, I am pleased that there is consensus so far that dimension attributes must not have a keyPosition attribute as this would be pointless. Let me reiterate the desire for a timely clarification in the standard. Your thoughtful remarks below are consistent with your earlier, invaluable comment in which you stated that there should be one obvious way to derive the key positions from the dataset by accessing the keyPosition attribute of each dimension rather than falling back on some pre-fetched DSD if the keyPosition attribute is missing. However, in your below remarks you downnplay the practical relevance of a mandatory keyPosition attribute. Please find my dissenting replies below each of your comments. To summarize, I am convinced that there is a strong case for mandating the keyPosition Attribute unless Service implementors proved that this would entail disproportionate costs on their part. Best regards Leo

On 04/08/2017, Xavier Sosnovsky ***@***.***> wrote: I have mixed feelings about this. On the one hand, I understand the use case, i.e. dimensions can be split and so you have no way to know the order unless you have the DSD.

Sure. However, there are a lot more use-cases: Even when all dimensions but TIME or TIME_PERIOD are at series level, there is no way to determine the key position other than falling back on the DSD, unless the keyPosition attributes are present. The standard does not enforce dimension order at any level. So a REST service implementor is perfectly in line with the standard if she disregards the dimension order provided by the DSD.

On the other hand: * You get the SDMX-JSON file via a RESTful data query, which means you most likely know the right dimension order already, else you would not have been able to retrieve the data in the first place.

Yes and no. The ECB is a great example where your assumption may hold as one can easily query the SDMXML 2.1-formatted DSD and thereafter the SDMXJSON-formatted dataset. However, in practice SDMX services tend to be less generous: OECD and ABS spring to mind: They run an SDMX 2.0 service plus an SDMXJSON 2.1 service. So at client side one has to implement both SDMXML-2.0 structure messages and SDMXJSON 2.1 datamessages to figure out the dimension order. This amounts to a burden which I consider unacceptable. In practice, data consumers will most likely make up a request using the OECD's query browser thereby determining the dimension order in passing. But this does not mean that the SDMXJSON client will be aware of this information, say, to render a dataset in a predictable way.

* In addition, in a streaming scenario, it's unlikely that the web service provider will be able to make this kind of clever optimisation, such as, for example, attaching some dimensions to the dataset level. That means that, in practice, I would expect most implementers to return all the dimensions at the series level (except the special *dimensionAtObservation* of course), unless the client request a flat representation of the data, in which case this issue of splitting dimensions would not exist anyway, as they would all be attached to the observations.

While I am unfamiliar with implementors' thoughts on this, I would rather hope that a sensible implementor does make use of such obvious optimizations, e.g., when the REST query contains a key with a single dimension value such as FREQ='A'. However, even if you are right for most cases, it is difficult from a client's perspective to cater for the few residual cases when her ambition is to write good software.

So, in my opinion, we could live without a keyPosition, at least in the first release of the spec (i.e. until it has been proven that enough implementers can't live without), but I don't have a strong preference.

A relevant question in this context would be: What are the costs for service implementors of mandating the keyPosition attribute? Without a clear answer to this question I consider it awkward to put the burden of proof on client implementors even though I believe to have abundantly proven that costs for client implementors are substantial.

…

[sosna]

Hi Leo,

Thanks for your very useful and clear comments, as always!

I disagree with a few statements but, overall, let me reiterate that I'm fine with your proposal re. the keyPosition for dimensions, even more so when providers of great libraries like you have a clear preference :).

The few statements with which I disagree are:

• The standard does not enforce dimension order at any level. I would say that this is then a documentation issue that we need to address because series keys, heavily used by many SDMX implementers, wouldn't work if the order was random.
• Regarding the obvious optimization that you mentioned, I will confess it was not implemented in the ECB web service, as far as I can remember. And, to be fair, I'm not convinced it would have made much of a difference, especially with compressed data messages, at least for service providers. For service consumers, I suspect the opinions might be split, with some in favour (as you mentioned, it's a nice implementation) and some against (they would prefer not to bother and simply read sequentially).

But as you can see from the comments above, these are minor indeed and I'm fine with your proposal, as stated at the beginning.

[dr-leo]

Dear Jens and Xavier,

let me clarify that contrary to Jens's initial comment TimeDimension does have a
key position attribute. TimeDimension inherits from Dimension the only specialty being that its role is always None.
This confirms that in the json format key position should be mandated for every Dimension including TimeDimension.

The following code example shows the key positions of all dimensions of an arbitrary
DSD taken from the ECB:

from pandasdmx import *
resp = Request('ecb').datastructure('ECB_EXR1')
dsd = resp.datastructure.ECB_EXR1

{id : dim._position for id, dim in dsd.dimensions.items()}
{'CURRENCY': 2,
 'CURRENCY_DENOM': 3,
 'EXR_SUFFIX': 5,
 'EXR_TYPE': 4,
 'FREQ': 1,
 'TIME_PERIOD': 6}

I stay away from any politics, such as whether the TWG is dominated by representatives of data providers or any other constituency
who may be biased
by how they have prototyped their sdmx-json API. From a client implementor's perspective I am naturally interested in
the most rigorous standard so as to avoid ambiguities and divergent implementations across agencies.

Leo

[dosse]

The SDMX-TWG decided: keyPosition becomes mandatory for all dimensions including special dimensions such as the Time dimension. It returns the information on the order of dimensions in the Data Structure Definition. It is not to be used for attributes.
Documentation, examples and schema have been updated.