SDMX 3.0: implement "feature 001 Improve the exchange of reference metadata" for SDMX-ML messages about sdmx-ml

[dosse]

In order to support the new requirement, it is proposed to:

structure message changes:

Reference Metadata against Structure Metadata

• The Provision Agreement now only links to a Dataflow, and a new Object called Metadata Provision is introduced which enables linking to Metadataflows.
• Restrict the MSD to a single Report structure, by removing the Report structure container. The Metadata Attributes are now a direct child of the MSD and not children of Report Structure containers.
• The Metadata Target is moved it to the Metadataflow/Metadata Provision and is no longer a part of the MSD.
• The Metadata Target is simplified, only allowing a references to identifiable structure(s), which may include wildcarded references. The Metadata Provision can also include zero to many Metadata Targets in order to further restrict the targets of the Metadataflow.
• The reported MetadataSet includes an absolute reference to the Metadataflow and/or Metadata Provision which it is reported against, and one or more target structures, which must comply with the restrictions imposed by the Metadataflow and/or the Metadata Provision.
• The role of the content constraint for a metadata collection is not to restrict targets, but restrict content for each Metadata Attribute.

Reference Metadata against Datasets

• The ability for the DSD to optionally describe the relationship between the MSD Metadata Attribute and the Dimensions against which metadata can be reported. Where the absence of this relationship definition is interpreted as the Metadata Attribute being able to be reported against any combination of Dimensions.
• The AttributeRelationship provides the ability to make Dimension references optional:

<str:AttributeRelationship>
  <str:Dimension optional="true">
    <Ref id="REF_AREA"/>
  </str:Dimension>
  <str:Dimension>
    <Ref id="INDICATOR"/>
  </str:Dimension>
</str:AttributeRelationship>

• XHTML is introduced as an allowable data type. The already used XHTMLType allows for mixed content of text and XHTML tags.
• Non-enumerated text Attributes (HTML/String) also support multilingual text.

--> Implemented in structure schemas by J

data message changes:

Reference Metadata against Structure Metadata

• Removing the Report structure reference and the AttributeSet container from the MetadataSet. The Metadata Attributes are now a direct MetadataSet
• The reported MetadataSet includes an absolute reference to the Metadataflow and/or Metadata Provision which it is reported against (a simple structure reference!), and one or more target structures (now also simple structure references!), which must comply with the restrictions imposed by the Metadataflow and/or the Metadata Provision.

<MetadataSet id="ALB" providerID="ALB" version="1.0.0">
	<Metadataflow agencyID="OECD" id="ALB_DQAF" version="1.0.0"/>
	<Target type="Dataflow" aganecyID="OECD" id="GDP" version="1.0.0"/>
	<Attr id="ATTRIBUTE_1">
		Lorem Ipsum
		<Attr id="ATTRIBUTE_1_1">Lorem Ipsum<Attr/>
			<Attr id="ATTRIBUTE_1_2">
				<StructuredText xml:lang="en">
					<div xmlns="http://www.w3.org/1999/xhtml">
						<p>Lorem Ipsum</p>
					</div>
				</StructuredText>
			</Attr>
		</Attr>
	</Attr>
</MetadataSet>

<MetadataSet id="ALB" providerID="ALB" version="1.0.0">
	<MetadataProvision agencyID="OECD" id="ALB_DQAF" version="1.0.0"/>
	<Target type="Dataflow" aganecyID="OECD" id="GDP" version="1.0.0"/>
	<Attr id="ATTRIBUTE_1">
		Lorem Ipsum
		<Attr id="ATTRIBUTE_1_1">Lorem Ipsum<Attr/>
			<Attr id="ATTRIBUTE_1_2">
				<StructuredText xml:lang="en">
					<div xmlns="http://www.w3.org/1999/xhtml">
						<p>Lorem Ipsum</p>
					</div>
				</StructuredText>
			</Attr>
		</Attr>
	</Attr>
</MetadataSet>

• Only the GenericMetadata message is updated for SDMX3.0 because it allows a direct XML database storage with a single schema and it is in general not more verbose than the StructureSpecificMetadata message. The StructureSpecificMetadata message is deprecated and not updated to SDMX 3.0.

Reference Metadata against Datasets

• Values of MSD Metadata Attributes referenced by DSD are allowed to be transmitted with or without the data (and so allow including only relevant dimensions)
• For data messages we extend this possibility to transmit any attribute values with or without the data (thus separately).
  ° current syntax

<Series FREQ="A" INDICATOR="A" REF_AREA="DZA">
    <Obs TIME_PERIOD="2010-08" OBS_VALUE="111" OBS_STATUS="A"/>
</Series>
<Series FREQ="A" INDICATOR="A" REF_AREA="CZE">
    <Obs TIME_PERIOD="2010-08" OBS_VALUE="222" OBS_STATUS="A"/>
</Series>
<Series FREQ="A" INDICATOR="A" REF_AREA="ROM">
    <Obs TIME_PERIOD="2010-08" OBS_VALUE="333" OBS_STATUS="A"/>
</Series>

<Obs FREQ="A" INDICATOR="A" REF_AREA="DZA" TIME_PERIOD="2010-08" OBS_VALUE="111" OBS_STATUS="A"/>
<Obs FREQ="A" INDICATOR="A" REF_AREA="CZE" TIME_PERIOD="2010-08" OBS_VALUE="222" OBS_STATUS="A"/>
<Obs FREQ="A" INDICATOR="A" REF_AREA="ROM" TIME_PERIOD="2010-08" OBS_VALUE="333" OBS_STATUS="A"/>

° New syntax for attributes (e.g. metadata attributes): a subset of dimensions is sufficient for the new Attr object containing partial-key or dataset-level attributes

<Series FREQ="A" INDICATOR="A" REF_AREA="DZA">
    <Obs TIME_PERIOD="2010-08" OBS_VALUE="111" OBS_STATUS="A"/>
</Series>
<Series FREQ="A" INDICATOR="A" REF_AREA="CZE">
    <Obs TIME_PERIOD="2010-08" OBS_VALUE="222" OBS_STATUS="A"/>
</Series>
<Series FREQ="A" INDICATOR="A" REF_AREA="ROM">
    <Obs TIME_PERIOD="2010-08" OBS_VALUE="333" OBS_STATUS="A"/>
</Series>
<Attr INDICATOR="A" UNIT_MEASURE="CPW" UNIT_MULT="0"/>

<Obs FREQ="A" INDICATOR="A" REF_AREA="DZA" TIME_PERIOD="2010-08" OBS_VALUE="111" OBS_STATUS="A"/>
<Obs FREQ="A" INDICATOR="A" REF_AREA="CZE" TIME_PERIOD="2010-08" OBS_VALUE="222" OBS_STATUS="A"/>
<Obs FREQ="A" INDICATOR="A" REF_AREA="ROM" TIME_PERIOD="2010-08" OBS_VALUE="333" OBS_STATUS="A"/>
<Attr INDICATOR="A" UNIT_MEASURE="CPW" UNIT_MULT="0"/>

° New syntax for attributes without data: a subset of dimensions is sufficient for the new Attr object containing partial-key or dataset-level attributes

<Attr INDICATOR="A" UNIT_MEASURE="CPW" UNIT_MULT="0"/>

• XHTML is introduced as an allowable data type.

<Series FREQ="A" INDICATOR="A" REF_AREA="DZA">
	<Obs TIME_PERIOD="2010-08" OBS_VALUE="111" OBS_STATUS="A"/>
		<HTML_ATTR>
			<StructuredText xml:lang="en">
				<div xmlns="http://www.w3.org/1999/xhtml">
					<p>Lorem Ipsum</p>
				</div>
			</StructuredText>
		</HTML_ATTR>
	</Obs>
</Series>

<Attr INDICATOR="A" UNIT_MEASURE="CPW" UNIT_MULT="0"/>
	<HTML_ATTR>
		<StructuredText xml:lang="en">
			<div xmlns="http://www.w3.org/1999/xhtml">
				<p>Lorem Ipsum</p>
			</div>
		</StructuredText>
	</HTML_ATTR>
</Attr>

• Non-enumerated text Attributes (HTML/String) also support multilingual text.

<Series FREQ="A" INDICATOR="A" REF_AREA="DZA">
	<Obs TIME_PERIOD="2010-08" OBS_VALUE="111" OBS_STATUS="A"/>
		<LOCAL_ATTR>
			<Text xml:lang="en">Lorem Ipsum</Text>
			<Text xml:lang="fr">Lorem Ipsum</Text>
		</LOCAL_ATTR>
	</Obs>
</Series>

<Attr INDICATOR="A" UNIT_MEASURE="CPW" UNIT_MULT="0"/>
	<LOCAL_ATTR>
		<Text xml:lang="en">Lorem Ipsum</Text>
		<Text xml:lang="fr">Lorem Ipsum</Text>
	</LOCAL_ATTR>
</Attr>

• Attributes also support multiple values (from "feature 003 Microdata exchange").

<Attr INDICATOR="A" UNIT_MEASURE="CPW" UNIT_MULT="0"/>
	<MULTI_ATTR>
		<Value>AAA</Value>
		<Value>BBB</Value>
	</MULTI_ATTR>
</Attr>

[dosse]

@jgager @agent96 I see the following differences between the new file "SDMX_3-0-0_SECTION_3A_PART_IV_DATA.doc" and the above specs:

• providerID has been renamed to agencyID: I thought that the metadataset's providerID is different from a normal agency. Could you please confirm?
• MetadataProvision has been renamed to MetadataProvisionAgreement: OK
• MetadataStructure is still in the IM but should be removed, because the MSD doesn't contain any target definitions. Only either Metadataflow or MetadataProvisionAgreement must be referenced.
• Attr has been renamed to Att: Why?
• The MetadataSet is allowed to have several targets. Is this really allowed?

[dosse]

Released