iati / iati-schemas

Schema development for the International Aid Transparency Initiative.

International Aid Transparency Initiative (IATI) XML Schemas

IATI Support <[email protected]>

Release 2.03, Unreleased

This directory contains version 2.03 of the IATI XML schemas. More information about these schemas is available at http://iatistandard.org

This directory contains the following files:

File	Description
README.rst	This file.
CHANGES.txt	Change log.
iati-activities-schema.xsd	XML Schema for describing aid activities (e.g. projects).
iati-organisations-schema.xsd	XML schema for describing aid organisations and their budgets.
iati-common.xsd	A supplementary schema with common IATI markup. Must be in the same directory as the above.
xml.xsd	A supplementary schema that must be in the same directory as the above.
iati-registry-record-schema.xsd	An application-specific extension schema for importing documents into the IATI registry.
tests/	Unit tests for the schemas.

Dev Installation

# Create and start a virtual environment
$ python3 -m venv pyenv
$ source pyenv/bin/activate

# Install Python package dependencies
$ pip install -r requirements-dev.txt

Tests

There are two types of tests:

Legacy tests

A series of short XML documents are included in the tests/**/should-pass and tests/**/should-fail directories (and sub-directories). These files test elements and attributes added pre-version 2.03 and offer a simple pass/fail check when parsed against the schemas in this distribution.

These tests can be run using:

$ xmllint --schema http://www.w3.org/2001/XMLSchema.xsd --noout *.xsd
$ ./tests/run-tests.sh

Tests for elements/attributes added in v2.03

A new test structure was added at version 2.03, with additional test cases added to tests/should-pass and tests/should-fail directories (and sub-directories). Directory structure is based upon the schema. Each element has a folder based on its name and is nested accordingly. Attributes also have folders but their folder names are prepended by an underscore.

All XML test cases in the should-fail directory are tested to be valid XML but invalid against the distributed IATI schemas. The expected reason for failure (for example, a missing required attribute) in each test case is asserted.

We use pytest as a test runner to check the validity of these files against the defined schema.

$ pytest

Test file naming syntax

should-fail test file names should begin with one of the following strings, depending on the expected failure type:

• err-not-iati-xml-missing-required-element
• err-not-iati-xml-missing-attribute
• err-not-iati-xml-forbidden-attribute
• err-not-iati-xml-incorrect-datatype

Additional test descriptions may be added to the file name following an underscore, like so:

err-not-iati-xml-missing-required-element_document-link-with-multiple-description-elements.xml

The files are now parsed with etree.fromstring. If an XMLSchemaError is raised during the parsing, the beginning of the file name is compared against Regex for the four possible error types above. For example, the error raised from err-not-iati-xml-missing-required-element_document-link-with-multiple-description-elements.xml is expected to match the Regex:

Element '[a-z-]*?': ((Missing child element\(s\)\.)|(This element is not expected\.)) Expected is( one of)? \( [#\*a-z-\s,]*? \)\.

Acknowlegements

David Megginson <[email protected]> for his original work on the IATI Schemas and continued support and involvement.

iati-schemas's Issues

Percentages as positive decimal

Our proposal on percentages as decimals (instead of integers) is here:
http://wiki.iatistandard.org/standard/revision/1.03/changes_specification?&#allow_percentages_to_be_specified_as_decimals

However, percentages should be positive (I think!), do we need a more sophisticated definition where we set minInclusiveValue=0 as well like this example:
http://stackoverflow.com/questions/3853476/how-do-i-define-custom-datatype-in-xsd

Use decimal instead of integer for currency values

Details: http://wiki.iatistandard.org/standard/revision/1.03/changes_specification?&#currency_values_as_decimal

Create Capital Spend element

See line 198 of this:
https://docs.google.com/spreadsheet/ccc?key=0AnWngmdQt3stdDcwRHJxaXgtLWVoY2N5VWt4ck1oeGc#gid=1

Update schema header

The version and date need changing for 1.04

Enhance Description of Transaction Type Disbursement

Raised from Zendesk ticket #1047. It is currently not clear what is meant by the text in brackets that describes the Transaction Type of Disbursement on the Transaction Type Codelist. Please could this be clarified

Update documentation

Boxes highlighted in yellow need to be altered in the schema from here:
https://docs.google.com/spreadsheet/ccc?key=0AnWngmdQt3stdDcwRHJxaXgtLWVoY2N5VWt4ck1oeGc#gid=1

Convert percentages from integer to decimal

Details: http://wiki.iatistandard.org/standard/revision/1.03/changes_specification?&#allow_percentages_to_be_specified_as_decimals

Recipient-region @vocabulary

Bill - please confirm.

Add @vocabulary to recipient-region
Line 80
the spreadsheet
https://docs.google.com/spreadsheet/ccc?key=0AnWngmdQt3stdDcwRHJxaXgtLWVoY2N5VWt4ck1oeGc#gid=1

Add a Schema Version Attribute to the root element

Details: http://wiki.iatistandard.org/standard/revision/1.03/changes_specification?&#add_a_schema_version_attribute_to_the_root_element

Create New CRS elements

Start at line 311 of this:
https://docs.google.com/spreadsheet/ccc?key=0AnWngmdQt3stdDcwRHJxaXgtLWVoY2N5VWt4ck1oeGc#gid=1

reporting-org description text

The description text for this element is currently:

The organisation issuing a report (not necessarily involved in the organisation or activity being described). May be a donor, fund, agency, etc. Specifying the @Identifier and @ROLE attributes is strongly recommended. May contain the organisation name as content. For the value of the @type attribute, see http://iatistandard.org/codelists/organisation-type For the value of the @ref attribute, see the list of officially-registered organizations at http://iatistandard.org/codelists/organisation

Changes

1 - @Identifier is not an attribute - change to @ref

2 - The last sentence should be revised to point people to the guidance:

So - new text proposal:

The organisation issuing a report (not necessarily involved in the organisation or activity being described). May be a donor, fund, agency, etc. Specifying the @ref and @ROLE attributes is strongly recommended. May contain the organisation name as content. For the value of the @type attribute, see http://iatistandard.org/codelists/organisation-type For guidance on constructing the value of the @ref attribute, see http://iatistandard.org/org-ref

NDP example for result > indicator /@measure

http://iatistandard.org/activities-standard/result/indicator/

Text describes NDP, but this is not on the code list

Update CHANGES.txt for 1.04

Create country-budget element

See line 139 of this:
https://docs.google.com/spreadsheet/ccc?key=0AnWngmdQt3stdDcwRHJxaXgtLWVoY2N5VWt4ck1oeGc#gid=1

reporting-org descriptive text needs review

"Specifying the @ref and @ROLE attributes..."

reporting-org does not have a @ROLE attribute.

Also we might want to move descriptions about attributes out of element descriptions and into descriptions connected to the attribute itself.

@version attribute description

The description of @Version is different between iati-activities and iati-activity:

iati-activities

A number indicating the IATI specification version in use.

iati-activity

A number indicating the IATI specification version in use. Defaults to "1.0" if not specified. It is required to specify this attribute if the document is using features specific to an IATI specification other than the initial 1.0 version.

Should we:

a - align the description around the latter
b - add to iati-activity that the @Version here can differ from at iati-activities (or not?)

Introduce a New Transaction Type For Internally Sourced Funds

Raised from Zendesk ticket #1047. Currently, it is not necessary for organisations to report or include funding that comes from their own fundraising ( or similar internal) sources. Therefore should such funding be reported on (and do we perhaps need a new transaction type called locally raised funds, or something)

Better Describe the Roles of Organisations Publishing On Behalf Of Others

This issue has been raised from zendesk ticket 18. See https://iati.zendesk.com/agent/#/tickets/18

We need to include additional fields within the schema to better describe the role of a publisher who is publishing data on behalf of others. The Activity files needs to support secondary publishers. There are three scenarios that need to be covered:

1. Aggregator publishes a donor activity that it has imported (unchanged) from another aggregator
2. Aggregator publishes a donor activity that it has imported (unchanged) from the donor
3. Aggregator publishes a donor activity that it has imported and modified

Scope of activity

Bill please confirm!

Add activity Scope element.

Lines 71-73 of the spreadsheet
https://docs.google.com/spreadsheet/ccc?key=0AnWngmdQt3stdDcwRHJxaXgtLWVoY2N5VWt4ck1oeGc#gid=1

Splling mistake on xsd - activity-scope

iati-activities-schema.xsd

<xsd:element name="activity-scope" type="codeType">
xsd:annotation
<xsd:documentation xml:lang="en">
What geographical area does the activity encompass? eg. Global, Regional, Multi-National, National, Multiple (sub-national) aministrateive areas, etc
/xsd:documentation
/xsd:annotation
/xsd:element

"aministrateive areas, etc" should be "administrative areas, etc"

Contact Details - missing elements.

In the forum post on this we have:

We have added the following:

Contact Type (with an associated code list)
Job Title
Website
The ability to report multiple languages has been added to:

Organisation Name
Person Name
Job Title
Mailing Address

.....

Looking at the 1.03 dev schema it looks to me as though:

1. Job title has been added, but Contact Type and Website have not

Also it seems that the elements are all of 'plainType' I think they should have been changed to 'textType' to enable multiple languages to be used as discussed above.

However, something I'm not clear on is the multilingual changes needed - is it
just changing the 'type=' bit?

common schema - percentage

Description for percentage in common schema states:

The percentage of the budget allocated to this item. Content
must be a positive real number between 0 and 100, with no
percentage sign.

I think this has to be updated to reflect the changelog in 1.03:

Where used, the @percentage attribute is now designated as a decimal value and no longer as a positive Integer

1.04 Tests - do they need to be improved?

Test for pass and fail XML for the new elements in 1.04 should be written.

Activity Date explanation

Currently the text within the activity-schema for activity-date is:
Planned starting date for the activity in ISO 8601 date
format, e.g. "2010-10-01". For the value of the @type
attribute, see
http://iatistandard.org/codelists/activity_date_type

The text content may contain a general date text (e.g. 2011Q1)
for recording less specific dates such as month, quarter, or
year.

I don't think that makes sense (activity dates are not just planned starting dates).
The text in the spreadsheet on line 40 doesn't help (I think this is about changing the final line):
https://docs.google.com/spreadsheet/ccc?key=0AnWngmdQt3stdDcwRHJxaXgtLWVoY2N5VWt4ck1oeGc#gid=1

Bill - can you check pls.

fss not in schema CHANGES.txt

It looks like the addition of the new optional element fss is not in the CHANGES.txt - maybe on the assumption that is it a child element of crs-add?

Licence information

How are the iati-schema licenced? Do they need to be, etc, etc

Description For Transcation Type Incoming Funds Is Incorrect

Raised from Zendesk ticket #1047. Currently the description for Incoming Funds on the codelist for Transaction Type states that IF transations only relate to 'external' funds. This is apprantly not the case and IF transactions can be used to describe transfers between accounts or projects within a single organisation

country-budget-items allows a mix of text and iati sub elements

In the new schema, XML like this is valid:

<country-budget-items vocabulary="1">
   IATI
  <budget-item code="1.3.3" percentage="100">
    Accountability - planning
    <description>Accountability - planning</description>
  </budget-item>
</country-budget-items>

This is because country-budget-items has mixed="true". This is different to most other elements that have subelements in the schema. Is this a bug?

Release date needs to be added to each of the 4 schema documents

When the release date of the 1.03 version is agreed, this needs to go into the documentation notes of each of the four schema.

country-budget-items/budget-item@percentage

This attribute expects percentage, even when there is only one in use.

This should act in the same way as country and sector, in that percentage is only expected when there are multiples of the element within the activity

Grammar in transaction documentation

https://github.com/davidmegginson/iati-schemas/blob/0fdda51c83ad34213966f1fb4f1063314717229f/iati-activities-schema.xsd#L655

allows uniquely identifying a transaction should maybe be allows unique identification of a transaction

@version attribute on iati-activity

        A number indicating the IATI specification version in use.
        Defaults to "1.0" if not specified.  It is required to
        specify this attribute if the document is using features
        specific to an IATI specification other than the initial
        1.0 version.

Should this be on both iati-activities and iati-activity. The behaviour if one is defined and the other isn't is not well defined. Moreover, it is not clear if documents with iati-activity elements with different version attributes are permitted, since the documentation says iati-activity@version refers to whether the document uses certain features.

Additionally, the documentation says that it defaults to '1.0'. This version of the schema is not available online.

Should crs-add/fss/forecast have a value date?

Every other use of a monetary amount with a @Currency code includes a value date, but the standard doesn't require one for crs-add/fss/forecast (line 339ff). Can you find out if we should include it here as well? I won't implement it right now in #4 , but can add it if Bill would like to include it.

README.txt is out of date

Still says 1.02

Description of attributes should be moved into attributes

crs-add/loan-status/repayment-type

I'm seeing
crs-add/loan-status/repayment-type
in the XML

I think this should not be there.

this IS correct under loan-terms:
crs-add/loan-terms/repayment-type

@bill-anderson & @davidmegginson Should we remove this element?

Description on location-type is inacurate

This is apparently not accurate
"The type of location (e.g. "PCL" for a political entity), from an IATI-defined subset of the UCPD/Aiddata code list. See http://iatistandard.org/codelists/location_type"

New git branches

David: I've created two new git branches:

• relaxng - my old work in progress on switching to RelaxNG (saved for future use)
• pre-version1.03 - branch for our work getting ready for version1.03 (rather than using master)

Are these OK with you? Please feel free to rename them, if not.

Some notes and questions on the 1.04 schema changes

Country, adm1, adm2 descriptions. These are different in the spreadsheet to those currently in the schema. Given that we are deprecating these attributes, do we want to alter the text to the text that is in the spreadsheet?

Point is marked up as 'point' in the schema instead of 'Point' as specified in the spreadsheet. Is that ok?
point has no description in the spreadsheet, should it have?

Have added 'Deprecated in 1.04' to descriptions of elements and attributes that
are now deprecated, but there is no other indication. Should we add a deprecated="true/false" attribute to attributes/elements?

@currency is used in organisation xml, but talks only about iati-activities

https://github.com/IATI/IATI-Schemas/blob/master/iati-common.xsd#L201

" This is required for all currency amounts unless the iati-activity/@default-currency attribute is specified."

Should be amended to mention that this is true for IATI Activity files, whereas for IATI Organisation files, it is iati-organisation/@default-currency attribute

Push version numbers in files that have not changed

Should we change the revision metadata in schema files that have not changed:
iati-organisations-schema.xsd has not changed but the iati-common.xsd it relies on has.

Similarly the registry-record schema has not changed.

@type description

The @type description, via the common/xsd is:

Free text describing the type of thing being referenced.

However, for instances such as organisation type or document type, we do wish that the code list is utilised. Is this description therefore in need of a fix?

Remove @version and @last-modified attributes

These are no longer necessary, as version control is done through github.

reporting-org documentation mentions non-existent @role

@ROLE is mentioned in the reporting-org documentation (https://github.com/IATI/IATI-Schemas/blob/master/iati-common.xsd#L86), but it does not have an attribute by this name.

Remove hardcoded urls from the schema

This will be replaced by a separate mapping file - https://github.com/IATI/IATI-Codelists/blob/master/mapping.xml

I will also look into creating a new annotated schema, with extra information about codelists, and from https://github.com/IATI/IATI-Extra-Documentation

Review and update schema text

Specifically, the text that is used for SSOT documentation.

Ensure this is precise and concise

Remove links to codelists - see: #53

Currency documentation

From:
http://support.iatistandard.org/entries/21908708-Allow-decimal-transaction-values

there is a note that says we will:
I propose we alter the guidance in the (iati-common) schema to:

[add] A three letter ISO 4217 code for the original currency of the amount.
[remove] An ISO 639 code for the original currency of the amount.

Update README.txt for 1.04

Markup for contact-info/person-name/job-title

According to the spreadsheet at https://docs.google.com/a/acclar.org/spreadsheet/ccc?key=0AnWngmdQt3stdDcwRHJxaXgtLWVoY2N5VWt4ck1oeGc#gid=1 there will be a new element called

content-info/person-name/job-title

However, since person-name also allows text content, this will make for messy markup. Can we make that an attribute instead? e.g.

content-info/person-name/@job-title

Capitalisation of repository title

This is inconsistent with all our other IATI- repositories, should it be renamed?