iati / iati-dashboard Goto Github PK

http://jakevdp.github.io/blog/2013/12/19/a-d3-viewer-for-matplotlib/

E.g. % of ODA

Access to a (legible) change log for each report

Start with validation / download / XML errors

• Date | name of publisher added / removed from list

The y axis scale on the failed files graphs starts at 550 and should be at 0?

Problems:

• Dates get cropped at the moment
• Graphs are non interactive static images

The Javascript ordering of the table by file size does not work as expected because values are in KB and MB

• Might be nice to also see the number of files on the registry (as we say the number that have failed - out of what?)
• Similar for publishers - show number of publishers on registry.

NB both of the above should probably be shown elsewhere as well

• Graphs don't need to to be quite as big IMHO

This could get silly on activity features perhaps, but

• Number of activities (may be silly and meaningless, but people want to know!)
• Number of activities per publisher

We could easily get into:
Number of activities with Commitments, no. in each country, no. without reporting-org ref etc, but I think we should maybe hold back on that for now and work out what is useful for us to know internally first to drive date quality

.. to relevant publisher page

Would be useful to link the name in the Group by Publisher groupings - ie:

aa (1)

Add page(s) for this.

E.g.  1.0 on http://bjwebb.github.io/IATI-Dashboard/versions.html

• Name
• reporting-org reference(s)
• Dates - registered / first published / last updated (?)
• Other issues (the codelist stuff?)
• Link with data.tickets- a feed of tickets associated with their Registry ID?
• Invalid XML
• Download errors

Links to packages add a .xml to the URL, which returns a 404

eg:

http://iatiregistry.org/dataset/aet-activities.xml

should be

http://iatiregistry.org/dataset/aet-activities

Monitor who is publishing over HTTPS as opposed to HTTP

Probably need to know by publisher first of all
Then perhaps a number of files and maybe even the number of activities.

This may represent a percentage of the data that is not easily machine readable (or at least puts up barriers to re-use)

Alongside the registry ID, we need to also display the publisher name.

This could apply to all lists/pages - start with the Publisher table first?

Quantity:

• How many published organisations in total
• How many new each month (annual avg)
• How many activities across how many files

Coverage:

• How many different countries are publishing & how many from each
• What % of potential publishing organisations are publishing from each country (income > £10,000,000)

Quality:

• What % of published files ‘validate’ or have no ‘errors’
• What % of published files contain more than n different elements
• What % of publisher files match what is in their Implementation Schedule

Timliness:

• What % of publishers consistently republish regularly (>3 in a row) on time
• What is the avg. time taken for a new publisher to publish (from commit to publish)

A page similar to our current open milestones page, but with a list of closed milestones (and closed issues without a milestone)

Click through to list of publishers needed

This top-line figure is a bit meaningless as it covers different periods for different publishers, and also includes double counting.

A more accurate (tho' still fuzzy) picture would be an annual calculation of Disbursements + Expenditures - Incoming Funds.

More info about 'Files'

• Graph of IATI versions in use (maybe also numbers) - important thing is to track adoption of/movement of publishers to new versions.
• Number of files CKAN is reporting are available (v's number we are testing)
• Size of full data set (again graph over time to show how it grows or shrinks)
• Number of files < 1MB, Between 1 and 5MB, 5 and 10, 10 and 20, over 20. (discuss maybe) - again I think seeing this over time is good.
• Numbers of files per publisher - list (and group maybe)

See also:
http://bntest.vm.bytemark.co.uk/david/iati/validate/results/
I think both those graphs are useful

Also maybe:

• Number of files that were updated 'last night' (again a graph over time)
• Number of new files added last night (graph over time)
• Number of deletions (graph)

Each page/table should have a link to the source data. This will likely involve restructuring the output of https://github.com/Bjwebb/IATI-Stats

http://bjwebb.github.io/IATI-Dashboard/elements.html

There are two many links in the navbar to fit in one line. Maybe they should be grouped into submenus, e.g.:

• Headline stuff: Publishers, Files
• Data quality: Download errors - org xml
• Exploring data: Elements - Booleans
• Github: Milestone view

Might make sense to order the table on the page http://bjwebb.github.io/IATI-Dashboard/files.html with the largest first, as that is the primary focus

With all the key figures on that one page.

Need to go through each page and determine where and how we link back to the dashboard publisher specific page

For each report/page/graph it is going to be useful to access the historical figures

eg:

• date
• number of publishers

This way, we can then utilise it for specific date points.

This would be the raw data to begin with

Via @stevieflow "A way to annotate aspects / add notes, especially if we have contacted them etc"

Just a note that (as you know) installation fails on:
Traceback (most recent call last):
File "make_html.py", line 14, in
ckan = json.load(open('./ckan.json'), object_pairs_hook=OrderedDict)
IOError: [Errno 2] No such file or directory: './ckan.json'

To fix:
echo '{}' > ckan.json;

ie. add codelist name column to table on http://bjwebb.github.io/IATI-Dashboard/codelists.html

It might be good to know:
How many activities are reporting a certain element across
a) the entire dataset
b) at individual publisher level
c) perhaps at county or region level

This is complicated by hierarchy, so not every activity needs to report in exactly the same way.

Typical questions we might want to answer would be:
How many activities (maybe at the lowest reported hierarchy) are reporting budgets/results/

This idea does need more work to be a more generic proposal, but I think getting e.g. budgets and results reporting right might be useful in itself.

Bar graph/list of licence types (left hand side)
Line graph / list of publishers that have non-open licences (right hand side)

Might be worth knowing who is using hierarcy and what levels they are using.

E.g. Water Aid is using 4
DFID uses 1 and 2

• Schema versions
• Files With Invalid Root Elements
• Invalid XML files
• Download errors
• Non open files / publishers

Just leaving these notes somewhere:
I had this issue on Ubuntu server 12.04.

I had to install libfreetype6-dev and libpng-dev from the repositories. I was using a virtualenv and installing matplotlib using pip when I ran into this issue. (same as
http://stackoverflow.com/questions/8359383/error-while-installing-matplotlib )

Then to build the virtual environment (note needed to add 'distribute')
virtualenv pyenv; source pyenv/bin/activate; easy_install -U distribute; pip install -r requirements.txt

On the table on http://bjwebb.github.io/IATI-Dashboard/files.html add a column for publisher - which is the publisher id, which then links to their page on the dashboard

We can also then group files by publisher

E.g. spaces or /

If you have the latest datastore, you can see one example using http://localhost:5000/api/1/access/activity.xml?transaction_receiver-org.receiver-activity-id=GB-CHC-326568-RND11UKAidMatch%20