okfn / opendataindex Goto Github PK
View Code? Open in Web Editor NEWOpen Data Index website
Home Page: http://index.okfn.org/
Open Data Index website
Home Page: http://index.okfn.org/
Seeing as the new index templates will translated from Jinja > Liquid, there will be some amount of tweaking and enhancement in the process. This seems to me like a good opportunity to upgrade the base styles from Bootstrap 2 to Bootstrap 3.
Data used by the index will be stored as data files locally in the repo in CSV format (maybe JSON too if demand).
An executable script needs to be able to build these data files from the spreadsheet database of the instance.
Extras:
What data do we have in the end?
entries.csv # original entries probably plus contributor info
places.csv # all places
datasets.csv # all datasets
questions.csv
summary.csv
Places table
id | name | slug | rank | score (out of 100) | [[ {datasetid}_rank | score_previous | score_change | score_change_rank (most-improved) | rank_previous | rank_change | score_{yyyy} | rank_{yyyy} | ]]
Datasets
| original columns | average_score | average_percent | no_responses |
Questions
| original columns | yes_count | no_count | na_count
Summary
| id | title | year | value | type (?) | details | (i18n values e.g. title@de etc)
e.g.
places_count, Number of Places, 249
Add a dedicated page for direct downloads of data.
Downloads will probably be direct downloads of the source data files. And/or, the API results. Alternatively, we could just provide zipped versions that are built when we build the other sources of the site.
A chloropeth map would provide a fairly instant overview of the state of open data in the global Index.
Especially with the new country additions, it would be easier to gain an overview of the current state of play, when compared to using the existing data table.
Think about the home page and user journey
With current settings, the site is generated with > 4500 pages.
To create these pages, it takes Pelican about 5 minutes.
The file-watching dev server takes the same.
Without putting time into addressing this issue, my current strategy is:
datasets
, historical
and places
directories out of content/pagesThis is far from ideal, but in practice, works quite well.
Some possible solutions to this problem:
url-usable, lower-case etc
future: probably merge this with id column (but do that after launch probably!)
Looks like this is covered with the supported plugins for GitHub Pages:
https://help.github.com/articles/using-jekyll-plugins-with-github-pages
Depends on discussion around #4.
I've started some work on this.
The Open Data Index is a static site, generated by Pelican.
It accesses Index data via a Pelican plugin written for the purpose, called datastore.
The datastore is simply a collection of CSV files, which are in turn a snapshot of the Open Data Census database + some post-porcessing.
The CSV files can be implemented with additional meta data to meet the Tabular Data Package specification.
Re-theme site in line with old (or updated) theme
Additional questions: do we have a bespoke logo (of course based on OK one ...)?
What is the overall tagline / text logo we incorporate on embeddable stuff?
Ranking is implicit in the default view of the current "country" table: https://index.okfn.org/country/
Ranking should be made explicit, and used throughout the site (e.g.: overview pages per country).
It may also be possible and useful to provide additional ranking per dataset. This could lead to additional storytelling with the data (e.g.: "Denmark ranks in the top 5 on the Open Data Index, but when looking at Government Spending data, it falls well behind Russia, which is #30 in the Index.")
This is a simple first pass - mainly copying over and checking we know how to do the jekyll. (Later we will do a proper redesign)
/place/:placeid
e.g. - note we use fullname as url ID rather than 2-digit
/place/australia
In the place page index.html
---
layout: place
placeid: au
---
Then _layouts/place.html
- cf https://github.com/okfn/okfn.github.com/blob/master/_layouts/person.html
---
layout: default
breadcrumbs: Place /
---
{% include partials/data_glance.html place="{{page.placeid}}" year="2014" data_glance_class="row" data_point_class="col-md-3" %}
The census submission form (for the OKFN Open Data Census 2014) looses information entered in the "Title and short description" field under the "Publicly available?" criteria.
Example: the "Title and short description" field is empty even though content was entered before the form was submitted: http://global.census.okfn.org/submission/0b992a16-9304-4d18-bf4f-78a6bea5a687
Data from the datastore plugin needs to be localisable (dataset info, questions, etc.). It can't be localised via the existing methods in Pelican. So, we need function to call on datastore data, when generating the site, to handle this.
To be clear, we have translation of strings in templates, and translations of normal "pages" in markdown, already covered with Pelican/Babel/Jinja etc.
Currently, country score are a number, like "940" for the UK. It is not clear what this number means. 940/1000? Something else?
Country scores should be displayed as percentages, which are easy to understand (e.g.: "The UK tops the Open Data Index with 94% open data")
atm we have plurals but not census site has singular ...
GitHub Pages offers no way to configure CORS headers.
This means that the API we expose cannot be used by third parties (anyone calling the API from a different domain than the one that serves the API).
Amazon S3 and Google Cloud Storage both allow setting of CORS headers.
Seeing as we are not on Jekyll anyway, GitHub Pages is now just a place that we push built files too, so the difference between serving this files from GitHub Pages, Amazon S3 or Google Cloud Storage is negligible in terms of cost, development time, and ease of use.
These days, I use Google Cloud services rather than AWS (I like the tooling). However, I suggest we move to whichever one is preferred on the OK network.
Info analysis / design of the (chloropeth) map
By default goes on the front page but can be developed as its own little module and included …
Generate API endpoints in JSON from Pelican.
Also note that the data
directory has our raw data sources in CSV format, and could feasibly be used directly as an API (for example).
Currently, data is exposed as an archive file of the whole data package, and various CSV and JSON endpoints.
I've missed the specific use case of exposing the raw data as it is with a data package file.
This is quite easy to remedy:
As raised in #21.
develop
branch, for all development work (and to create feature branches from, etc.)master
as the branch to manage production deployments fromUsing GitHub Pages, we can't use custom plugins in the automated build process.
So, any i18n implementation would require a local build before pushing to GitHub (would probably need a branch per language, and to deploy localised sites at {LANGUAGE}.index.okfn.org).
I see there are attempts at localisation in the existing code. I'm not sure if this is a consideration in building the new Index on GitHub Pages.
Proposed information architecture for the new Index site.
Based on the existing implementation at https://index.okfn.org and new user stories.
/ # overview page (probably the table or map - switchable possibly)
/place # overview of places (??) # may not have this as just the front page (??)
/place/:placeid
/place/:placeid/:datasetid
/place/:placeid/:datasetid/:year # less important right now
/dataset/ # overview of all datasets e.g. overall stats ....
/dataset/:datasetid
/dataset/:datasetid/:year (???)
# generic pages
/about
/about/methodology
/…
# downloading data
/data # maybe download
The various routes/entry points for exploring the data of the Index.
The home page should introduce the project and get right to the high level insights of the data.
The current home page has more of a focus on the mechanics behind the collection of the data.
Suggest to rethink the home page as a combination of https://index.okfn.org/ and https://index.okfn.org/country/. The current /country/ page is really the global overview, and this should be front and centre of the Index.
Similar to existing country pages, presents an overview of country performance (e.g.: https://index.okfn.org/country/overview/Australia/).
In addition:
Similar to existing dataset pages, presents an overview of dataset performance (e.g.: https://index.okfn.org/country/dataset/timetables).
In addition:
Similar to existing country/dataset pages, presents a detail of this dataset in this country (e.g.: https://index.okfn.org/country/Australia/timetables)
In addition:
Now that the index will contain the data of more than one census, we can think about ways to show historical data.
At the simplest level, we can implement historical routes for all of the above routes: Country, Dataset and Country/Dataset.
As a special case, the "current" census, which is available on the "overview" routes, would have its year route redirected to overview (e.g.: /Australia/Timetables/2014 -> /Australia/Timetables/overview, if the current census is 2014).
Topical pages transcend the base Country/Dataset paths for exploring data in the Index, and present additional ways of using the information.
An overview page presenting an analysis of change in the open govt. data landscape, as reflected by the Index, over the last year. Probably a combination of text, plus visualisations focused in displaying change, and linking through to other parts of the site.
Dumps of the raw dataset for download and further use. The data will be relatively static, but is subject to change on occasion if problems are reported in the dataset.
Suggest to have a timestamp of when the data was generated for the current build of the site, and include that as part of the filename for downloading data.
General informational pages that support the core site.
Generic pages will never be generated from "data" in a build process. Rather, they should be fairly simple .md pages that are editable by content editors. If new generic pages are added, they should be discoverable on build and routed accordingly.
In order for Pelican to build out the full site, it needs the appropriate source files in place ('source files' here refers to all files located under the content
directory).
We require thousands of source files for an instance of the Index. The source files required are also dependant on the data that will be available to display in the Index.
Hence, we will have script that automates the building of the source files, designed to run whenever a new 'data snapshot' is made (#5) (new data in the data
directory).
places
overviewplace
detail current yearplace
detail historical yearsplace/dataset
detail current yearplace/dataset
detail historical yearsdatasets
overviewdataset
detail current yeardataset
detail historical yearshistorical
overviewsAs far as I see, we have two main topics to discuss before closing on a development plan for visualisations in our current scope:
For any of the above:
This is a meta issue to collect all bugs, enhancements and suggestions made/found in various other places.
Based on this UI mock
Based on spec in #15.
Bugs will be treated in #41.
This issue is a place for any and all notes/comments on UI/UX.
I think we'll try to keep everything in list form, organised by area/feature of the site. The initial list comes from things discussed by John and I today, and by Rufus and I yesterday.
@johnmartin I extended on some things. Please have read, and add/comment for anything I may have missed from our talk.
Note that this list is also after John's main work on theme implementation, so please refer to it:
Currently identical to Place overview
Nice if font awesome had them but may need to use the noun project or similar (and store locally in e.g. assets/icons/
)
Jekyll pros:
Pelican pros
Conclusion: seems like pelican is a good idea ...
Some scenarios are presented here:
https://docs.google.com/document/d/1So6gt-8IJNBHUVFW5yuawIeiSiK3Y-OZ4XaidKbLEh8/edit
We can write the spec for a comparative visualisation that goes with the new site functionality. The ideas in the above document are great, but very ties to the way the 2013 Index looks, and in particular the table view of the data.
Placeholder issue for implementing a launch able UI for the new Index. Depends on #37.
Enhancements:
Was looking around to see if anyone had ported Jekyll's Data Files functionality to Pelican, and found that you have a datastore plugin.
I was wondering if you'd consider both:
So that others can benefit from this. The code looks like exactly what I'm looking for.
Thanks for your consideration.
Thomas
Just a first pass:
Do not too much more as we will likely redesign a bit
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.