opendataservices / cove Goto Github PK

There's various classes we can use from twitter bootstrap to make our error page look more like an error.

(To get an error page currently, you can upload a file that is not JSON or XLSX).

It should handle these properly if uploaded, and also offer them as an option for conversion from JSON.

Number of unique contracting processes (unique OCIDs)
Earliest/latest publication date

If I use the URL https://github.com/OpenDataServices/flatten-tool/raw/master/flattentool/tests/fixtures/xlsx/basic.xlsx

I get 'Unrecognised file type.'

It seems that mime_type is application/octet-stream

https://github.com/OpenDataServices/flatten-tool/blob/master/flattentool/tests/fixtures/tenders_releases_2_releases.json?raw=true fails
https://raw.githubusercontent.com/OpenDataServices/flatten-tool/master/flattentool/tests/fixtures/tenders_releases_2_releases.json works

This is most likely because the redirect is not being followed.

"We can only process json, xls, xlsx and csv files."

Once a file has been successfully accepted and converted, the 'explore' page has a unique URL, allowing for that page to be shared with others.

We could make this a feature of the 'explore' page.

Note also that the page may cease to exists (e.g. if and when a file is deleted)
See: #38

https://github.com/OpenDataServices/cove#introduction

When a file is uploaded and converted we could show a bit more useful info on that page

Currently we don't really say what files can be uploaded to Cove (XLSX and JSON, but not currently CSV, or XLS or ODS). Relates to #54

When I spin up the server locally I see:

lib/python3.4/site-packages/django/templatetags/future.py:25: RemovedInDjango19Warning: Loading the url tag from the future library is deprecated and will be removed in Django 1.9. Use the default url tag instead.
RemovedInDjango19Warning)

On the 'explore' page we could be showing the size of the file that was provided

Might also want to show the size of the files that have been converted and made available for download

This will allow other sites to link to Cove's view of a particular file.

This is about constructing a link using the URL of the data, instead of pasting the URL into the box on the input page (rather than being about Cove's own shareable URL).

(because it is just an empty file)

Currently schemas are fetched from external urls each time they are needed. This has overhead each time they are used, and will cause the application to break if GitHub or the OCDS standard website go down.

I think the alternatives are:

• Have the application fetch the schemas from remote urls, but cache them. The tricky bit here is knowing if/when the schemas need updating.
• Include the schemas in the GitHub repository. This is messy, but may be the easiest option, and the most obvious way to ensure that the application will work offline.

Whichever mechanism we choose should also be used by the tests, see #16

Using django-environ we can get settings values from environment variables.

The most important use of this atm is to allow DEBUG to be set to False on the server.

I've made the necessary change to the code, eb2127f , but I also need to enable the repository via the coverall.io web interface - https://coveralls.io/ - which is currently down.

Once a user has uploaded a file, they may want it removed from our servers immediately.

We could provide a 'Delete' button (on the 'explore' page?) to enable this.

This needs more thought as e.g.
users are not logged in (they might want to come back later to delete)
Anyone can visit the explore URL and delete it

django-template-i18n-lint may be a useful tool to check for this:

$ pip install django-template-i18n-lint
$ django-template-i18n-lint cove
cove/templates/explore.html:14:46:XLSX (
cove/templates/explore.html:15:45:JSON (
cove/templates/explore.html:19:45:XLSX (
cove/templates/explore.html:20:41:JSON (
cove/templates/base.html:38:45:Go
cove/templates/base.html:63:29:Footer text here. Running version

We could think about whteher we want this to run automatically, e.g. as part of our tests.

On the explore pages we are just printing out strings of dates at the moment.
We should turn these into date objects so we can manipulate them.

This builds upon #9

This may want its own page, linked to from the validation box on the main explore page.

The other two inputs have required=required, which enables the html5 prompts to fill the field in. The file input doesn't have this, only Django's prompt once the page has actually been submitted.

(and vice-versa)

I think we should probably disable this, and possibly redirect to the correct URL (and template).

We need to think about whether it's possible to offer a single CSV download sometime, or whether this should always be multiple downloads, or a zip of CSVs etc.

In order to test the multilingual functionality, I created a Spanish translation using google translate (there's an option to upload a .po file and machine translate all of that).

There's also a French translation, which is similar, but with slightly more manual translations that are likely to be wrong (#19 (comment)).

Do we want to keep the google translated translations as a starting point, or remove them for release, and wait until we have manually translated copies?

This may want its own page, much like #46

We should be able to generate a report that gives us some idea about:

• How many 'files' have been uploaded/pasted/fetched via URL (recently, in the last week, etc)
• What kind of conversions have people been requesting?

Other things we might want to think about:
Average size of files, time to process, number of subsequent downloads of converted files.

We can use this thread to add more ideas.

Currently we just get a 500 error.

There's no very security sensitive use of this yet, but we should set it via an environment variable just in case.

The Footer needs some copy including link to github, etc

When you open another fold of the accordian, it doesn't close the other tabs any more.

Where possible we should be using URLs created locally, so that running the tests doesn't depend on having an internet connection, and the availability of third party services.

These should include the use of the application without javascript.

Spend some time considering what further steps might be taken around securing the application as it is an application that accepts file input. (e.g. file size, rate limits, server resources)

When people submit data to us (upload, paste, url) we should delete that data (and the derived converted data) after a period of x days (lets choose 7).

We will retain meta-data about that file e.g. was it an upload, url, or paste, when did it occur, did it convert or error, and maybe more...

BUT 7 days after upload/paste/pull from URL it (and it's derivatives) will be deleted from our server.

Once successfully uploaded, a page is created with a unique URL with 'explore' information and links to downloads. This URL could be used by people to share the results. After 7 days I expect that this page/URL will 404 (will it?)

Files that are uploaded, but fail to convert will also need to be deleted. (should this be immediately, or also after 7 days?)
(I expect that there is no easily discoverable URL for these files, although if you know the filename and the code for this application it can easily be deduced)

Associated with this are the following tasks:

• We need to tell people of our deletion policy in Terms and conditions
• We need to tell people about our deletion policy on the front page/footer i.e. be explicit somewhere!
• We need to put a message on the 'explore' page showing when a file will be deleted (and that that URL/page will be come a 404 once the file is deleted) ( we might also show 'time until deletion', but this is not necessary)
• Provide a 404 message for that URL when the file is gone

Currently to interact with the accordian, you must click on the text header/icon. Might be easier to use if the entire header bar was clickable.

We need some details for terms and conditions to be linked from the footer

Currently this is Django's default error page.

The traceback should still be accessible for debugging purposes. For now, it is probably easiest to emit this to stdout.

Additionally, it would be useful to display extra information for a JSON file, even if the conversion to spreadsheet failed, so maybe the error should be part of the explore.html template.

We may want to have a click through from the explore pages to individual releases/grants etc

Need to decide what to click through on (ocid?), and what view we get of the release/grant.

We're currently using:
Open Contracting Data Explorer

We have one suggestion of: Open Contracting Data Tool

We also have a faked up French translation in place that needs altering or removing, which could be done at the same time

This is multiple CSV "sheets" describing the data, not upload of multiple different datasets (e.g. for comparison) which would be a seperate issue.

We could provide a series of upload boxes for people to do this, or ask them to be uploaded in a zip archive.

Currently many things are loaded into memory, when they could be streamed. We need to test how large files need to be for this to cause problems (and then decide whether this is an acceptable limit or not).