akvo / akvo-flow-dataman Goto Github PK

This involves having the correct access rights to S3 buckets depending on the instance which the device is associated with. The files are retrieved from S3 and verified as correctly retrieved. They also need to be unpacked since they are zip archive files.

Requests for answers from the API currently return a value that contains the path of the image as it was on the device. This is not useful to an API user as they would not be able to view these images. We need to return the proper URL of the image that makes it retrieveable.

As an initial step to the implementation of the dataman service, we will create the skeleton of the application and setup the deployment infrastructure and continuous integration in order to start with automated builds and deployments right from the beginning.

The data files from devices contain different formats and versions for each of the format. These need to be correctly processed and the various entities related to surveys and form responses created and stored.

Note: We need to correctly process the files depending on the format. Perhaps we should also document the formats of these files somewhere.

Data files uploaded by the devices to S3 need to be correctly retrieved. We need an endpoint to which the notification of an available device file is sent. The notification end point needs to be as robust and available as possible to ensure that devices get confirmation that their files have been correctly uploaded as soon as possible. The check for presence of a file is done based on the MD5 hash retrieved from S3 and that received by the notification end point from the device (via GAE). In this first phase the notification is expected to come through from GAE meaning we need to have the endpoint prepared to receive notifications from multiple GAE instances, however, we should prepare this endpoint to receive the notification directly from devices.