moja-global / flint.cloud Goto Github PK

Description

Right now, we are getting CORS errors over the deployment of FLINT.Cloud and FLINT-UI. The error can be traced over to these references:

• FLINT.Example:

  FLINT.Cloud/local/rest_api_flint.example/app.py

  Line 14 in da310e4

  CORS(app, origins=['http://127.0.0.1:8080/', 'http://127.0.0.1:8000',

• REST API GCBM:

  FLINT.Cloud/local/rest_api_gcbm/app.py

  Line 23 in da310e4

  CORS(

We need to add *.azurewebsites.net to this so that we can mitigate this issue.

Currently, the Google Cloud Storage bucket name is hardcoded with the name simulation_data_flint-cloud this might give an error something like this

│ Error: googleapi: Error 409: Sorry, that name is not available. Please try a different one., conflict
│
│   with google_storage_bucket.simulation_data_flint-cloud,
│   on main.tf line 259, in resource "google_storage_bucket" "simulation_data_flint-cloud":
│  259: resource "google_storage_bucket" "simulation_data_flint-cloud" {

This issue is the exact same issue as #19, but in the Local and Cloud versions of the GCBM REST API.

Fix: Please have a look at #20.

Files to be changed:

Cloud:

• app.py
• requirements.txt

Local:

• app.py
• requirements.txt

Currently, the contribution section does not provide details about the commit/branch conventions followed.
It can be made similar to : https://github.com/moja-global/FLINT-UI

Presently, the /dynamic endpoint is used to run the simulation. The endpoint can be renamed to something more intuitive like /run

GDAL 3.4.1 is built in the flint.base image along with it's python bindings on L157, but python doesn't know where to find it.

import gdal 
ModuleNotFoundError: No module named 'gdal'

If I append the osgeo package (the parent of GDAL) to PYTHONPATH I get a different error:

export PYTHONPATH=$PYTHONPATH:/usr/local/lib/python3/dist-packages/osgeo

import gdal
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3/dist-packages/osgeo/gdal.py", line 2039, in <module>
    from . import ogr
ImportError: attempted relative import with no known parent package

Description

subprocess.run uses a default of check=False, which means that a nonzero exit code will be ignored by default, instead of raising an exception.

Is your feature request related to a problem? Please describe.
We need a way for instances of FLINT Docker containers to communicate with the outside world.

Describe the solution you'd like
For this, I propose a multi-layer architecture. The top layer would be user-facing. The second layer would be a REST API (which may or may not scale as per the number of requests) and the final layer would be of ephemeral FLINT containers.

gRPC would be ideal for establishing communication between the FLINT containers. It's very lightweight, fast, and also language agnostic. The FLINT Docker images already come with Python preinstalled. To set up a gRPC server inside the containers, only two Python packages need to be installed.

Describe alternatives you've considered
One way could be to expose a REST API from within the container itself. However, that would inefficient since the API server would be replicated for every instance of the container. All the heavy lifting (computational load) is handled by the FLINT instances only, so ideally we want the REST API to scale independently from FLINT instances.

Additional context
I've created a basic proof-of-concept for the FLINT containers with a gRPC server here: https://github.com/arnav-t/FLINT-gRPC-POC
This shows how we can invoke FLINT from outside using gRPC.

Currently, the swagger.json file of the gcbm file contains the /rothc endpoint description. There is no endpoint of /rothc currently present for the rest api gcbm.

Path of the file:
local\rest_api_gcbm\static\swagger.json

Add the path to Dockerfile in the README

Reported in moja-global/FLINT-UI#258 and on Slack - there seems to be something wrong with the integration of moja.canada into our GCBM endpoints.

The last known working version was #63 but CI was not configured at this time. @HarshCasper - I believe you manually pushed a copy from a forked repository, but it's hard to tell which image it was.

The issue may have occured in #65, #70 or #88, and it is not clear what the differences between these builds are. The mono-Dockerfile in #63 moved some Poco build arguments into the base image which are recreated in moja.canada - which may be the culprit (but also may not be).

Help dissecting out these changes is very much appreciated.

Currently, on running the command docker build --build-arg BUILD_TYPE=RELEASE --build-arg NUM_CPU=4 -t gcbm-api . generates

The image does not build successfully

The test_cases of rest_api_example routes are missing. These are some following endpoints of the rest_api_example.

The Problem:

I am working on a proof of concept of a user interface for FLINT Example REST API.

I am trying to connect the basic UI made by @waridrox to the FLINT Example but the request gets blocked by the Cross-Origin Resource Sharing (CORS) policy.
Please go through this great document on MDN to understand the CORS policy and its implications.

This is the console log:

xhr.js?ec6c:177 GET http://127.0.0.1:8080/spec net::ERR_FAILED
dispatchXhrRequest @ xhr.js?ec6c:177
xhrAdapter @ xhr.js?ec6c:13
dispatchRequest @ dispatchRequest.js?c4bb:52
Promise.then (async)
request @ Axios.js?5e65:61
Axios.<computed> @ Axios.js?5e65:76
wrap @ bind.js?24ff:9
apiRoute_spec @ App.vue?26cd:53
click @ App.vue?edc5:27
invokeWithErrorHandling @ vue.esm.js?efeb:1872
invoker @ vue.esm.js?efeb:2197
original._wrapper @ vue.esm.js?efeb:7609
:8000/#/:1 Access to XMLHttpRequest at 'http://127.0.0.1:8080/spec' from origin 'http://127.0.0.1:8000' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.
App.vue?26cd:55 Error: Network Error
    at createError (createError.js?16d0:16)
    at XMLHttpRequest.handleError (xhr.js?ec6c:84)

Proposed Solution:

This issue can be solved in two ways:

Both the solutions involve enabling CORS, but I do not know about the severity of the security risks we may encounter.

1. Client side: by using a Chrome extension, but this will be too much work for the average user, which defeats the purpose of building a user interface for the FLINT.
2. Server side: by making a few changes in the API itself. We can use the Flask-CORS library with the API to enable CORS. In this way the user would not have to make any environment changes on their machine.

This is how the console looks after hitting the /spec endpoint from @waridrox 's UI via Axios with Flask-CORS:

@Tlazypanda @arnav-t @aornugent @mfellows @abhi211199
As the user would be working on the local Docker environment as well as the FLINT Cloud environment, is the solution sustainable from a security standpoint? Please give your views on this! Thank you very much!

cc: @HarshCasper @waridrox

The command:

Here in Flask.example REST API Setup
In order to run the REST API, please follow the following steps: -

1. docker build -t FLINT-api .

2. docker run --rm -p 8080:8080 FLINT-api

These commands do not work properly.

Gives :
"invalid argument "FLINT-api" for -t, --tag" flag: invalid reference format: repository name must be lowercase.
See 'docker build --help'.

• The instructions present https://github.com/moja-global/FLINT.Cloud/tree/master/local/rest_api_gcbm are not corresponding to the latest repository structure
• Store the endpoints in a tabular format similar to https://github.com/moja-global/FLINT.Cloud

Reproducible research requires publishing all code and data necessary to derive the original results. Independently replicating results reinforces our belief that the analysis is valid and correct. However data are often not well supported by existing version control system.

Data version control (DVC) and it's companion software Continuous Machine Learning (CML) provide a framework to continuosly reproduce analyses using Github Actions. I'd like to demonstrate this functionality using the FLINTcloud images for other users to incorporate them in their implementations. It is unlikely, however, that we will want this feature to be active on the FLINTcloud repo - rather the results of this issue should be published as documentation or a case-study.

The first step is to develop a 'run-gcbm' stage - where the GCBM Demo Run is executed within a single Python script. This stage, the input data and the output results, are added to DVC creating a dvc.yaml file and a dvc.lock file. When the DVC configuration is pushed to a repository, the the checksums of the model inputs are commited to version control, allowing us to track when they change. Additional pre- and post-processing stages can be added to develop a complete DVC pipeline.

In a specific implementation (e.g moja.belize) users may choose to store their input data on remote storage like Google Drive, an S3 bucket or Azure Blob. This allows the data to be accessed remotely by anyone that pulls the repository, enabling them to easily reproduce the results. Further, a CML Github Action can be configured to reproduce the results on every pull request, allowing us to track changes in data inputs and model results over time.

Implementing this second phase will require that the FLINTcloud Docker images be hosted on a central repository, so that they can be used by the Github Actions Runner. Actions for setup-cml and setup-dvc are available, which can be installed as part of the workflow, meaning that we don't need to add CML or DVC as a dependency of our project.

In summary, we want to:

• Publish the FLINTcloud images on a container registry
• Develop a small run-gcbm.py script that executes the GCBM demo run
• Add the run-gcbm stage to DVC
• Host the input files on Google Drive
• Demonstrate the CML Github Action
• Document the workflow to be used in other implementations

The documentation of the flint cloud is missing. The curl file only contains a little information.
Missing content of the documentation.

• Project Structure
• Internal working.
• Required output of the API.

When there is no simulation exists this endpoint returns an empty list with a suitable message. The following is the message.
Message:
{ "data": [], "message": "To create a new simulation, create a request at gcbm/new. To access the results of the existing simulations, create a request at gcbm/download." }

It is bad practice to return an empty list in the response. The following message is optimal in the place of the above message.
{ "message": "No simulations have been run yet. Please run a simulation by making a request to gcbm/new." }

The unzipping instruction for the GCBM_demo_run.zip to setup the GCBM REST API is incorrect in the present version of the README

We sometimes use this image for running automated tests. There's an example of our CML workflow here: moja-global/GCBM.Belize#7

Because the rest_api_gcbm image just a light wrapper over the standard GCBM build, I think it's ok to add npm here and save the installing it in every workflow. Please add npm to the rest_api_gcbm Dockerfile.

Automated tests using pytest will be written for the REST API Endpoints

Description

Add a root endpoint to the rest_api_flint.example API. It can be as simple as adding:

@app.route('/', methods=['GET'])
def home():
    return 'FLINT.Example API'

Test it on the Docker container and submit a patch.

/gcbm/download api is working even for non-exist simulation titles and returns a corrupted output.zip file.

There is only run4 simulation is present, but it is producing the simulation for the "run47" title (This is not created)

Output.zip

I am following the steps to setup FLINT cloud on GCP. I have :

In step 3 : In main.tf, change the project variable to your project ID. Change any other variables if necessary.

What are the other variables needed to be changed ?
Error might be due to this

Variation 1

1. Created a service account with Owner permissions
2. Replace the project id with my project id
3. Run terraform apply

The following is the result:

1. The Storage and Pub/Sub resources are created as expected

google_cloud_run_service.fc-ingress: Still creating... [19m10s elapsed]
google_cloud_run_service.fc-cr-processor: Still creating... [19m20s elapsed]
google_cloud_run_service.fc-ingress: Still creating... [19m20s elapsed]
google_cloud_run_service.fc-cr-processor: Still creating... [19m30s elapsed]
google_cloud_run_service.fc-ingress: Still creating... [19m30s elapsed]
google_cloud_run_service.fc-cr-processor: Still creating... [19m40s elapsed]
google_cloud_run_service.fc-ingress: Still creating... [19m40s elapsed]
google_cloud_run_service.fc-cr-processor: Still creating... [19m50s elapsed]
google_cloud_run_service.fc-ingress: Still creating... [19m50s elapsed]

│ Error: Error creating Service: googleapi: Error 409: Resource 'fc-ingress' already exists.
│
│   with google_cloud_run_service.fc-ingress,
│   on main.tf line 133, in resource "google_cloud_run_service" "fc-ingress":
│  133: resource "google_cloud_run_service" "fc-ingress" {
│
╵
╷
│ Error: Error creating Service: googleapi: Error 409: Resource 'fc-cr-processor' already exists.
│
│   with google_cloud_run_service.fc-cr-processor,
│   on main.tf line 190, in resource "google_cloud_run_service" "fc-cr-processor":
│  190: resource "google_cloud_run_service" "fc-cr-processor" {
│
╵
╷
│ Error: Error creating Subscription: googleapi: Error 409: Resource already exists in the project (resource=large-simulations-sub).
│
│   with google_pubsub_subscription.large-simulations-sub,
│   on main.tf line 268, in resource "google_pubsub_subscription" "large-simulations-sub":
│  268: resource "google_pubsub_subscription" "large-simulations-sub" { 

Variation 2

1. Created a service account with Owner permissions
2. Replace the project id with my project id
3. Replace the account id of the newly created service account
4. Run terraform apply

│ Details:
│ [
│   {
│     "@type": "type.googleapis.com/google.rpc.ResourceInfo",
│     "resourceName": "projects/moja-global/serviceAccounts/[email protected]"
│   }
│ ]
│ , alreadyExists
│
│   with google_service_account.fc-sa,
│   on main.tf line 81, in resource "google_service_account" "fc-sa":
│   81: resource "google_service_account" "fc-sa" {
│
╵
╷
│ Error: Error creating Subscription: googleapi: Error 409: Resource already exists in the project (resource=large-simulations-sub).
│
│   with google_pubsub_subscription.large-simulations-sub,
│   on main.tf line 268, in resource "google_pubsub_subscription" "large-simulations-sub":
│  268: resource "google_pubsub_subscription" "large-simulations-sub" {

Input data for simple modules like RothC are sent to the FLINTcloud backend in JSON format. The existing config example has several fields for each input data type when only one is necessary.

We can standardise this by consolidating all of the input data fields to only have one data value called data which is the default data_property used in FLINT.

For example:

Can be changed to:

 "rainfall": {
        "transform": {
          "allow_nulls": true,
          "library": "moja.flint.example.base",
          "type": "TimeSeriesTransform",
-          "data_property": "data_month_avg",
+          "data": [ 0.0, 24.00, 23.00, 25.00, 55.00, 78.00, 77.00, 89.00, 77.00, 63.00, 52.00, 39.00, 29.00, 24.70, 24.60, 22.90, 20.90, 18.70, 15.20, 14.20, 14.80, 17.60, 20.60, 22.90, 24.00, 22.60, 23.50, 31.00, 22.00, 19.00, 16.50, 13.00, 13.20, 18.00, 20.50, 22.40, 27.00, 23.00, 24.00, 26.00, 21.00, 18.60, 16.00, 14.00, 14.10, 17.20, 20.00, 23.00, 26.00 ],
-          "data_orig": [ 0.0, 24.00, 23.00, 25.00, 55.00, 78.00, 77.00, 89.00, 77.00, 63.00, 52.00, 39.00, 29.00, 24.70, 24.60, 22.90, 20.90, 18.70, 15.20, 14.20, 14.80, 17.60, 20.60, 22.90, 24.00, 22.60, 23.50, 31.00, 22.00, 19.00, 16.50, 13.00, 13.20, 18.00, 20.50, 22.40, 27.00, 23.00, 24.00, 26.00, 21.00, 18.60, 16.00, 14.00, 14.10, 17.20, 20.00, 23.00, 26.00 ],
-          "data_month_avg": [ 0.0, 24.00000, 23.00000, 25.00000, 55.00000, 78.00000, 77.00000, 89.00000, 77.00000, 63.00000, 52.00000, 39.00000, 29.00000, 24.70000, 24.60000, 22.90000, 20.90000, 18.70000, 15.20000, 14.20000, 14.80000, 17.60000, 20.60000, 22.90000, 24.00000, 22.60000, 23.50000, 31.00000, 22.00000, 19.00000, 16.50000, 13.00000, 13.20000, 18.00000, 20.50000, 22.40000, 27.00000, 23.00000, 24.00000, 26.00000, 21.00000, 18.60000, 16.00000, 14.00000, 14.10000, 17.20000, 20.00000, 23.00000, 26.00000, 23.57500, 23.77500, 26.22500, 29.72500, 33.57500, 31.17500, 32.55000, 29.77500, 28.95000, 28.27500, 26.82500, 26.50000 ],
-          "data_lastyearcopy": [ 0.0, 24.00, 23.00, 25.00, 55.00, 78.00, 77.00, 89.00, 77.00, 63.00, 52.00, 39.00, 29.00, 24.70, 24.60, 22.90, 20.90, 18.70, 15.20, 14.20, 14.80, 17.60, 20.60, 22.90, 24.00, 22.60, 23.50, 31.00, 22.00, 19.00, 16.50, 13.00, 13.20, 18.00, 20.50, 22.40, 27.00, 23.00, 24.00, 26.00, 21.00, 18.60, 16.00, 14.00, 14.10, 17.20, 20.00, 23.00, 26.00, 23.00, 24.00, 26.00, 21.00, 18.60, 16.00, 14.00, 14.10, 17.20, 20.00, 23.00, 26.00 ]
        }

This can be repeated for all timeseries data inputs. Once complete, you should test the updated config and verify that the simulation still runs.

Presently, in the docs, interaction with the GCBM example endpoints is possible only using the Postman collection provided. I would like to provide a command-line alternative using cURL to interact with the endpoints.

#63 included an updated Docker build for the GCBM container and #65 + #70 published them on the Github Container Repository.

However, #29 also publishes an image for the GCBM. The build in the FLINT.cloud repository can be simplified to draw from the correct source.

It's slightly circular that moja.canada pulls from FLINT.cloud then FLINT.cloud pulls from moja.canada, but this is necessary until #116 is merged upstream.

I tried to run the docker build -t flint-api . command to build the project locally, and I got the error above.

Currently, the SQLite version is causing issues when running the GCBM Belize project inside a FLINT Cloud container. Newer functions lie RANK/PARTITION do not work with the old version.

See the issue here for more details of the errors.

As discussed with @mfellows @aornugent on Slack, SQLite needs to be be updated to either this or a newer version:
https://www.sqlite.org/2018/sqlite-autoconf-3260000.tar.gz

@Tlazypanda - now that you've the first version of FLINTcloud documentation in #13, can you please show @sarthakkundra how to deploy new documentation on docs.moja.global?

@sarthakkundra we need to document the process itself (and then deploy the deployment instructions) so that other projects can migrate their documentation to a central repository.

The /upload endpoint present in the rest_api_gcmb accepts inputs present in the GCBM_Demo_Run.

The pre-processing script generates all the JSON files present in the /config folder.

The pre-processing script has three folders, one meant for the JSON and tiff files for Classifiers, one for disturbances, and one for annual_mean_temperature and initial_age. Based on this, we are planning to have three endpoints: /classifiers, /disturbances, /miscellaneous and /database (for the mapping --> spinup.json & variables.json.

In the end, it should look something like this:

inputs/
         name-of-simulation
                 disturbances
                 classifiers
                 miscellaneous

Though we are trying to generate all the JSONs that are being used as inputs, we were quite unsure about having to deal with the JSON files present in the /layers/tiled folder(since most of them are boilerplate but still can be parsed from their respective tiff files using raster objects). We will include that in the script as well.

@Namyalg has already proposed an API design for the configuration of the tables (from the SQL DB file) here. We shall be pushing our code to this particular repository trying to implement a basic design and later wrap it up for FLINT.Cloud.

CC: @aornugent @Namyalg

There are two folders rest_api_* that also exist in local/rest_api_* creating duplication and confusion.

I propose removing the top level folders and working within the local folder going forward.

Currently, the rest_api_flint.example has tests for /spec and /version, I would like to extend testing for all the other endpoints

Describe the bug

Binding to all network interfaces can potentially open up a service to traffic on unintended interfaces, that may not be properly documented or secured. This can be prevented by changing the code so it explicitly only allows access from localhost.

When binding to 0.0.0.0, you accept incoming connections from anywhere. During development, an application may have security vulnerabilities making it susceptible to SQL injections and other attacks. Therefore when the application is not ready for production, accepting connections from anywhere can be dangerous.

It is recommended to use 127.0.0.1 or a local host during the development phase. This prevents others from targeting your application and executing SQL injections against your project.

Currently, when the GCBM example simulation is run locally, fails. However then test suite for does not report a failure.
The behaviour observed is as follows :

1. /gcbm/new and /gcbm/upload function as expected in the test suite and locally
2. On running /gcbm/dynamic locally, the endpoint returns (200, {"Run started"}). Soon after the simulation fails. This is not covered in the test suite.
3. On a failed simulation, an output folder is created, however it is empty

Currently, the FLINT example does not have a cURL-based setup. The cURL setup will help users interact easily with the endpoints

I will work on refactoring the README.md, make it easier to read and navigate similar to https://github.com/moja-global/FLINT-UI and https://github.com/moja-global/community-website

Add the address http://localhost:8000 in the origins list in the CORs configurations as many people open the FLINT UI on localhost and not 127.0.0.1

The API requests do not work from localhost currently.

I have been trying to set up FLINT.Cloud on my local machine to test around, and I was trying to run the point and rothc examples as per the README.md examples.
On using the endpoints, I am facing timeouts even after waiting for almost 10 minutes.

For context, here are some images of my command line from running the Docker commands and the Postman endpoints I am hitting.

On using the /help/all endpoint, I do get a valid response, telling me that the

Here is an image of Postman sending a request to the /rothc endpoint for the last 10 minutes.

Please advise on what to do in this case.

Added workflows auto-labeller.yaml
Added github-issue labeller ci

@arnav-t has done some early analyses of the costs for different sized GCBM runs here:

• https://docs.google.com/spreadsheets/d/1F1se1fBGAQoFr0egH7lxJx2UDqvSDzXXQMpS9sFCUfM/edit?usp=drivesdk
• https://docs.google.com/spreadsheets/d/14qTzqExD_yjTZUR80xfxKbo6FBLFeUw-qT2M_6dJjYg/edit?usp=drivesdk

We should put these in the README/wiki alongside deployment instructions.

We are slowly adding different deployment scenarios. These should be organised by provider, then FLINT implementation:

| - local
|    | - example
|    | - gcbm
| - gcp
|   |  - layered
| - aws
...

I noticed that there is no Dockerfile or anything in the README that runs the Python tests during development. I would like to take this up but I just want to find out if it is worth working on or looking into.

The new /download endpoint in #7 is super cool, but I wondered if it's possible to make the use of S3 buckets optional. This would be useful for people that want to rent a VM or host a bare-metal server that has permanent volumes for storage.

Perhaps we can start by describing what we expect the default behaviour to be, then think about a design where the S3 credentials are added in without users having to edit the API source code directly. It's probably worth thinking through user patterns before beginning to code - it might be that users will always want to use S3 in which case this issue is moot.

Currently, the inputs to the model at the /gcbm/upload route are uploaded via a POST request, through this issue, a docker volume mounted will be referenced and used to prepare the inputs for the model

Describe how we are communicating and why it is not optimal.
As per the discussion on slack and from @mfellows and @aornugent suggestions, the gcbm endpoint has to be dynamic in nature. This entails all the -

• config files: All files in config directory (.json and .cfg files)
• db: Input databse file (gcbm_input.db)
• input: All files in layers/tiled directory (.tif files)

So the proposed solution is to have these as the parameters for the api endpoint and create a directory structure from this within the mounted volume (local run data shared by @mfellows).

Needs further discussion and work to properly perfect this to an extent that users find this pellucid.

Presently, the /gcbm/status route returning the message {"finished": message"}. This return statement can be renamed to {"status": message"} inorder to make more intutitive

As mentioned in the documentation, the endpoint /gcbm in the GCBM REST API demo return 404 not found