chicago / sgim_results

The Sustainable Green Infrastructure Monitoring (SGIM) project uses sensors, provided by Opti, that measures water diverted into the ground. This repo contains the ETL code which updates Chicago's open data portal. It can be reused to connect Opti sensors to Socrata open data portals. Depends on the "open-data-etl-utility-kit" repo.

Sustainable Green Infrastructure Monitoring ETL

The Sustainable Green Infrastructure Monitoring (SGIM) project uses sensors, provided by Opti, that measures water diverted into the ground. This repo contains the ETL code which uploads data from Opti's sensors to Socrata open data portals. It is a generalized program from the same code that uploads SGIM data to Chicago's open data portal.

This ETL depends on the open source "Open Data ETL Utility Kit" framework. The Utility Kit uses the open source Kettle data integration program to help automate uploads to a Socrata open data portal. This repository uses that Utility Kit and is further preconfigured to work with Opti's APIs.

Installation

This ETL requires you first install and configure the "Open Data ETL Utility Kit". Download that repository and follow the installation instructions.

ETL Installation

Place this repository in the ./open-data-etl-utility-kit/ETL/ directory. You may name the folder whatever you prefer, though you can see our recommended naming conventions.

Opti Credentials

Opti uses a token to authenticate users of their API. Obtain your token and place it in the credentials_sample.csv document and save it as credentials.csv. The ETL is pre-configured to read from the credentials.csv for authentication.

Configuring Automation

The bash script, ggws-77ih.sh, will run the ETL process. Open the bash script and see the instructions to configure the file names to match your directory.

Use a task scheduler, such as crontab to set a regular time to run the ETL. See the recommended configuration to setup a regular updating process while also logging the outcome of each update.

Windows Users

Automation is currently initiated by a Bash script, such as Cygwin to automate the process.

Structure

Below are the principal components of the repository and a brief description of their role.

• SGIM_Results_ggws-77ih.ktr - Main file which contains the ETL. This can be opened in Kettle (Spoon) and viewed.
• SGIM-Admin_DataStreams_Original.txt - Shows all of the streams available through Opti's ETL.
• SGIM-Admin_DataStreams_Edited.csv - The actual streams that will be used in the ETL. The list in the repository is the sensors currently displayed on Chicago's portal.
• ggws-77ih.sh - A bash script which runs the ETL.
• credentials_sample.csv - A sample of how credentials should be specified, however, it is not used in the ETL. Please see instructions to configure credentials.

Other files located in the repository not listed here either indirectly support the workflow or are adminstrative in nature. Note that .gitignore includes credentials.csv (excludes it from commits). You may wish to change this setting for internal-only forks of this repository within your organization but should be cautious if you later want to contribute back to this public repository.

License

See the license file.

Contributing

Contributions such as pull requests or opening issues are welcomed. If you plan on submitting a pull request, please complete a Contributor License Agreement.

sgim_results's Issues

Create README file

• A lengthier description of the project
• Instructions on setting-up the ETL

Instructions for Contributors

• Create CONTRIBUTING.md - Create a separate contribution instructions to provide instructions to potential contributors. Ensure the instructions refer to the need to sign a CLA
• Create CLAHub - Create the CLA for this repo in CLAHub. Integrate it with GitHub so it is required before accepting a pull request.