ibmstreams / streamsx.waterconservation.starterkit

Starter kit for smart and connected sprinkler system using Apache Edgent, Streaming Analytics and Insights for Weather

streamsx.waterConservation.starterKit

Notice

This starter kit can no longer be deployed since the Weather Company service is no longer available on IBM Cloud.

You can still modify the code and use its architecture as a reference.

Overview

Water scarcity is one of the most pressing problems in the world right now. According to the World Wildlife Fund (WWF), 70% of our planet is covered by water. However, only 3% of it is fresh water, and only one-third of that is available for consumption. Many water systems are stressed due to the growing human population and over-consumption. Agriculture consumes more water than any other source. Much of the water used in agriculture is wasted due to inefficient irrigation systems. To alleviate this problem, many governments impose restrictions on water usage, especially in drought-stricken states like California. However, these bans are difficult to enforce. For example, some people can afford to violate the restrictions and simply pay the fines.

To improve irrigation efficiency and properly enforce water usage restrictions, this starter kit is a prototype for a smarter, connected sprinkler system.

Water Conservation Application Architecture

The starter kit demonstrates the reference architecture of using Apache Edgent and how it can work in conjunction with a centralized analytics system.

This system consists of two main analytics components:

• Analytics on edge devices: We used a Raspberry Pi to simulate a smart sprinkler device. On the device, local soil conditions like moisture level are continuously collected and analyzed using Apache Edgent. When the device detects that the soil is too dry and requires watering, it sends a request to the centralized analytics system for permission to turn on the sprinkler. The smart sprinkler system will not turn on the sprinkler unless it is approved by the centralized analytics system.
• Centralized streaming analytics system: In the centralized analytics system, we used the Streaming Analytics service to analyze the incoming water requests in real time. To approve a water request, the application checks the weather forecast for the next two days using the Weather Company Data API. If there is not enough precipitation in the forecast, it checks to see if a water ban is currently in effect. The Streams application will approve the water request only if there is not enough precipitation in the forecast and if there is no water ban in effect.

For rapid application development, we leveraged other IBM Bluemix services. These services handle the infrastructure and connectivity, allowing us to focus on developing the analytics and business logic of the application. For the device to communicate with the centralized analytics system, we used the Watson Internet of Things Platform. This service handles application connectivity for us, allowing the Apache Edgent application to communicate with the IBM Streams application easily via MQTT.

To visualize the data, we implemented a visualization server using the SDK for Node.js service, and we used React.js to implement a web-based dashboard.

This smart sprinkler system now enables more efficient water irrigation based on current soil conditions. The prototype only takes the moisture level into account, but the system could be extended to monitor and analyze other important factors like moisture evaporation rate, soil mineral content, or irrigation requirements based on the type of vegetation being planted. Controlling the sprinklers in a centralized manner enabled us to implement business rules for managing water resources and properly enforcing water restrictions when necessary.

See the Google+ Hangout Event for more detail.

NOTE: The jobs in the centralized streaming analytics system are configured to run for one hour only. You can start or stop the jobs manually under the 'Settings' menu.

Services Used in this Starter Kit

Analytics on Edge Device

The moisture sensing simulator application is written using Apache Edgent. The simulator can be run on any workstation. It can also be run with a moisture sensor and buzzer on a Raspberry Pi.

*Apache Edgent is an effort undergoing Incubation at The Apache Software Foundation (ASF), sponsored by the Incubator.

Centalized Streaming Analytics System

The centralized streaming analytics system uses the following Bluemix services:

• Weather Company Data for IBM Bluemix
• Watson IoT Platform
• Streaming Analytics Service
• SDK for Node.js

An demonstration of the centralized analytics system is hosted here: http://waterconservation.mybluemix.net/.

Getting Started

To try out this starter kit:

1. Deploy the centralized analyltics application onto Bluemix, following the instructions below.
2. Follow instructions here to set up device configuration file.
3. Download a copy of the Apache Edgent application from the release page. Run the application on your workstation or on a Raspberry Pi.

Deploy to Bluemix

Prerequisites

Create a Bluemix account.

Option 1: Automatic Deployment

Click the button below to fork into IBM DevOps Services and deploy your own copy of this application on Bluemix, with a DevOps toolchain already configured.

We are deploying a number of applications onto Bluemix. This process will take about 15 minutes to complete.

Option 2: Manual Deployment

In Bluemix:

1. Create a Streaming Analytics Service with the name Streaming-Analytics
2. Create a Weather Company Data Service with the name Weather-Company-Data
3. Create a Watson IoT Platform Service with the name Internet-of-Things-Platform

Then:

• cd nodejs
• npm install
• npm run build
• cd ..
• cf push -n <host prefix>

Deploy locally

• cd nodejs
• modify app/server/config/application-local.js, and fill in the bluemix services credentials
• npm install
• npm start

By default, the server starts on localhost:3000

Running the Apache Edgent Application

Refer to com.ibm.streamsx.smartsprinkler.edgent/README.md for further information.

Project File Structure

root
│   manifest.yml (Bluemix deployment configuration file)
|   README.md (The file you are reading now)
|   LICENSE.md (License Terms)
|   copysabs.sh (Scripts for copying streams SAB files into nodejs folder)
│
├───.bluemix (Bluemix pipeline configuration file)
├───readmeImg (Image files for README.md)
├───com.ibm.streamsx.smartsprinkler.iot (Streams project for connecting to IOT)
├───com.ibm.streamsx.smartsprinkler.streams (Streams project for making water decisions)
├───com.ibm.streamsx.smartsprinkler.edgent (Java project for gathering sensor data on Raspberry Pi)
└───nodejs (Node.js project for dashboard)

Component Dependencies

The Java application running on Raspberry Pi require:

• Pi4J library
• Edgent library

The components required by the Node.js app are documented in the package.json in the nodejs directory. The list include:

• async
• body-parser
• bunyan
• chart.js
• d3
• express
• http-proxy
• http-status-codes
• lodash
• material-ui
• moment
• multer
• react
• react-chartjs
• react-dom
• react-sparklines
• react-tap-event-plugin
• request
• socket.io
• superagent
• babel-core
• babel-eslint
• babel-loader
• babel-preset-es2015
• babel-preset-react
• eslint
• eslint-loader
• eslint-plugin-react
• react-hot-loader
• webpack
• webpack-dev-server

License

The source code for the app is available under the Apache license, which is found in LICENSE in the root of this repository.

streamsx.waterconservation.starterkit's Issues

Vulnerability in dependency "webpack-dev-server" found in nodejs/package.json

Remediation

Upgrade webpack-dev-server to version 3.1.11 or later. For example:

"dependencies": {
"webpack-dev-server": ">=3.1.11"
}

or…

"devDependencies": {
"webpack-dev-server": ">=3.1.11"
}

Always verify the validity and compatibility of suggestions with your codebase.
Details
CVE-2018-14732
More information
low severity
Vulnerable versions: < 3.1.11
Patched version: 3.1.11

An issue was discovered in lib/Server.js in webpack-dev-server before 3.1.11. Attackers are able to steal developer's code because the origin of requests is not checked by the WebSocket server, which is used for HMR (Hot Module Replacement). Anyone can receive the HMR message sent by the WebSocket server via a ws://127.0.0.1:8080/ connection from any origin.

edgent smart sprinkler app does not connect to Bluemix stage 1 env

Yesterday I deployed the water conservation sample in the Bluemix stage1 env. I had to make a few changes to work in stage1 env (with the help of Kendrick) but in the end the Bluemix app deployed OK. However, when I try to run the edgent smart sprinkler app, it tries to connect to the wrong mqtt host. It is missing the "staging" from the host name. How can I change that?

C:\Dev_Streams\WaterIvanAuto>java -jar smartsprinkler.edgent-1.1.0.jar
UI host configure to: http://WaterIvanAuto.stage1.mybluemix.net
Running Simulation: true
Moisture: 946.9408616119974
Sprinkler: false
Avg: 946.9408616119974
Aug 23, 2016 8:58:14 AM com.ibm.iotf.client.AbstractClient createClient
INFO: pool-1-thread-1-2822ab31-0af4-4e3b-8240-314af158b3da: Org ID = cvsmok
Client ID = d:cvsmok:iotSprinkler:rPiSprinkler1
Aug 23, 2016 8:58:15 AM com.ibm.iotf.client.AbstractClient connect
INFO: pool-1-thread-1-2822ab31-0af4-4e3b-8240-314af158b3da: Connecting client d:cvsmok:iotSprinkler:rPiSprinkler1 to ssl://cvsmok.messaging.internetofthings.ibmcloud.com:8883 (attempt #1)...

My mqtt host name: "mqtt_host": "cvsmok.messaging.staging.internetofthings.ibmcloud.com",

change name of the default branch

as per guidelines the "main" branch shall no longer have the name "master"

Vulnerabilities found in eslint

Details

WS-2018-0347

Vulnerable versions: < 4.18.2
Patched version: 4.18.2

A vulnerability was descovered in eslint before 4.18.2. One of the regexes in eslint is vulnerable to catastrophic backtracking.

WS-2018-0592

Vulnerable versions: < 4.18.2
Patched version: 4.18.2

A vulnerability was descovered in eslint before 4.18.2. One of the regexes in eslint is vulnerable to catastrophic backtracking.

Remediation

Upgrade eslint to version 4.18.2 or later. For example:

"dependencies": {
  "eslint": ">=4.18.2"
}

or…

"devDependencies": {
  "eslint": ">=4.18.2"
}

Always verify the validity and compatibility of suggestions with your codebase.

Handle vulnerability in package.json dependency

GitHub sent this warning:

We found a potential security vulnerability in one of your dependencies.

A dependency defined in nodejs/package.json has known security vulnerabilities and should be updated.

Vulnerability found in 'superagent' refered by nodejs/package.json

Remediation:

Upgrade superagent to version 3.7.0 or later. For example:

"dependencies": {
"superagent": ">=3.7.0"
}

or…

"devDependencies": {
"superagent": ">=3.7.0"
}

Always verify the validity and compatibility of suggestions with your codebase.

Details

CVE-2017-16129
More information
low severity
Vulnerable versions: < 3.7.0
Patched version: 3.7.0

The HTTP client module superagent is vulnerable to ZIP bomb attacks. In a ZIP bomb attack, the HTTP server replies with a compressed response that becomes several magnitudes larger once uncompressed. If a client does not take special care when processing such responses, it may result in excessive CPU and/or memory consumption. An attacker might exploit such a weakness for a DoS attack. To exploit this the attacker must control the location (URL) that superagent makes a request to.

Starter kit deployment issues

1- Weather Data service is no longer free, so automatic Deploy To IBM Cloud will fail for anyone with a non-paid IBM account
2- Streaming Analytics service V1 has been removed, meaning no new instances can be created. Changing the starter kit to use V2 service and API requires some work.

I am going to mark this and the IoT starter kit as unavailable for now until this is resolved.

@petenicholls FYI

Web app hosted locally or on bluemix does not load

Followed the instructions in the readme to deploy locally.
My web page loads at localhost:3000 but I have a blank page with JS error TypeError: Super expression must either be null or a function, not undefined in the console.

Running at https://streamsxwaterconservationstarterkit-20170919180133191.mybluemix.net with the same error.
Tried to debug it but it seems to be a very generic error message so I have not found an answer. Seems to be a React issue.

waterconservation.mybluemix.net is down

Visiting that link, I get

404 Not Found: Requested route ('waterconservation.mybluemix.net') does not exist.

Update Deploy to Bluemix URL #14

Update Deploy to Bluemix URL #14
See pull request #14