jxmot / node-dht-udp

SensorNet 2 of 3 - The server side to the esp8266-dht-udp project

node-dht-udp

This is the server component for the SensorNet project.

History

After the development on the esp8266-dht-udp project reached the point were a real server was necessary the test code was used as a starting point for this server.

Overview

This server has been implemented as a NodeJS application. It is required to listen for UDP packets that contain sensor data from the ESP8266 devices and then forward the data to a database.

It also listens for multi-cast UDP packets. They are used to convey the status of the ESP8266 devices.

Application Requirements

This NodeJs application must -

• Be as thin as possible. The intended target platform is the Tessel 2. However this application can be run on any NodeJS installation.
• Utilize a database where clients can be notified in real time when new records are written to the database. At this time Firebase is the only solution that offers that feature.
• Acts only as a means to forward data to the database. The sensor devices define the layout and contents of the data records. The only alteration to the data records is the addition of a timestamp.

Updated Requirements

As detailed below I decided to store the data in an SQL database. This may unfortunately prohibit the application from running on a thin platform. In order to accomplish this -

• The application may require some extra configuration to run on the thin platform. However it might encounter restrictions that are yet to be determined.
• The SQL database would have to reside on a separate server. The thin platform does not have the resources to run an SQL database.

Details

Database Candidates

Initially I decided to use one of two potential candidates for a database to store sensor data and status. My first choice was Firebase due to a number of its features -

• JSON data can used directly when writing data to a node.
• The client side can receive notifications when new data has been saved.

Choosing SQL vs NoSQL

I have some primary criteria for choosing SQL vs NoSQL -

• Will the individual records be dynamic in regards to the fields they contain? YES - use NoSQL like MongoDB or Firebase. NO The SQL might be the way to go. SQL databases require well defined schemas. And are not flexible, if a change is necessary it can be painful to migrate existing data to a new schema.
• How much overhead can you tolerate in your application? None, or very little. - The something like MongoDB or Firebase might be preferable. The advantage there is textual JSON can be used directly to write records to the database. In my humble opinon NoSQL database require less code to implement.
• Are queries (simple and complex) required? YES - The SQL is the better choice. Tip: MySQL Workbench is extremely useful for testing SQL statments and procedures before they end up in your code. On the other hand, you'll do your query development and testing in your code for the NoSQL alternatives.
• What level of security is required by your application? None or very little - then a NoSQL like MongoDB is ok. Basic to strong security is requried - Then SQL or a NoSQL database like Firebase should be sufficient.
• If you're developing for a customer (or employer) what are they currently using? Generally speaking it's better to "go with the flow" and work with what they have been using.

Other than the above I also look at what are the feature that my application will require. For example, my current project requried -

1. Control over the quantity of data that is stored and kept.
2. The client side of the application requires notification when a new record is written to the database.
3. Some basic queries are required.
4. The business logic side of the application (typically a NodeJS app) needs to have a small footprint. In other words, it may be required to run on a thin client.... like a Arduino platform or a Tessel 2.
5. There cannot be any monetary cost for hosting the application.

First I started with Firebase (NoSQL) but I quickly discovered that #1 would incur a monetary cost if #4 was satisfied. At that point I switch over to a SQL database (hosted on equipment in my home) and used Socket.io to satisfy #2, which Firebase does quite easily.

Firebase Requirements

This application makes use of the Firebase REST API. And it does not require any packages other than the native NodeJS packages.

In order to run for more than a relatively short period of time it is necessary to use some means to limit the amount of accumulated sensor data and status records. The method used here are Firebase Cloud Functions. For details please review this README.

MySQL Requirements

The same requirement exists for MySQL. A way to limit the quantity of data is required. Initially I had investigated the use of trigger functions as a way to run some SQL to delete specific rows. However trigger functions are not allowed to alter the contents of a table.

The solution I used was to -

• Utilize a timestamp as the primary key.
• Implement a long interval timer under NodeJS
  • When the timer expires it will delete a range of rows specified in a where clause by a timestamp value.

Details regarding this can be found in the MySQL README in this repository.

Running the Application

Depending upon which database type (Firebase or MySQL) is used determines a few things -

Firebase * It is not necessary to install any NodeJS packages. * In order to limit the quantity of saved data Firebase Cloud Functions are required. * This requires a billing account and will incur some monetary costs.

MySQL * It will be necessary to install some external NodeJS packages : npm install * The current web client will require some modifications.

To choose between Firebase and MySQL open the /servercfg.js file and edit it to read as -

    db : {
        type : 'mysql'
    }

Or -

    db : {
        type : 'firebase'
    }

Run the application with - node server.js.

Configuration

File Naming Convention

Some configuration files may contain sensitive information that should not be placed into a public repository. In order to prevent them from getting into the repository their filenames begin with an underscore. This is accomplished with an entry in this repository's .gitignore file. However there are example configuration files provided that to not have the underscore in their names.

Server/Client Configuration

The primary configuration is kept in servercfg.js -

module.exports = {

    server : {
        host : '0.0.0.0',
        port : 48431,
        reply: false
    },

    multi : {
        addr : '224.0.0.1',
        port : 54321,
    },
    
    db : {
        type : 'mysql'
    }
};

The server object represents the IP address and port on which the server will listen for sensor data. The use of host : '0.0.0.0' indicates that all network interfaces are to be used. If you need to listen on a specific interface then replace 0.0.0.0 with the IP address of that interface.

The multi object represents the address and port that will be used by the multi-cast client when listening for multi-cast UDP packets.

The db object is described above.

Firebase Configuration

This application utilizes a total of three configuration files -

• firebase-rest-config.js -
  • _firebase-config.js (see example_firebase-config.js)
  • _firebase-paths.js (see example_firebase-paths.js)

MySQL Configuration

For use of MySQL a single configuration file is used. The file is _dbcfg.js (see example_dbcfg.js) and it contains -

• A parms object used by the mysql package.
• Table names and column definitions.
• Data purge (row deletion) configuration object.

Details regarding this can be found in the MySQL README in this repository.

Future Development

Databases

In order to avoid the monetary costs of using Firebase alternative solutions are required. I'll need to see if any can fulfill these requirements -

• Limiting the quantity of records automatically, Firebase accomplishes this with Firebase Functions.
• Notifying the clients when a new record has been created (or updated?). On the client side this is easily done.

MySQL

A method utilizing websockets will be investigated for communicating current updates to sensor data.

MongoDB

Not sure if what I need is possible with a Mongo database. Further investigation is necessary, but is not likely to proceed at this time.

---

node-dht-udp's Issues

gather and relay additional data

• Sensors :

  • chart data : requires interaction with client

• Weather :

  • sunrise/sunset, moon phase, rise and set
  • alerts : smart, does not retrieve if no clients are connected. saves last alert less than X hours old.
  • almanac

Displayed timestamps are incorrect

The displayed(textual) timestamps are incorrect when the time is between 00:00(midnight) and 01:00. For example, 00:30 is seen as 24:00

Purge - give option to independently enable/disable status purges

Status purge is currently commented out.

Create a flag for disabling it in _dbcfg.js and mysql/index.js line ~142

Data purge failure

After the purge timer expires and calls /mysql/index.js:purgedata() an exception occurs :

TypeError: Cannot read property 'col' of undefined
    at Timeout.purgedata [as _onTimeout] (e:\Workspaces\IoT\ESP8266\arducam_K0061_ESP8266_KIT\repos\node-dht-udp\mysql\index.js:229:29)
    at ontimeout (timers.js:384:18)
    at tryOnTimeout (timers.js:244:5)
    at Timer.listOnTimeout (timers.js:214:5)

update all code to the ES6 specification where possible

All js files are affected.

ES6 Compatibility (6.10.2/3 in use on my platforms) : http://node.green/#ES2016

improve method for collecting weather data

Weather data collection is currently periodic only. Change it to be a combination of time base and periodic.

• On the first read of current/forecast data obtain the time stamp.
• Set an interval = the difference between now and the next update(usually timestamp + 1 hour)
• Add 5min to the interval and wait, reuse this interval

Preserve oldest sensor status on purges

The status purge erases the oldest sensor status and the client field goes blank. Improve the purge process so that it will retain the oldest status record.

README updates

• update the firebase doc, some info is missing
• create the MySQL README doc and update the link in the main README.

update socket.io version

server & client

Improve logging

Move log option code and flag checking out of modules, move it into Log.js.

Relay sensor historical data to client(s)

Modify the socket.io server so that it will respond to requests for historical data. Clients will use the existing socket connection and issue a history request.

Major Tasks:

• Create a testing client that can display graphs of sensor data COMPLETE
• Modify this app so that it can receive and respond to requests from the client
  • Recognizes requests COMPLETE
  • Assembles a database query COMPLETE - retrieves sensor data, added new columns to config table for color selection
  • Runs the query and returns the result to the client COMPLETE

Reduce number of log records

Remove "tracing" messages from the log by modifying 'log()` to take a 2nd argument to use in determining whether or not to write the record to the log.

Replace NOAA as weaher data source

The NOAA data is unreliable :

• Updates fail to occur for hours or days
• Data is incomplete or invalid
• Data is not correct

find replacement for string.js

https://www.npmjs.com/package/string

It has a vulnerability that has not been fixed. Find an alternative.

Add next scheduled purge info to purge status

This will also require changes in the client.

Fix Log.js path use

Modify lib/Log.js to accept a configured path for the log files. Will need code to split the path+filename so that on log file rollovers the timestamp can be prepended to the log file name.

Receive and monitor heartbeat messages from devices

This issue is related to -
jxmot/esp8266-dht-udp#5

After successful device startup the server will expect a heartbeat message from any device that has gone idle for a configured period of time.

Implement endpoint(s) for use by Alexa skill(s)

Implement a means to respond to http queries sent from AWS(requestor) on behalf of an Alexa skill.

Requirements:

• Must obtain necessary data with very little impact on the node-dht-udp application
• Must verify that requestor is valid. The requestor must supply a valid ID within the query that it sends.
• Will not access the MySQL database to obtain the necessary data. Instead it will "watch" a JSON formatted text file for changes. The node-dht-udp application will write to the file when ever data is updated. This is the same data block used in resending to new clients when they connect.
• Most aspects of this application will be configurable:
  • Address & port to listen on
  • Device<->Path table
  • Data file : path + name + extension
  • Unique requestor ID(s)
• Data request paths handled:
  • Request all sensors
  • Request a specific sensor
  • Request weather data (if stored in JSON file), must consider:
    • Service format
    • Forecast data period for a format
    • Observation data for a format
• Will not handle any "commands", this application will only provide data

Collect thermostat settings & state and broadcast to connected clients

Collect Data:

• environment - temp, humidity
• settings :
  • heat/cool/off
  • setpoint
• state
  • heat/cool/off
    
    
    
• Operations:
  • Use smallest possible payloads via UDP.
  • Settings data is sent on client connection, and only to that client.
  • Last known state is sent on client connection, and only to that client.
  • State changes are broadcast to all connected clients.
  • Setting updates are broadcast to all connected clients.
  • Environment changes are broadcast to all connected clients.
    
    
    
• Payload Contents:
  • time stamp
  • current temp
  • current humidity
  • payload type : setting/state/null(environment change)
    • setting
      • mode - heat/cool/off
      • setpoint
    • state
      • mode

Sensor statistics are not being updated after start-up

The symptoms are seen in the sensornet_chart client:

• In the date picker the oldest available date does not update
• The node app requires restart to correct

FIX: Must call mysql/index.js:sensorStats() after a data "purge" occurs.

Document new behavior

Document the changes and behavior created for #32 Relay sensor historical data to client(s)

Send SMS Alerts

To Do :

• Find a low-cost SMS service
• Create simple test/proof-of-concept
• Integrate and design -
  • Alert types -
    • over/under temp/humidity
    • error/recovery
    • priority statuses
• Investigate 2-way comm

Provide devices with the IP addresss of the server.

This issue is related to -
jxmot/esp8266-dht-udp#6

When requested via a UDP multi-cast message the server will reply with a packet containing the IP address it uses for UDP single-cast message. The device(s) will then send UDP single-cast messages to that address.

Fix Node deprecation warnings

After a recent update to my server's NodeJS(10 to 12) a few deprecation warnings have appeared. Need to identify all and their origins.

Look info packages loaded with NPM, they are likely to need updates.

Enable status data purge on start-up

Modify the app to purge the status data on start up just like it's done for the sensor data. Also review the purge timers and determine if an adjustment is necessary.

Detect and indicate trends in sensor data to the clients

After the sensor data record is written, and before it's sent to the client(s) compare current data to last data and indicate a positive, negative, or no change in a trend field in the outgoing data. It will contain one of the following :

• -1 = negative
• 0 = no change
• 1 = positive

MongoDB Investigation

Investigate use of MongoDB, update README as necessary.

Rename this repository

This repo is to be renamed as sensornet-server.

Collect weather service data and broadcast to clients

Collect Data:

• temperature, humidity, conditions, warnings/watches, alerts

Forward/Broadcast to all connected clients using smallest possible payloads via UDP.

Payload Contents:

• time stamp
• payload type : each type object can be can be different, common fields will reside in the parent object.
  • conditions
  • watch/warning
  • alert