Coder Social home page Coder Social logo

emoncms / demandshaper Goto Github PK

View Code? Open in Web Editor NEW
15.0 15.0 19.0 1.91 MB

Appliance, Smartplug, OpenEVSE demand scheduler: Find the best time to run household loads

License: GNU Affero General Public License v3.0

PHP 71.71% JavaScript 22.35% CSS 2.81% Shell 0.54% Hack 0.02% Python 2.58%

demandshaper's Introduction

Emoncms

Emoncms is an open-source web application for processing, logging and visualising energy, temperature and other environmental data and is part of the OpenEnergyMonitor project.

Emoncms

Requirements

  • PHP (tested with 8.1.12)
  • MySQL or MariaDB (tested with 10.5.15)
  • Apache (tested with 2.4.54)
  • Redis* (tested with 6.0.16)

*Redis is recommended because it reduces the number of disk writes and therefore prolongs disk life (noticeably on SD cards e.g. Raspberry Pi). Some input-processors also require Redis and fail silently if Redis is not installed. Some environments such as shared hosting or as far as we have tried Windows servers don't support Redis hence why Emoncms has a fall back mode that allows core operation without Redis.

Documentation

View the Emoncms documentation at: https://docs.openenergymonitor.org/emoncms

Design

Emoncms timeseries database design (feed storage)

Other

Emoncms Terminology

  • Input: An incoming datasource. Each input has an associated "node" identifier and a "key" sub-identifier. Inputs are entry points, only the last value and time of the input is recorded. To record historic data a feed needs to be created from an input.
  • Input: Node: A grouping identifier for an input or feed.
  • Input: Key: A sub-identifier for items within each Node.
  • Input process list (or input processing): A list of processes* performed sequentially on each input value as it is received on that input.
  • Process: A function that can be attached to the process list of an input to change the value or to save the value to a feed*.
  • Feed: A place where data is recorded, a time-series of datapoints. The standard time-series databases used by Emoncms are PHPFina and PHPTimeSeries and were written as part of the Emoncms project.
  • For a description of what each input process does in Emoncms, see the helper note within the Emoncms input processing configuration interface.

Emoncms.org API Reference

Install

Emoncms is designed and tested to run on either Ubuntu Linux (Local, Dedicated machine or VPS) or RaspberryPi OS. It should work on other Debian Linux systems though we dont test or provide documentation for installation on these.

We do not recommend and are unable to support installation on shared hosting or XAMPP servers, shared hosting in particular has no or limited capabilities for running some of the scripts used by emoncms. There is now a large choice of low cost miniature Linux VPS hosting solutions that provide a much better installation environment at similar cost.

Recommended:

docker standalone container

An easy way to start with emoncms is to use the all-in-one docker container

A pipeline using github actions is producing builds with latest emoncms stable version for different architectures : amd64, arm64, arm/v7

These docker images, based on the alpine linux distribution, are designed for iot. Images are tagged using alpine and emoncms versions, for example alpine3.19_emoncms11.4.11.

The images have onboard :

  • the mariadb and redis databases,
  • the mosquitto mqtt broker,
  • the main modules : graph, sync, backup, dashboard and app,
  • the workers : emoncms_mqtt, service-runner and feedwriter.

You can easily :

  • deactivate the low-write
  • use an external broker.

To pull the latest image for testing :

sudo docker pull alexjunk/emoncms

More on https://emoncms-docker.github.io

Experimental

not currently up to date

Multi-platform using Docker Container

Modules

Modules can be installed by downloading or git cloning into the emoncms/Modules folder. Be sure to check for database updates in Administration menu after installing new modules. The following core modules are included on the emonSD image:

There are many other available modules such as the event module and openbem (open source building energy modelling module): check out the Emoncms repo list.

Branches

  • master - The latest and greatest developments. Potential bugs, use at your own risk! All pull-requests should be made to the master branch.

  • stable - emonPi/emonBase release branch, regularly merged from master. Slightly more tried and tested. See release change log.

Tools

  • PHPFina data file viewer - Easily explore phpfina timeseries feed engine data files directly without a full Emoncms installation. Useful for checking backups and archived data.

Android App

Google Play

GitHub Repo

Development Forum

More information

demandshaper's People

Contributors

borpin avatar dakaix avatar danbates2 avatar emrysr avatar glynhudson avatar jpalo avatar reedy avatar sijones avatar trystanlea avatar weebull avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

demandshaper's Issues

device won't delete

Just a note.
Can't seem to delete a device.
Helpful for dev to be able to do this.

ubuntu problem

root@ubuntu:~#  service demandshaper status
โ— demandshaper.service - Emoncms Demandshaper service
   Loaded: loaded (/lib/systemd/system/demandshaper.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Mon 2020-11-02 01:10:10 EET; 4s ago
     Docs: https://github.com/emoncms/demandshaper/blob/master/readme.md
  Process: 2230 ExecStart=/usr/bin/php /opt/emoncms/modules/demandshaper/demandshaper_run.php (code=exited, status=255)
 Main PID: 2230 (code=exited, status=255)

Nov 02 01:10:10 ubuntu systemd[1]: demandshaper.service: Main process exited, code=exited, status=255/n/a
Nov 02 01:10:10 ubuntu systemd[1]: demandshaper.service: Failed with result 'exit-code'.





root@ubuntu:~#  php /opt/emoncms/modules/demandshaper/demandshaper_run.php
PHP Warning:  Uncaught ErrorException: require_once(Modules/device/device_model.php): failed to open stream: No such file or directory in /opt/emoncms/modules/demandshaper/demandshaper_run.php:72
Stack trace:
#0 /opt/emoncms/modules/demandshaper/demandshaper_run.php(72): exceptions_error_handler()
#1 /opt/emoncms/modules/demandshaper/demandshaper_run.php(72): require_once()
#2 {main}
  thrown in /opt/emoncms/modules/demandshaper/demandshaper_run.php on line 72

Warning: Uncaught ErrorException: require_once(Modules/device/device_model.php): failed to open stream: No such file or directory in /opt/emoncms/modules/demandshaper/demandshaper_run.php:72
Stack trace:
#0 /opt/emoncms/modules/demandshaper/demandshaper_run.php(72): exceptions_error_handler()
#1 /opt/emoncms/modules/demandshaper/demandshaper_run.php(72): require_once()
#2 {main}
  thrown in /opt/emoncms/modules/demandshaper/demandshaper_run.php on line 72
PHP Fatal error:  main(): Failed opening required 'Modules/device/device_model.php' (include_path='.:/usr/share/php') in /opt/emoncms/modules/demandshaper/demandshaper_run.php on line 72

Fatal error: main(): Failed opening required 'Modules/device/device_model.php' (include_path='.:/usr/share/php') in /opt/emoncms/modules/demandshaper/demandshaper_run.php on line 72

EmonCMS error

Error when loading demandshaper web page:
Screenshot 2019-11-05 at 17 47 38

any ideas as to what this is caused by?

Message: TypeError: result.ctrl_mode is undefined
Route: Modules/demandshaper/general.js?v=10
Line: 481
Column: 21
Cheers
Tim Robinson

Demand shaper fills emoncms.log with errors

Every few seconds the demandshaper create an error log in emoncms.log, the entries are not really error and should not really be logged using ERROR level. Is the demand shaper.log sufficient? Maybe demand shaper does not need to log anything to emocnms.log unless there is an actual error?

tail emoncms.log -n10
2020-06-07 02:05:00.044|ERROR|demandshaper_run.php|EVSE timeleft: 3109.0909090909
2020-06-07 02:16:00.066|ERROR|demandshaper_run.php|Recalculating EVSE schedule based on emoncms input: 0
2020-06-07 02:16:00.066|ERROR|demandshaper_run.php|EVSE timeleft: 3109.0909090909
2020-06-07 02:27:00.044|ERROR|demandshaper_run.php|Recalculating EVSE schedule based on emoncms input: 0
2020-06-07 02:27:00.044|ERROR|demandshaper_run.php|EVSE timeleft: 3109.0909090909
2020-06-07 02:38:00.414|ERROR|demandshaper_run.php|Recalculating EVSE schedule based on emoncms input: 0
2020-06-07 02:38:00.415|ERROR|demandshaper_run.php|EVSE timeleft: 3109.0909090909
2020-06-07 02:49:00.667|ERROR|demandshaper_run.php|Recalculating EVSE schedule based on emoncms input: 0
2020-06-07 02:49:00.667|ERROR|demandshaper_run.php|EVSE timeleft: 3109.0909090909
2020-06-07 03:00:00.021|ERROR|demandshaper_run.php|Recalculating EVSE schedule based on emoncms input

Compatability with OpenEVSE WiFi V4.0.0+

Version V4.0.0+ of OpenEVSE WiFi firmware has changed control of the EVSE timer schedule from the OpenEVSE controller (RAPI commands) to the OpenEVSE WiFi.

V4.0.0+ WiFi firmware has been shipping on new units since March 2021.

It's important that the time schedule is not set via RAPI when running V4.0.0+ WiFi FW since this causes a race condition conflict. Here is the new HTTP POST schedule API Demand Shaper should use when setting the time schedule:

https://github.com/OpenEVSE/ESP32_WiFi_V4.x/blob/master/test/schedule.http

The version of the WiFi firmware can be found via /status output

RAPI commands will soon be disabled on the next release of controller firmware

Demand shaper interfering with OpenEVSE

It's been reported that Demand Shaper on the emonPi can automatically setup and communicate with the OpenEVSE without user input which can be very confusing: https://community.openenergymonitor.org/t/eco-pv-divert-not-working-after-upgrade-to-v4/17780/27

Short term: Maybe we need a clear way to disable demand shaper in the shape of a big disable toggle. Demand shaper should be disabled by default. When enabled users should be warned not to use the OpenEVSE interface.

Long term: Look at integrating demand shaper into the OpenEVSE interface on the ESP32 OpenEVSE/openevse_esp32_firmware#4

Can't set time using Chrome (desktop) or Firefox (mobile)

Using Chrome desktop version I can't seem to increment the "complete by" time, getting these errors:
screenshot 2019-03-02 at 22 27 41

On firefox mobile I get the following error when loading the page:
screenshot 2019-03-02 at 22 27 15

I managed to set the time using Chrome mobile, the '+' and '-' buttons would not work on the interface but tapping the time brought up the Android time picker

Disabling "OK to interupt" doesn't re-enable balancing.

With the following state, the "Time left: is 1h50m", there's "105 mins left to run", it's scheduled for "3:15 to 5:00" (also 105 minutes, 1h45), and there is 3x30 minute segments highlighted to run.

Minor Issue: I'd expect any segment used, even partially, to be highlighted.

Screenshot 2020-03-03 at 00 49 16

Settings:

  • Battery Capacity: 27kWh
  • AC Charge Rate: 7.4 kW
  • Balancing %: 95%
  • Balancing Time: 90 mins
  • Current SoC: 91%
  • Target Soc: 100%
  • OK to Interrupt: False
  • Time to complete: 16:00

Switching "Ok to Interrupt" to true causes a recalculation where balancing is ignored, and a single 30m segment is scheduled.

Screenshot 2020-03-03 at 01 02 06

Main Issue: Returning "Ok to Interrupt" to False leaves us with the same solution, and only a single 30m segment scheduled.

Screenshot 2020-03-03 at 01 14 02

Changing the target SoC adds the balancing time back.

Demandshaper side bar menu missing on sidebar branch

The demand shaper makes use of the side bar is show a menu with the various items which the demandshaper can be used with e.g:

fa30cd7fa1f7ed6653aae87c85f3cda7e2db2c2a

After switching to the sidebar branch the demandshaper sidebar is no longer present, replaced with bullet points at the top of the page. Sidebar functionality should be resorted similar to App module

Screenshot 2019-04-08 at 10 29 42

Nordpool not working

Hi,
I started fixing issue with Nordpool signal as the demandshaper_run.php always set the smart schedule to start at (complete_time-run_period), and end at (complete_time) regardless of actual signal prices, but ended up in a situation that caused stuck relay error. I assume there is some issue in the code causing rapid cycle of turning charging on/off at the time when smart schedule has started charging. Looking at the charging history, it hasn't charged at all during that time AND I see the scheduled start time being changed...as if it would modify the smart schedule during charging.

If you could take a look at this https://github.com/jpalo/demandshaper/tree/nordpoolfixes. It fixes the smart schedule timing issue (as there isn't always 24h future signal data available), but would appear to cause issues.

Top banner 'Devices'

It is confusing to use 'Devices' in the top banner for the Demandshaper module, when there is a 'device' module and devices in that context mean something different.

image

Custom Day-Ahead feed

Hello,

How to integrate this custom day-ahead prices feed into DS?

It comes as the following from the API, price per hour, 48 hours (today + tomorrow):
{ "price": { "1661724000000": 1155.6, "1661727600000": 1064.25, "1661731200000": 1023.49, "1661734800000": 1018.99, "1661738400000": 1032.44, "1661742000000": 1152.1, "1661745600000": 1390.4, "1661749200000": 1438.06, "1661752800000": 1482.36, "1661756400000": 1426.25, "1661760000000": 1310.93, "1661763600000": 1214.57, "1661767200000": 1371.04, "1661770800000": 1371.04, "1661774400000": 1379.86, "1661778000000": 1409.02, "1661781600000": 1507.51, "1661785200000": 1584.85, "1661788800000": 1638.77, "1661792400000": 1703.53, "1661796000000": 1630.65, "1661799600000": 1564.66, "1661803200000": 1478.69, "1661806800000": 1379.78, "1661810400000": 1207.71, "1661814000000": 1163.86, "1661817600000": 826.85, "1661821200000": 803.85, "1661824800000": 870.38, "1661828400000": 1198.04, "1661832000000": 1475.32, "1661835600000": 1521.75, "1661839200000": 1500, "1661842800000": 1214.57, "1661846400000": 757.24, "1661850000000": 817.26, "1661853600000": 1200.04, "1661857200000": 1477, "1661860800000": 1463.66, "1661864400000": 1510.6, "1661868000000": 1563.71, "1661871600000": 1586, "1661875200000": 1750, "1661878800000": 1831.3, "1661882400000": 1745.13, "1661886000000": 1623.03, "1661889600000": 1246.43, "1661893200000": 1142.73 } }

Exception thrown

Demandshaper sometimes throws this exception.

This was seen on a restart of the service after a clean install, on Ubuntu, of stable branch - but it does not always happen (sadly).

Suggest code is inserted to catch the exception cleanly.

Oct 01 21:14:20 ubunt-emon demandshaper[1971]: PHP Fatal error:  Uncaught ErrorException: Invalid argument supplied for foreach() in /opt/emoncms/modules/demandshaper/demandshaper-module/demandshaper_model.p
Oct 01 21:14:20 ubunt-emon demandshaper[1971]: Stack trace:
Oct 01 21:14:20 ubunt-emon demandshaper[1971]: #0 /opt/emoncms/modules/demandshaper/demandshaper-module/demandshaper_model.php(64): exceptions_error_handler(2, 'Invalid argumen...', '/opt/emoncms/mo...', 64,
Oct 01 21:14:20 ubunt-emon demandshaper[1971]: #1 /opt/emoncms/modules/demandshaper/demandshaper_run.php(361): DemandShaper->set(1, Object(stdClass))
Oct 01 21:14:20 ubunt-emon demandshaper[1971]: #2 {main}
Oct 01 21:14:20 ubunt-emon demandshaper[1971]:   thrown in /opt/emoncms/modules/demandshaper/demandshaper-module/demandshaper_model.php on line 64
Oct 01 21:14:20 ubunt-emon demandshaper[1971]: Fatal error: Uncaught ErrorException: Invalid argument supplied for foreach() in /opt/emoncms/modules/demandshaper/demandshaper-module/demandshaper_model.php:64
Oct 01 21:14:20 ubunt-emon demandshaper[1971]: Stack trace:
Oct 01 21:14:20 ubunt-emon demandshaper[1971]: #0 /opt/emoncms/modules/demandshaper/demandshaper-module/demandshaper_model.php(64): exceptions_error_handler(2, 'Invalid argumen...', '/opt/emoncms/mo...', 64,
Oct 01 21:14:20 ubunt-emon demandshaper[1971]: #1 /opt/emoncms/modules/demandshaper/demandshaper_run.php(361): DemandShaper->set(1, Object(stdClass))
Oct 01 21:14:20 ubunt-emon demandshaper[1971]: #2 {main}
Oct 01 21:14:20 ubunt-emon demandshaper[1971]:   thrown in /opt/emoncms/modules/demandshaper/demandshaper-module/demandshaper_model.php on line 64

MQTT setting of OpenEVSE demandshaper mode

It would be useful to allow a home automation system to control the demandshaper operating mode for OpenEVSE to prevent a schedule affecting other operation. One example would be setting to On for pre-heating.

406 error when not logged in

If the demandshaper page is visited when the user is not logged in, the following 406 error appears. The demandshaper should re-direct to the Emoncms login like other modules do with the message "authentication required".

Screenshot from 2020-09-17 10-55-54

Integrating EV SOC (State of Charge) data: OVMS

For demandshaper to be useful for EV charge scheduling the demand shaper ideally needs to know the current SOC (state of charge of an EV). Unfortunately it's not possible to obtain SOC data via the J1772 EV charging protocol.

Various EV's have different proprietary API's retrieve SOC. The API's are often unpublished and reverse engineered. We will try and add support for all EV's (starting the Tesla). However, as a first implementation we have tested integrating with OVMS (Open vehicle Monitoring system). OVMS is an open-source hardware/software telematics platform that supports many types of EV's and is totally independent to the manufacturer.

See OVMS user guide

Data can be retrieved from OVMS via their mobile app, MQTT or HTTP rest API (if using dexters-web OVMS server, default in Europe). See issue openvehicles/Open-Vehicle-Monitoring-System-3#178

Here is an example API call to retrieve OVMS data:

https://dexters-web.de/api/call?fn.name=ovms/export&fn.vehicleid=XXXXXX&fn.carpass=XXXXX&fn.format=csv&fn.types=D,S&fn.last=1
Example responce:

#FILE,d.csv
rec_time,0,doors1,doors2,lockunlock,tpem,tmotor,tbattery,trip,odometer,speed,parktimer,ambient,doors3,staletemps,staleambient,vehicle12v,doors4,vehicle12v_ref,doors5,tcharger,vehicle12v_current
2019-03-13 17:44:08,0,192,0,5,11,0,12,0,434390,0,0,11,0,1,1,14.41,0,12.6,0,0,0,18
#FILE,s.csv
rec_time,0,soc,units,linevoltage,chargecurrent,chargestate,chargemode,idealrange,estimatedrange,chargelimit,chargeduration,chargeb4,chargekwh,chargesubstate,chargestateN,chargemodeN,chargetimer,chargestarttime,chargetimerstale,cac100,charge_etr_full,charge_etr_limit,charge_limit_range,charge_limit_soc,cooldown_active,cooldown_tbattery,cooldown_timelimit,charge_estimate,charge_etr_range,charge_etr_soc,idealrange_max,chargetype,chargepower,battvoltage,soh
2019-03-13 17:44:08,0,28.0,M,0,0,stopped,standard,24,0,0,0,0,0,13,21,0,0,0,0,57.22,0,0,0,0,-1,0,0,0,0,0,0,0,0.00,368.50,86
rec_time 0 soc units linevoltage chargecurrent chargestate chargemode idealrange estimatedrange chargelimit chargeduration chargeb4 chargekwh chargesubstate chargestateN chargemodeN chargetimer chargestarttime chargetimerstale cac100 charge_etr_full charge_etr_limit charge_limit_range charge_limit_soc cooldown_active cooldown_tbattery cooldown_timelimit charge_estimate charge_etr_range charge_etr_soc idealrange_max chargetype chargepower battvoltage soh
17:44:08 0 28.0 M 0 0 stopped standard 24 0 0 0 0 0 13 21 0 0 0 0 57.22 0 0 0 0 -1 0 0 0 0 0 0 0 0.00 368.50 86
rec_time 0 doors1 doors2 lockunlock tpem tmotor tbattery trip odometer speed parktimer ambient doors3 staletemps staleambient vehicle12v doors4 vehicle12v_ref doors5 tcharger vehicle12v_current
2019-03-13 17:44:08 0 192 0 5 11 0 12 0 434390 0 0 11 0 1 1 14.41 0 12.6 0 0 0 18

openevse unexpected gap in charging

I've noticed a couple of times now an unexpected gap during charging using demandshaper.

Here is some information from the most recent occurrence:

demandshaper set for 80% by 11:00 with no interruption
battery state input not available, so default initial charge 20%
calculated 3h9m 12kWh and selected 01:45 to 05:00

car set to terminate at 80% from initial charge of 18%

charging ran 01:45 to 05:01 and 05:30 to 05:38 (ended by car)

annotated log:

2019-11-14 00:06:09.948 | openevse set timer 01 45 05 00
2019-11-14 02:01:00.771 | openevse set timer 02 00 05 15
2019-11-14 02:31:00.628 | openevse set timer 02 30 05 45
2019-11-14 03:01:00.945 | openevse set timer 03 00 06 15
2019-11-14 03:31:00.055 | openevse set timer 03 30 06 45
2019-11-14 04:01:00.580 | openevse set timer 04 00 07 15
2019-11-14 04:31:00.079 | openevse set timer 04 30 07 45
2019-11-14 05:01:00.010 | openevse set timer 05 30 08 45 - expected 05 00 08 15
2019-11-14 06:01:00.105 | openevse set timer 06 00 09 15
2019-11-14 06:31:00.582 | openevse set timer 06 30 09 45
2019-11-14 07:01:00.872 | openevse set timer 07 00 10 15
2019-11-14 07:31:00.218 | openevse set timer 07 45 11 00 - looks wrong (but car already charged)
2019-11-14 08:01:00.548 | openevse set timer 08 00 11 00
2019-11-14 08:31:00.814 | openevse set timer 08 30 11 00
2019-11-14 09:01:00.443 | openevse set timer 09 00 11 00
2019-11-14 09:31:00.943 | openevse set timer 09 30 11 00
2019-11-14 10:01:00.740 | openevse set timer 10 00 11 00
2019-11-14 10:31:00.842 | openevse set timer 10 30 11 00
2019-11-14 10:34:00.715 | openevse set timer 10 30 11 00 - repeat
2019-11-14 11:01:00.990 | openevse set timer 11 00 14 15 - not expected
2019-11-14 11:01:00.991 | openevse schedule complete
2019-11-14 11:02:00.998 | openevse set timer 01 45 05 00

24 hour continous rescheduling

Is it possible to set a device schedule to 24/7 and do continuous rescheduling?
At the moment It doesn't matter how I set it, the data after "Complete by:" gets grayed out (not considered by the algo).

The idea is to run 4 big water heaters (4 tons total) 24/7 when the electricity pricing is lower throughout the 24 hours.
Can DS also monitor the temperature reported and use that along with the mWh pricing?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.