Coder Social home page Coder Social logo

emonscripts's Introduction

Emoncms Stack Installer & Update scripts

Including: emoncms core, emoncms modules, emonhub, apache2, mysql, redis, mqtt, dependencies and configuration.

Install the Emoncms Stack

To install the stack see:
https://github.com/openenergymonitor/EmonScripts/blob/master/docs/install.md

Update the Emoncms Stack

Run the update via the Emoncms administration page, alternatively for command line level update see:
https://github.com/openenergymonitor/EmonScripts/blob/master/update/readme.md

Understand the EmonPi file System

Before diving into these scripts, there are some things you need to know about architecture : main dependencies, systemd services, modules, structuring variables

read more about EmonPi File System

emonscripts's People

Contributors

alexandrecuer avatar borpin avatar bwduncan avatar cduffers avatar emonfan avatar glynhudson avatar reedy avatar slyremarks avatar trystanlea avatar wthomson avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

alexandrecuer pb66 isc-konstanz borpin mminehannz sammcgeown bgraabek oppti cduffers scotty-j gui28347 eqilibruim501 bwduncan lanarf smoothville adarga-polytech roy13 jjbotulinum emonfan fluppie janesak1977 miguelalonso abbbbbottttt yn1v lukobladi babadochia jwomack ausil reedy

emonscripts's Issues

the config.ini in update may not reflect the installed locations

There are 3 hardcoded paths in the updae/config.ini - these may not reflect the choices made during the installation process.

EmonScripts/update/config.ini

Line 5 in e3fee4d

openenergymonitor_dir=/opt/openenergymonitor

The update/config.ini file should be added to the '.gitignore' and this file generated generated as part of the installation process with the selected paths.

An alternative solution might be to simply copy the install config.ini to the update folder.

There is a similar issue with the backup script IIRC.

Errors on update

On a non-pi system, the following errors were seen on update of Emoncms

sudo: /opt/openenergymonitor/emonpi/lcd/emonPiLCD_detect.sh: command not found
sudo: /opt/openenergymonitor/emonpi/lcd/emonPiLCD_detect.sh: command not found
/opt/openenergymonitor/EmonScripts/update/main.sh: line 33: [: ==: unary operator expected
/opt/openenergymonitor/EmonScripts/update/main.sh: line 33: [: ==: unary operator expected

No rfm module is fitted (not a Pi).

Hardware detected: rfm2pi

Start emoncms update:
-------------------------------------------------------------
Update Emoncms Core
-------------------------------------------------------------

Traceback (most recent call last):
  File "/opt/openenergymonitor/EmonScripts/common/./md5sum.py", line 16, in 
    with open(file_name) as file_to_check:
IOError: [Errno 2] No such file or directory: '/var/www/emoncms/settings.php'
current settings.php md5: 
Default settings.php md5: 59e80f731935867f5f93753d49c0fe86
settings.php HAS been user modified

How does EmonCMS know what hardware it is on and what subsystems have been installed?

Update message on checking config.ini

@TrystanLea The message on checking the config.ini is fine in that it interrupts the install, but the user will have no idea where the file is nor how to trigger the install to start again.

EmonScripts/install/main.sh

Line 30 in e22ed87

echo "Please review config.ini and restart the build script to continue"

I suggest 2 commands one for editing the file and another for running the install are included in the message.

echo "cd $openenergymonitor_dir/EmonScripts/install/"
echo "nano config.ini"
echo "./main.sh"

My bash is not good enough to know if that is right to include a variable 😀.

Install onto Ubuntu Server using scripts

Install went reasonably well. The key issue was with the services and the User in service-runner and the LOG file creation and the use of the user 'pi'.

This could be solved by either creating a specific user (perhaps oem) or on installing the Ubuntu Server create it with the user pi.

Note, even though emonhub was not selected to be installed, it still appears in the left menu.

config.ini executeable

EmonScripts/install/config.ini

Line 3 in e22ed87

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

I note that there are executable statements in a file with an extension .ini.

It seems to me this is a bad idea.

I think the settings (the ini part - i.e. the values) are separated from a config.sh file to process the config.

this will make modifying the settings easier.

Substitue emoncms_datadir in settings.ini file.

Currently the datadir is hard coded into the emonpi.settings.ini so takes no account of a setting in the install/config.ini

Use $emoncms_datadir in the same way as the other directories are used.

EmonScripts/install/emoncms_core.sh

Line 34 in b255689

sed -i "s~EMONCMS_DIR~$emoncms_dir~" $emoncms_www/settings.ini

This should be done here:

EmonScripts/install/emoncms_core.sh

Line 45 in b255689

for engine in "phpfina" "phpfiwa" "phptimeseries"; do

If only 2 folders are to be referenced in the settings.ini then consideration should be made as to whether they are created. As the recommendation is not to use phpfiwa on an EmonPi, I suggest that phpfiwa should not be created.

Could there be an engine selection process as part of the install?

Apache install raises an error as the document root does not exist

EmonScripts/install/apache.sh

Line 18 in 3cb8ddc

# Configure virtual host

Configuring the virtual host at the point when Apache is installed raises errors.

It might be better to do this when emoncms is installed especially as the root folder might be a different path and so the virtual host conf file would need to be amended to suit.

EmonScripts/install/emoncms_core.sh

Line 11 in b255689

# Install emoncms core repository with git

introduction of a global variable git_repo

@TrystanLea : Would you be willing to introduce a new variable in config.ini, in order to be able to test forks or branches more easily and also to separate the config from the code, even if I understand that for you https://github.com/emoncms and https://github.com/openenergymonitor are constants
it would be a question of creating a variable named git_repo, which we could define the following way :

declare -A git_repo
git_repo[emoncms_core]=https://github.com/emoncms/emoncms.git
git_repo[config]=https://github.com/emoncms/config.git
git_repo[graph]=https://github.com/emoncms/graph.git
git_repo[dashboard]=https://github.com/emoncms/dashboard.git
git_repo[app]=https://github.com/emoncms/app.git
git_repo[wifi]=https://github.com/emoncms/wifi.git
git_repo[device]=https://github.com/emoncms/device.git
git_repo[demandshaper]=https://github.com/emoncms/demandshaper.git
git_repo[sync]=https://github.com/emoncms/sync.git
git_repo[backup]=https://github.com/emoncms/backup.git
git_repo[usefulscripts]=https://github.com/emoncms/usefulscripts.git
git_repo[postprocess]=https://github.com/emoncms/postprocess.git
git_repo[emonhub]=https://github.com/emoncms/emonhub.git
emonhub_branch=master
.....

the scripts that would need to be modified would be all the "git clone" scripts (emoncms_core.sh, emoncms_modules.sh, emonhub.sh) maybe we could extend it to emonsd.sh, emonpilcd.sh, firmware.sh, wifiap.sh, just to be clean

if somebody wants to test its own fork, he just needs to modify config.ini without going in the sh files...

if this seems OK for yu, I can propose a PR...if it is not suitable, no problem

Alex

Errors on non-pi systems

Hi @TrystanLea Just installed on an Ubuntu system. had an error because of hardcoded user name so tried again with the user pi created.

1. No log window

image

2. Demandshaper not in menu

Just realised it has moved 🤦‍♂

3. Dashboards not imported

4. Cannot create new dashboards

Apache conf files have hardcoded paths.

Both of these conf files have hard coded locations for log and root www.

In addition, Apache raises an error as the www folder is not created. Suggest log & www folder are checked for and created here or all folders are created by top level script first.

EmonScripts/install/apache.sh

Line 15 in 79dc389

sudo cp $openenergymonitor_dir/EmonScripts/defaults/apache2/emonsd.conf /etc/apache2/conf-available/emonsd.conf

EmonScripts/install/apache.sh

Line 19 in 79dc389

sudo cp $openenergymonitor_dir/EmonScripts/defaults/apache2/emoncms.conf /etc/apache2/sites-available/emoncms.conf

[doc] panorama of all services and modules involved in EmonCMS

I've read those times a lot of threads related to emoncms settings, to general architecture, etc, etc
As I was a bit lost, I began to write a small doc to have a better view of all services and modules involved in EmonCMS (global vars used, log, conf, etc,etc)
it is actually not perfect, maybe wrong in some cases.....feedback most welcome if things are misunderstood :-)
I submit the result here, in the EmonScripts repository, as most things related to installation are now gathered in this repo......
Alex

EmonCMS file system
EMONCMS_WWW /var/www/emoncms location of the core package and of the main modules (admin, dashboards, config)
EMONCMS_DATADIR /var/opt/emoncms timeseries directories
EMONCMS_DIR /opt/emoncms location of symlinked modules (postprocess, sync) and of uploaded tar.gz archives when importing a backup
OPENENERGYMONITOR_DIR /opt/openenergymonitor location of EmonScripts, emonhub and of created backup archives
EMONCMS_LOG_LOCATION /var/log/emoncms log location for emoncms logs
emoncms.log is main emoncms log, managed with the EmonLogger PHP class

main dependencies
web server Apache2  
relational databases mysql or maria  
key value database  redis-server data buffering for timeseries and backgroud processing for service-runner
message broker mosquitto  
data engine language php with extensions pear (for pecl needs)
libapache2-mod-php
mysql
gd, opcache, dev, common
mbstring
mcrypt
redis
mosquitto (https://github.com/mgdm/Mosquitto-PHP)
curl  

Nota : mbcrypt DEPRECATED > to be replaced (sodium?)

EmonCMS systemd services

all systemd services have to be symlinked in /etc/systemd/system

emoncms_mqtt.service -> /var/www/emoncms/scripts/services/emoncms_mqtt/emoncms_mqtt.service
emonhub.service -> /opt/openenergymonitor/emonhub/service/emonhub.service
feedwriter.service -> /var/www/emoncms/scripts/services/feedwriter/feedwriter.service
service-runner.service -> /var/www/emoncms/scripts/services/service-runner/service-runner.service

Maria and Redis services are in lib/systemd/system but they are compiled...

mysqld.service -> /lib/systemd/system/mariadb.service
mysql.service -> /lib/systemd/system/mariadb.service
redis.service -> /lib/systemd/system/redis-server.service
phpmqtt input (PHP) listen to payloads published to the mosquitto broker on topic emon
scriptpath EMONCMS_WWW/scripts/services/emoncms_mqtt/emoncms_mqtt.php
Servicepath EMONCMS_WWW/scripts/services/emoncms_mqtt/emoncms_mqtt.service
globals $log , $mysqli, $redis, $user, $feed_settings, $mqtt_server $mqttsettings, $mqtt_client,
$feed, $input, $process
non globals $connected, $subscribed, $last_retry, $last_heartbeat, $count
log uses emoncms main log file
journalctl -f -u emoncms_mqtt
specific conf file no
feedwriter (PHP) writes input datas from redis-buffer to disk in order to feed the timeseries
scriptpath EMONCMS_WWW/scripts/feedwriter.php
Servicepath EMONCMS_WWW/scripts/services/feedwriter/feedwriter.service
globals $log, $feed_settings, $mysqli, $redis
$user, $feed
log uses emoncms main log file
journalctl -f -u feedwriter
specific conf file no
Service-runner (Python) trigger background workers from modules (update, backup, sync, postprocess)
scriptpath EMONCMS_WWW/scripts/services/service-runner/service-runner.py
Servicepath EMONCMS_WWW/scripts/services/service-runner/service-runner.service
log uses systemd log file
journalctl -f -u service-runner
specific conf file no
emonhub (Python) listen on serial port or ethernet and publish to mosquitto broker on topic emon
scriptpath OPENENERGYMONITOR_DIR/emonhub/src/emonhub.py
Servicepath OPENENERGYMONITOR_DIR/service/emonhub.service
log log file is in /var/log/emonhub
journalctl -u emonhub -n 30 –no-pager
specific conf file /etc/emonhub/emonhub.conf
/etc/emonhub/emonhub.env

EmonCMS modules

admin module integrated in the core package - does not have its own git repo
place EMONCMS_WWW/Modules/admin
globals $mysqli,$session,$route,$updatelogin,$allow_emonpi_admin, $redis, $openenergymonitor_dir, $admin_show_update, $path
$log, $log_location, $log_enabled, $log_level
background workers none
log uses emoncms main log file
specific conf file no
config web interface to supervise and configure emonhub
place EMONCMS_WWW/Modules/config
globals $route, $session, $redis, $homedir
non globals $emonhub_config_file, $emonhub_logfile, $restart_log
Also in model : $logfile, $config_file, $restart_log_name
background workers restart.sh
log /var/opt/restart.log > to change
specific conf file no
dashboard Dashboard Module
place EMONCMS_WWW/Modules/dashboard
globals $mysqli, $session, $route, $path
background workers no
log no
specific conf file no
graph main graph module with averaging on the fly
place EMONCMS_WWW/Modules/graph
globals $session,$route,$mysqli,$redis, $path
background workers no
log no
specific conf file no
backup import/export all timeseries+mysql database
place EMONCMS_DIR/modules/backup
globals $route, $session, $path, $redis, $linked_modules_dir, $log_location
background workers emoncms-import.sh
emoncms-export.sh
emoncms-copy.sh ?
service-runner bash file why ? All things related to service-runner are in controller
log EMONCMS_LOG_LOCATION/exportbackup.log
EMONCMS_LOG_LOCATION/importbackup.log
specific conf file config.cfg which has to be set up during installation
sync connect to distant EmonCMS machines to retrieve feeds
IOT flavoured alternative to backup
place EMONCMS_DIR/modules/sync
globals $linked_modules_dir,$path,$session,$route
$mysqli,$redis,$user,$feed_settings,$log_location
$log in model
background workers emoncms-sync.sh launching sync_run.php in CLI
log EMONCMS_LOG_LOCATION/sync.log
also uses emoncms main log file
specific conf file no
postprocess postprocess feeds - calculation module
place EMONCMS_DIR/modules/postprocess
globals $linked_modules_dir,$session,$route,$mysqli,$redis,$feed_settings, $log_location
background workers postprocess.sh launching postprocess_run.php in CLI
Postprocess_run.php scans the redis postprocessqueue and start process functions
log EMONCMS_LOG_LOCATION/postprocess.log
specific conf file no

Review partition sizes

re PR #66 (documentation)
re Issue #58

https://community.openenergymonitor.org/t/emonsd-17oct19-image-partition-sizes/12282/

The current script/image fixes the size of the rootfs and only works for 16GB cards.

Suggest the script is amended to fix the Ext2 partition to the end of the card at a max size.

#EmonSD Fix end of Rootfs
if [ $ROOT_DEV_SIZE -gt 16000000 ]; then
  #16GB card or above, assign last 10GB for data 
  TARGET_END=$(($ROOT_DEV_SIZE - 20972544 - 1 ))
elif [ $ROOT_DEV_SIZE -gt 8000000 ]; then
  #8GB card, assign last 4GB for data 
  TARGET_END=$(($ROOT_DEV_SIZE - 7341056 - 1 ))
fi

More discussion on community thread.

I'm not certain this is a robust solution.

wifi on first build

On first build, configuring wifi does not appear to work - perhaps relating from an initial configuration with the AP enabled..

May need:

wpa_cli -i wlan0 reconfigure

rebooting the pi after initial configuration works

Post script steps to complete image

after reboot emonpiupdate.log will be present, remove to ensure emonpifirstupdate runs next time...

1. Set WiFi country code

sudo raspi-config

2. Prepare wifiAP:

sudo systemctl disable hostapd.service 
sudo wifiAP start
sudo reboot

3. Remove emoncms logs

sudo rm /var/log/emoncms/emonpiupdate.log
sudo rm /var/log/emoncms/emonupdate.log
sudo rm /var/log/emoncms/emoncms.log

4. Disable SSH

sudo update-rc.d ssh disable
sudo invoke-rc.d ssh stop
sudo dpkg-reconfigure openssh-server
sudo halt

5. Check that emonpiupdate.log has been removed before creating image.

6. Update emonSD-DDMMMYY file on boot partition and in safe-update list on master branch

/var/log/rotated_logs make cause log2ram to fail?? Remove using SD card reader

7. Create image:

sudo dd if=/dev/sdb of=emonSD-17Oct19.img bs=4M

/var/log filling up

/var/log is filling up again. https://community.openenergymonitor.org/t/var-log-filling-up-again/13808/

This is because the logrotate.log file is not being rotated!

I have added a rotation spec into my 00_defaults file to see what happens. Not sure how it will react to rotating itself.

However, you need to

sudo truncate -s 0 /var/log/logrotate/logrotate.log

the file first otherwise the logrotate fails as there is not enough room to write the file.

Will need to be added to the update script.

Review emonSDexpand

In import/restore documentation see note:

Note: If a new emonSD pre-built-SDcard image has been written to an SD card larger than 4GB the read-write ~/data partition should be expanded to fill the SD card to create sufficient space to import a backup. Do not use Raspbian raspi-config, instead connect via SSH) and run $ sudo emonSDexpand and follow prompts.

Create scheme for default configs

The current config.ini file needs to be excluded from the repo via .gitignore so changes do not prevent a later update of the repo.

I suggest that by default there is no config.ini.

I suggest there is a pi.config.ini and perhaps a linux.config.ini with settings likely to work for a Linux system other than Raspbian.

On running init.sh if the question is no review, a pi is assumed and the pi.config.ini is copied to config.ini and install proceeds.

If a review is required, the user is told to copy either of the base ini files and modify as required.

As an aside, I also think some of the later True/False options need to be brought to the top and the modules bit pushed down.

Installation on a DigitalOcean droplet

Note: For a single user with moderate use, a small VM on a cloud service is likely plenty of performance for an emoncms installation. For applications with a large number of users and feeds the performance of emoncms will depend a lot on how the cloud provider implements storage. Networked disk drives will result in slow performance due to the network latency associated with the way the emoncms feed engines are implemented. VM's on machines with local storage should provide much better performance.

1. Click on 'Create' to create a new cloud server
2. Select Ubuntu, standard $5/month, region e.g London then Create Droplet
3. DigitalOcean email root login details, ssh in using these credentials.
4. Change root password
5. Follow Initial Server Setup with Ubuntu 18.04

adduser oem
usermod -aG sudo oem
ufw allow OpenSSH
ufw enable

logout of root and login with oem.

6. Disable root login:

sudo nano /etc/ssh/sshd_config
PermitRootLogin no
sudo service ssh restart

7. Allow created user to run sudo without password.

sudo visudo

Add to end of visudo file:

oem ALL=(ALL) NOPASSWD:ALL

Save and exit.

8. Start the script

wget https://raw.githubusercontent.com/openenergymonitor/EmonScripts/master/install/init.sh
chmod +x init.sh
./init.sh

9. Modify config.ini for non emonSD installation.

Disable emonhub installation

install_emonhub=false

Disable firmware, emonpilcd, emonsd, wifiap installation

install_firmware=false
install_emonpilcd=false
install_emonsd=false
install_wifiap=false

Set user as applicable and set emonSD_pi_env=0

user=oem
hostname=emonpi
emonSD_pi_env=0

Comment out config, wifi and setup module:

#emoncms_modules[config]=stable
#emoncms_modules[wifi]=stable
#emoncms_modules[setup]=stable

10. Run main installation script:

./main.sh

11. Allow access to port 80 in firewall:

sudo ufw allow 80/tcp

EmonPiFileSystem explanation - links to services

@alexandrecuer

Firstly this is excellent!

I think this should be just 'EmonCMS File System' as it is not specific to the Pi (with specific Pi stuff noted later).

Secondly, I don't think this statement is accurate (couldn't find a better way of linking to it).
all systemd services have to be symlinked in /etc/systemd/system

What actually happens is that the install script creates a link, for the service file to /lib/system/systemd/

EmonScripts/common/install_emoncms_service.sh

Line 53 in 776c0e0

sudo ln -s $servicepath /lib/systemd/system

The command sudo systemctl enable <service is the process that then creates the link in /etc/systemd/system. Thus, if the user wants to override this service, they can.

It's quite an important distinction 😀.

I've found a few things that will need specific differences to support non-pi environments.

Write load from php sessions

Not sure why I didnt notice this in previous testing but there seems to be quite a bit of write load resulting from updating php sessions. Placing the session folder in a tmpfs makes a big difference:

tmpfs           /var/lib/php/sessions  tmpfs   nodev,nosuid,size=1M,mode=1777  0  0

non-pi logrotate file permissions incorrect

@TrystanLea

EmonScripts/install/non_emonsd.sh

Line 8 in b804e32

sudo chown root /etc/logrotate.d/emoncms

On running the logrotate on Ubuntu the following error is seen

brian@ubunt-emon:/var/log/emoncms$ sudo /usr/sbin/logrotate /etc/logrotate.conf
error: Ignoring emoncms because of bad file mode - must be 0644 or 0444.

But I could not fix it for some reason...

Changed mode and works fine. Probably the same problem for all these logrotate configurations.

Test with

sudo /usr/sbin/logrotate /etc/logrotate.conf

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.