rdnfn / beobench Goto Github PK

View Code? Open in Web Editor NEW

37.0 37.0 4.0 2.72 MB

A toolkit providing easy and unified access to building control environments for reinforcement learning (RL).

Home Page: https://beobench.readthedocs.io

License: MIT License

Makefile 2.77% Python 87.99% Dockerfile 3.09% Jupyter Notebook 6.15%

building-control building-energy-optimization reinforcement-learning reinforcement-learning-environments

beobench's People

Contributors

Stargazers

Watchers

Forkers

enjeeneer david-woelfle xkunw jimmy-inl

beobench's Issues

Enable passing of pip requirements file for custom agent scripts

Problem

Experiments in energym containers do not seem to perform as well as they potentially could.

Solution

Evaluate what takes all the time. Potential reasons:

The reset() calls cause the entire simulation to be reset and warm-up again after each iteration

Problem

The documentation is currently still a bit unfocused and does not capture the project targets well. It does not show well enough what beobench is useful for, and how to use it.

Solution

Rewrite the documentation (including main readme and sphinx) to be more focused. The following points should be addressed:

Add more info on available environments (at least Table and link to detailed description in packages)
#45
Change overall focus to easy access to frameworks and cross-framework experiments
#46
#47
#48
Add description of setting up dev environment, and other dev guidance
Remove/replace boilerplate dev guidance

Add tutorial on using beobench to run the same experiment in multiple frameworks

On arm64 experiment containers fail due to missing pypi ray package

Problem

In arm64 architecture-based containers (e.g. on M1 macs) the experiment containers fail due to the ray package not being available (they only provide x86 wheels via pypi).

Solution

There are a few different approaches to tackle this problem:

~~Build custom ray package in container if on arm64 (see guide to building arm64 ray)~~
Run experiment containers in compatiblity mode via x86 (this is slow and can only be used for testing)

Edit:

The focus will now be on getting a development version running without necessarily going for full performance.

Potential solution:

Move build pipeline to docker buildx to add --platform linux/amd64 and use build contexts from local package data from submodule.

Problem

The currently recommended experiment tracking solution requires a login.

Solution

Add support for MLflow, based on RLlib support for the platform.

Create separate extended version of beobench

Problem

Currently Beobench is always installed with all possible dependencies. But the dependencies used vary drastically depending on whether it is called inside a container.

Solution

Create an extended version that downloads the full experiment dependencies (RLlib, torch) only when inside the container. This would then be installed using pip install beobench[extended].

Add support for non-gpu systems

Problem: Currently, beobench assumes that the host system has access to an nvidia gpu. This will not apply to many systems.

Solution: Enable execution of beobench experiments without GPUs, ideally via an optional argument in main API

In random agent script wandb full episode data logging skips a few steps

Problem

In random agent script wandb full episode data logging skips a few steps. This is because wandb counts the epsiode reward logging steps made prior to the full data logging.

Potential Solution

Add another metric to log that shows timestep and day (proportional).

Move config files to beobench_contrib

Problem

Currently, there are two locations with experiment files.

Potential Solution

Move all experiment files to folder in beobench_contrib.

Add support for wandb sweeps API

Problem

Given the new yaml based config files there is no longer an obvious way to do hyperparameter tuning (as was ray tune syntax before).

Potential Solution

Test the use of wandb sweeps with Beobench configs and add guidance to docs about usage.

Todos:

get basic experiment to run
implement #54 to enable full control via sweeps
make sure that wandb inside experiment container adds data to the sweep

Edit: the last point seems easy but adds some complexity. W&B uses wandb.init() in the experiment script to setup the right environment for the sweep. There are two reasons why this is non-trivial in Beobench:

We are using RLlib, and thus are not called init itself explicitly (although the logger callback allows passing through arguments)
RLlib call is running inside a docker container, making it quite removed from the original sweep agent, which likely runs outside the docker container.

Moving forward: try to figure out how the wandb init method knows which sweep it belongs to and move the right env variables into docker container. Possible solutions

Run sweep agents directly inside experiment containers (thanks to the config file this should keep pretty much all information)
Transfer wandb env variables that may define sweep into docker container.

Resources:

Edit 2:
This is hard because the wandb agent assumes everything happens in the same process. As an alternative we might want to consider the distributed training group feature.

Enable mounting agent dir

Problem

Describe what you were trying to get done.
Tell us what happened, what went wrong, and what you expected to happen.

Potential Solution

Describe a potential solution to the problem, if you have any.

Add automatic testing

Problem

Currently there are not automatic test running that verify that a PR to the main branch does not introduce any new bugs.

Solution

Use GitHub actions to add automatic testing to repo.

Docker-in-docker does not work on remote machine

Problem

Currently the docker does not work in the remote devcontainer. Both the remote and the standard devcontainer have no functioning docker on the remote machine. Locally docker is available in the devcontainer. Likely a problem with the docker installation on the remote machine.

Solution

Check problem with remote docker installation.

Add support for multiple --config args

Problem

Currently Beobench can only use a single config argument. However, it may be desirable to mix different configs, as long as they are not contradicting each other. This may help with varying the agent config for multiple experiments in the same environments.

Potential Solution

Allow there to be multiple calls of --config in the CLI API, and enable passing a list of configs in the python API.

Alternatively, for consitency maybe both should be passed as lists.

Improve support for custom agent

Problem

The support for custom agents can be improved.

Solution

Add the following features for custom agents:

Access to env_creator script via beobench package, e.g. via beobench.inside_container.create_env()
#49

Add section on managing docker disk usage

Problem

Beobench is able to use a lot of disk space. We should add a section in the docs that elaborates how to view the docker related disk usage and how to remove older build cache entries.

Resources:

Potential Solution

Add info to docs.

Add beobench_contrib as submodule

Problem

beobench_contrib is currently a different repo, completely disconnected from the main repo.

Solution

Add beobench_contrib as a submodule.

Pros

Simple integration of data from contrib into the docs

Cons

Additional complexity for the developement process
More difficult to maintain long-term compared to simpler separate setup

Alternatives

git subtree, but this is not supported by vscode as of now: microsoft/vscode#39152

Resources

Add observation and action spaces to env list in docs

Problem

Similar to how the building type is shown in the env list, it would be useful to have access to observation and action spaces for each env.

Potential Solution

Add dictionaries of these to the google sheets table, and then parse them into the rst document from there.

Add Energym obs and act spaces too google sheet
Similarly, add Sinergym
Similarly, add BOPTEST
Parse the spaces into docs rst file

Make command line options possible to set via config yaml/dict

Problem

Currently there are two types of Beobench config: (1) parameters that are directly passed to the run method and (2) parameters that are passed inside the yaml/dict config parameter. Ideally, we want that all parameters can also be set inside the config. This will help make experiments more reproducable.

Potential Solution

Add logic that allows the parameters to be set via config, but also overwritten by the main parameters. This will also mean adding a new section to config yaml besides agent and env. This new section could be called general or other

Add support for custom reward function

Problem

Currently user cannot define their own reward function

Solution

Create wrapper function that updates default environment reward functions with user specified function.

Add tutorial on custom agents

Add baseline control script with agent from energym package

Problem

Limited availability of good baseline agents.

Potential Solution

Add agent script for energym envs that uses the energym baseline agents.

See https://bsl546.github.io/energym-pages/sources/notebooks/Apartments2Thermal.html

Add demo of using beobench cross-framework

Description

Add demo (notebook) that uses beobench to run the same method on environment from different frameworks. The notebook should show:

How few lines of code this requires.
Plots of the experimental results.

Ideally #22 will also be adressed, so that the resulting demo is easy to follow.

Add Solara integration

Problem

Currently, there is now official support for Solara environments but these Envs could serve as great simple test cases for agents.

Potential Solution

Add Solara integration to beobench contrib.

Add citation to gym integration readme pages

Enable installing agent requirements

Problem

Currently agent scripts can only use already installed dependencies.

Potential Solution

Add option to install requirements from file and, in a second step, ideally also integrate this into the build process. For example by building the final container with sha hash of saved config attached to name. Note that this may not be necessary.

Add supported integration data into package to remove need for github clone and improve versioning

Problem

Currently the supported integrations are temporarily cloned every time an experiment image is built. This means that whatever version is available on beobench_contrib branch is used instead of the git commit of the submodule. This means that the general beobench versioning does not apply to the integrations, but for simplicity it would be desirable to add them to the main versioning. It will also make developing and improving the integrations easier.

Potential Solution

Add integration data to package.

Add more info to gym intergration specific config arguments to docs

Problem

Currently it is necessary to look at the code for each integration to figure out what the different config parameters are.

Solution

Automatically parse the docstrings from the integrations and show them in pseudo yaml.

Add visualisation tools

Problem

It can be difficult to debug RL agents just based on iteration/metrics.

Potential Solution

Build flexible visualisation tool to be able to inspect episode logs from RLlib but also any other library.

Remove unnecessary beobench installations from gym integration base image dockerfiles

Problem

Currently the experiment images are installing beobench twice, once for the base image, and again for the complete image.

Solution

Remove beobench-specific installation steps in base image dockerfiles.

Create interactive mode

Problem

Currently, the user can only test software in experiment containers via pre-written scripts. It might be worthwhile to add a simple API to get interactive shell access to experiment containers.

Potential Solution

Add API for starting interactive shell.

Let energym integration use env name argument

Problem

Currently the env name needs to be set twice in configurations using the energym integration

Solution

Use the config["env"]["name"] parameter inside the configuration.

In RLlib agent the full episode data logging does not work for long runs

Problem

The RLlib agent script is unable to process multiple output files. Multiple files are given by RLlib when the previous output file hits a size limit. This limit is hit in longer runs (>15k steps).

Potential Solution

This is relatively easy to solve. Just load and order the different files from the output dir.

Add list of all available environments to docs

Description

Create a list of all available environments in Beobench.

This would likely be based on a .csv file.

Enable full observation state extraction from energym integration

Problem

Only the normalised observation state is currently accessible to the agent, and thus saveable via wandb.

Potential Solution

Pass the full observation space via the info parameter.

step_size not passed on to energym by integration

Problem

There appears to be a mismatch between the user-defined step size and the actual step size of the simulation. The plot below shows the solar radiation, and indicates that a day is about 480 steps (step_size 3min), as opposed to the 96 steps that the configured step_size of 15min would mean.

Potential Solution

As far as I can tell the step_size parameter is not actually passed on to the Energym env making command, thus it defaults to another step_size. Creating the relevant logic should be sufficient to fix this.

Add multi-stage build process for experiment containers

Problem

Currently experiment containers also define beobench-specific parameters such as making the env_creator available. This make maintaining the containers as beobench evolves difficult. Example: integrating the usage of extended dependencies.

Solution

Build a base version of container, and then add beobench-specific features on top of that in a separate image that builds on the base.

Add wandb support for random agent script to be able to add its performance for reference

Problem

The random agent script currently does not support logging metrics via wandb.

Potential Solution

Add basic wandb integration into the script.

Energym integration experiments in `Apartments2Thermal` env crash.

Problem

When running the experiment defined in the config here, the simulation appears to warm up but then crashes once the agent takes an action with an error message that ends as follows:

  File "/usr/local/lib/python3.8/dist-packages/ray/rllib/env/vector_env.py", line 233, in vector_step
    obs, r, done, info = self.envs[i].step(actions[i])
  File "/opt/beobench/experiment_setup/energymGymEnv.py", line 239, in step
    observations = self.env.step(action)
  File "/usr/local/lib/python3.8/dist-packages/energym-0.1-py3.8.egg/energym/envs/apartments2/apartments2.py", line 620, in step
    out = super().step(inputs=inputs)
  File "/usr/local/lib/python3.8/dist-packages/energym-0.1-py3.8.egg/energym/envs/env_fmu_eplus.py", line 125, in step
    output = super().step(inputs)
  File "/usr/local/lib/python3.8/dist-packages/energym-0.1-py3.8.egg/energym/envs/env_fmu.py", line 448, in step
    input_list.append(inputs[key][p])
IndexError: invalid index to scalar variable.

I think most likely there is a problem with the size of the action space in the Beobench integration, and therefore the (action) inputs are of the wrong format downstream.

Edit:

Example of action that causes this type of error

OrderedDict([
    ('Bd_Ch_EV1Bat_sp', array([-0.67184997], dtype=float32)), 
    ('Bd_Ch_EV2Bat_sp', array([-0.79589915], dtype=float32)), 
    ('Bd_Pw_Bat_sp', array([0.7324372], dtype=float32)), 
    ('P1_T_Thermostat_sp', array([-0.9726137], dtype=float32)), 
    ('P1_onoff_HP_sp', 1),
    ('P2_T_Thermostat_sp', array([-0.911861], dtype=float32)),
    ('P2_onoff_HP_sp', 0),
    ('P3_T_Thermostat_sp', array([-0.40071118], dtype=float32)),
    ('P3_onoff_HP_sp', 0),
    ('P4_T_Thermostat_sp', array([0.84636706], dtype=float32)),
    ('P4_onoff_HP_sp', 1)
])

Edit 2:

To test this problem, run the following command in the root of the beobench repository on the (up-to-date) dev-general branch:

beobench run -c beobench/data/configs/rewex01_test01.yaml -d .[extended] --docker-shm-size 4gb

Note that this doesn't currently work on ARM64-based systems because of an error with the energym simulator.

Edit 3:

The problem is most likely the discrete inputs that present in the ApartmentThermal2 that are not present in the previously tested MixedUse env.

There is a difference between discrete spaces in energym and gym, energym wraps discrete action scalars in list, gym does not. See:

Create integration with energym

Feature Description

Make energym package accessible via beobench.

Minimum features:

action convertor function
obs convertor function
reset function
default reward functions (that works out of the box)

Later:

norm obs wrapper
norm action wrapper

Later later:

discrete action wrapper

For Arduin:

set up functioning docker container for energym

Questionable warm-up period in Energym integration

Problem

When running rewex01_test01.yaml, The output of the Energym simulations suggest that the simulations runs for an entire year before the agent takes the first action. Is this possible? See output below. To reproduce it run the following command in the repo dir on the dev/general branch:

beobench run -c ./beobench/data/configs/rewex01_test01.yaml -d .[extended]

Reading input and weather file for preprocessor program.
The IDF version of the input file ///root/Energym_runs/1648646930_1219589//resources//Apartments2_heavy_insulated_pump.idf starts with 9
Successfully finish reading weather file.
This is the Begin Month: 1
Time (0) set is smaller than minimun allowed (1 day). Day will be set to 1.
This is the Day of the Begin Month: 1
This is the End Month: 12
This is the Day of the End Month: 31
Day of week was left blank in input file.
This is the New Day of Week:  
Running EPMacro...
ExpandObjects Started.
No expanded file generated.
ExpandObjects Finished. Time:     0.475
EnergyPlus Starting
EnergyPlus, Version 9.5.0-de239b2e5f, YMD=2022.03.30 13:28
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Adjusting Air System Sizing
Adjusting Standard 62.1 Ventilation Sizing
Initializing Simulation
Reporting Surfaces
Beginning Primary Simulation
Initializing New Environment Parameters
Warming up {1}
Instantiating FunctionalMockupUnitExport interface
ExternalInterface initializes.
Number of outputs in ExternalInterface = 68
Number of inputs  in ExternalInterface = 13
Calculating Detailed Daylighting Factors, Start Date=01/01
Warming up {2}
Warming up {3}
Warming up {4}
Warming up {5}
Warming up {6}
Starting Simulation at 01/01/2017 for ALL_YEAR
ExternalInterface starts first data exchange.
Updating Shadowing Calculations, Start Date=01/21/2017
Updating Detailed Daylighting Factors, Start Date=01/21
Continuing Simulation at 01/21/2017 for ALL_YEAR
Updating Shadowing Calculations, Start Date=02/10/2017
Updating Detailed Daylighting Factors, Start Date=02/10
Continuing Simulation at 02/10/2017 for ALL_YEAR
Updating Shadowing Calculations, Start Date=03/02/2017
Updating Detailed Daylighting Factors, Start Date=03/02
Continuing Simulation at 03/02/2017 for ALL_YEAR
Updating Shadowing Calculations, Start Date=03/22/2017
Updating Detailed Daylighting Factors, Start Date=03/22
Continuing Simulation at 03/22/2017 for ALL_YEAR
Updating Shadowing Calculations, Start Date=04/11/2017
Updating Detailed Daylighting Factors, Start Date=04/11
Continuing Simulation at 04/11/2017 for ALL_YEAR
Updating Shadowing Calculations, Start Date=05/01/2017
Updating Detailed Daylighting Factors, Start Date=05/01
Continuing Simulation at 05/01/2017 for ALL_YEAR
Updating Shadowing Calculations, Start Date=05/21/2017
Updating Detailed Daylighting Factors, Start Date=05/21
Continuing Simulation at 05/21/2017 for ALL_YEAR
Updating Shadowing Calculations, Start Date=06/10/2017
Updating Detailed Daylighting Factors, Start Date=06/10
Continuing Simulation at 06/10/2017 for ALL_YEAR
Updating Shadowing Calculations, Start Date=06/30/2017
Updating Detailed Daylighting Factors, Start Date=06/30
Continuing Simulation at 06/30/2017 for ALL_YEAR
Updating Shadowing Calculations, Start Date=07/20/2017
Updating Detailed Daylighting Factors, Start Date=07/20
Continuing Simulation at 07/20/2017 for ALL_YEAR
Updating Shadowing Calculations, Start Date=08/09/2017
Updating Detailed Daylighting Factors, Start Date=08/09
Continuing Simulation at 08/09/2017 for ALL_YEAR
Updating Shadowing Calculations, Start Date=08/29/2017
Updating Detailed Daylighting Factors, Start Date=08/29
Continuing Simulation at 08/29/2017 for ALL_YEAR
Updating Shadowing Calculations, Start Date=09/18/2017
Updating Detailed Daylighting Factors, Start Date=09/18
Continuing Simulation at 09/18/2017 for ALL_YEAR
Updating Shadowing Calculations, Start Date=10/08/2017
Updating Detailed Daylighting Factors, Start Date=10/08
Continuing Simulation at 10/08/2017 for ALL_YEAR
Updating Shadowing Calculations, Start Date=10/28/2017
Updating Detailed Daylighting Factors, Start Date=10/28
Continuing Simulation at 10/28/2017 for ALL_YEAR
Updating Shadowing Calculations, Start Date=11/17/2017
Updating Detailed Daylighting Factors, Start Date=11/17
Continuing Simulation at 11/17/2017 for ALL_YEAR
Updating Shadowing Calculations, Start Date=12/07/2017
Updating Detailed Daylighting Factors, Start Date=12/07
Continuing Simulation at 12/07/2017 for ALL_YEAR
Updating Shadowing Calculations, Start Date=12/27/2017
Updating Detailed Daylighting Factors, Start Date=12/27
Continuing Simulation at 12/27/2017 for ALL_YEAR
Writing tabular output file results using comma format.
Writing tabular output file results using HTML format.
 ReadVarsESO program starting.
 ReadVars Run Time=00hr 02min 10.71sec
 ReadVarsESO program completed successfully.
 ReadVarsESO program starting.
 Requested ESO file=Apartments2_heavy_insulated_pump.mtr
 does not exist.  ReadVarsESO program terminated.
 ReadVarsESO program terminated.
EnergyPlus Run Time=00hr 22min 33.38sec
EnergyPlus Completed Successfully.
Reading input and weather file for preprocessor program.
The IDF version of the input file ///root/Energym_runs/1648648289_4677490//resources//Apartments2_heavy_insulated_pump.idf starts with 9
Successfully finish reading weather file.
This is the Begin Month: 1
Time (0) set is smaller than minimun allowed (1 day). Day will be set to 1.
This is the Day of the Begin Month: 1
This is the End Month: 12
This is the Day of the End Month: 31
Day of week was left blank in input file.
This is the New Day of Week:  
Running EPMacro...
ExpandObjects Started.
No expanded file generated.
ExpandObjects Finished. Time:     0.479
EnergyPlus Starting
EnergyPlus, Version 9.5.0-de239b2e5f, YMD=2022.03.30 13:51
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Adjusting Air System Sizing
Adjusting Standard 62.1 Ventilation Sizing
Initializing Simulation
Reporting Surfaces
Beginning Primary Simulation
Initializing New Environment Parameters
Warming up {1}
Instantiating FunctionalMockupUnitExport interface
ExternalInterface initializes.
Number of outputs in ExternalInterface = 68
Number of inputs  in ExternalInterface = 13
Calculating Detailed Daylighting Factors, Start Date=01/01
Warming up {2}
Warming up {3}
Warming up {4}
Warming up {5}
Warming up {6}
Starting Simulation at 01/01/2017 for ALL_YEAR
ExternalInterface starts first data exchange.
Note: RLlib beobench integration not available.
Random agent: creating env.
[OK] fmi2Instantiate: The Resource location of FMU with instance name %s is %s.

[WARNING] fmi2Instantiate: Argument loggingOn is set to %d
. This is not supported. loggingOn will default to '0'.

[OK] The current working directory is %s

[OK] fmi2Instantiate: Path to fmuUnzipLocation %s

[OK] fmi2Instantiate: Path to fmuResourceLocation %s

[OK] Command executes to copy content of resources folder: %s

[OK] fmi2Instantiate: Path to model description file is %s.

[OK] fmi2Instantiate: The FMU modelIdentifier is %s.

[OK] fmi2Instantiate: The FMU modelGUID is %s.

[OK] fmi2Instantiate: Slave %s is instantiated.

[OK] fmi2Instantiate: Instantiation of %s succeded.

[OK] fmi2EnterInitializationMode: The sockfd is %d.

[OK] fmi2EnterInitializationMode: The port number is %d.

[OK] fmi2EnterInitializationMode: This hostname is %s.

[OK] fmi2EnterInitializationMode: TCPServer Server waiting for clients on port: %d.

[OK] fmi2EnterInitializationMode: The number of input variables is %d.

[OK] fmi2EnterInitializationMode: The number of output variables is %d.

[OK] Get input file from resource folder %s.

[OK] Searching for following pattern %s

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Found matching file %s.

[OK] done searching pattern %s

[OK] Get input file from resource folder %s.

[OK] Searching for following pattern %s

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Found matching file %s.

[OK] done searching pattern %s

[OK] Get input file from resource folder %s.

[OK] Searching for following pattern %s

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Found matching file %s.

[OK] done searching pattern %s

[OK] This version uses the **energyplus** command line interface to  call the EnergyPlus executable. **RunEPlus.bat** and **runenergyplus** , which were used in earlier versions, were deprecated as of August 2015.
[OK] fmi2EnterInitializationMode: The connection has been accepted.

[OK] fmi2EnterInitializationMode: Slave %s is initialized.

Random agent: resetting env.
[OK] fmi2Terminate: fmiFreeInstanceSlave must be called to free the FMU instance.

[OK] fmi2FreeInstance: The function fmi2FreeInstance of instance %s is executed.

[OK] freeInstanceResources: %s will be freed.

[OK] fmi2Instantiate: The Resource location of FMU with instance name %s is %s.

[WARNING] fmi2Instantiate: Argument loggingOn is set to %d
. This is not supported. loggingOn will default to '0'.

[OK] The current working directory is %s

[OK] fmi2Instantiate: Path to fmuUnzipLocation %s

[OK] fmi2Instantiate: Path to fmuResourceLocation %s

[OK] Command executes to copy content of resources folder: %s

[OK] fmi2Instantiate: Path to model description file is %s.

[OK] fmi2Instantiate: The FMU modelIdentifier is %s.

[OK] fmi2Instantiate: The FMU modelGUID is %s.

[OK] fmi2Instantiate: Slave %s is instantiated.

[OK] fmi2Instantiate: Instantiation of %s succeded.

[OK] fmi2EnterInitializationMode: The sockfd is %d.

[OK] fmi2EnterInitializationMode: The port number is %d.

[OK] fmi2EnterInitializationMode: This hostname is %s.

[OK] fmi2EnterInitializationMode: TCPServer Server waiting for clients on port: %d.

[OK] fmi2EnterInitializationMode: The number of input variables is %d.

[OK] fmi2EnterInitializationMode: The number of output variables is %d.

[OK] Get input file from resource folder %s.

[OK] Searching for following pattern %s

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Found matching file %s.

[OK] done searching pattern %s

[OK] Get input file from resource folder %s.

[OK] Searching for following pattern %s

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Found matching file %s.

[OK] done searching pattern %s

[OK] Get input file from resource folder %s.

[OK] Searching for following pattern %s

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Read directory and search for *.idf, *.epw, or *.idd file.

[OK] Found matching file %s.

[OK] done searching pattern %s

[OK] This version uses the **energyplus** command line interface to  call the EnergyPlus executable. **RunEPlus.bat** and **runenergyplus** , which were used in earlier versions, were deprecated as of August 2015.
[OK] fmi2EnterInitializationMode: The connection has been accepted.

[OK] fmi2EnterInitializationMode: Slave %s is initialized.

Random agent: taking action.
OrderedDict([('Bd_Ch_EV1Bat_sp', array([-0.5792162], dtype=float32)), ('Bd_Ch_EV2Bat_sp', array([-0.79893726], dtype=float32)), ('Bd_Pw_Bat_sp', array([-0.55746967], dtype=float32)), ('P1_T_Thermostat_sp', array([-0.83537054], dtype=float32)), ('P1_onoff_HP_sp', 1), ('P2_T_Thermostat_sp', array([-0.4884674], dtype=float32)), ('P2_onoff_HP_sp', 0), ('P3_T_Thermostat_sp', array([-0.346657], dtype=float32)), ('P3_onoff_HP_sp', 1), ('P4_T_Thermostat_sp', array([-0.31184208], dtype=float32)), ('P4_onoff_HP_sp', 0)])
Random agent: taking action.
OrderedDict([('Bd_Ch_EV1Bat_sp', array([0.78278834], dtype=float32)), ('Bd_Ch_EV2Bat_sp', array([0.2201611], dtype=float32)), ('Bd_Pw_Bat_sp', array([0.27057496], dtype=float32)), ('P1_T_Thermostat_sp', array([0.64747214], dtype=float32)), ('P1_onoff_HP_sp', 1), ('P2_T_Thermostat_sp', array([-0.22706172], dtype=float32)), ('P2_onoff_HP_sp', 1), ('P3_T_Thermostat_sp', array([0.6409295], dtype=float32)), ('P3_onoff_HP_sp', 1), ('P4_T_Thermostat_sp', array([0.51800954], dtype=float32)), ('P4_onoff_HP_sp', 0)])
Random agent: taking action.
OrderedDict([('Bd_Ch_EV1Bat_sp', array([0.7659943], dtype=float32)), ('Bd_Ch_EV2Bat_sp', array([-0.04078925], dtype=float32)), ('Bd_Pw_Bat_sp', array([0.7055465], dtype=float32)), ('P1_T_Thermostat_sp', array([-0.50344086], dtype=float32)), ('P1_onoff_HP_sp', 1), ('P2_T_Thermostat_sp', array([0.8202817], dtype=float32)), ('P2_onoff_HP_sp', 1), ('P3_T_Thermostat_sp', array([-0.8721977], dtype=float32)), ('P3_onoff_HP_sp', 0), ('P4_T_Thermostat_sp', array([-0.52730197], dtype=float32)), ('P4_onoff_HP_sp', 0)])
Random agent: taking action.
/opt/beobench/experiment_setup/energymGymEnv.py:119: DeprecationWarning: invalid escape sequence \d
  self.temps = list(filter(lambda t: match("Z\d\d_T", t), self.obs_keys))
/opt/beobench/experiment_setup/energymGymEnv.py:393: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  def compute_reward(self, observation: dict) -> np.float:
/usr/local/lib/python3.8/dist-packages/energym-0.1-py3.8.egg/energym/schedules/EVSchedule.py:60: FutureWarning: Series.dt.weekofyear and Series.dt.week have been deprecated. Please use Series.dt.isocalendar().week instead.
  self.schedule["Week"] = self.schedule["Time"].dt.week
/usr/local/lib/python3.8/dist-packages/energym-0.1-py3.8.egg/energym/schedules/EVSchedule.py:60: FutureWarning: Series.dt.weekofyear and Series.dt.week have been deprecated. Please use Series.dt.isocalendar().week instead.
  self.schedule["Week"] = self.schedule["Time"].dt.week
/usr/local/lib/python3.8/dist-packages/energym-0.1-py3.8.egg/energym/schedules/EVSchedule.py:177: FutureWarning: Passing method to DatetimeIndex.get_loc is deprecated and will raise in a future version. Use index.get_indexer([item], method=...) instead.
  indices = [self.schedule.index.get_loc(dt, method="nearest") for dt in dts]
/opt/beobench/experiment_setup/energymGymEnv.py:432: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  return np.float(reward)
OrderedDict([('Bd_Ch_EV1Bat_sp', array([0.44146132], dtype=float32)), ('Bd_Ch_EV2Bat_sp', array([-0.7559141], dtype=float32)), ('Bd_Pw_Bat_sp', array([-0.4943884], dtype=float32)), ('P1_T_Thermostat_sp', array([0.12277272], dtype=float32)), ('P1_onoff_HP_sp', 1), ('P2_T_Thermostat_sp', array([0.23925698], dtype=float32)), ('P2_onoff_HP_sp', 1), ('P3_T_Thermostat_sp', array([0.2703274], dtype=float32)), ('P3_onoff_HP_sp', 0), ('P4_T_Thermostat_sp', array([-0.6166424], dtype=float32)), ('P4_onoff_HP_sp', 0)])
Random agent: taking action.
OrderedDict([('Bd_Ch_EV1Bat_sp', array([0.37055752], dtype=float32)), ('Bd_Ch_EV2Bat_sp', array([-0.3510263], dtype=float32)), ('Bd_Pw_Bat_sp', array([-0.2277605], dtype=float32)), ('P1_T_Thermostat_sp', array([-0.86376804], dtype=float32)), ('P1_onoff_HP_sp', 1), ('P2_T_Thermostat_sp', array([-0.9350178], dtype=float32)), ('P2_onoff_HP_sp', 1), ('P3_T_Thermostat_sp', array([0.7757111], dtype=float32)), ('P3_onoff_HP_sp', 1), ('P4_T_Thermostat_sp', array([-0.08315059], dtype=float32)), ('P4_onoff_HP_sp', 1)])
Random agent: taking action.
OrderedDict([('Bd_Ch_EV1Bat_sp', array([-0.01111914], dtype=float32)), ('Bd_Ch_EV2Bat_sp', array([0.9637071], dtype=float32)), ('Bd_Pw_Bat_sp', array([-0.87789124], dtype=float32)), ('P1_T_Thermostat_sp', array([0.20853722], dtype=float32)), ('P1_onoff_HP_sp', 1), ('P2_T_Thermostat_sp', array([-0.8076891], dtype=float32)), ('P2_onoff_HP_sp', 1), ('P3_T_Thermostat_sp', array([-0.843262], dtype=float32)), ('P3_onoff_HP_sp', 0), ('P4_T_Thermostat_sp', array([-0.63166785], dtype=float32)), ('P4_onoff_HP_sp', 1)])
Random agent: taking action.
OrderedDict([('Bd_Ch_EV1Bat_sp', array([-0.7527687], dtype=float32)), ('Bd_Ch_EV2Bat_sp', array([-0.873507], dtype=float32)), ('Bd_Pw_Bat_sp', array([0.37959477], dtype=float32)), ('P1_T_Thermostat_sp', array([-0.37768596], dtype=float32)), ('P1_onoff_HP_sp', 0), ('P2_T_Thermostat_sp', array([-0.23146532], dtype=float32)), ('P2_onoff_HP_sp', 0), ('P3_T_Thermostat_sp', array([-0.99944913], dtype=float32)), ('P3_onoff_HP_sp', 1), ('P4_T_Thermostat_sp', array([0.00051912], dtype=float32)), ('P4_onoff_HP_sp', 0)])
Random agent: taking action.
OrderedDict([('Bd_Ch_EV1Bat_sp', array([-0.8320181], dtype=float32)), ('Bd_Ch_EV2Bat_sp', array([0.09077284], dtype=float32)), ('Bd_Pw_Bat_sp', array([0.17681736], dtype=float32)), ('P1_T_Thermostat_sp', array([0.9338784], dtype=float32)), ('P1_onoff_HP_sp', 0), ('P2_T_Thermostat_sp', array([0.8568551], dtype=float32)), ('P2_onoff_HP_sp', 1), ('P3_T_Thermostat_sp', array([0.93643355], dtype=float32)), ('P3_onoff_HP_sp', 0), ('P4_T_Thermostat_sp', array([0.09125433], dtype=float32)), ('P4_onoff_HP_sp', 1)])
Random agent: taking action.
OrderedDict([('Bd_Ch_EV1Bat_sp', array([-0.9112033], dtype=float32)), ('Bd_Ch_EV2Bat_sp', array([0.9594012], dtype=float32)), ('Bd_Pw_Bat_sp', array([-0.80443734], dtype=float32)), ('P1_T_Thermostat_sp', array([-0.92528975], dtype=float32)), ('P1_onoff_HP_sp', 0), ('P2_T_Thermostat_sp', array([0.8676486], dtype=float32)), ('P2_onoff_HP_sp', 1), ('P3_T_Thermostat_sp', array([-0.13093314], dtype=float32)), ('P3_onoff_HP_sp', 0), ('P4_T_Thermostat_sp', array([0.20742875], dtype=float32)), ('P4_onoff_HP_sp', 1)])
Random agent: taking action.
OrderedDict([('Bd_Ch_EV1Bat_sp', array([0.5088712], dtype=float32)), ('Bd_Ch_EV2Bat_sp', array([-0.16283542], dtype=float32)), ('Bd_Pw_Bat_sp', array([-0.20831573], dtype=float32)), ('P1_T_Thermostat_sp', array([-0.73916245], dtype=float32)), ('P1_onoff_HP_sp', 1), ('P2_T_Thermostat_sp', array([-0.64466465], dtype=float32)), ('P2_onoff_HP_sp', 0), ('P3_T_Thermostat_sp', array([0.02287476], dtype=float32)), ('P3_onoff_HP_sp', 0), ('P4_T_Thermostat_sp', array([0.4030808], dtype=float32)), ('P4_onoff_HP_sp', 1)])
Random agent: completed test.
Error: The server closed the socket while the client was reading.
Error: The server closed the socket while the client was reading.
**FATAL:Error in ExternalInterface: Check EnergyPlus *.err file.
EnergyPlus Run Time=00hr 00min 38.69sec
Program terminated: EnergyPlus Terminated--Error(s) Detected.
Note: RLlib beobench integration not available.

Potential Solution

I am not sure if this is really a bug or whether this output can be explained otherwise somehow.

Reintroduce support for env and method arguments

Problem

With the introduction of yaml-based config files, support for the --env and --method options in the CLI and API were deprecated.

Potential Solution

Reintroduce the two options building on the new config structure.

Energym infos currently log actions not taken by agent

Problem

The plot below below shows that the actions of a random agent over a day of 3 min steps. Both discrete and continuous actions appear to be reasonably random apart from acts.Bd_DisCh_EV1Bat_sp. Turns out this is because the dict value of the action actually just set within the env.env (the core energym environment), our random agent is not even aware of these actions - it's more of an artefact of the logging method.

Potential Solution

We may want to do a deep copy of the actions before adding them to info in order to avoid this

Update logo to new version

Improve experiment definition parsing

Problem

Currently the experiment definition parsing is not very intuitive.

Solution

Create a solid, transparent and extendable experiment parsing pipeline.

Related work

wandb sweeps (example yaml, tutorial, parameter docs)
ray (ray cluster config docs, first yaml parsing, yaml parser for grid search, rllib example config,search space API docs)

Features

Allow users to specify hyperparameter search yaml syntax (but default to sweeps (?))
Enable pre-configured envs and agents (e.g. for a benchmark or for a baseline agent).
Change name of experiment file to just config (?)

Example yaml

agent:
    type: rllib
    config:
        run_or_experiment: PPO
        <remaining rllib agent config apart from env>
        ...
env:
    gym: energym
    config:
        name: env_name_within_gym
         <usual config>

Open questions

Should the env params be set by agent script, or should this be directly set by beobench in container (i.e. all envs in container have certain fixed config)? The latter seems too inflexible, especially since some parts of the env config could be considered agent-related (i.e. normalisation).
What library should be used to check config values/add defaults (q: do we need to check data types, valid values -- is that even feasible?):
- munch
- dataclass (built-in)
How well does the sweeps API match the search spaces in RLlib? Should sweeps be used inside the container? (what would that mean when using alternative tracking options)
Would it be possible to have beobench be called from tuning library (tune or sweeps)? i.e. is it possible to make beobench tuning library agnostic?
- This would require for an efficient start of new experiments within the same container (potentially with the option to start each experiment in new container if desired).

Status

5 March (on dev/general):
- config argument added to scheduler.run() method, but it does
- template yaml config added

Add CI deployment to Pypi

Problem

Currently publishing version to Pypi is done manually.

Potential Solution

Add GitHub action to do this automatically, triggered by adding a release tag on the main branch. It would also make sense to add at leat some rudimentary automatic testing so that the triggered releases are stable (see #25).

For options see https://github.com/rdnfn/beobench/actions/new.

Fix broken links in main readme env list

Problem

Currently the main readme contains a number of outdated links to env descriptions. Update the env list to reflect the current state of the documentation.

Simplify command line interface

Description

Currently calling the command line is a bit unwieldy:

python -m beobench.experiment.scheduler

This could be replaced with something like:

beobench run --method ppo --env boptest.bestest_hydronic_v0

How

Following the click guide on integrating with Setup tools, use the entry_points argument of the setuptools.setup() to remove python -m from CLI calls.

TODOs

Update click command
Add env
and method args to CLI and Python API
Fix no_container command to work with new CLI

Fix API section in docs

Problem

Currently, the API section in the docs is not actually populated with the docstrings from the package

Solution

Use a long-term maintainable way to add the docstrings as required.