spine-tools / spine-toolbox

Spine Toolbox is an open source Python package to manage data, scenarios and workflows for modelling and simulation. You can have your local workflow, but work as a team through version control and SQL databases.

Spine Toolbox

Link to the documentation: https://spine-toolbox.readthedocs.io/en/latest/?badge=latest

Spine Toolbox is an open source Python package to manage data, scenarios and workflows for modelling and simulation. You can have your local workflow, but work as a team through version control and SQL databases.

Programming language

• Python 3.8*
• Python 3.9
• Python 3.10
• Python 3.11

*Python 3.8.0 is not supported (use Python 3.8.1 or later).

License

Spine Toolbox is released under the GNU Lesser General Public License (LGPL) license. All accompanying documentation, original graphics and other material are released under the Creative Commons BY-SA 4.0 license. Licenses of all packages used by Spine Toolbox are listed in the Spine Toolbox User Guide.

Attribution

If you use Spine Toolbox in a published work, please cite the following publication (Chicago/Turabian Style).

Kiviluoma Juha, Pallonetto Fabiano, Marin Manuel, Savolainen Pekka T., Soininen Antti, Vennström Per, Rinne Erkka, Huang Jiangyi, Kouveliotis-Lysikatos Iasonas, Ihlemann Maren, Delarue Erik, O’Dwyer Ciara, O’Donnel Terence, Amelin Mikael, Söder Lennart, and Dillon Joseph. 2022. "Spine Toolbox: A flexible open-source workflow management system with scenario and data management" SoftwareX, Vol. 17, 100967, https://doi.org/10.1016/j.softx.2021.100967.

Installation

We provide three options for installing Spine Toolbox. The first two options also require that you follow the Pre-installation steps:

• Python/pipx (we intend to make stable releases every month or so)
• From source files (this is the cutting edge - and more likely to have bugs)
• Windows installation package (these are quite old - not recommended)

Pre-installation

These steps apply to both Python/pipx option and to From source files option.

1. If you don't have Python installed, please install e.g. Python 3.11 from Python.org.

2. Test that python is now in your PATH. Open a new terminal (e.g. Command Prompt) window and type

   python --version
   

   The output should be the Python version. If it did not work (e.g. the output is 'python' command not found or similar or the Microsoft Store opens), you need to add the Python installation folder into the PATH environment variable. On Windows, this can be done from Windows Control Panel (use search with 'environment'). See specific instructions for all operating systems from this link.

3. Install Git. Windows version can be found here: https://git-scm.com/download/win.

4. Test that Git is now in your PATH. Open a new terminal after installing Git and type

   git --version
   

   If the output from this command is not the Git version, you need to add <git-install-path>/Git/cmd into your PATH environment variable.

5. Upgrade pip to the latest version (pip is a package manager for Python)

   python -m pip install --upgrade pip
   

Installation with Python and pipx

This works best for users that just want to use Spine Toolbox and keep it updated with new releases.

Once you have completed the Pre-installation steps, do the following:

1. Install pipx. pipx helps in creating an isolated environment for Spine Toolbox to avoid package conflicts. Open a terminal and run

    python -m pip install --user pipx
   

   After pipx has been installed, run

    python -m pipx ensurepath
   

2. Restart the terminal or re-login for the changes of the latest command to take effect.

3. Choose which Spine Toolbox version to install. Latest release version from PyPi is installed using

    python -m pipx install spinetoolbox
   

   or the latest development version using

    python -m pipx install git+https://github.com/spine-tools/spinetoolbox-dev
   

That’s it! To launch Spine Toolbox, open a terminal and run

spinetoolbox

If for some reason the command is not found, the executable can be found under ~/.local/bin (%USERPROFILE%\.local\bin on Windows).

To upgrade Spine Toolbox to the latest release, open a terminal and run

python -m pipx upgrade spinetoolbox

or upgrade the development version with

python -m pipx upgrade spinetoolbox-dev

Installation from sources using Git

This option is for developers and other contributors who want to debug or edit Spine Toolbox source code. Once you have completed the Pre-installation steps, do the following:

1. Clone this repository. Open a terminal, use the cd command to change the current directory to some data directory (preferably not a OneDrive folder on Windows), and run

   git clone https://github.com/spine-tools/Spine-Toolbox.git
   

   This creates a Spine-Toolbox directory into your current directory.

2. Use the cd command to go to Spine-Toolbox dir

   cd ./Spine-Toolbox
   

3. Create a Python virtual environment using

    python -m venv .venv
   

   This creates a .venv directory into the Spine-Toolbox directory, but you can create the virtual environment anywhere on your system.
   
   

   Optional: Instead of venv, one can also use a miniconda environment. You can download miniconda from here. Note: Anaconda environments are not supported. Create a new Python 3.11 miniconda environment without linking packages from the base environment using

    conda create -n spinetoolbox python=3.11
   

4. Activate the venv environment on Windows (provided that you are in Spine-Toolbox directory) using

   cd ./.venv/Scripts
   activate
   

   Or on bash, zsh

   source .venv/bin/activate
   

   or on a Conda prompt (if you are using a miniconda environment instead of venv)

   conda activate spinetoolbox 
   

   Make sure that the terminal prompt now indicates the active environment.

5. Use the cd .. command to go back to the Spine-Toolbox directory (the one that contains requirements.txt) if needed.

6. Install Spine Toolbox along with its dependencies with

    python -m pip install -r requirements.txt
   

7. (Optional) Install additional development packages with

    python -m pip install -r dev-requirements.txt
   

8. Once the installation has finished, you can now launch Spine Toolbox using

   spinetoolbox
   

To upgrade, pull the latest changes from the Spine Toolbox repository (while in Spine-Toolbox directory)

git pull
git fetch --tags origin

and run (to upgrade the other Spine Toolbox packages)

python -m pip install -U -r requirements.txt

NOTE: The version number of Spine packages are automatically generated from Git tags during the python -m pip install ... step. This makes it important to fetch new tags from GitHub before running the python -m pip install ... when upgrading. This is true also for editable installations.

Additional comments for developers

If you are alternating between multiple branches, you should repeat the pip install -e . step after switching a branch. This ensures the version number is updated correctly. On the other hand if you do not need the version number to be precise, it is okay to skip the step.

To illustrate with an example, consider there are two feature branches P and Q that you are working on. Both are based on master after the release 0.6.18.

 0.6.18
---o----o--o--o  P
    \
     o--o--o--o  Q

Then the versions are:

• P: 0.7.0.dev3+g<commit-hash-P>.d<commit-date-P>,
• Q: 0.7.0.dev4+g<commit-hash-Q>.d<commit-date-Q>.

The version should be available inside Python as:

from spinetoolbox.version import version_tuple

print(version_tuple)
# P: (0, 7, 0, 'dev3', 'g<commit-hash-P>.d<commit-date-P>')
# Q: (0, 7, 0, 'dev4', 'g<commit-hash-Q>.d<commit-date-Q>')

So if your code uses any of the components of the version_tuple that might change, you should re-run pip install, otherwise this can be ignored safely.

Windows 64-bit installer package

There are old Windows installer packages available for a quick install, but they are at this point (26.1.2023) quite obsolete and cannot be recommended for anything but a quick look at how Spine Toolbox looks and feels (although even that has changed). Download the installer package from here, run it, and follow the instructions to install Spine Toolbox.

About requirements

Python 3.8.1-3.11 is required. Python 3.8.0 is not supported due to problems in DLL loading on Windows.

See the files pyproject.toml and requirements.txt for packages required to run Spine Toolbox. (Additional packages needed for development are listed in dev-requirements.txt.)

The requirements include three packages (spinedb_api, spine_engine, and spine_items), developed by the Spine project consortium.

Building the User Guide

You can find the latest documentation on readthedocs. If you want to build the documentation yourself, source files for the User Guide can be found in docs/source directory. In order to build the HTML docs, you need to install the optional requirements (see section 'Installing requirements' above). This installs Sphinx (among other things), which is required in building the documentation. When Sphinx is installed, you can build the HTML pages from the user guide source files by using the bin/build_doc.bat script on Windows or the bin/build_doc.sh script on Linux and Mac. After running the script, the index page can be found in docs/build/html/index.html. The User Guide can also be opened from Spine Toolbox menu Help->User Guide (F2).

Troubleshooting

Obscure crashes that may produce a traceback related to PySide6's model classes

The first thing is to make sure that you are not using Anaconda. Only Miniconda is supported. Anaconda's base environment includes Qt related packages and we suspect that they leak some shared Qt libraries into environments, even when specifically requesting that base environment packages should not be linked to environment packages. See also Problems in starting the application below.

Installation fails

Please make sure you are using Python 3.8, 3.9, 3.10, or 3.11 to install the requirements.

'No Python' error when installing with pipx

If you see the following error when running the command python -m pipx install spinetoolbox

No Python at 'c:\python38\python.exe'
Fatal error from pip prevented installation. Full pip output in file:

Where c:\python38\python.exe may be some other path. To fix this, delete the folder C:\Users\<user name>\.local\pipx\shared and run the python -m pipx install spinetoolbox command again.

Installation fails on Linux

If Python runs into errors while installing on Linux systems, running the following command in a terminal may help:

$ sudo apt install libpq-dev

Problems in starting the application

If there are problems in starting Spine Toolbox, the chances are that the required packages were not installed successfully. In case this happens, the first thing you should check is that you don't have Qt, PyQt4, PyQt5, PySide, PySide2, and PySide6 packages installed in the same environment. These do not play nice together and may introduce conflicts. In addition, make sure that you do not have multiple versions of these Qt related packages installed in the same environment. The easiest way to solve this problem is to create a blank (e.g. venv virtual environment) Python environment just for PySide6 applications and installing the requirements again.

Warning: Using the conda-forge channel for installing the requirements is not recommended.

The required qtconsole package from the conda-forge channel also installs qt and PyQt packages. Since this is a PySide6 application, those are not needed and there is a chance of conflicts between the packages.

Note: Python 3.8.0 is not supported. Use Python 3.8.1 or later.

Recorded Webinars showing the use of Spine Tools

Spine Toolbox: Data, workflow and scenario management for modelling

Wednesday Sep 8, 17:00-18:30 CEST (11:00-12:30 EDT) - Organized together with G-PST

Spine Toolbox is an open source software to manage data, scenarios and workflows for modelling and simulation. You can have your local workflow, but work as a team through version control and SQL databases. This webinar gives a quick overview of the different functionalities and showcases them through two examples.

Recording Chapters:

• 00:00-00:05 Relation to G-PST Pillar 5 (Clayton Barrows, NREL)
• 00:05-00:35 Building a workflow in Spine Toolbox PDF, (Juha Kiviluoma, VTT)
• 00:35-00:55 Example workflow from Canada to manage lots of input sources PDF, (Madeleine McPherson and Jake Monroe, University of Victoria)
• 01:05-01:15 Example workflow from EU project TradeRES to serve multiple models (Milos Cvetkovic and Jim Hommes, TU Delft)
• 01:15-01:30 Moderated Q&A

SpineOpt: A flexible energy system modelling framework in Julia

Tuesday Sep 7, 14:00-15:30 CEST (8:00-9:30 EDT) - Organized together with EERA ESI

The importance of operational details in planning future energy systems has been rapidly increasing. One driver for this is the increasing role of variable power generation, which requires that the energy system models consider higher temporal granularity, longer time series and new aspects in power system stability. Another important driver is a consequence of sector coupling through the electrification of transport, space heating and industries. As the other sectors become more integrated with electricity, they need to be modelled at a granularity that is sufficient for describing the potential flexibility they can bring to the power system dominated by variable and uncertain power generation.

This webinar will present the open source Julia based energy system modelling framework SpineOpt, which has been built with these challenges in mind. It can represent different energy sectors using representation typically available only in sector specific models and with highly adaptable temporal and stochastic structures available both for planning and operations. More information at https://spine-project.github.io/SpineOpt.jl/latest/index.html.

Recording Chapters:

• 00:00-00:08 Introduction PDF
• 00:08-00:31 Basic elements of SpineOpt and the flexible spatial structure PDF
• 00:31-00:40 Adaptable temporal and stochastic structures PDF
• 00:50-01:30 Representation of different energy sectors & Attempts to make the model faster PDF
• 01:25-01:35 Q&A + Step-by-step 10 minute demo on how to build a simple model with SpineOpt using Spine Toolbox

Demonstration of Spine modelling tools through selected case studies

Thursday Sep 9, 14:00-16:00 CEST (8:00-10:00 EDT)

Full recording, all presentations slides PDF

Over the past 4 years, the EU project Spine has developed a set of open-source tools for modelling complex energy systems. This webinar demonstrates the Spine software through six selected case studies, covering topics such as sector coupling, co-optimization of operation and investments, stochastic modelling, and rolling horizon optimization. Each subsection described below consists of a brief introduction followed by a live demonstration of the particular case, where some of the outstanding features of Spine are highlighted and discussed.

Recording Chapters:

• 00:05-00:15 Introduction to Spine: This section uses a simple example to demonstrate the SpineOpt modelling principle. First, the user defines the different objects in their system, such as units and nodes, as well as the relationships between them, such as which units are connected to which nodes. Then, they specify values for certain pre-defined parameters such as node demand, unit capacity, cost, and conversion ratio. The resulting dataset is passed to SpineOpt which generates the corresponding optimisation model, optimizes it, and delivers the results.
• 00:15-00:30 Hydro: This section demonstrates hydropower modelling in Spine as performed in Case study A5. The objective is to model part of the Swedish hydropower system, namely the Skellefte river with its 15 power stations, by coupling the river system with the power system. The model maximizes income from electricity supply over one week with an hourly resolution, while respecting basic hydrological constraints.
• 00:29-00:45 Building heating: This section demonstrates building heating modelling in Spine as performed in Case study A4. The objective is to model the Finnish power and district heating system coupled with an electrically heated residential building stock. The result is a rolling unit commitment and economic dispatch model, that optimizes system performance over one year at hourly resolution.
• [00:45-00:57] Break
• 00:57-01:06 Gas grid: This section demonstrates gas grid modelling in Spine as performed in Case study A2. The objective is to model a natural gas transmission system with pressure-driven gas and couple it with an electricity system to capture the flexibility provided by the gas network. The result is a dispatch model that co-optimizes operations in both systems over a day at hourly resolution.
• 01:06-01:22 Stochastic: This section demonstrates stochastic modelling in Spine using a simple example system. Three different stochastic structures are demonstrated: purely deterministic, stochastic fan, and converging fan.
• 01:22-01:35 Power grid investments: This section demonstrates power grid investment modelling in Spine as performed in case study C2. The objective is twofold: (i) to model the Nordic synchronous power system (Norway, Sweden, Finland, and Denmark) with high operational detail; and (ii) to find optimal transmission line investment decisions over a long-term horizon of 10 years, for three different wind penetration scenarios.
• 01:35-01:40 Q&A

SpineInterface: How to quickly and easily create new optimization models in Julia Friday

Friday Sep 10, 14:00-15:30 CEST (8:00-9:30 EDT)

Full recording, all presentations PDF

Creation of new optimisation models requires a lot of work to get the data to the models and the results out of the models. Spine Toolbox is an open source data and workflow management tool to assist with these tasks and can work with models written in any language. Meanwhile, SpineInterface is a Julia package that links the data management capabilities of Spine Toolbox with Julia/JuMP modelling environment in a very direct way.

The data interfaces of Spine Toolbox together with SpineInterface simplifies the process of developing optimization models by allowing the model developer to focus on the constraint equations. The required data structures and data are defined with a graphical interface in Spine Toolbox and are immediately available to the model developer inside the constraint equation code without any action or code required by the model developer. SpineInterface supports the full range of data parameter types supported by Toolbox and provides a mechanism for representation of time and time-based data, either time series, time patterns or arbitrarily varying temporal data.

This session will be of interest to model developers and/or students who want a significant head start in developing optimization models. The sessions will also be of interest to model developers who may wish to translate existing models developed in other platforms such as GAMS, into the Spine framework using SpineInterface. The power of SpineInterface will be demonstrated in an interactive session where the full modelling workflow will be illustrated from data structure design and implementation to constraint equation development.

Agenda:

• Overview of SpineInterface
• Toolbox concepts and data structures including the Spine data API
• SpineInterface: convenience functions and accessing Spine Toolbox data in Julia
• Defining a model data structure in Spine Toolbox
• Building and solving an optimization model using SpineInterface
• Q&A + live demo 00:45-01:21

Contribution Guide

All are welcome to contribute!

See detailed instructions for contribution in Spine Toolbox User Guide.

Below are the bare minimum things you need to know.

Setting up development environment

1. Install the developer requirements:

    python -m pip install -r dev-requirements.txt
   

2. Optionally, run pre-commit install in project's root directory. This sets up some git hooks.

Coding style

• Black is used for Python code formatting. The project's GitHub page includes instructions on how to integrate Black in IDEs.
• Google style docstrings

Linting

It is advisable to run pylint regularly on files that have been changed. The project root includes a configuration file for pylint. pylint's user guide includes instructions on how to integrate the tool in IDEs.

Unit tests

Unit tests are located in the tests directory. You can run the entire test suite from project root by

python -m unittest

Reporting bugs

If you think you have found a bug, please check the following before creating a new issue:

1. Make sure you’re on the latest version.
2. Try older versions.
3. Try upgrading/downgrading the dependencies
4. Search the project’s bug/issue tracker to make sure it’s not a known issue.

What to put in your bug report:

1. Python version. What version of the Python interpreter are you using? 32-bit or 64-bit?
2. OS. What operating system are you on?
3. Application Version. Which version or versions of the software are you using? If you have forked the project from Git, which branch and which commit? Otherwise, supply the application version number (Help->About menu).
4. How to recreate. How can the developers recreate the bug? A screenshot demonstrating the bug is usually the most helpful thing you can report. Relevant output from the Event Log and debug messages from the console of your run, should also be included.

Feature requests

The developers of Spine Toolbox are happy to hear feature requests or ideas for improving existing functionality. The format for requesting new features is free. Just fill out the required fields on the issue tracker and give a description of the new feature. A picture accompanying the description is a good way to get your idea into development faster. But before you make a new issue, please check that there isn't a related idea already open in the issue tracker.

 

---

	This work has been partially supported by EU project Mopo (2023-2026), which has received funding from European Climate, Infrastructure and Environment Executive Agency under the European Union’s HORIZON Research and Innovation Actions under grant agreement N°101095998.
	This work has been partially supported by EU project Spine (2017-2021), which has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 774629.

spine-toolbox's Issues

Redo UI

In GitLab by @PekkaSavolainen on Apr 19, 2018, 10:58

UI should be redone to fix the graphics glitches in the current QGraphicsView. The glitches are probably caused by QGraphicsProxyWidgets. QGraphicsView (project view or whatever) should look like this:

When a user selects an item, the current subwindow stuff should be shown in its own widget in the Spine Toolbox main window.

Undo-Redo functionality in Design View

In GitLab by @manuelma on Apr 21, 2018, 13:13

I don't know how difficult is to implement this or if there are ready-made libraries that we can use, but it would be really nice. Let's say the user removed a project item or a link by mistake, they should be able to get it back with ctrl-z

Change Tool template

In GitLab by @PekkaSavolainen on Mar 2, 2018, 18:31

User should be able to change the Tool template associated to a Tool without destroying the Tool and making a new Tool.

Tool - Tool interface

In GitLab by @PekkaSavolainen on Mar 1, 2018, 16:30

Tools should be able to find input files that are output files of another tool and then execute the preceding tool before executing itself.

Bug when julia subprocess fails to start

In GitLab by @manuelma on Mar 30, 2018, 08:32

If Julia is not properly installed in the sytem and the user executes a Julia tool, the Julia subprocess will terminate with an error and its _process attribute will be set ot None. If then the user fixes the problem with Julia and tries to execute the Julia tool again, the Toolbox will try and check if the Julia subprocess is already running and will call state() on a None object.

Feature request: view and edit connections by drawing links between items on the mdiArea

In GitLab by @manuelma on Apr 6, 2018, 15:27

I think this would be very nice to attract users. I am currentyl working on it and the first results are up on the experimental branch...

Tool details

In GitLab by @PekkaSavolainen on Feb 7, 2018, 11:33

User should see at least the executable model main program file, required input files and output files from the Tool item in UI without looking at the JSON Tool Template definition file.

Change Data Store into Data Collection

In GitLab by @PekkaSavolainen on Jan 24, 2018, 10:20

New name for Data Store is Data Collection. Change all references to Data Store -> Data Collection. Also Add Data Store icon should be changed.

Data Interface impossible to remove from the Design view

Once the Data Interface is added in the Design View it can't be removed. If the item is right-clicked the menu doesn't appear and the terminal returns: "2019-06-24 11:34:26 ERROR: Unknown item type:Data Interface".
I'm using julia 1.1.1 | Spine Toolbox v0.2.4 | spinedatabase_api v0.0.27 |Python 3.7.1 64-bit | Qt 5.9.6 | PyQt5 5.9.2 | Windows 10

Support for tabular datapackage in the Spine Toolbox

In GitLab by @manuelma on Apr 2, 2018, 10:23

It would be nice if Spine Toolbox could provide some support to create datapackages for Data connections to CSV files. Datapackages are great for fitting tabular data into Spine as the metadata they provide is so rich. Perhaps the key aspect is the ability to specify foreign keys, since these can be turned into relationships when creating a Spine data object in EAV/CR format.

An idea for the implementation is in this branch. To try the functionality, just create a new Data connection and add some CSV files to it. Note that you will need to install the Python datapackage module as explained in the README.

Set name and description for Data Stores, Data Connections, Tools, and Views

In GitLab by @PekkaSavolainen on Jan 16, 2018, 14:10

Ask user to enter name and description for new items. No duplicate names allowed.

Namechange: Tool Candidates -> Tool Templates

In GitLab by @PekkaSavolainen on Feb 7, 2018, 11:16

Tool Template is a better name for executable models that the Tool item represents.

Project directory structure

In GitLab by @PekkaSavolainen on Jan 16, 2018, 14:26

Make first version of directory structure for projects. Each item should have its own directory with input and output folders. This is likely to change in the future. Note: All items must have a unique name.

e.g.

Projects/
   project_1/
      data_connection_1/
         input/
         output/
      tool_1/
         input/
         output/

Linking multiple data connections to tools

In GitLab by @PekkaSavolainen on Mar 1, 2018, 16:27

At the moment, tools only search input files from their immediate input data connections. They should be able to find input files from the input items of those data connections and beyond. Watch out for infinite loops while searching for files due to connections looping back to the source and also feedback loops.

Add executable tool to project

In GitLab by @PekkaSavolainen on Jan 23, 2018, 14:24

Implement possibility to add an executable tool (GAMS model, Julia Model, or other) to Tool items.

Support for input files for Data Connections

In GitLab by @PekkaSavolainen on Jan 16, 2018, 11:33

User should be able to add input files for Data Connections

Code refactoring. Move item model related methods to own class

In GitLab by @PekkaSavolainen on Jan 19, 2018, 14:37

QStandardItemModel stuff should be in its own class. Maybe inherit QStandardItemModel or make a new model from QAbstractItemModel. Move all methods that handle item model stuff to this class.

Visualize project data processing chain

In GitLab by @PekkaSavolainen on Jan 16, 2018, 15:43

All connected items in the project should be visualized (automatically) when the user connects items to other items.
e.g.
Data Connection -> Tool -> Data Store -> View

UI basics

In GitLab by @PekkaSavolainen on Dec 15, 2017, 14:44

Make a Widget for the following items:

• Tool
• Data Store/Collection
• Data Connection
• View

Make a way for the user to add these to Spine Toolbox.

Widget for making Tool Templates

In GitLab by @PekkaSavolainen on Feb 7, 2018, 11:20

User should not have to make Tool Template definition files (JSON) by hand. There should be a widget for doing this in Spine Toolbox.

Saving Tool output files

In GitLab by @PekkaSavolainen on Jan 16, 2018, 11:38

Tool item should be able to save output files to a data store or a data connection. What happens to Tool output files if there is no data store or data connection connected to it?

For version 0.1 the result files can be saved from the work folder to project folder. Just make a new directory for each tool that finishes. Timestamp when the tool finished is the result directory name. All tools have their own directory under __results__ directory.

Example directory structure for project called spine1:

SPINETOOLBOX/projects/ spine1/ __results__/ tool1/ 2018-04-30T10.11.32/ 2018-04-30T12.22.10/ tool2/ 2018-04-30T12.28.27/

NOTE1: results directory should have two underscores on both sides.
NOTE2: This prevents making a project item called __results__ but I guess that is something we can live with for now.

@manuelma

Executing GAMS models

In GitLab by @PekkaSavolainen on Feb 1, 2018, 16:38

Implement functionality for 'Execute' button of Tool items.

Fix Data Store file refresh issue

In GitLab by @PekkaSavolainen on May 4, 2018, 13:00

Check if Data Store items have the same problem as Data Connection items described in Issue #41.

@manuelma

Tool configuration assistant incompatible with julia 1.1.1.

The tool configuration assistant is unable to run and return this error:
"SpineModel.jl requires Julia version 1.x, whereas current version is 1.1.1."
I'm using julia 1.1.1 | Spine Toolbox v0.2.4 | spinedatabase_api v0.0.27 |Python 3.7.1 64-bit | Qt 5.9.6 | PyQt5 5.9.2 | Windows 10

RunTimeError when a Julia Tool is executed without REPL

In GitLab by @PekkaSavolainen on Apr 19, 2018, 11:02

When a Julia Tool execution is finished, this happens:

Traceback (most recent call last):
File "C:\data\GIT\SPINETOOLBOX\spinetoolbox\tool_instance.py", line 136, in julia_tool_finished
self.tool_process.repl_finished_signal.disconnect(self.julia_tool_finished)
RuntimeError: Failed to disconnect signal repl_finished_signal(int).

Save application window size and location between sessions

In GitLab by @PekkaSavolainen on Jan 16, 2018, 11:29

Application should remember the window size and location from the last time it was opened.

Fix Julia Tool execution

In GitLab by @PekkaSavolainen on Mar 29, 2018, 17:37

Spine Toolbox needs to know when a script execution has finished in the Julia REPL. Without this information, Spine Toolbox cannot start another Tool or do anything with the results.

Another option is to fix the performance issues when not using Julia REPL. I.e. running scripts as "julia script.jl". The issue is best summarized in link:
https://stackoverflow.com/questions/17056824/overhead-in-running-julia-from-command-line?rq=1

Feature request: wildcards in Tool Template inputfiles

In GitLab by @manuelma on Apr 15, 2018, 12:11

The user should be able to specify something like:

{
  ...
  "inputfiles" : ["datapackage.json", "*.csv"]
  ...
}

Then, whatever .csv files are present in the Tool's inputs should be copied into the Work directory.

System Settings window can't be enlarged

The size of the system settings window can't be enlarged and the current size makes it hard to use.

Refresh Tool when Tool Template changes

In GitLab by @PekkaSavolainen on Feb 7, 2018, 11:27

If user makes changes to the Tool Template when Spine Toolbox is running, the changes are not reflected to Tools that already use this Tool Template. At the moment, the user must restart the application after changing Tool Templates for the changes to take effect.

Refresh Tools button that reloads the Tool Templates into ToolTemplateModel is a possible solution to this issue.

Interface to Spine data structure

In GitLab by @PekkaSavolainen on Dec 15, 2017, 14:52

Make a first version of Spine data structure classes. Needs somekind of interface class structure to the actual data.

Open questions:

• Where is the Spine data structure data? Local computer or on a server?
• How is the Spine data structure stored? SQLite, MySQL?

Problems with installation of SpineToolbox

Hi @manuelma
I tried installing the toolbox on a relatively clean conda env "spinetoolbox" (there is just spinedb-api installed).

I get following error:

(spinetoolbox) steffen@steffen-NG-VN7-591G-767P:~/SPINE/Spine-Toolbox$ pip install -r requirements.txt
Obtaining file:///home/steffen/SPINE/Spine-Toolbox (from -r requirements.txt (line 8))
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing wheel metadata ... done
Requirement already satisfied: spinedb_api from git+https://github.com/Spine-project/Spine-Database-API.git#egg=spinedb_api in /home/steffen/ELDEST/Spine-Database-API (from -r requirements.txt (line 1)) (0.7.14)
Collecting spine_engine
Cloning https://github.com/Spine-project/spine-engine.git to /tmp/pip-install-68yfmq02/spine-engine
Running command git clone -q https://github.com/Spine-project/spine-engine.git /tmp/pip-install-68yfmq02/spine-engine
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing wheel metadata ... done
Collecting cchardet!=2.1.6
Using cached https://files.pythonhosted.org/packages/c6/20/905b6c5664736d884a40ac3b1204ab874c3c4a8ce86f7b2e28abc1fc6ee4/cchardet-2.1.5-cp37-cp37m-manylinux1_x86_64.whl
Collecting matplotlib!=3.2.1
Using cached https://files.pythonhosted.org/packages/f1/b9/712584f12f840968a14969a1fe298ffdeaa9c4b484b3bfd973c74c4a481d/matplotlib-3.3.1-cp37-cp37m-manylinux1_x86_64.whl
Collecting gdx2py>=2.0.4
Downloading https://files.pythonhosted.org/packages/57/96/aba6646dbd902a60957b6b9e87947a77479baf232c8f2c14bcefb85b155f/GDX2py-2.1.1-py3-none-any.whl
Collecting pyodbc>=4.0.23
Downloading https://files.pythonhosted.org/packages/81/0d/bb08bb16c97765244791c73e49de9fd4c24bb3ef00313aed82e5640dee5d/pyodbc-4.0.30.tar.gz (266kB)
|████████████████████████████████| 276kB 2.2MB/s
Collecting jsonschema==2.6
Using cached https://files.pythonhosted.org/packages/77/de/47e35a97b2b05c2fadbec67d44cfcdcd09b8086951b331d82de90d2912da/jsonschema-2.6.0-py2.py3-none-any.whl
Collecting datapackage>=1.2.3
Downloading https://files.pythonhosted.org/packages/12/3b/89542ab6c64d455cd4aca22261f61892260797d546048cbb537309b4dc0a/datapackage-1.15.0-py2.py3-none-any.whl (85kB)
|████████████████████████████████| 92kB 6.4MB/s
Requirement already satisfied: python-dateutil>=2.8.0 in /home/steffen/anaconda3/envs/spinetoolbox/lib/python3.7/site-packages (from spinetoolbox==0.4.3->-r requirements.txt (line 8)) (2.8.1)
Requirement already satisfied: sqlalchemy>=1.2.6 in /home/steffen/anaconda3/envs/spinetoolbox/lib/python3.7/site-packages (from spinetoolbox==0.4.3->-r requirements.txt (line 8)) (1.3.19)
Requirement already satisfied: numpy>=1.15.1 in /home/steffen/anaconda3/envs/spinetoolbox/lib/python3.7/site-packages (from spinetoolbox==0.4.3->-r requirements.txt (line 8)) (1.19.1)
Collecting ijson>=2.6.1
Downloading https://files.pythonhosted.org/packages/6b/fc/2b2fadd440dc2730c38c0b563d697358aab2bc750e035f2857a310e59406/ijson-3.1.1-cp37-cp37m-manylinux2010_x86_64.whl (129kB)
|████████████████████████████████| 133kB 8.3MB/s
Collecting pyside2<5.15,>=5.14
Downloading https://files.pythonhosted.org/packages/76/67/1a9667e7aa16fffaadd224b9a3d5504b634a928085a4b7728d95e0c67d6d/PySide2-5.14.2.3-5.14.2-cp35.cp36.cp37.cp38-abi3-manylinux1_x86_64.whl (166.0MB)
|████████████████████████████████| 166.0MB 160kB/s
Collecting pandas>=0.24.0
Downloading https://files.pythonhosted.org/packages/61/ed/10112535645ad7afd26c3b9defd20a32d9e42340b19c4f73ff26ccad06ee/pandas-1.1.1-cp37-cp37m-manylinux1_x86_64.whl (10.5MB)
|████████████████████████████████| 10.5MB 4.2MB/s
Collecting scipy>=1.1.0
Using cached https://files.pythonhosted.org/packages/65/f9/f7a7e5009711579c72da2725174825e5056741bf4001815d097eef1b2e17/scipy-1.5.2-cp37-cp37m-manylinux1_x86_64.whl
Collecting psycopg2>=2.7.4
Using cached https://files.pythonhosted.org/packages/fd/ae/98cb7a0cbb1d748ee547b058b14604bd0e9bf285a8e0cc5d148f8a8a952e/psycopg2-2.8.6.tar.gz
ERROR: Command errored out with exit status 1:
command: /home/steffen/anaconda3/envs/spinetoolbox/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-68yfmq02/psycopg2/setup.py'"'"'; file='"'"'/tmp/pip-install-68yfmq02/psycopg2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-68yfmq02/psycopg2/pip-egg-info
cwd: /tmp/pip-install-68yfmq02/psycopg2/
Complete output (23 lines):
running egg_info
creating /tmp/pip-install-68yfmq02/psycopg2/pip-egg-info/psycopg2.egg-info
writing /tmp/pip-install-68yfmq02/psycopg2/pip-egg-info/psycopg2.egg-info/PKG-INFO
writing dependency_links to /tmp/pip-install-68yfmq02/psycopg2/pip-egg-info/psycopg2.egg-info/dependency_links.txt
writing top-level names to /tmp/pip-install-68yfmq02/psycopg2/pip-egg-info/psycopg2.egg-info/top_level.txt
writing manifest file '/tmp/pip-install-68yfmq02/psycopg2/pip-egg-info/psycopg2.egg-info/SOURCES.txt'

Error: pg_config executable not found.

pg_config is required to build psycopg2 from source.  Please add the directory
containing pg_config to the $PATH or specify the full executable path with the
option:

    python setup.py build_ext --pg-config /path/to/pg_config build ...

or with the pg_config option in 'setup.cfg'.

If you prefer to avoid building psycopg2 from source, please install the PyPI
'psycopg2-binary' package instead.

For further information please check the 'doc/src/install.rst' file (also at
<https://www.psycopg.org/docs/install.html>).

----------------------------------------

ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Add Spine logo and fix bug in About window

In GitLab by @PekkaSavolainen on Dec 15, 2017, 13:37

Add support for Julia tools

In GitLab by @PekkaSavolainen on Dec 19, 2017, 18:25

Projects

In GitLab by @PekkaSavolainen on Dec 15, 2017, 13:29

Add support for saving/loading projects.

Add support for GAMS tools

In GitLab by @PekkaSavolainen on Dec 15, 2017, 13:27

Support for adding a (dummy) GAMS program to a tool.

Event log

In GitLab by @PekkaSavolainen on Jan 16, 2018, 11:22

A widget that prints information to user.

Settings window

In GitLab by @PekkaSavolainen on Jan 16, 2018, 11:20

Implement a settings window.

Data Connection - Tool interface

In GitLab by @PekkaSavolainen on Jan 16, 2018, 11:35

Tool should be able to check data connections that are connected to it for input files that the tool requires.

opening the getting_started.html fails

After following the installation instructions and starting Spine-Toolbox, I clicked on the "Getting Started" guide in the Event Log.

This unfortunately fails, yielding this message in the Event Log:
[16-11-2020 12:51:16] Opening path [...]\Spine-Toolbox\docs\build\html\getting_started.html failed

and this message in the terminal:
ShellExecute 'file:///[...]Spine-Toolbox%5Cdocs%5Cbuild%5Chtml%5Cgetting_started.html' failed (error 2).

I'm running Windows 10, 64 bit, and Python 3.7.9, on the latest master branch.

The folder docs/build does not exists. I've also tried to python setup.py install but that didn't help either.
Any suggestions?

After having installed the dev-requirements.txt and ran make html in the docs directory, it seems to work.
Is this the intended behavior?

Fix Data Connection file refresh issue

In GitLab by @PekkaSavolainen on May 4, 2018, 12:57

Data files in data connection directory should be refreshed every time a data connection item is selected.

This should also happen when the user presses Open button -> Adds a file to data connection folder and then goes back to Spine Toolbox folder.

@manuelma

Display archetype properties

In GitLab by @jkiviluo on May 4, 2018, 22:16

Units and connections will be connected to an archetype that defines how the unit or the connection will behave in the Spine Model (what constraints it will use and when). The user should have a nice view on the unit and the archetype showing what the unit will do. It would also be really nice to be able to test the unit directly (run a mini-model as a test). Version 1.x.

Save project as... changes only the save file name

In GitLab by @PekkaSavolainen on Jan 16, 2018, 15:40

Save project as... should give the option to change the project name. After the name has been changed, check that all project items are owned by the new project.

Why the toolbox needs the work directory?

In GitLab by @manuelma on Apr 9, 2018, 11:43

I've been meaning to ask this, is it a GAMS-related issue?
To me it seems simpler to copy files only to the project folder and only once, and use that as the working directory. What am I missing?

Connecting blocks to other blocks

In GitLab by @PekkaSavolainen on Dec 19, 2017, 18:29

For example build support for a connection from data connection -> data store -> tool

Widget for showing Tool code

In GitLab by @PekkaSavolainen on May 4, 2018, 13:56

There needs to be a window that shows the code of a Tool (main script) when a Tool is selected. Maybe a QTextBrowser could do the job. Do not enable editing the code in Spine Toolbox yet (a separate issue is needed for this).

@manuelma

missing install dependencies

In the installation process, I get the following error, although the installation completes:

ERROR: mkdocs 1.1 requires Jinja2>=2.10.1, which is not installed.

Additionally, when I try to run the toolbox my installation is missing shiboken2:

(spinetoolbox) ➜  Spine-Toolbox git:(master)  python -m spinetoolbox

Traceback (most recent call last):
  File "/Users/cbarrows/miniconda/envs/spinetoolbox/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Users/cbarrows/miniconda/envs/spinetoolbox/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/cbarrows/Documents/repos/Spine-Toolbox/spinetoolbox/__main__.py", line 20, in <module>
    from .main import main
  File "/Users/cbarrows/Documents/repos/Spine-Toolbox/spinetoolbox/main.py", line 27, in <module>
    from .ui_main import ToolboxUI
  File "/Users/cbarrows/Documents/repos/Spine-Toolbox/spinetoolbox/ui_main.py", line 41, in <module>
    from .widgets.custom_menus import (
  File "/Users/cbarrows/Documents/repos/Spine-Toolbox/spinetoolbox/widgets/custom_menus.py", line 24, in <module>
    from ..plotting import plot_pivot_column, plot_selection, PlottingError, PivotTablePlottingHints
  File "/Users/cbarrows/Documents/repos/Spine-Toolbox/spinetoolbox/plotting.py", line 29, in <module>
    from .widgets.plot_widget import PlotWidget
  File "/Users/cbarrows/Documents/repos/Spine-Toolbox/spinetoolbox/widgets/plot_widget.py", line 19, in <module>
    from matplotlib.backends.backend_qt5agg import NavigationToolbar2QT as NavigationToolBar
  File "/Users/cbarrows/miniconda/envs/spinetoolbox/lib/python3.7/site-packages/matplotlib/backends/backend_qt5agg.py", line 11, in <module>
    from .backend_qt5 import (
  File "/Users/cbarrows/miniconda/envs/spinetoolbox/lib/python3.7/site-packages/matplotlib/backends/backend_qt5.py", line 15, in <module>
    import matplotlib.backends.qt_editor.figureoptions as figureoptions
  File "/Users/cbarrows/miniconda/envs/spinetoolbox/lib/python3.7/site-packages/matplotlib/backends/qt_editor/figureoptions.py", line 12, in <module>
    from matplotlib.backends.qt_compat import QtGui
  File "/Users/cbarrows/miniconda/envs/spinetoolbox/lib/python3.7/site-packages/matplotlib/backends/qt_compat.py", line 145, in <module>
    _setup_pyqt5()
  File "/Users/cbarrows/miniconda/envs/spinetoolbox/lib/python3.7/site-packages/matplotlib/backends/qt_compat.py", line 79, in _setup_pyqt5
    import shiboken2
ModuleNotFoundError: No module named 'shiboken2'

Finally, after installing shiboken2, I still get the following error:

(spinetoolbox) ➜  Spine-Toolbox git:(master)  python -m spinetoolbox        

Traceback (most recent call last):
  File "/Users/cbarrows/miniconda/envs/spinetoolbox/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Users/cbarrows/miniconda/envs/spinetoolbox/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/cbarrows/Documents/repos/Spine-Toolbox/spinetoolbox/__main__.py", line 20, in <module>
    from .main import main
  File "/Users/cbarrows/Documents/repos/Spine-Toolbox/spinetoolbox/main.py", line 27, in <module>
    from .ui_main import ToolboxUI
  File "/Users/cbarrows/Documents/repos/Spine-Toolbox/spinetoolbox/ui_main.py", line 41, in <module>
    from .widgets.custom_menus import (
  File "/Users/cbarrows/Documents/repos/Spine-Toolbox/spinetoolbox/widgets/custom_menus.py", line 24, in <module>
    from ..plotting import plot_pivot_column, plot_selection, PlottingError, PivotTablePlottingHints
  File "/Users/cbarrows/Documents/repos/Spine-Toolbox/spinetoolbox/plotting.py", line 29, in <module>
    from .widgets.plot_widget import PlotWidget
  File "/Users/cbarrows/Documents/repos/Spine-Toolbox/spinetoolbox/widgets/plot_widget.py", line 22, in <module>
    from .plot_canvas import PlotCanvas
  File "/Users/cbarrows/Documents/repos/Spine-Toolbox/spinetoolbox/widgets/plot_canvas.py", line 23, in <module>
    from pandas.plotting import register_matplotlib_converters
  File "/Users/cbarrows/miniconda/envs/spinetoolbox/lib/python3.7/site-packages/pandas/__init__.py", line 55, in <module>
    from pandas.core.api import (
  File "/Users/cbarrows/miniconda/envs/spinetoolbox/lib/python3.7/site-packages/pandas/core/api.py", line 15, in <module>
    from pandas.core.arrays import Categorical
  File "/Users/cbarrows/miniconda/envs/spinetoolbox/lib/python3.7/site-packages/pandas/core/arrays/__init__.py", line 9, in <module>
    from pandas.core.arrays.datetimes import DatetimeArray
  File "/Users/cbarrows/miniconda/envs/spinetoolbox/lib/python3.7/site-packages/pandas/core/arrays/datetimes.py", line 45, in <module>
    from pandas.core.arrays import datetimelike as dtl
  File "/Users/cbarrows/miniconda/envs/spinetoolbox/lib/python3.7/site-packages/pandas/core/arrays/datetimelike.py", line 49, in <module>
    from pandas.tseries import frequencies
  File "/Users/cbarrows/miniconda/envs/spinetoolbox/lib/python3.7/site-packages/pandas/tseries/frequencies.py", line 30, in <module>
    from pandas.tseries.offsets import (
  File "/Users/cbarrows/miniconda/envs/spinetoolbox/lib/python3.7/site-packages/pandas/tseries/offsets.py", line 1297, in <module>
    class CustomBusinessMonthEnd(_CustomBusinessMonth):
  File "/Users/cbarrows/miniconda/envs/spinetoolbox/lib/python3.7/site-packages/pandas/util/_decorators.py", line 334, in __call__
    func.__doc__ = dedent(self.join.join(docitems))
TypeError: sequence item 0: expected str instance, NoneType found

Toolbar for adding Data Stores, Data Connections and other items

In GitLab by @PekkaSavolainen on Jan 16, 2018, 11:25

Remove push buttons and put them in a toolbar. Also, Remove selected item and Remove all items from project buttons should be in the toolbar.

Support for tool execution in a subprocess

In GitLab by @PekkaSavolainen on Dec 15, 2017, 13:31

Implement classes to handle running an external model (tool) in a subprocess.

Remove Conn. buttons from all project item widgets

In GitLab by @PekkaSavolainen on May 4, 2018, 12:44

Remove Conn. buttons from all item widgets. They are not needed anymore.

@manuelma