iiasa / ixmp

The ix modeling platform for integrated and cross-cutting scenario analysis

ixmp: the ix modeling platform

The ix modeling platform (ixmp) is a data warehouse for high-powered scenario analysis, with interfaces to Python and R for efficient scientific workflows and effective data pre- and post-processing, and a structured database backend for version-controlled data management. In the name, “ix” stands for “integrated” and “cross (x) cutting”.

The MESSAGEix modeling framework is built on top of the ix modeling platform.

This repository contains the core and application programming interfaces (API) for the ix modeling platform (ixmp), as well as a number of tutorials and examples for a generic model instance based on Dantzig's transport problem.

Documentation

Complete documentation of the ixmp API is available for current and past versions at: https://docs.messageix.org/ixmp/. This includes:

• Installation. Most users will have ixmp installed automatically as a dependency when installing MESSAGEix. To install ixmp from source code, or to use ixmp from R, see ‘Installation’ or INSTALL.rst.
• Tutorials. Introductory tutorials are provided in both Python and R; see ‘Tutorials’ or tutorial/README.rst.
• How to cite ixmp when using it in published scientific work. See ‘User guidelines and notice’ or NOTICE.rst.
• Changelog and migration notes for each new release: see “What's New”.

Other forms of documentation:

• The online documentation is built automatically from the contents of the ixmp GitHub repository.
• For documentation of a specific release, e.g. v3.2.0, use the chooser in the bottom sidebar.
• For offline use, the documentation can be built from the source code. See doc/README.rst for further details.
• The MESSAGEix API documentation links to the ixmp documentation in many places, for convenience.

License

Copyright © 2017–2024 IIASA Energy, Climate, and Environment (ECE) program

ixmp is licensed under the Apache License, Version 2.0 (the "License"); you may not use the files in this repository except in compliance with the License. You may obtain a copy of the License in LICENSE or at https://www.apache.org/licenses/LICENSE-2.0.

ixmp's Issues

Include simple-install for *nix users

It would be nice to have an install script for *nix users similar to the one for the message_ix repository, but for users who only want to install ixmp as a stand-alone project.

Switch to R package ``Rixmp``

The R package should replace the current ixmp/ixmp.R implementation.

The new version should also implement the separation between the back-end and MESSAGE-specific settings similar to the Python ixmp package and the message_ix package in the "message_ix" repository.

Clone across platforms (database instances) not operational

The current implementation of Scenario.clone() does not actually save the scenario to another database instance, only the Python-class internal platform argument is changed.

Oracle error when trying to update timeseries

If timeseries have already been uploaded e.g. for 1990 to 2110, and the same timeseries are then updated e.g. for 2020-2100 then the following error occurs:

2018-07-06 09:54:39,554 ERROR at.ac.iiasa.ixmp.database.DbDAO:766 - Error writing the timeseries data to the IXMP database!
java.sql.BatchUpdateException: ORA-00001: unique constraint (ENE_IXMP.SYS_C0065938) violated

    at oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:12296)
    at oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:246)
    at at.ac.iiasa.ixmp.database.OracleDAO.saveTimeseriesToDB(OracleDAO.java:187)
    at at.ac.iiasa.ixmp.objects.Scenario.commit(Scenario.java:2014)
    at at.ac.iiasa.ixmp.objects.Scenario.commit(Scenario.java:1906)
    at at.ac.iiasa.ixmp.objects.MsgScenario.commit(MsgScenario.java:1316)
    at at.ac.iiasa.ixmp.objects.MsgScenario.commit(MsgScenario.java:1301)

error in read_sol_from_gdx

Another issue which I have encountered, but which I havent been able to replicate:

When a scenario has been run, an error occurs where the scenario is not checked when trying to upload the solution from the gdx file.

--- Restarting execution
--- MESSAGE-MACRO_run.gms(4794) 402 Mb
--- Reading solution for model MESSAGE_MACRO
--- MESSAGE-MACRO_run.gms(4509) 403 Mb
    +++ Convergence criteria satisfied after 8 iterations +++
--- MESSAGE-MACRO_run.gms(4941) 406 Mb
--- GDX File /h/u045sumida/fricko/.local/lib/python3.7/site-packages/message_ix/model/output/MsgOutput_MESSAGEix-GLOBIOM_BvR_OFR_noBCCS.gdx
--- MESSAGE-MACRO_run.gms(4949) 406 Mb
    +++ End of MESSAGEix-MACRO run - have a nice day! +++
*** Status: Normal completion
--- Job MESSAGE-MACRO_run.gms Stop 11/20/18 16:39:38 elapsed 0:31:26.838
2018-11-20 16:39:39,455 DEBUG at.ac.iiasa.ixmp.database.DbDAO:1744 - loading elements for set 'map_temporal_hierarchy' from database...
2018-11-20 16:39:39,461 DEBUG at.ac.iiasa.ixmp.database.DbDAO:1744 - loading elements for parameter 'duration_time' from database...
2018-11-20 16:39:39,506  INFO at.ac.iiasa.ixmp.objects.Scenario:1346 - this Scenario does not contain any elements in a Variable or Equation
--- Warning: The GAMS version [25.1.2] differs from the API version [24.8.3].
Model: MESSAGEix-GLOBIOM_BvR_OFR
Scenario: noBCCS
kwargs: {'run': ['all'], 'exclusion': {'exclusion_type': 'noBECCS'}, 'base': {'model': 'MESSAGEix-GLOBIOM_BvR_OFR', 'scenario': 'NPi2020_Net0'}}
default_kwargs: {'solve': {'model': 'MESSAGE-MACRO'}, 'macro': {'copy_demand': 'True'}, 'reporting': 'True', 'clone': {'first_model_year': 2030}, 'upload': 'True'}
Traceback (most recent call last):
  File "runMESSAGEix.py", line 155, in <module>
    run(model, scenario, kwargs, default_kwargs)
  File "runMESSAGEix.py", line 91, in run
    scen.solve(**sol_kwa)
  File "/h/u045sumida/fricko/.local/lib/python3.7/site-packages/message_ix/core.py", line 289, in solve
    return super(Scenario, self).solve(model=model, **kwargs)
  File "/h/u045sumida/fricko/.local/lib/python3.7/site-packages/ixmp/core.py", line 1006, in solve
    var_list, equ_list, check_solution)
  File "/h/u045sumida/fricko/.local/lib/python3.7/site-packages/ixmp/core.py", line 937, in read_sol_from_gdx
    check_solution)
jpype._jexception.IxExceptionPyRaisable: at.ac.iiasa.ixmp.exceptions.IxException: This Scenario is not checked out - no changes to discard!

filters dictionary values defined as string throw error

When using a filter to retrieve par/var, the values must be defined as lists. i.e. filters={'key': [value]}.
This should be changed so that the values can be passed as strings and are converted to lists

Question: parallel run

Is it possible run message parallel with python multiprocess? Or what is the best way to run message parallel?

jvm args fail if available ram is too small.

In short, line 40 in core.py will return 0 if it is given a value less than 1. We should have a common practice for this use case. Solving this will likely also solve #29

Remove Scenario from DB

Hi,
Is it possible to delete a Scenario from the database? Because the HyperSQL database becomes very large after a sensitivity analysis with 1000 runs.

Drop support for Python 2.7

See: https://python3statement.org. There will be no Python 2.8.

Parallel to iiasa/message_ix#177, and since ixmp 0.2 has been released, we should drop support for Python 2.7. This would include:

• Disabling CI for Python 2.7
• Removing code marked with # py2 compat or similar.

scenario.clone() returns the default scenario

I observe an unexpected behavior when I clone an ixmp Scenario class.
For example, I load a Scenario that has a version number of 42 and clone this via: scenario.clone()
Then, when I check the version of the cloned scenario (via scenario.version), it is 20, while I expect that it would be 43. But when I look at the database scenario list (via mp.scenario_list()), I see that the cloned scenario has got a version of 43. More importantly, the cloned scenario is not actually cloned from scenario version 42! It is indeed my scenario version 20 loaded. When I look further to the database, I see that this scenario version 20 is the default scenario.
So, in short, it seems when I clone scenario version 42, it will be saved as 43 in the database, but scenario.clone() method returns my default scenario instead (and not scenario version 43). I hope I'm not missing something out.
ixmp.version: 0.1.3.post.dev21

Support databases other than JDBC

Are there plans to support open-source relational database management systems (e.g. MySQL)?

Address thread safety of read-only operations

Per our discussion in #97, read-only operations (e.g., just reading scenario data from a database) is at the moment not thread safe. In essence, there is a check_out() occuring in order to check whether the current model schema is at the expected version. We should address this either by guaranteeing thread saftey of read-only ops or by offering a thread-safe version (through key-word argument or something else).

Move `clone()` with kwarg `first_model_year` to `message_ix.Scenario`

Calling the function clone() with a kwarg first_model_year on a non-MESSAGE-scheme Scenario will raise an error message, because it only applies there. Therefore, this option should be moved to the message_ix.Scenario class.

Support non-GAMS models

Scenario.solve() ends with the following lines:

ixmp/ixmp/core.py

Lines 1234 to 1238 in aa7b2fe

	# write to gdx, execture GAMS, read solution from gdx
	self.to_gdx(ipth, ingdx)
	run_gams(model_file, args)
	self.read_sol_from_gdx(opth, outgdx, comment,
	var_list, equ_list, check_solution)

In order to support other, non-GAMS models, the following changes could be made:

1. Extend and generalize the code in ixmp.model_settings, adding:
   • a new ixmp.model.Model abstract class,
   • an ixmp.model.GAMSModel subclass.
2. Change Scenario.to_gdx(…) to GAMSModel.write_data(scenario, …).
3. Change run_gams(…) to GAMSModel.run(scenario, …).
4. Change Scenario.read_sol_from_gdx(…) to GAMSModel.read_solution(scenario, …).

After this, users would be able to run non-GAMS models by subclassing ixmp.Model() and implementing the methods write_data(), run(), and read_sol_from_gdx().

As identified here in the discussion of #108.

Integrate `nodes` from MESSAGE-scheme `Scenario` set and the `timeseries` data

At the moment, defining a new element in the set node of a MESSAGE-scheme ixmp.Scenario does not add an entry to the table IAMC_NODES, meaning that one cannot directly use that node in the add_timeseries() command.

Remove solution error message

The exception error for removing solution (for example, when trying to check out a scenario that contains a solution) suggests using the command 'removeSolution()'. However, the actual command doing this task now is 'remove_sol()'. It seems the text in the error message needs to be updated.

Update conda deps to include xlrd and xlsxwriter

And also remove these from existing CIs.

test_utils.py pollutes source directory

ixmp/tests/test_utils.py

Lines 9 to 20 in fb195fd

	def make_obs(fname, exp, **kwargs):
	utils.pd_write(exp, fname, index=False)
	obs = utils.pd_read(fname, **kwargs)
	os.remove(fname)
	return obs
	def test_pd_io_csv():
	fname = 'test.csv'
	exp = pd.DataFrame({'a': [0, 1], 'b': [2, 3]})
	obs = make_obs(fname, exp)
	pdt.assert_frame_equal(obs, exp)

Calling make_obs() with 'temp.csv' (a filename with no directory component) causes the file to be created in the directory where tests are invoked. Instead, the pytest tmp_path fixture should be used.

Default logLevel DEBUG or INFO?

Inside the PyCharm Console, I always get messages at the DEBUG level:

However, when I look to ixmp/ixmp/utils.py is this for me not clear, because the level is only set there to INFO:

def logger():
    """Access global logger"""
    global _LOGGER
    if _LOGGER is None:
        logging.basicConfig()
        _LOGGER = logging.getLogger()
        _LOGGER.setLevel('INFO')
    return _LOGGER

Is there another possibility to change the logging level?

TimeSeries.commit() overwrites TimeSeries.version() method with a variable

As noted in #108.

Per this comment, it should be preferred to call self._jobj.getVersion() every time.

Return dataframe of mapping sets, parameters etc. with ordered columns

Currently, calling Scenario.par() returns a pd.DataFrame with columns in no particular order.

This should be changed to using the order of Scenario.idx_names() + ['value', 'unit'] (for parameters) and ['lvl', 'mrg'] for variables and equations, respectively.

`remove_solution` does not remove timeseries with `meta=0`

This basically keeps all reported/post process timeseries in the clone/removed solution scenario. Thus if reporting is run again, stale values could be still included

Disable console output

Hi,
is there a easy opition to disable all output from the console?

for e.g.
... DEBUG at.ac.iiasa.ixmp.database.DbDAO:1695 ...
--- Job MESSAGE_run.gms ...

errors when running concurrently

When trying to load a given scenario in more than one thread, we see the following issue:

foo.py contains:

import ixmp

p = ixmp.Platform()
p.Scenario('MESSAGEix-GLOBIOM_BvR_OFR1', 'NPi2020_Net0')

Then we run:

In [1]: import multiprocessing, subprocess

In [2]: n=2; pool=multiprocessing.Pool(n); pool.map(subprocess.call, [('python', 'foo.py')] * n)
INFO:root:launching ixmp.Platform using config file at '/home/gidden/work/iiasa/message/properties_files/enedb.properties'
INFO:root:launching ixmp.Platform using config file at '/home/gidden/work/iiasa/message/properties_files/enedb.properties'
Nov 21, 2018 9:39:29 AM org.flywaydb.core.internal.util.VersionPrinter printVersion
INFO: Flyway 4.2.0 by Boxfuse
Nov 21, 2018 9:39:29 AM org.flywaydb.core.internal.util.VersionPrinter printVersion
INFO: Flyway 4.2.0 by Boxfuse
Nov 21, 2018 9:39:30 AM org.flywaydb.core.internal.dbsupport.DbSupportFactory createDbSupport
INFO: Database: jdbc:oracle:thin:@gp3.iiasa.ac.at:1521:GP3 (Oracle 12.1)
Nov 21, 2018 9:39:30 AM org.flywaydb.core.internal.dbsupport.DbSupportFactory createDbSupport
INFO: Database: jdbc:oracle:thin:@gp3.iiasa.ac.at:1521:GP3 (Oracle 12.1)
Nov 21, 2018 9:39:30 AM org.flywaydb.core.internal.command.DbValidate validate
INFO: Successfully validated 11 migrations (execution time 00:00.030s)
Nov 21, 2018 9:39:30 AM org.flywaydb.core.internal.command.DbValidate validate
INFO: Successfully validated 11 migrations (execution time 00:00.031s)
2018-11-21 09:39:31,049  INFO at.ac.iiasa.ixmp.Platform:111 - Welcome to the IX modeling platform!
2018-11-21 09:39:31,050  INFO at.ac.iiasa.ixmp.Platform:112 -  connected to database 'ixENE-DB@GP3'...
/home/gidden/.local/lib/python3.5/site-packages/ixmp/core.py:171: UserWarning: The constructor `mp.Scenario()` is deprecated, please use `ixmp.Scenario(mp, ...)`
  warnings.warn('The constructor `mp.Scenario()` is deprecated, '
2018-11-21 09:39:31,112  INFO at.ac.iiasa.ixmp.objects.Scenario:218 - loading Scenario 'MESSAGEix-GLOBIOM_BvR_OFR1|NPi2020_Net0' by version id (version: 1, runid: 18489)...
2018-11-21 09:39:31,113  INFO at.ac.iiasa.ixmp.Platform:111 - Welcome to the IX modeling platform!
2018-11-21 09:39:31,115  INFO at.ac.iiasa.ixmp.Platform:112 -  connected to database 'ixENE-DB@GP3'...
/home/gidden/.local/lib/python3.5/site-packages/ixmp/core.py:171: UserWarning: The constructor `mp.Scenario()` is deprecated, please use `ixmp.Scenario(mp, ...)`
  warnings.warn('The constructor `mp.Scenario()` is deprecated, '
2018-11-21 09:39:31,179  INFO at.ac.iiasa.ixmp.objects.Scenario:218 - loading Scenario 'MESSAGEix-GLOBIOM_BvR_OFR1|NPi2020_Net0' by version id (version: 1, runid: 18489)...
2018-11-21 09:39:31,541  INFO at.ac.iiasa.ixmp.objects.Scenario:241 - done loading Scenario from the database!
2018-11-21 09:39:31,571  INFO at.ac.iiasa.ixmp.objects.MsgScenario:457 -  spatial and temporal disaggregation maps assignment
2018-11-21 09:39:31,590  INFO at.ac.iiasa.ixmp.objects.MsgScenario:573 -  period mapping assignment
2018-11-21 09:39:31,612  INFO at.ac.iiasa.ixmp.objects.Scenario:241 - done loading Scenario from the database!
2018-11-21 09:39:31,613  INFO at.ac.iiasa.ixmp.objects.MsgScenario:599 -  duration period assignment
2018-11-21 09:39:31,636  INFO at.ac.iiasa.ixmp.objects.Scenario:1826 - committing changes of Scenario 'MESSAGEix-GLOBIOM_BvR_OFR1|NPi2020_Net0' to the database (runid: 18489)...
Traceback (most recent call last):
  File "foo.py", line 4, in <module>
    p.Scenario('MESSAGEix-GLOBIOM_BvR_OFR1', 'NPi2020_Net0')
  File "/home/gidden/.local/lib/python3.5/site-packages/ixmp/core.py", line 174, in Scenario
    return Scenario(self, model, scen, version, scheme, annotation, cache)
  File "/home/gidden/.local/lib/python3.5/site-packages/ixmp/core.py", line 460, in __init__
    self._jobj = mp._jobj.getScenario(model, scenario)
jpype._jexception.IxExceptionPyRaisable: at.ac.iiasa.ixmp.exceptions.IxException: This Scenario is currently locked by user gidden
2018-11-21 09:39:32,907  INFO at.ac.iiasa.ixmp.objects.Scenario:601 - done updating Scenario 'MESSAGEix-GLOBIOM_BvR_OFR1|NPi2020_Net0' to the database (runid: 18489, version: 1)!
2018-11-21 09:39:32,912  INFO at.ac.iiasa.ixmp.objects.MsgScenario:457 -  spatial and temporal disaggregation maps assignment
2018-11-21 09:39:32,912  INFO at.ac.iiasa.ixmp.objects.MsgScenario:573 -  period mapping assignment
2018-11-21 09:39:32,913  INFO at.ac.iiasa.ixmp.objects.MsgScenario:599 -  duration period assignment
/home/gidden/.local/lib/python3.5/site-packages/ixmp/core.py:468: UserWarning: Using `ixmp.Scenario` for MESSAGE-scheme scenarios is deprecated, please use `message_ix.Scenario`
  warnings.warn('Using `ixmp.Scenario` for MESSAGE-scheme scenarios '
Out[2]: [0, 1]

Configuration & data are not in expected (XDG) locations

Several related issues here that could be fixed together.

1. ixmp-config places a file at ~/.local/ixmp/config.json. This does not conform to the XDG base directory specification. The file should be at either:
   • ~/.local/share/ixmp/config.json — i.e. using $XDG_DATA_HOME or its default.
   • ~/.config/ixmp.json or ~/.config/ixmp/config.json — i.e. using $XDG_CONFIG_HOME or its default.
2. Likewise the localdb directory created with ixmp.Platform(dbtype='HSQLDB') is incorrectly at ~/.local/ixmp/localdb instead of ~/.local/share/ixmp/localdb.
3. ixmp.config.get() raises an exception if the configuration file is missing. Sensible defaults should be used instead.

problem with init_par if index set is more than 1 character long

it is not possible to initialize a new parameter if the index set is not a single letter set, taking example from the transport tutorial:

scen.init_set("i")
scen.add_set("i", ["seattle", "san-diego"])
scen.init_set("j")
scen.add_set("j", ["new-york", "chicago", "topeka"])
scen.init_par("a", idx_sets="i")
scen.add_par("a", "seattle", 350, "cases")
scen.add_par("a", "san-diego", 600, "cases")
scen.init_par("b", idx_sets="j")
b_data = [
{'j': "new-york", 'value': 325, 'unit': "cases"},
{'j': "chicago", 'value': 300, 'unit': "cases"},
{'j': "topeka", 'value': 275, 'unit': "cases"}
]
b = pd.DataFrame(b_data)
scen.add_par("b", b)

works properly, but when the set is a string with mode than one character it does not.

scen.init_set("iii")
scen.add_set("iii", ["seattle", "san-diego"])
scen.init_set("jjj")
scen.add_set("jjj", ["new-york", "chicago", "topeka"])
scen.set('iii')
scen.init_par("a", idx_sets="iii")
scen.add_par("a", "seattle", 350, "cases")
scen.add_par("a", "san-diego", 600, "cases")
scen.init_par("b", idx_sets="jjj")
b_data = [
{'j': "new-york", 'value': 325, 'unit': "cases"},
{'j': "chicago", 'value': 300, 'unit': "cases"},
{'j': "topeka", 'value': 275, 'unit': "cases"}
]
b = pd.DataFrame(b_data)
scen.add_par("b", b)

the tutorial with this issue is attached

py_transport.zip

Execute tutorial notebooks during install-and-test in temporary db

Currently, when installing and running unit tests, the tutorial notebooks are executed using the local default database. Over time, this db will become clogged by thousands of transport problems. It would be better to use temporary database instead.

This could be achieved by replacing the line
"mp = ix.Platform(dbtype='HSQLDB')"
in any notebook by:
"mp = ix.Platform(dbprops=os.path.expanduser(os.path.join('~', '.local', 'ixmp', 'unittest')), dbtype='HSQLDB')".

Where all files starting with unittest in ~\.local\ixmp will be deleted after the last unit test.

Note that changing this in the notebook itself would defeat the purpose...

Add feature to unlock `Timeseries`/`Scenarios` in a database

When checking out an ixmp.Scenario and not committing or discarding (or running into an error), the run-id is locked in the database instance. We need an elegant way to 'unlock' the run-id.

Follow-up: this should only be available to admins on central/shared database instance?

Script installation error on Windows when executing install.bat

If Anaconda has been installed as a system-wide application as opposed to a user-specific application in Windows, then installing ixmp (and message_ix) via install.bat results in an error related to permissions. The issue arises when installing scripts. See example below:

Installing import-timeseries-script.py script to C:\ProgramData\Anaconda3\Scripts error: [WinError 5] Access is denied: 'C:\\ProgramData\\Anaconda3\\Scripts\\import-timeseries-script.py'

In order to circumvent this behavior, the user must execute the install.bat with admin privelages.

Oracle Java change license - OpenJDK

We have received a notification from the IT department that Java will be charged to universities in the next few days. We should switch to OpenJDK if possible. Will ixmp work with it?

Issue when adding bound_emission parameter

Dear all,
I tried to add a cap on the GHGs emissions by adding the the bound_emission parameter:
bound_emission = pd.DataFrame({
'node': country,
'type_emission': 'GHGs',
'type_tec': 'all',
'type_year': horizon[1:],
'value': emission_cap_year,
'unit': 'kg/kWa',
})
scenario.add_par('bound_emission', bound_emission)
However I get the following error message:

"at.ac.iiasa.ixmp.exceptions.IxExceptionPyRaisable: at.ac.iiasa.ixmp.exceptions.IxException: The index set 'type_year' does not have an element '2020'!"

Best
jasper

Error checking for conflicts

Hi there,
I tried to install ixmp from source and got the error below. Prior to this attempt, I uninstalled ixmp from conda. Any ideas of how to fix this problem? Thanks!

H:\Things in C Drive\IIASA\Energy Group\MESSAGE\GitHub\ixmp>pip install .
Processing h:\things in c drive\iiasa\energy group\message\github\ixmp
Requirement already satisfied: JPype1>=0.6.2 in c:\users\hunt\appdata\local\cont
inuum\anaconda3\lib\site-packages (from ixmp==0.1.3.post0.dev23) (0.6.3)
Requirement already satisfied: pandas in c:\users\hunt\appdata\local\continuum\a
naconda3\lib\site-packages (from ixmp==0.1.3.post0.dev23) (0.23.4)
Requirement already satisfied: xlsxwriter in c:\users\hunt\appdata\local\continu
um\anaconda3\lib\site-packages (from ixmp==0.1.3.post0.dev23) (0.9.6)
Requirement already satisfied: xlrd in c:\users\hunt\appdata\local\continuum\ana
conda3\lib\site-packages (from ixmp==0.1.3.post0.dev23) (1.0.0)
Requirement already satisfied: numpy>=1.9.0 in c:\users\hunt\appdata\local\conti
nuum\anaconda3\lib\site-packages (from pandas->ixmp==0.1.3.post0.dev23) (1.15.4)

Requirement already satisfied: pytz>=2011k in c:\users\hunt\appdata\local\contin
uum\anaconda3\lib\site-packages (from pandas->ixmp==0.1.3.post0.dev23) (2016.10)

Requirement already satisfied: python-dateutil>=2.5.0 in c:\users\hunt\appdata\l
ocal\continuum\anaconda3\lib\site-packages (from pandas->ixmp==0.1.3.post0.dev23
) (2.6.0)
Requirement already satisfied: six>=1.5 in c:\users\hunt\appdata\local\continuum
\anaconda3\lib\site-packages (from python-dateutil>=2.5.0->pandas->ixmp==0.1.3.p
ost0.dev23) (1.10.0)
Building wheels for collected packages: ixmp
Building wheel for ixmp (setup.py) ... done
Stored in directory: C:\Users\hunt\AppData\Local\Temp\pip-ephem-wheel-cache-0h
whhx3j\wheels\fd\f1\5e\914b1324ef66c00c49912bcdc21453232d0e98f4b3ad9cd13c
Successfully built ixmp
Error checking for conflicts.
Traceback (most recent call last):
File "c:\users\hunt\appdata\local\continuum\anaconda3\lib\site-packages\pip_v
endor\pkg_resources_init_.py", line 2897, in _dep_map
return self._dep_map
File "c:\users\hunt\appdata\local\continuum\anaconda3\lib\site-packages\pip_v
endor\pkg_resources_init.py", line 2691, in getattr
raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "c:\users\hunt\appdata\local\continuum\anaconda3\lib\site-packages\pip_v
endor\pkg_resources_init_.py", line 2888, in _parsed_pkg_info
return self.pkg_info
File "c:\users\hunt\appdata\local\continuum\anaconda3\lib\site-packages\pip_v
endor\pkg_resources_init.py", line 2691, in getattr
raise AttributeError(attr)
AttributeError: _pkg_info

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "c:\users\hunt\appdata\local\continuum\anaconda3\lib\site-packages\pip_i
nternal\commands\install.py", line 503, in _warn_about_conflicts
package_set, dep_info = check_install_conflicts(to_install)
File "c:\users\hunt\appdata\local\continuum\anaconda3\lib\site-packages\pip_i
nternal\operations\check.py", line 108, in check_install_conflicts
package_set, _ = create_package_set_from_installed()
File "c:\users\hunt\appdata\local\continuum\anaconda3\lib\site-packages\pip_i
nternal\operations\check.py", line 47, in create_package_set_from_installed
package_set[name] = PackageDetails(dist.version, dist.requires())
File "c:\users\hunt\appdata\local\continuum\anaconda3\lib\site-packages\pip_v
endor\pkg_resources_init.py", line 2635, in requires
dm = self.dep_map
File "c:\users\hunt\appdata\local\continuum\anaconda3\lib\site-packages\pip_v
endor\pkg_resources_init.py", line 2899, in _dep_map
self.__dep_map = self.compute_dependencies()
File "c:\users\hunt\appdata\local\continuum\anaconda3\lib\site-packages\pip_v
endor\pkg_resources_init.py", line 2908, in _compute_dependencies
for req in self.parsed_pkg_info.get_all('Requires-Dist') or []:
File "c:\users\hunt\appdata\local\continuum\anaconda3\lib\site-packages\pip_v
endor\pkg_resources_init.py", line 2890, in parsed_pkg_info
metadata = self.get_metadata(self.PKG_INFO)
File "c:\users\hunt\appdata\local\continuum\anaconda3\lib\site-packages\pip_v
endor\pkg_resources_init.py", line 1410, in get_metadata
value = self._get(self.fn(self.egg_info, name))
File "c:\users\hunt\appdata\local\continuum\anaconda3\lib\site-packages\pip_v
endor\pkg_resources_init.py", line 1522, in _get
with open(path, 'rb') as stream:
FileNotFoundError: [Errno 2] No such file or directory: 'c:\users\hunt\appdat
a\roaming\python\python36\site-packages\ixmp-0+unknown.dist-info\METADATA'

Installing collected packages: ixmp
Found existing installation: ixmp 0+unknown
Could not install packages due to an EnvironmentError: [Errno 2] No such file or
directory: 'c:\users\hunt\appdata\roaming\python\python36\site-packages
\ixmp-0+unknown.dist-info\RECORD'

You are using pip version 19.0.3, however version 19.1.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' comm
and.

H:\Things in C Drive\IIASA\Energy Group\MESSAGE\GitHub\ixmp>

Incorrect semantics in Platform, TimeSeries, Scenario

Copied from #108.

• Some of these would require backwards-incompatible renaming of methods (e.g. items 3, 6) and would need to occur in the next major version (1.0)
• Some could be implemented right away (e.g. item 7; or items 4/5 with an alias to the old name, to be deprecated).

Methods in the wrong place

For a clean class hierarchy, since TimeSeries is the parent of Scenario, then TimeSeries methods and code should not depend on things implemented in Scenario.

1. TimeSeries.checkout() (1) calls Scenario.has_solution() and (2) raises an exception with the text "This Scenario…"—but it is the parent class of Scenario. Done in #270.
2. TimeSeries.add_timeseries() docstring references "MESSAGE-Scheme scenarios". Fixed with #120.
3. TimeSeries.timeseries() takes kwargs regions, units, years. AFAICT e.g. 'years' is not necessarily a set in an ixmp model; only in MESSAGEix. TimeSeries data has a 'year' dimension by default; the existence (or not) of set(s) that correspond to years in a model/Scenario is distinct.

Pythonic semantics

4. Platform.scenarios_list() → list_scenarios(). Methods should be named "[verb] [noun]"; only attributes/properties should have "[noun]" names.
5. TimeSeries.add_timeseries() → add_data(). Repeating the class name is confusing; this makes the user imagine doing ts1, ts2 = TimeSeries(), TimeSeries(); ts1.add_timeseries(ts2).
6. Scenario.add_set() is a misnomer: this method "adds elements to an existing set"; it does not "add a new set". On the other hand, Scenario.init_set() actually "adds a new set" to the Scenario definition. I'd propose:
   • Scenario.add_set() = add a new set.
   • Scenario.add_set_elements() = add elements to an existing set.
7. Scenario.item(), .element() → rename to ._item(), ._element(). The docstrings state these are "internal function"s; Python convention is to indicate this with leading _ on the name. Removed with the implementation of the Backend API.
8. Scenario.add_par() will operate fine if given only key, and not val. In this case, the values are actually contained in key → rename this argument to key_or_val.

Ease-of-use:

9. Platform.__del__() should invoke close_db() when necessary. Done in #298.
10. Scenario.clone() should raise a warning if platform==self.platform; the user might think they are cloning elsewhere, but have made a coding error (see also #101, #109).
11. Scenario.solve(): the 'model' keyword is required, but it could be inferred.

Failure to start jvm on computers with little RAM

The default assumption of 1GB RAM for the JVM instance seems to cause problems on some old PCs and laptops. This should be replaced by a more modest default value.

Also, the shipped libraries for the Java API are 64-bit only - this should be stated clearly in the system requirements.

IxException not handled correctly in Scenario.remove_solution()

When trying to modify a scenarios with a solution, it throws an error saying:

at.ac.iiasa.ixmp.exceptions.IxExceptionPyRaisable: at.ac.iiasa.ixmp.exceptions.IxException: This Scenario has elements in a variable or equation! Use 'removeSolution()' or clone(False)'before editing!

removeSolution() is deprecated. Actual name is remove_sol().

add deprecation warning to `ixmp.Scenario(..., scheme='MESSAGE')`

Using the class ixmp.Scenario for MESSAGE-scheme scenarios is deprecated and should be switched to the MESSAGE-specific sub-class message_ix.Scenario.

use versioneer for standard versioning

Previously, we tried to import the package to get the version number in setup.py. This broke our conda build. It appears that versioneer can fix this. See, e.g., xarray.

Seeing non-zero ACT value for a technology without any commodity linked to it

I have a strange observation, which I want to clarify.

By mistake, I didn't put any 'renewable_potential' values for the historical hydro tec I defined ("hydro_hist") for the entire time horizon, which means its input commodity ("hydro_c_hist") dedicated to the tec has no ground. And input and output parameters were defined only from 1965 and on. But there was historical_new_capacity starting from 1950.

Then in the result, I found that ACT has a value for 2020 for the hydro capacity of year_vtg 1960. (lifetime 60 years). I guess there is a loophole where ACT can still show up when all the input/output/renewable_potential are not defined.
Shouldn't it still give zero ACT if there is no potential defined for modeling years?

Document expected usage of `default` characteristic of scenarios, especially in use with `clone()`

Per our discussion in #97, it would be useful to have a documentation page that covers aspects of how versioning is done on model/scenario data. There are two main topics I see that could be covered at the moment.

First is the application of the default characteristic. How is this updated? What happens if there is no default version? How does this interact with clone()?

Second is the possible results from clone() with various arguments. E.g., what if I only give model, only scenario or both? What if model and scenario are the same as the copy-ee?

This could be pretty simple page enumerating all possible procedures with expected outcomes.

Extend varchar for scenario names

Currently, the scenario name length is restricted to 64 characters and some fun combinatorics for sensitivity analysis may result in longer names.

Add unit tests for R API

Now that we have implemented the rixmp package (see PR #19), we should add unit tests to make sure that future changes do not cause problems with existing features.

pandas.*.ix is depreciated

/home/khaeru/vc/iiasa/ixmp/ixmp/core.py:730: DeprecationWarning:                                                        
.ix is deprecated. Please use                                                                                             
.loc for label based indexing or                                                                                         
.iloc for positional indexing                                                                                            
                                                                                                                      
See the documentation here:                                                                                       
http://pandas.pydata.org/pandas-docs/stable/indexing.html#ix-indexer-is-deprecated                                     
  jPar.addElement(to_jlist(key.ix[i], idx_names),

Update MESSAGEix scheme defaults in the ixmp core

To merge the PRs from the MESSAGEix hackathon, a number of changes are required in the MESSAGE-scheme implementation on the Java side. This should also bump the version number of the MESSAGE scheme settings.

Scenario.var_list() and .var() not covering all variables

I find that I cannot read the contents of some new variables (e.g. REN) using the two function calls.

Using `iloc[]` fails if pandas auto-creates index from `ints`

PR #12 causes the message_ix tutorials to fail, because the pandas.DataFrame auto-sets a column of ints as index.

Feature: read_sql_query()

Hi @gidden @danielhuppmann,

I have a short question. Is there a possibility to read SQL query into a DataFrame from the Platform? Similar like the pandas command read_sql_query().

allow for caching subsets of data

For reporting subsets of data (e.g., only one region), we will not want to pull all data into memory. However, the caching system currently only supports that. Key lines are:

    def element(self, ix_type, name, filters=None, cache=None):
        """internal function to retrieve a dataframe of item elements"""
        item = self.item(ix_type, name)

        # if dataframe in python cache, retrieve from there                                                                                                                       
        if name in self._pycache:
            return filtered(self._pycache[(ix_type, name)], filters)

        # if no cache, retrieve from Java with filters                                                                                                                            
        if filters is not None and not self._cache:
            return _get_ele_list(item, filters, **self._java_kwargs[ix_type])

        # otherwise, retrieve from Java and keep in python cache                                                                                                                  
        df = _get_ele_list(item, None, **self._java_kwargs[ix_type])

        # save if using memcache                                                                                                                                                  
        if self._cache:
            self._pycache[(ix_type, name)] = df

        return filtered(df, filters)

Here we would need to update how it works if cache is True and filters exist. Probably very low priority.

Solve error under Windows 10

I get the following error when I try to solve the model with: scen.solve(model='MESSAGE):

subprocess.CalledProcessError: Command 'gams "MESSAGE.gms" --in="MESSAGE_in.gdx" --out="MESSAGE_out.gdx" LogOption=4' returned non-zero exit status 6.

model_file in ixmp/core is gives the following string MESSAGE.gms. In a former version model was a path to MESSAGE_run.gms .

Current version:
model = os.path.join(ix.default_paths.MODEL_DIR, '{}_run.gms'.format(model))

Older version:
model_file= model_file or config.model_file.format(model=model)

I am not sure if I made a error during the installation process or this is a general error.

Performance: Reporting upload of timeseries

Uploading of the full reporting as timeseries takes close up to 40 minutes if uploading values from 1990-2110. It seems that despite passing a single dataframe containing all the data, the timeseries are processed by individual variables.

Here the log for uploading values from 2020-2110 - time required: 38 minutes

Starting to upload timeseries
2018-07-06 09:16:42,076 DEBUG at.ac.iiasa.ixmp.objects.Scenario:289 - loaded all timeseries data for 'Resource|Extraction' (16)...
2018-07-06 09:16:47,111 DEBUG at.ac.iiasa.ixmp.objects.Scenario:289 - loaded all timeseries data for 'Resource|Extraction|Coal' (17)...
...
...
2018-07-06 09:54:34,186 DEBUG at.ac.iiasa.ixmp.objects.Scenario:289 - loaded all timeseries data for 'Water Withdrawal|Liquids|Gas|w/o CCS' (1322)...
2018-07-06 09:54:35,598 DEBUG at.ac.iiasa.ixmp.objects.Scenario:289 - loaded all timeseries data for 'Water Withdrawal|Liquids|Oil' (1323)...
Finished uploading timeseries
<<<