jgcri / gcamreader

Python package for reading GCAM output databases

gcamreader: A python package for importing GCAM data

gcamreader provides functions for reading data from output databases produced by GCAM

Purpose

gcamreader was created to:

• Use XML queries to extract data into a Pandas DataFrame from a GCAM XML database

• Integrate GCAM with other Python packages

Install gcamreader

pip install gcamreader

gcamreader's Issues

Switch from `unittest` to `pytest`

Switch testing suite from unittest to pytest. I like this because of the simple usage of assert with a condition following for testing rather then using assert* where * is wild to be bound by only what unittest offers in the way of tests. Lots of other benefits too and super stable. Here are some examples...https://docs.pytest.org/en/latest/example/index.html

Fix nested CDATA bug

Remote database access will fail if the query contains a CDATA section. There's a fix in JGCRI/rgcam#29. Once that's been validated we need to port it over here.

Problem about duplicate in results when query "elec gen by gen tech"

When querying "elec gen by gen tech", the results will be duplicated and do not match the results of the model interface. I find the following problems have appeared in both wind and solar technology queries. I don't know how to avoid the following problems:

Here is the queries XML：

<?xml version="1.0" encoding="UTF-8"?>
<queries>
     <aQuery>
     <all-regions/>
               <supplyDemandQuery title="elec gen by gen tech">
                    <axis1 name="technology">technology</axis1>
                    <axis2 name="Year">physical-output[@vintage]</axis2>
                    <xPath buildList="true" dataName="output" group="false" sumAll="false">*[@type='sector' (:collapse:) and
               (@name='electricity' or @name='elect_td_bld' or @name='industrial energy use')]/
               *[@type='subsector' and (@name='wind')]/*[@type='technology' and not (@name='electricity' or @name='elect_td_bld')]/
               *[@type='output' and (@name='electricity' or @name='elect_td_bld')]/
               physical-output/node()</xPath>
                    <comments/>
                </supplyDemandQuery>
     </aQuery>
</queries>

Add basic documentation

At the very least, the landing page should have installation instructions and a tour of the main functions.

XML Query structure

Is there a doc containing schema for XML queries or a guide on how to write them, for interacting with the GCAM basex db?

The sample_queries.xml given in example runs fine, but unclear why something like Main_queries.xml doesn't have same behaviour? (or if it's even the same type of query?)

Looked in other repos & docs but haven't found anything concrete - can anyone point me in the right direction? Thanks!

Add license and disclaimer

Since we are using GCAM jars in this, I think this should adopt the GCAM license and disclaimer. Please add them.

Add functionality tests

Add tests to evaluate high-level model functionality.

"CalledProcessError" when query "LDV energy by primary fuel"

when querying "LDV energy by primary fuel" with

conn.runQuery([i for i in queries if i.title == "LDV energy by primary fuel"][0])

The following error occured:
ERROR_log.txt

Other queries work well (Besides, the "LDV" query works fine in the interactive mode).

The query for "LDV energy by primary fuel" is this (which is copied from the main_query.xml file)

  <aQuery>
       <region name="China" />
        <supplyDemandQuery title="LDV energy by primary fuel">
               <axis1 name="Primary Fuel">input[@name]</axis1>
               <axis2 name="Year">demand-physical[@vintage]</axis2>
               <xPath buildList="true" dataName="input" group="true" sumAll="false"><![CDATA[
               declare function local:append-heirarchy($parent as node(), $append as node()*) as node() {
	       	 		 let $scn := $parent/ancestor::scenario,
	       			   	  $rgn := $parent (: /ancestor::region :)
	       			   return
	       			   	  document { element scenario {
	       			 	  					$scn/@*,
	       			 						element region {
	       			 							$rgn/@*,
	       			 							$append
	       			 						}
	       			 	  				}
	       				}
	       	 		 (: I can get by with just the scenario and region
	       			 let $new_node := element {local-name($parent)} {$parent/@*, $append} 	
	       	 		 return
	       	 		 if(local-name($parent) != 'scenario')
	       	 		 then local:append-heirarchy($parent/parent::*, $new_node)
	       	 		 else document { $new_node } :)
	       	 	 };  
	       	 	 declare function local:trace-inputs($outputName as xs:string, $currTree as node(), $outputs as node()*, $is_usa as xs:boolean) as node()* {
	       			if( exists(index-of(('biomass',
'traded unconventional oil', 'regional corn for ethanol', 'regional biomassOil', 'regional sugar for ethanol', 'regional sugarbeet for ethanol'),
$outputName)) or not($currTree/*[@type='sector' and @name=$outputName]) or sum($outputs//text()) < 0.001)
then 
if(not($is_usa) and string-length($currTree/@name) = 2) then
local:trace-inputs($outputName, $currTree/parent::*/*[@type='region' and @name='USA'], $outputs, true())
else
element input {
	       						attribute name {$outputName},
	       						attribute type {'input'},
	       						$outputs
	       					}
	       				else
	       			let $useOutputs := $currTree//output-primary[@type='output' and @name=$outputName]/physical-output,
	       			    $sectorSums := for $out in $useOutputs
						   let $currInputs := $out/../following-sibling::input-energy/demand-physical[@vintage=$out/@vintage]
						   return 
							for $currInput in $currInputs
						        return
						        element sectorSum {
							        attribute name { $currInput/../@name},
								attribute year { $currInput/@vintage},
							element output-sum {
								text { $out }
							},
							element input-sum {
								text { $currInput }
							}
	       					   }
				   return for $sector in distinct-values($sectorSums/@name)
	       			       return local:trace-inputs($sector, $currTree, for $out in $outputs
	       								      let $totalOutputSum := sum($useOutputs[@vintage=$out/@vintage]),
	       								      $inputSum := sum($sectorSums[@name=$sector and @year=$out/@vintage]/input-sum)
	       								      where $inputSum > 0
	       								      return element { local-name($out) } {
	       										$out/@*,
	       										text{ $out * ($inputSum
	       											     div $totalOutputSum) }
	       								      }, $is_usa)
		};
		declare function local:run-input-by-primary($scenarios as xs:string*, $regions as xs:string*, $collection as xs:string) as node()* { 	
			 	 unordered { 	
			 	 let $regionsG := if(not($regions[1] = 'Global'))
			 	 		  then $regions
			 	 		  else distinct-values(collection($collection)/scenario/world/*[@type='region']/@name)
			 	 return
			 	 for $scenario in $scenarios, 	    
			 	 $region in $regionsG 	
			 	 let $scenario_split := tokenize($scenario, ' '), 	    
				 $currTree := collection($collection)/scenario[@name = $scenario_split[1] and @date = $scenario_split[2]]/world/*[@type='region' and @name=$region],
				 $currInputs := $currTree/*[@type='sector' and @name='trn_pass_road']/*[@type='subsector' and @name='LDV']//*[@type='input']
				 return 
				    for $inputName in distinct-values($currInputs/@name)
				    return local:append-heirarchy($currTree, local:trace-inputs($inputName, $currTree, $currInputs[@name=$inputName]/demand-physical, false()))//text()
			 	 } 
	 	 };
		 local:run-input-by-primary((:scenarios:), (:regions:), (:collection:))
               ]]> 
               </xPath>
               <comments/>
        </supplyDemandQuery>
   </aQuery>          

pkg_resources is deprecated

Update code to deal with:
https://setuptools.pypa.io/en/latest/pkg_resources.html

subprocess.run() doesn't exist prior to python 3.5

We need an alternate implementation for older python versions. This probably also means wrapping the subprocess calls in an adaptor function.

query file from Gcam v6 folder is incompatible with this package

I saw from the jupyter notebook regarding a "sample-query.xml" file used to run queries. This file was not supplied. I used the Main_queries.xml file obtained from the GCAM v6 download folder. Using that xml file generates an empty query list with no queries in it. There are no error messages.

Remove default model output years from model_interface.properties

When these years are not present in the database, they should not be included in the output tables.

Replicate rgcam tests

We should be able to replicate all of the rgcam query_mi tests:
https://github.com/JGCRI/rgcam/blob/master/tests/testthat/test_querymi.R

Which should get us pretty good coverage (with the obvious caveat that we can't really test remote DB connections still)

Ability to break out multiple scenarios from a GCAM database

Add ability to read multiple scenarios from a single database as rgcam provides.

gcam_reader does not currently support this.

Error when extract policy cost

Hi, I'm trying to extract policy cost results using gcamreader, but errors happen when applying runQuery. The query file is modified ref to example_queries and shows as follows:

<?xml version="1.0" encoding="UTF-8"?>
<queries>
  <queryGroup name="Policy Costs">
     <aQuery>
       <all-regions/>
         <costCurveQuery title="Policy Cost By Period">
            <axis1 name="Region">Curve</axis1>
            <axis2 name="Year">DataPoint</axis2>
            <xPath buildList="true" dataName="Cost" group="false" sumAll="false">PointSet/DataPoint/y/text()</xPath>
            <comments/>
         </costCurveQuery>
    </aQuery> 
  </queryGroup> 
</queries>   

the error when I run gcamreader:

I appreciate any suggestions and help to solve it!