jgcri / gcamreader Goto Github PK
View Code? Open in Web Editor NEWPython package for reading GCAM output databases
License: Other
Python package for reading GCAM output databases
License: Other
Switch testing suite from unittest
to pytest
. I like this because of the simple usage of assert with a condition following for testing rather then using assert* where * is wild to be bound by only what unittest
offers in the way of tests. Lots of other benefits too and super stable. Here are some examples...https://docs.pytest.org/en/latest/example/index.html
Since we are using GCAM jars in this, I think this should adopt the GCAM license and disclaimer. Please add them.
Hi, I'm trying to extract policy cost results using gcamreader, but errors happen when applying runQuery. The query file is modified ref to example_queries and shows as follows:
<?xml version="1.0" encoding="UTF-8"?>
<queries>
<queryGroup name="Policy Costs">
<aQuery>
<all-regions/>
<costCurveQuery title="Policy Cost By Period">
<axis1 name="Region">Curve</axis1>
<axis2 name="Year">DataPoint</axis2>
<xPath buildList="true" dataName="Cost" group="false" sumAll="false">PointSet/DataPoint/y/text()</xPath>
<comments/>
</costCurveQuery>
</aQuery>
</queryGroup>
</queries>
the error when I run gcamreader:
I appreciate any suggestions and help to solve it!
I saw from the jupyter notebook regarding a "sample-query.xml" file used to run queries. This file was not supplied. I used the Main_queries.xml file obtained from the GCAM v6 download folder. Using that xml file generates an empty query list with no queries in it. There are no error messages.
Update code to deal with:
https://setuptools.pypa.io/en/latest/pkg_resources.html
We need an alternate implementation for older python versions. This probably also means wrapping the subprocess calls in an adaptor function.
when querying "LDV energy by primary fuel" with
conn.runQuery([i for i in queries if i.title == "LDV energy by primary fuel"][0])
The following error occured:
ERROR_log.txt
Other queries work well (Besides, the "LDV" query works fine in the interactive mode).
The query for "LDV energy by primary fuel" is this (which is copied from the main_query.xml
file)
<aQuery>
<region name="China" />
<supplyDemandQuery title="LDV energy by primary fuel">
<axis1 name="Primary Fuel">input[@name]</axis1>
<axis2 name="Year">demand-physical[@vintage]</axis2>
<xPath buildList="true" dataName="input" group="true" sumAll="false"><![CDATA[
declare function local:append-heirarchy($parent as node(), $append as node()*) as node() {
let $scn := $parent/ancestor::scenario,
$rgn := $parent (: /ancestor::region :)
return
document { element scenario {
$scn/@*,
element region {
$rgn/@*,
$append
}
}
}
(: I can get by with just the scenario and region
let $new_node := element {local-name($parent)} {$parent/@*, $append}
return
if(local-name($parent) != 'scenario')
then local:append-heirarchy($parent/parent::*, $new_node)
else document { $new_node } :)
};
declare function local:trace-inputs($outputName as xs:string, $currTree as node(), $outputs as node()*, $is_usa as xs:boolean) as node()* {
if( exists(index-of(('biomass',
'traded unconventional oil', 'regional corn for ethanol', 'regional biomassOil', 'regional sugar for ethanol', 'regional sugarbeet for ethanol'),
$outputName)) or not($currTree/*[@type='sector' and @name=$outputName]) or sum($outputs//text()) < 0.001)
then
if(not($is_usa) and string-length($currTree/@name) = 2) then
local:trace-inputs($outputName, $currTree/parent::*/*[@type='region' and @name='USA'], $outputs, true())
else
element input {
attribute name {$outputName},
attribute type {'input'},
$outputs
}
else
let $useOutputs := $currTree//output-primary[@type='output' and @name=$outputName]/physical-output,
$sectorSums := for $out in $useOutputs
let $currInputs := $out/../following-sibling::input-energy/demand-physical[@vintage=$out/@vintage]
return
for $currInput in $currInputs
return
element sectorSum {
attribute name { $currInput/../@name},
attribute year { $currInput/@vintage},
element output-sum {
text { $out }
},
element input-sum {
text { $currInput }
}
}
return for $sector in distinct-values($sectorSums/@name)
return local:trace-inputs($sector, $currTree, for $out in $outputs
let $totalOutputSum := sum($useOutputs[@vintage=$out/@vintage]),
$inputSum := sum($sectorSums[@name=$sector and @year=$out/@vintage]/input-sum)
where $inputSum > 0
return element { local-name($out) } {
$out/@*,
text{ $out * ($inputSum
div $totalOutputSum) }
}, $is_usa)
};
declare function local:run-input-by-primary($scenarios as xs:string*, $regions as xs:string*, $collection as xs:string) as node()* {
unordered {
let $regionsG := if(not($regions[1] = 'Global'))
then $regions
else distinct-values(collection($collection)/scenario/world/*[@type='region']/@name)
return
for $scenario in $scenarios,
$region in $regionsG
let $scenario_split := tokenize($scenario, ' '),
$currTree := collection($collection)/scenario[@name = $scenario_split[1] and @date = $scenario_split[2]]/world/*[@type='region' and @name=$region],
$currInputs := $currTree/*[@type='sector' and @name='trn_pass_road']/*[@type='subsector' and @name='LDV']//*[@type='input']
return
for $inputName in distinct-values($currInputs/@name)
return local:append-heirarchy($currTree, local:trace-inputs($inputName, $currTree, $currInputs[@name=$inputName]/demand-physical, false()))//text()
}
};
local:run-input-by-primary((:scenarios:), (:regions:), (:collection:))
]]>
</xPath>
<comments/>
</supplyDemandQuery>
</aQuery>
When querying "elec gen by gen tech", the results will be duplicated and do not match the results of the model interface. I find the following problems have appeared in both wind and solar technology queries. I don't know how to avoid the following problems:
Here is the queries XML:
<?xml version="1.0" encoding="UTF-8"?>
<queries>
<aQuery>
<all-regions/>
<supplyDemandQuery title="elec gen by gen tech">
<axis1 name="technology">technology</axis1>
<axis2 name="Year">physical-output[@vintage]</axis2>
<xPath buildList="true" dataName="output" group="false" sumAll="false">*[@type='sector' (:collapse:) and
(@name='electricity' or @name='elect_td_bld' or @name='industrial energy use')]/
*[@type='subsector' and (@name='wind')]/*[@type='technology' and not (@name='electricity' or @name='elect_td_bld')]/
*[@type='output' and (@name='electricity' or @name='elect_td_bld')]/
physical-output/node()</xPath>
<comments/>
</supplyDemandQuery>
</aQuery>
</queries>
Is there a doc containing schema for XML queries or a guide on how to write them, for interacting with the GCAM basex db?
The sample_queries.xml given in example runs fine, but unclear why something like Main_queries.xml doesn't have same behaviour? (or if it's even the same type of query?)
Looked in other repos & docs but haven't found anything concrete - can anyone point me in the right direction? Thanks!
Remote database access will fail if the query contains a CDATA section. There's a fix in JGCRI/rgcam#29. Once that's been validated we need to port it over here.
When these years are not present in the database, they should not be included in the output tables.
Add ability to read multiple scenarios from a single database as rgcam
provides.
gcam_reader
does not currently support this.
We should be able to replicate all of the rgcam
query_mi tests:
https://github.com/JGCRI/rgcam/blob/master/tests/testthat/test_querymi.R
Which should get us pretty good coverage (with the obvious caveat that we can't really test remote DB connections still)
At the very least, the landing page should have installation instructions and a tour of the main functions.
Add tests to evaluate high-level model functionality.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.