djq
is the DREQ JSON Query tool which allows you to query the DREQ
for variable mappings, and also provides an interface to explore
algorithms to implement such mappings. You can hand it requests
specifying MIPs and experiments within those MIPs and it will reply
with lists of variables. You should be able to install it from its
setup.py
. It requires the dreqPy
package from the DREQ (but is
not fussy about version), nose to run tests, and will also need access
to the a SVN checkout of the DREQ itself: you will almost certainly
need to teach it where this is.
Because the mapping from MIPs and experiments to variables is not very
well-defined by the DREQ, djq
also provides a simple interface which
allows you to define your own mapping function: this function doesn't
need to know anything about djq
at all other than how it is called, and
can be loaded dynamically from a module at run time, so no modifications
are needed to djq
in order to provide an alternative mapping function . It
is possible to specify which function or module to use both from the Python API
and from the command line. Multiple such functions / modules can exist
concurrently. These functions / modules are called 'implementations' in the
code.
Once the set of variables is computed, it needs to be elaborated in various ways
before being turned into JSON. This is done by a 'JSONifier', and these are
also components which can be plugged in to djq
dynamically.
A tool is included, cci
, which allows you to directly compare
implementations.
There are four command line programs associated with djq
.
djq
is the main thing: you can use it to make queries and get answers;cci
compares implementations -- you can give it the names of one or two implementation modules and will tell you whether they differ in the variables they compute, giving a metric of similarity going from 0.0 (completely different) to 1.0 (identical);all-requests
is a little program which reads the DREQ, and then generates a request for every experiment in every MIP, which can be fed to djq to run a really comprehensive set of queries;scatter-replies
is another little program which will read a set of replies fromdjq
and spit them out into files named after the MIPs and experiments.
As an example, using cci
to compare the two implementations bundled
with djq
, which correspond to what Martin described in his document
and what he used at to generate his spreadsheets, both at the time
djq
was originally written, the results vary from 0.0 to 1.0 for
different pairs of MIP and experiment.
dqi
is the DREQ Query Interface. You do not need this to use djq
,
although some djq
back ends may need it (they do not currently).
small
contains some small, more-or-less ad-hoc programs which are
related to the CMIP6 DREQ. These are almost entirely undocumented and
may or may not work.
See the change log, which contains at least an entry for each release, and often also any changes which matter since the most recent release.
djq
- Some general documentation
- (No documentation for
dqi
yet)
- The CMIP6 data request or DREQ (redirection page)
- Subversion repo for the DREQ, also browsable
If you use Travis CI, you should be able to persuade it to run tests:
look at at .travis.yml
. These previously worked, but Travis CI
needs access to the repo to do them, and it doesn't have access to the
Met Office repo (nor should it, I think).
The tests should pass unless there are serious bugs in djq
itself.
It formerly ran a hairy sanity test which compared what it computes
against spreadsheets included with the DREQ. Unfortunately the DREQ
is so unstable that these essentially never passed, so these are no
longer included in the Travis CI tests.
All the documentation is in
Markdown (and
specifically GitHub flavoured
Markdown):
this should be fairly readable as plain text. README.md
files are
the entry points, and any extended documentation is in subdirectories
called doc
.
You can use grip to view the
pretty version of the documentation locally. It can be installed with
pip
:
$ pip install grip
[...]
$ grip
* Running on http://localhost:6419/ (Press CTRL+C to quit)
grip -b
is also useful (opens a browser tab).
See its
documentation. Note
that grip
works by using GitHub's API to format the markdown files,
and so sends their content to GitHub: it's not suitable for sensitive data.
© British Crown Copyright 2016, 2018, Met Office. See LICENSE.md for license details.