jmespath / jmespath.test

JMESPath compliance tests.

JMESPath

JMESPath (pronounced "james path") allows you to declaratively specify how to extract elements from a JSON document.

For example, given this document:

{"foo": {"bar": "baz"}}

The jmespath expression foo.bar will return "baz".

JMESPath also supports:

Referencing elements in a list. Given the data:

{"foo": {"bar": ["one", "two"]}}

The expression: foo.bar[0] will return "one". You can also reference all the items in a list using the * syntax:

{"foo": {"bar": [{"name": "one"}, {"name": "two"}]}}

The expression: foo.bar[*].name will return ["one", "two"]. Negative indexing is also supported (-1 refers to the last element in the list). Given the data above, the expression foo.bar[-1].name will return "two".

The * can also be used for hash types:

{"foo": {"bar": {"name": "one"}, "baz": {"name": "two"}}}

The expression: foo.*.name will return ["one", "two"].

Installation

You can install JMESPath from pypi with:

pip install jmespath

API

The jmespath.py library has two functions that operate on python data structures. You can use search and give it the jmespath expression and the data:

>>> import jmespath
>>> path = jmespath.search('foo.bar', {'foo': {'bar': 'baz'}})
'baz'

Similar to the re module, you can use the compile function to compile the JMESPath expression and use this parsed expression to perform repeated searches:

>>> import jmespath
>>> expression = jmespath.compile('foo.bar')
>>> expression.search({'foo': {'bar': 'baz'}})
'baz'
>>> expression.search({'foo': {'bar': 'other'}})
'other'

This is useful if you're going to use the same jmespath expression to search multiple documents. This avoids having to reparse the JMESPath expression each time you search a new document.

Options

You can provide an instance of jmespath.Options to control how a JMESPath expression is evaluated. The most common scenario for using an Options instance is if you want to have ordered output of your dict keys. To do this you can use either of these options:

>>> import jmespath
>>> jmespath.search('{a: a, b: b}',
...                 mydata,
...                 jmespath.Options(dict_cls=collections.OrderedDict))


>>> import jmespath
>>> parsed = jmespath.compile('{a: a, b: b}')
>>> parsed.search(mydata,
...               jmespath.Options(dict_cls=collections.OrderedDict))

Custom Functions

The JMESPath language has numerous built-in functions, but it is also possible to add your own custom functions. Keep in mind that custom function support in jmespath.py is experimental and the API may change based on feedback.

If you have a custom function that you've found useful, consider submitting it to jmespath.site and propose that it be added to the JMESPath language. You can submit proposals here.

To create custom functions:

• Create a subclass of jmespath.functions.Functions.
• Create a method with the name _func_<your function name>.
• Apply the jmespath.functions.signature decorator that indicates the expected types of the function arguments.
• Provide an instance of your subclass in a jmespath.Options object.

Below are a few examples:

import jmespath
from jmespath import functions

# 1. Create a subclass of functions.Functions.
#    The function.Functions base class has logic
#    that introspects all of its methods and automatically
#    registers your custom functions in its function table.
class CustomFunctions(functions.Functions):

    # 2 and 3.  Create a function that starts with _func_
    # and decorate it with @signature which indicates its
    # expected types.
    # In this example, we're creating a jmespath function
    # called "unique_letters" that accepts a single argument
    # with an expected type "string".
    @functions.signature({'types': ['string']})
    def _func_unique_letters(self, s):
        # Given a string s, return a sorted
        # string of unique letters: 'ccbbadd' ->  'abcd'
        return ''.join(sorted(set(s)))

    # Here's another example.  This is creating
    # a jmespath function called "my_add" that expects
    # two arguments, both of which should be of type number.
    @functions.signature({'types': ['number']}, {'types': ['number']})
    def _func_my_add(self, x, y):
        return x + y

# 4. Provide an instance of your subclass in a Options object.
options = jmespath.Options(custom_functions=CustomFunctions())

# Provide this value to jmespath.search:
# This will print 3
print(
    jmespath.search(
        'my_add(`1`, `2`)', {}, options=options)
)

# This will print "abcd"
print(
    jmespath.search(
        'foo.bar | unique_letters(@)',
        {'foo': {'bar': 'ccbbadd'}},
        options=options)
)

Again, if you come up with useful functions that you think make sense in the JMESPath language (and make sense to implement in all JMESPath libraries, not just python), please let us know at jmespath.site.

Specification

If you'd like to learn more about the JMESPath language, you can check out the JMESPath tutorial. Also check out the JMESPath examples page for examples of more complex jmespath queries.

The grammar is specified using ABNF, as described in RFC4234. You can find the most up to date grammar for JMESPath here.

You can read the full JMESPath specification here.

Testing

In addition to the unit tests for the jmespath modules, there is a tests/compliance directory that contains .json files with test cases. This allows other implementations to verify they are producing the correct output. Each json file is grouped by feature.

Discuss

Join us on our Gitter channel if you want to chat or if you have any questions.

jmespath.test's Issues

Invalid benchmark expressions

First of all, thanks @jamesls for creating this great little query language and the detailed specifications!

I liked the various implementations of JMESPath so much, that I decided I wan't to use it in C++ too, so I created my own implementation. The library is nearly finished, I just need to get some packaging code in place and improve the user documentation.
It runs all the compliance tests except the benchmarks. The problem is that benchmarks.json contains two expressions which according to the current specifications are invalid.
The first one is: sum(z, y, x, w, v, u, t, s, r, q, p, o, n, m, l, k, j, i, h, g, f, e, d, c, b, a), the specifications of the sum function says it should accept only a signle array argument.
The second one is: foo[bar > baz][qux > baz], but filter expressions should start with [?.

I'm not sure if these tests made it into benchmarks.json by error (in which case I will create a PR to fix the tests), or if they are written for some yet unreleased version of the specifications?

Filters not working intuitively

Maybe it is my misunderstanding, but with this data:

 {
            "a": {
                "b": [{
                    "c": {
                        "d": {
                            "e": "foo"
                        }
                    },
                    "something": "bar"
                }, {
                    "c": {
                        "d": {
                            "e": "miss"
                        }
                    }
                }]
            }
}

With the following query, nothing returns:

a.b[*].c.d[?e == 'foo']

But this does work:

a.b[*].c.d|@[?e == 'foo']

And in JSON Path (JayWay implementation for example):

a.b[*].c.d[?(@.e == "foo")] is the equivalent.

Is this intentional? they seem functionally equivalent but obviously are not. This fails in the playground (JavaScript), but also fails in the Java implementation.

Is this a spec issue? As designed? Common bug in all implementations? Where do we report something like this that seems to be prevalent across all of JMES?

Questions about MultiSelect list

In MultiSelect list, it says "Within the start and closing characters is one or more non expressions ..." Should that read "one or more expressions"? Or is a word missing after "non"?

In the syntax.json test file, there is a test case

"given": {"type": "object"}
"comment": "Multi-select of a list using an identifier index"
"expression": "foo[abc]"
"error": "syntax"

While this can't succeed, what in the specification makes this a syntax error? The grammar is written as

multi-select-list = "[" ( expression *( "," expression ) ) "]"

Thanks!
Daniel

jmespath.spec?

It's hard right now to submit a proposed change to jmespath because I need to send a PR to jmespath.test then one to boto/jmespath. I suggest moving the spec, docs, and test cases to jmespath.spec (or something similar) so that language proposals, new functions, etc can be managed in a single place.

Tests using U+D800 to U+DFFF

I notice there are some tests that ask the library to encode Unicode values in the UTF-16 surrogate range. (example)

I don't know much about Unicode but my crude understanding is that this should be an invalid range for encoding. Is the JMESPath spec including these values because it specifically wants to work with them? (I'm working with a library that refuses to encode those values based on the spec and wondering if I need to hack around my library or if I can convince you to modify the JMESPath spec.)

wildcard issue

jmespath.search("a..c", { "a" : { "b" : { "c" : { "d" : "AAAA" } } } } )
[{'d': 'AAAA'}]
jmespath.search("a..c.d", { "a" : { "b" : { "c" : { "d" : "AAAA" } } } } )

"a..c" works but "a..c.d" does not work. It should work like "a.b.c.d" -> "AAAA" back.