jg-rp / python-jsonpath

A flexible JSONPath engine for Python with JSON Pointer and JSON Patch

Python JSONPath

A flexible JSONPath engine for Python.
We follow RFC 9535 and test against the JSONPath Compliance Test Suite.

---

Table of Contents

• Install
• Links
• Related projects
• Examples
• License

Install

Install Python JSONPath using pip:

pip install python-jsonpath

Or Pipenv:

pipenv install -u python-jsonpath

Or from conda-forge:

conda install -c conda-forge python-jsonpath

Links

• Documentation: https://jg-rp.github.io/python-jsonpath/.
• JSONPath Syntax: https://jg-rp.github.io/python-jsonpath/syntax/
• Change log: https://github.com/jg-rp/python-jsonpath/blob/main/CHANGELOG.md
• PyPi: https://pypi.org/project/python-jsonpath
• Source code: https://github.com/jg-rp/python-jsonpath
• Issue tracker: https://github.com/jg-rp/python-jsonpath/issues

Related projects

• Python JSONPath RFC 9535 - An implementation of JSONPath that follows RFC 9535 much more strictly. If you require maximum interoperability with JSONPath implemented in other languages - at the expense of extra features - choose python-jsonpath-rfc9535 over python-jsonpath.

  python-jsonpath-rfc9535 matches RFC 9535's JSONPath model internally and is careful to match the spec's terminology. It also includes utilities for verifying and testing the JSONPath Compliance Test Suite. Most notably the nondeterministic behavior of some JSONPath selectors.

• JSON P3 - RFC 9535 implemented in TypeScript. JSON P3 does not include all the non-standard features of Python JSONPath, but does define some optional extra syntax.

Examples

JSONPath

import jsonpath

data = {
    "users": [
        {"name": "Sue", "score": 100},
        {"name": "John", "score": 86},
        {"name": "Sally", "score": 84},
        {"name": "Jane", "score": 55},
    ]
}

user_names = jsonpath.findall("$.users[[email protected] < 100].name", data)
print(user_names) # ['John', 'Sally', 'Jane']

JSON Pointer

We include an RFC 6901 compliant implementation of JSON Pointer. See JSON Pointer quick start, guide and API reference

from jsonpath import pointer

data = {
    "users": [
        {"name": "Sue", "score": 100},
        {"name": "John", "score": 86},
        {"name": "Sally", "score": 84},
        {"name": "Jane", "score": 55},
    ]
}

sue_score = pointer.resolve("/users/0/score", data)
print(sue_score)  # 100

jane_score = pointer.resolve(["users", 3, "score"], data)
print(jane_score)  # 55

JSON Patch

We also include an RFC 6902 compliant implementation of JSON Patch. See JSON Patch quick start and API reference

from jsonpath import patch

patch_operations = [
    {"op": "add", "path": "/some/foo", "value": {"foo": {}}},
    {"op": "add", "path": "/some/foo", "value": {"bar": []}},
    {"op": "copy", "from": "/some/other", "path": "/some/foo/else"},
    {"op": "add", "path": "/some/foo/bar/-", "value": 1},
]

data = {"some": {"other": "thing"}}
patch.apply(patch_operations, data)
print(data) # {'some': {'other': 'thing', 'foo': {'bar': [1], 'else': 'thing'}}}

License

python-jsonpath is distributed under the terms of the MIT license.

python-jsonpath's Issues

Filter expression efficiency

We currently evaluate filter sub-expressions for every preceding selector match, even if that sub-expression can't change. Here, _.names.size is resolved once for each item in users.

$.users[[email protected] > _.names.size].name

With the exception of custom mapping classes that generate dynamic values - which we probably don't care about within the scope of a single findall()/finditer() - the only filter expression type that needs to be reevaluated for every item is SelfPath (default @).

Just wanted to say thank you

Hi,

Just wanted to thank you for this awesome json path library. The best I've tried and the most pleasing to work with.

Keep up the good work!

Child pointers with the slash operator

Implement JSONPointer.__truediv__() to build child pointers from an existing JSON Pointer.

The `keys` function extension is not well-typed

The keys() functions extension does not inherit from FilterFunction and define its argument/return types.

We probably need to return a special "Nothing" value that plays nicely with contains and in, then return that if keys() is passed a non-mapping argument.

Filter function well-typedness

We don't enforce the spec's well-typeness rules when it comes to validating JSONPath queries containing filter function calls. This was a deliberate choice in an attempt at making user-defined function extension as simple as possible.

However, there is nothing stopping us from supporting well-typedness validation for those that do want it. That goes for standard filer functions and user-defined functions.

We should extend the class-based filter function interface to allow for adding parameter and return type information, and a JSONPathEnvironment option to enable/disable well-typedness validation.

Command Line Interface

Add a command line interface for:

• Finding objects in a JSON document given a JSONPath
• Resolving a JSON Pointer against a JSON document
• Applying a JSON Patch to a JSON document

Conditionally return target JSON root value

An empty JSONPath query (non-standard) or a query containing just the root identifier ($) will return the target JSON document in its entirety.

from jsonpath import findall

rv = findall("$", {'a': [1, 2, 3], 'b': 42})
print(rv)  # [{'a': [1, 2, 3], 'b': 42}]

But there is no way to conditionally return the target root (or an empty result/node list) using a filter selector. This is a known limitation in the IETF JSONPath Base. See ietf-wg-jsonpath/draft-ietf-jsonpath-base#390.

We can stray into Python and use jsonpath.match().

from jsonpath import match

data = {'a': [1, 2, 3], 'b': 42}
rv = [data] if match("$[@ < 50]", data) else []
# ...

But that is not great or always an option.

One solution is to add an additional root identifier. A root Identifier that is functionally equivalent to wrapping the target JSON value in a single-element array. I suggest the circumflex (U+005E ^ aka caret) character for this purpose. I've been calling this the "fake root identifier".

from jsonpath import findall

data = {'a': [1, 2, 3], 'b': 42}

# Implicitly wrap `data` in an array.
findall("^[[email protected] < 50]", data)  # [{'a': [1, 2, 3], 'b': 42}]
findall("^[[email protected] > 50]", data)  # []

# Same as explicitly wrapping `data`.
findall("$[[email protected] < 50]", [data])  # [{'a': [1, 2, 3], 'b': 42}]
findall("$[[email protected] > 50]", [data])  # []

String literal Unicode escape sequence parsing

Rather than messing around decoding, encoding then decoding string literals, we should pass them through json.loads(). It will handle UTF-16 encoded surrogate pairs and unnecessary escape characters. The latter is something the spec requires, and that we don't currently do.

python-jsonpath/jsonpath/parse.py

Lines 394 to 401 in 6ce5495

	if self.env.unicode_escape:
	name = (
	codecs.decode(
	stream.current.value.replace("\\/", "/"), "unicode-escape"
	)
	.encode("utf-16", "surrogatepass")
	.decode("utf-16")
	)

Also, we're currently only doing this for quoted property selectors, and not string literal in comparison expressions or filter function arguments.

jsonpath.match

Add a match() function to complement findall() and finditer().

jsonpath.match() will take the same arguments as findall() and finditer(), but return a JSONPathMatch for the first match found, or return None if there were no matches.

Note that None could be the legitimate result of a match, but a match() result will always be wrapped in a JSONPathMatch instance. So the matched object would be available as JSONPathMatch.obj.

Optionally disable decoding of UTF-16 escape sequences

Add an option to disable decoding of UTF-16 escape sequences when parsing a JSONPath. If client code has already handled the possibility UTF-16 code points, including surrogate pairs, disabling it could improve parsing performance.

JSONPointer and JSONPatch already have arguments to control UTF-16 decoding.

Remember to:

• Expose this option via the module-level find* functions
• Expose this option via the command line interface

`startswith` function extension

Include a function extension that accepts two string arguments and returns true if the second argument is a prefix of the first argument.

Existing match and search function extensions can achieve the same result, but carry less semantic meaning and are probably less readable.

The `keys` function extension is undocumented

We are missing documentation for the keys() function extension.

python-jsonpath/jsonpath/function_extensions/keys.py

Lines 7 to 8 in 8e72725

	def keys(obj: Mapping[str, object]) -> Optional[Tuple[str, ...]]:
	"""Return an object's keys, or `None` if the object has no _keys_ method."""

Include the CTS as a git submodule

Commit the JSONPath Compliance Test Suite as a submodule. This will allow us to test against it in GitHub workflows and make it easier to report accurate test coverage.

We'll need to cover git submodule usage in a "contributing" guide.

`match` and `search` flags

Add an optional flags argument to the built-in match() and search() filter functions.

This would be a deliberate deviation to the IETF JSONPath standard.

Current key identifier

Add _ as a handle for the current key when filtering a map-like object.

When filtering a map-like object, _ would be the key associated with @, or undefined if filtering a non-mapping object.

Arbitrary string without slashes returns document root

When given an arbitrary string without any slashes, pointer.ressolve() returns the document root. Only the empty string should resolve to the document root.

import jsonpath
print(jsonpath.pointer.resolve("noshuchthing", {"foo": [0, 1, 2]}))

Output:

{'foo': [0, 1, 2]}

We expect a JSONPointerKeyError.

jsonpath get keys not compatible

I have a json like this

{
    "task_tag": "blade_appearance_detect",
    "image_type": "base64",
    "extra_args": [
        {
            "model": "crack",
            "param": {
                "conf": 0.66,
                "iou": 0.25
            }
        },
        {
            "model": "damaged",
            "param": {
                "conf": 0.3,
                "iou": 0.25
            }
        }
    ]
}

I want param's all keys, expect ['conf', 'iou', 'conf', 'iou'],
I use jsonpath.findall('$..param~', ret) to get right result, but other syntax is "$..param.*~", not compatible.

Filter context selector as filter function argument

We're getting a JSONPathSyntaxError when trying to parse a JSONPath containing a filter context selector (default _) as a filter function argument.

Logical expressions and non-singular filter queries

We are erroneously raising a JSONPathSyntaxError when non-singular filter queries appear in logical expressions.

Example:

import jsonpath

data = [
  {
    "a": "b",
    "d": "e"
  },
  {
    "b": "c",
    "d": "f"
  }
]

query = "$[?@..* && @.b]"
jsonpath.findall(query, data)
# JSONPathSyntaxError: non-singular query is not comparable, line 1, column 8

This would be the correct behaviour in a comparison expression, as non-singular queries can not be compared, but we are applying the same logic to logical expressions, which is bad.

IETF JSONPATH Working Group Internet-Draft

All commits up to this point, including the syntax defined in jsonpath.bnf, where done without reference to the IETF JSONPATH Working Group Internet-Draft. I have only just become aware of it.

With the goal of aligning with the aforementioned document, we need to ..

• Adhere to the same naming conventions.
• Be able to disable features that we want to keep but are not in the draft.
• Configure the default environment to match the semantics described in the draft
• Implement features described in the draft that we have omitted (function extensions)
• Use all examples found in the draft document as test cases in this project (with appropriate licence notices)

Relative JSON Pointer

Add support for building JSONPointer instances from and existing pointer and a relative JSON Pointer.

See https://www.ietf.org/id/draft-hha-relative-json-pointer-00.html.

JSON Pointer parent

Add a parent() method to JSONPointer, returning a new JSONPointer pointing to the original pointer's parent.

Wrap JSONDecodeError

To make it clear where the error is coming from, wrap potential JSONDecodeErrors when giving findall(), finditer() etc. a string as a target document.

Root pointer from parts

When given an empty parts iterable, the class method JSONPointer.from_parts() currently returns a pointer with a string representation of /. It should be an empty string.

There's a similar situation with JSONPointer.from_match(), although there's technically no way to reference the document root using a valid JSONPath.

Filter query existence tests and logical expressions

When a singular filter query appears as part of a logical expression, we are incorrectly using the node's value rather than treating the query as an existence test.

Given the query $.users[[email protected] && @.score > 1] and the data:

{
  "users": [
    {
      "name": "Sue",
      "score": 100
    },
    {
      "name": "John",
      "score": 86,
      "admin": true
    },
    {
      "name": "Sally",
      "score": 84,
      "admin": false
    },
    {
      "name": "Jane",
      "score": 55
    }
  ],
  "moderator": "John"
}

We get

[
  {
    "name": "John",
    "score": 86,
    "admin": true
  }
]

And we should get

[
  {
    "name": "John",
    "score": 86,
    "admin": true
  },
  {
    "name": "Sally",
    "score": 84,
    "admin": false
  }
]

JSONPointer.exists

Add an exists() method to JSONPointer. exists() takes a JSON-like "document" and returns true if the document contains a value at the pointed to location, or false otherwise.

Custom JSONDecoder

Add support for specifying a custom JSONDecoder when calling findall() or finditer() with a string or file-like object containing a JSON document.

`typeof()` or `isinstance()` filter function

Add a non-standard typeof() or type() filter function. Such a function would return the type of its argument as a string.

We would default to using JSON terminology - "object", "array", "number" etc. - but allow package users to select Python equivalent type names or define custom aliases.

isinstance() or is() would be similar, but return a Boolean if the type of the first argument matches the second argument. This has the benefit of matching against several possible aliases for a type in one function call.

Filter expression operator precedence

When parsing JSONPath filter expressions, we are not giving the logical not operator (!) a higher precedence than logical or (||) and logical and (&&).

>>> import jsonpath
>>> str(jsonpath.compile("$[[email protected] || [email protected]]"))
"$[?!(@['a'] || !@['b'])]"

`JSONPathMatch` to JSON Pointer

Add support for converting a JSONPathMatch (as generated by finditer()) to a JSON Pointer.

Our JSON Pointer class will define get and set methods for retrieving and updating a single value in a JSON document or equivalent Python object.

There's no immediate plan to evaluate JSON Pointers given as strings.

It has become clear that a complete implementation of rfc6901 is for the best, plus rfc6902 (JSON Patch) for modifying JSON documents, with a Python API mirroring JSON Patch operations.

Keys selector

Add a non-standard "keys" or "property names" selector to retrieve keys/properties from an object/mapping/dict.

Some JSONPath implementations use ~ as a keys selector. In the absence of a better idea, we'll do the same.

Object data and `contains` and `in` membership operators

The non-standard contains and in membership operators currently work for any sequence.

python-jsonpath/jsonpath/env.py

Lines 551 to 554 in 880c123

	if operator == "in" and isinstance(right, Sequence):
	return left in right
	if operator == "contains" and isinstance(left, Sequence):
	return right in left

We should change that so they work for any container (any mapping is a container).

Bracketed segments can contain filter selectors

The spec allows bracketed segments to contain an arbitrary number of filter selectors alongside other selectors, separated by commas. We currently only recognise filters that appear on their own inside square brackets.

$.foo[?<expression>]

but not

$.foo['bar', ?<expression>, ?<expression>]

We should follow the spec in this regard.

More find* helpers

We have findall() and finditer(), add find_one, find_n / limit and skip / drop helpers for managing the results generated by finditer().

Also alias findall as find_all and finditer as find_iter.