jdevillard / jmespath.net Goto Github PK

Context: I have a scenario to evaluate multiple Jmespath on a single device context. As of the Jtoken is created for every Json per Rule.

E.g., On a JSON Document, I need to run 150000 Jmespath. Also, I need a way to run it on the Multiple documents based on the Request.

If there is a way provided to improve the JToken or reuse the Jtoken and provide the expression which I can cache will be helpful.

I can provide PR for the same.

The .Net Framework 4.5 and the dotnet standard 1.6 DLL is not fully signed. This can be validated using

> sn.exe -vf .\JmesPath.Net.dll

This library can be used in a Console Application (tested using a .Net Framework 4.5 and 4.6.1).
It cannot be used in a WebAPI Application, .Net Framework 4.5 or 4.6.1. The assembly is loaded in a AspNET Core 2.0.

Trying to install version 1.0.145 from nuget results in an error as it lists JmesPath.Parser >= 1.0.0 as a dependency, but that library has not been published to nuget.

https://www.nuget.org/packages/JmesPath.Net/1.0.145

https://www.nuget.org/packages/JmesPath.Parser/
Error 404 - Page Not Found
Looks like you might have gotten lost. Let's see if we can help you find your way!

Hi,

I'm trying to filter using this query: props[? id == 2 ] on the following json:

{
    "props": [
        {
            "id": 1,
            "name": "prop #1"
        },
        {
            "id": 2,
            "name": "prop #2"
        },
        {
            "id": 3,
            "name": "prop #3"
        }
    ]
}

If I'm trying it on https://jmespath.org/tutorial.html it works, but on JmesPath.net I need to put my number between back-ticks.
So to make my query work I need to transform it into props[? id == `2` ]
Is this intended ?

As reported here, JMESPath.Net does not handle Unicode surrogate pairs.

Consider this character: 😀 (0x1F600).

The following string "😀" is represented in JSON as a pair of encoded surrogate characters like "\uD83D\uDE00".

Internally, JMESPath.Net uses C# which encodes strings using UTF16-LE.
However, .NET strings do not handle surrogate pairs correctly.

Please, consider using the StringInfo class for proper support of Unicode in JMESPath.Net.

Are these supposed to be committed to the repo? They look like secrets that shouldnt be exposed.
Forgive me if I'm mistaken, just trying to keep people safe.

https://github.com/jdevillard/JmesPath.Net/blob/master/appveyor.yml#L18

https://github.com/jdevillard/JmesPath.Net/blob/master/appveyor.yml#L20

https://github.com/jdevillard/JmesPath.Net/blob/master/appveyor.yml#L22

https://github.com/jdevillard/JmesPath.Net/blob/master/appveyor.yml#L57

Hi there,

I believe there is a bug in the transform of the .NET code when using an expression with an && and a number comparison. Consider the following code

            string jsonString = @"{
                ""StringKey"": ""StringValue"",
                ""StringKey2"": ""StringValue2"",
                ""NumberKey"": 100}";
            string jmesExpression = "StringKey == 'StringValue' && NumberKey > '5'";
            var jmes = new JmesPath();

            var jmesResult = jmes.Transform(jsonString, jmesExpression);

            Console.WriteLine($"JMES result string = {jmesResult}");

            string jmesExpression2 = "StringKey == 'StringValue' && StringKey2 == 'StringValue2'";

            var jmesResult2 = jmes.Transform(jsonString, jmesExpression2);

            Console.WriteLine($"JMES result 2 string = {jmesResult2}");

            string jmesExpression3 = "NumberKey > '5'";

            var jmesResult3 = jmes.Transform(jsonString, jmesExpression3);

            Console.WriteLine($"JMES result 3 string = {jmesResult3}");

            string jmesExpression4 = "NumberKey == '100'";

            var jmesResult4 = jmes.Transform(jsonString, jmesExpression4);

            Console.WriteLine($"JMES result 4 string = {jmesResult4}");

If you run this, you will see that the first transform where the NumberKey is compared will return null and the second transform will return true. The third also returns null and the fourth returns false so there is something wrong with the number comparison.

I'd like to propose a slight re-architecture of this project to remove the dependency on Json.NET. It would then allow the project to be used and adapted (in a binary fashion) to other JSON libraries, including (but not limited to) Jil, Manatee Json, Jayrock JSON and the now system-supplied System.Text.Json.

I believe this is achievable in a rather straightforward fashion by separation of concerns. It will also enable many other scenarios, for example, pure validation without incurring the cost of building an entire AST.

Is this something that would be welcome? It may involve breaking changes (which means a major revision) so I prefer to ask before. I would also rather contribute and collaborate to maintain one project rather than forking my own version.

The canonical way to refer to the specification is JMESPath.
Please, consider changing the various occurrences of this casing in the documentation, project description on GitHub and so on.

Hi,

My expression is working fine with the online editor but when I used the same expression with the same data in c# it's giving me the below error.

• Version : JmesPath.Net - 1.0.153
• Framwork - Dot net core
• Language - C#

ERROR

Error(4, 76): syntax, near '$'.
at DevLab.JmesPath.JmesPathScanner.yyerror(String format, Object[] args)
at DevLab.JmesPath.JmesPathScanner.Scan()
at DevLab.JmesPath.JmesPathScanner.yylex()
at StarodubOleg.GPPG.Runtime.ShiftReduceParser`2.Parse()
at DevLab.JmesPath.Parser.Parse(Stream stream, Encoding encoding, IJmesPathGenerator generator)
at DevLab.JmesPath.JmesPath.Parse(Stream stream)
at DevLab.JmesPath.JmesPath.Parse(String expression)
at AzureFunctionDurableTic.AzureFunctionTIC.d__16.MoveNext() in /Users/atomar10/Projects/AzureFunctionDurableTic/AzureFunctionDurableTic/AzureFunctionTIC.cs:line 203

CODE

            var q2 = @"
                            in_network[].{
                                                      unique_id:join('_',[negotiation_arrangement,name,billing_code_type,billing_code_type_version,billing_code,description,negotiated_rates[].tin | [0],negotiated_rates[].negotiated_price.expiration_date | [0],negotiated_rates[].negotiated_price.negotiated_type | [0]]),                 
                                                      reporting_entity_name:$.reporting_entity_name,
                                                      name:name,                   
                                                      billing_code:billing_code,                   
                                                      billing_code_type:billing_code_type,
                                                      billing_code_type_version:billing_code_type_version,                   
                                                      description:description,                   
                                                      negotiation_arrangement:negotiation_arrangement,    
                                                                     negotiated_rates:negotiated_rates[*].{
                                                                        providers:providers,
                                                                        tin:tin,
                                                                        service_code:service_code,
                                                                        expiration_date:negotiated_price.expiration_date,
                                                                        negotiated_rate:negotiated_price.negotiated_rate,
                                                                        negotiated_type:negotiated_price.negotiated_type
                                                                    }                 
                                                }";

            return q2;

  var jmes = new JmesPath();
  var exp = jmes.Parse(expression).Transform(data);

Hi,

I would like you to consider adding a feature to register a custom JmesPathFunction, so it is possible to use JMESPath in more advanced scenarios.

It could be something simple as constructor overload that exposes Action<IRegisterFunctions> as a parameter.

Roughly speaking:

public JmesPath(Action<IRegisterFunctions> factory) {
  repository_ = JmesPathFunctionFactory.Create(evaluator_);
  factory.Invoke(repository_);
}

Please let me know if you accept contributions, I would be more than happy to add this feature.

When is passed invalid filter the exception is very generic, maybe we can improve it.

Exception has occurred: CLR/Newtonsoft.Json.JsonReaderException
An exception of type 'Newtonsoft.Json.JsonReaderException' occurred in Newtonsoft.Json.dll but was not handled in user code: 'JsonToken EndArray is not valid for closing JsonType Object. Path ''

There is a parser to convert JMESPath string to Expression but I can't find a way to do the opposite.

The idea would be to use the JMESPath string as a serialization method that can then be interpreted by any application written in any language.

I noticed that netstandard1.3 is removed as supported framework in this NuGet, can you please add this again?

RegexTest.cs(25,13): warning xUnit2004: Do not use Assert.Equal() to check for boolean conditions. [C:\projects\jmespath-net\tests\jmespathnet.tests\jmespathnet.tests.csproj]

Summary

I wish for the nuget packages to have the licence expression property set correctly and the repository url

Details

The licence expression property should be set to the correct licence type I.e. Apache-2.0 as this will enable analysis of licences in use to occur in external tools & the license type will be shown in Nuget etc. By providing the repo url it will make it easier to contribute and can be used by source-link.

The following test would fail:

        [Theory]
        [InlineData("[:30]", "'abc'", "'abc'")]
        public void JmesPathSliceExpression_Transform_String(string slices, string input, string expected)
            => Assert(slices, input, expected);

This is because string slice doesn't have bounds check.

https://github.com/jdevillard/JmesPath.Net/blob/master/src/jmespath.net/Expressions/JmesPathSliceProjection.cs#L85-L86

            for (var index = start; compare(index, stop); index += step)
                characters.Add(text.CodePoints[index]);

I have been trying to come up with appropriate JmesPath when there is a dot (".") in the key.

{
  "some.key": 10
}

I am not sure how should I access the value for such or complex cases where the key has a dot.

Hello @jdevillard ,
I think good ideia implements split into string.

What you think about?

Give this input:

{
              "payload": {
                "partId": "",
                "mimeType": "multipart/alternative",
                "filename": "",
                "headers": [
                  {
                    "name": "Delivered-To",
                    "value": "[email protected]"
                  },
                  {
                    "name": "To",
                    "value": "[email protected]"
                  }
                ]
              }
            }

and this expression
payload.headers[?not_null(name,'')=='Delivered-To'].value[?contains(not_null(@,''), '@uipath.rocks')]

the library returns an empty array [] but it should return
["[email protected]"]

Validated it with: https://jsoneditoronline.org/#left=local.xahute

When compare dates example : today >= '2018-07-17T08:00:00Z' not working except today == '2018-07-17T08:00:00Z'.

The default Unicode case mapping algorithm is implemented using ToLower() and not the ToLowerInvariant() method currently used.

I'm not asking for it to be done, but as you have in depth knowledge of JMESPath, is it conceivably possible for JMESPath queries to be mapped into LINQ Expression Trees*?

*There may be some operations that aren't coverted in System.Linq which could have custom Methods in a well-known namespace, I think that would be allowed under the scope of my question!

Hello - I'm considering using your class library but I'm having a hard time finding an actual copyright statement to include in the third party notices file being put together.

IANAL but at the bottom of the Apache license is a note "APPENDIX: How to apply the Apache License to your work." --- would it be possible to update the LICENSE file in your repo to be something like this (with the year and author's names of course)? It doesn't needs to contain the entire ASL verbatim...

Copyright [yyyy] [name of copyright owner]

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

I'd also suggest adding a copyright property in the project file so it ends up in the nuspec next time you publish.

Thanks a lot, and thanks for building this project! Also, if you don't feel like changing the LICENSE file, just let me know what the Copyright [yyyy] [name of copyright owner] should be I'll copy your answer into the notices file. Cheers!

Caused from trying to build this repo in vscode. What I need to do for setup?

A sub-expression is very similar to a pipe-expression with a few key differences:

• A sub-expression restricts the kinds of expressions allowed on its right-hand-side.
• A pipe-expression stops a projection on the left-hand-side from propagating to the right-hand-side.

However, in some cases, those differences can be ignored.

For instance, currently JMESPath.NET works like this:

• search( `null`.[@], {} ) -> null
• search( `null` | [@], {} ) -> [ null ]

Given those expressions do not involve any projection, they should have the same result.
This calls into question which is the correct result.

Compliance tests suggests the correct result is null and not [ null ].
The canonical JavaScript implementation agrees with this.

However, the specification calls out how a sub-expression (and thus, by extension) a pipe-expression should be implemented.

left-evaluation = search(left-expression, original-json-document)
result = search(right-expression, left-evaluation)

However, compliance tests cannot succeed by following this guideline.
For instance, this compliance test looks like so:

search( foo.[a || b], {"type": "object"} ) -> null

Which evaluates to:

• Left evaluates identifier foo which cannot be found in the original json document. Thus returns null.
• Right evaluates the multi-select-list against the value null.
• Inside this list, an or-expression evaluates identifiers a and b which cannot be found in the current value null. Thus it returns null.
• Therefore, the final result is a JSON array with a single element being the null value.

Instead, the correct implementation seems to be:

left-evaluation = search(left-expression, original-json-document)
if (left-evaluation == null)
    result = null
else
    result = search(right-expression, left-evaluation)

Given the following JSON:

{
	"matches": [
		{
			"comp_level": 5,
			"winning_alliance": 0,
			"key": "2024alhu_f1m1",
			"set_number": 1,
			"match_number": 1,
			"alliances": {
				"red": {
					"score": 120,
					"team_keys": [
						"frc5002",
						"frc4635",
						"frc2481"
					],
					"surrogate_team_keys": [],
					"dq_team_keys": []
				},
				"blue": {
					"score": 60,
					"team_keys": [
						"frc7111",
						"frc4265",
						"frc6517"
					],
					"surrogate_team_keys": [],
					"dq_team_keys": []
				}
			},
			"event_key": "2024alhu",
			"time": 1712436840,
			"actual_time": 1712438011,
			"predicted_time": 1712438103,
			"post_result_time": 1712438219,
			"score_breakdown": {
				"ValueKind": 1
			},
			"videos": [
				{
					"type": "youtube",
					"key": "K3Jv2DdIN1A"
				}
			]
		},
		{
			"comp_level": 5,
			"winning_alliance": 0,
			"key": "2024alhu_f1m2",
			"set_number": 1,
			"match_number": 2,
			"alliances": {
				"red": {
					"score": 94,
					"team_keys": [
						"frc5002",
						"frc4635",
						"frc2481"
					],
					"surrogate_team_keys": [],
					"dq_team_keys": []
				},
				"blue": {
					"score": 80,
					"team_keys": [
						"frc7111",
						"frc4265",
						"frc6517"
					],
					"surrogate_team_keys": [],
					"dq_team_keys": []
				}
			},
			"event_key": "2024alhu",
			"time": 1712438100,
			"actual_time": 1712439464,
			"predicted_time": 1712439426,
			"post_result_time": 1712439684,
			"score_breakdown": {
				"ValueKind": 1
			},
			"videos": [
				{
					"type": "youtube",
					"key": "ESV5X7EAmAQ"
				}
			]
		},
		{
			"winning_alliance": 0,
			"key": "2024alhu_qm1",
			"set_number": 1,
			"match_number": 1,
			"alliances": {
				"red": {
					"score": 68,
					"team_keys": [
						"frc1466",
						"frc9590",
						"frc4020"
					],
					"surrogate_team_keys": [],
					"dq_team_keys": []
				},
				"blue": {
					"score": 35,
					"team_keys": [
						"frc5410",
						"frc7717",
						"frc6107"
					],
					"surrogate_team_keys": [],
					"dq_team_keys": []
				}
			},
			"event_key": "2024alhu",
			"time": 1712325300,
			"actual_time": 1712325230,
			"predicted_time": 1712325300,
			"post_result_time": 1712325429,
			"score_breakdown": {
				"ValueKind": 1
			},
			"videos": [
				{
					"type": "youtube",
					"key": "gZe4iPmNwIw"
				}
			]
		}
	]
}

The JMESPath expression max_by(matches, &alliances.red.score) should return the single record:

{
  "comp_level": 5,
  "winning_alliance": 0,
  "key": "2024alhu_f1m1",
  "set_number": 1,
  "match_number": 1,
  "alliances": {
    "red": {
      "score": 120,
      "team_keys": [
        "frc5002",
        "frc4635",
        "frc2481"
      ],
      "surrogate_team_keys": [],
      "dq_team_keys": []
    },
    "blue": {
      "score": 60,
      "team_keys": [
        "frc7111",
        "frc4265",
        "frc6517"
      ],
      "surrogate_team_keys": [],
      "dq_team_keys": []
    }
  },
  "event_key": "2024alhu",
  "time": 1712436840,
  "actual_time": 1712438011,
  "predicted_time": 1712438103,
  "post_result_time": 1712438219,
  "score_breakdown": {
    "ValueKind": 1
  },
  "videos": [
    {
      "type": "youtube",
      "key": "K3Jv2DdIN1A"
    }
  ]
}

According to the tester on JMESPath.org.

However, when applied via JmesPath.Net, and exception is thrown:

System.Exception
  HResult=0x80131500
  Message=Error: invalid-type, the expression argument of function max_by should return a number or a string.
  Source=JmesPath.Net
  StackTrace:
   at DevLab.JmesPath.Functions.ByFunction.<EvaluateAsync>d__3.MoveNext()
   at DevLab.JmesPath.Functions.MaxByFunction.<ExecuteAsync>d__2.MoveNext()
   at DevLab.JmesPath.Expressions.JmesPathFunctionExpression.<TransformAsync>d__10.MoveNext()
   at DevLab.JmesPath.Expressions.JmesPathExpression.<TransformAsync>d__1.MoveNext()
   at DevLab.JmesPath.Expressions.JmesPathRootExpression.<TransformAsync>d__6.MoveNext()
   at DevLab.JmesPath.Expressions.JmesPathExpression.<TransformAsync>d__1.MoveNext()

  This exception was originally thrown at this call stack:
    DevLab.JmesPath.Functions.ByFunction.EvaluateAsync(DevLab.JmesPath.Expressions.JmesPathExpression, Newtonsoft.Json.Linq.JToken)
    DevLab.JmesPath.Functions.MaxByFunction.ExecuteAsync(DevLab.JmesPath.Functions.JmesPathFunctionArgument[])
    DevLab.JmesPath.Expressions.JmesPathFunctionExpression.TransformAsync(Newtonsoft.Json.Linq.JToken)
    DevLab.JmesPath.Expressions.JmesPathExpression.TransformAsync(DevLab.JmesPath.Expressions.JmesPathArgument)
    DevLab.JmesPath.Expressions.JmesPathRootExpression.TransformAsync(Newtonsoft.Json.Linq.JToken)
    DevLab.JmesPath.Expressions.JmesPathExpression.TransformAsync(DevLab.JmesPath.Expressions.JmesPathArgument)

Hello,

It would be great to have support for async custom functions, for example to lookup data in the database.
I looked at the code and have some changes ready, I'll make a pull request for it. Please let me know what you think.

The Expression type is the root of the parsed abstract syntax tree.
However it does not implement the Accept method to propagate visitors.

Also, consider exposing each JmesPathExpression-derived types internal structure.

Scenario

Using flatten operator [] or wildcard expression [*] on arrays, the result shouldn't contain elements which are null.

Starting with the the following json:

{"data":[null,["a",null,"b"],["c",["d",null,"e"],"f"]]}

Expected

data[] produces:

["a","b","c",["d",null,"e"],"f"]

data[*] produces:

[["a",null,"b"],["c",["d",null,"e"],"f"]]

map(&@,data) produces:

[null,["a",null,"b"],["c",["d",null,"e"],"f"]]

Actual

data[] produces:

[null,"a",null,"b","c",["d",null,"e"],"f"]

data[*] produces:

[null,["a",null,"b"],["c",["d",null,"e"],"f"]]

map(&@,data) is correct

I have some kind of concern about performance of .NET implementation. This part specifically.
It calculates hash code even if one or both arguments are of type JTokenType.Null.
Would it be nice to first check if they both have type JTokenType.Null or only one of them has?

Area of concern is the instantiation of the JmesPathFunctionFactory.

Please add info to the readme that you need to install the dotnet make tool globally to get it compiled.

First code example should be like:

const string input = "{ \"foo\": \"bar\" }";
const string expression = "foo";

var jmes = new JmesPath();
var result = jmes.Transform(input, expression);

No @-sign.

Querying this:

[
{
"Id": "ab9d98d3-f163-44b4-bb6a-739a82139208",
"SupportCode": "031eb155-cc36-460a-ad6b-164f9b4fd0f8",
"TicketNumber": 1
}
]
with:

[?TicketNumber==`1`].SupportCode | [0]

In c# (dotNET 6.0) with lines:

var jmesPath = new JmesPath();
var result = jmesPath.Transform(json, expression);

results in null.

When running in the 'playground', the result is as shown below:

Thanks.

These lines just trying to use obsolete function itself. Is it right?

Consider the following test:

[Fact]
public void TrimTest()
{
    const string json = "{\"a\": \"b c\"}";
    const string expression = "trim(a, 'c') | trim(@)";
    const string expected = "b";

    Assert(expression, json, expected);
}

This fails because the second argument (characters) is saved into local field and then used for the next invocations. If I change expression to trim(a, 'c') | trim(@, ' '), the test passes.

Hello,

has someone started on supporting System.Text.Json instead of/in addition to Newtonsoft.json? If not, I might give it a go myself.

I've seen #24 where @atifaziz has already laid the groundwork by separating out the parser from the rest.

Is your feature request related to a problem? Please describe.

I want to minimise dependencies in my project by utilising framework dependencies wherever possible

Describe the solution you'd like

I want the package to not have an explicit dependency on dependencies which can be provided by the framework ie System.Reflection.TypeExtensions, System.ValueTuple

Describe alternatives you've considered

Accept the additional dependency

Additional context

No response

I want this:
[
{
"index": 0,
"name": "a",
"state": "up"
},
{
"index": 1,
"name": "b",
"state": "down"
},
{
"index": 2,
"name": "c",
"state": "up"
}
]

From this:
{
"people": [
{
"name": "a",
"state": {"name": "up"}
},
{
"name": "b",
"state": {"name": "down"}
},
{
"name": "c",
"state": {"name": "up"}
}
]
}

But there is no '@.index' to allow this to work:
people[].{"index":@.index,"name":name, "state":state.name}

I realize this is not strictly a query function but AFAIK the items in a JSON list are ordered so it would not be arbitrary to associate a fixed index to each item.

Is there already a way to get this information?

If the Transform() is invoked on different json tokens, and expressions contain root-node expressions ($.foo.bar) it might lead to unexpected results, because the evaluator_'s Root property is updated inside Transform()/Parse() methods. Additional investigation needed.

The package currently targets netstandard 1.3. Some vulnerability checkers are flagging assemblies that are associated with .netstandard 1.3 as having critical vulnerabilities.

Would it be possible to target .netstandard 2.1?
I could provide a PR if you like and are willing to publish a new version.