JSON Slice
What is it?
JSON Slice is a Go package which allows to execute fast jsonpath queries without unmarshalling the whole data.
Sometimes you need to get a single value from incoming json using jsonpath, for example to route data accordingly or so. To do that you must unmarshall the whole data into interface{} struct and then apply some jsonpath library to it, only to get just a tiny little value. What a waste of resourses! Well, now there's jsonslice
.
Simply call jsonslice.Get
on your raw json data to slice out just the part you need. The []byte
received can then be unmarshalled into a struct or used as it is.
Getting started
1. install
$ go get github.com/bhmj/jsonslice
2. use it
import "github.com/bhmj/jsonslice"
import "fmt"
func main() {
var data = []byte(`
{ "sku": [
{ "id": 1, "name": "Bicycle", "price": 160, "extras": [ "flashlight", "pump" ] },
{ "id": 2, "name": "Scooter", "price": 280, "extras": [ "helmet", "gloves", "spare wheel" ] }
]
} `)
a, _ := jsonslice.Get(data, "$.sku[0].price")
b, _ := jsonslice.Get(data, "$.sku[1].extras.count()")
c, _ := jsonslice.Get(data, "$.sku[?(@.price > 200)].name")
d, _ := jsonslice.Get(data, "$.sku[?(@.extras.count() < 3)].name")
fmt.Println(string(a)) // 160
fmt.Println(string(b)) // 3
fmt.Println(string(c)) // ["Scooter"]
fmt.Println(string(d)) // ["Bicycle"]
}
Package functions
jsonslice.Get(data []byte, jsonpath string) ([]byte, error)
- get a slice from raw json data specified by jsonpath
Specs
See Stefan Gössner's article for original specs and examples.
Syntax features
-
Classic dot notation (
$.simple_key
) is limited to alphanumeric characters. For more complex cases use$['complex key!']
or$.'complex key!'
. -
A single index reference returns an element, not an array; a slice reference always returns array:
> echo '[{"id":1}, {"id":2}]' | ./jsonslice '$[0].id'
1
> echo '[{"id":1}, {"id":2}]' | ./jsonslice '$[0:1]
[1]
- Indexing or slicing on root node is supported (assuming json is an array and not an object):
./jsonslice '$[0].author' sample1.json
Expressions
Overview
$ -- root node (can be either object or array)
.node -- dot-notated child
.'some node' -- dot-notated child (syntax extension)
['node'] -- bracket-notated child
['foo','bar'] -- bracket-notated children (aggregation)
[5] -- array index
[-5] -- negative index means "from the end"
[1:9] -- array slice
[1:9:2] -- array slice (+step)
.* .[*] .[:] -- wildcard
..key -- deepscan
Functions
$.obj.length() -- number of elements in an array or string length, depending on the obj type
$.obj.count() -- same as above
$.val.size() -- value size in bytes (as is)
Slices
$.arr[start:end:step]
$.arr[start:end]
Selects elements from start
(inclusive) to end
(exclusive), stepping by step
. If step
is omitted or zero, then 1 is implied. Out-of-bounds values are reduced to the nearest bound.
If step
is positive:
- empty
start
treated as the first element inclusive - empty
end
treated as the last element inclusive start
should be less thenend
, otherwise result will be empty
If step
is negative:
- empty
start
treated as last element inclusive - empty
end
treated as the first element inclusive start
should be greater thenend
, otherwise result will be empty
Filters
[?(<expression>)] -- filter expression. Applicable to arrays only
@ -- the root of the current element of the array. Used only within a filter.
@.val -- a field of the current element of the array.
Filter operators
Operator | Description |
---|---|
== |
Equal to Use single or double quotes for string expressions. [?(@.color=='red')] or [?(@.color=="red")] |
!= |
Not equal to[?(@.author != "Herman Melville")] |
> |
Greater than[?(@.price > 10)] |
>= |
Greater than or equal to |
< |
Less than |
<= |
Less than or equal to |
=~ |
Match a regexp[?(@.name =~ /sword.*/i] |
!~ or !=~ |
Don't match a regexp[?(@.name !~ /sword.*/i] |
&& |
Logical AND[?(@.price < 10 && @isbn)] |
|| |
Logical OR[?(@.price > 10 || @.category == 'reference')] |
Examples
Assuming sample0.json
and sample1.json
in the example directory:
cat sample0.json | ./jsonslice '$.store.book[0]'
cat sample0.json | ./jsonslice '$.store.book[0].title'
cat sample0.json | ./jsonslice '$.store.book[0:-1]'
cat sample1.json | ./jsonslice '$[1].author'
cat sample0.json | ./jsonslice '$.store.book[?(@.price > 10)]'
cat sample0.json | ./jsonslice '$.store.book[?(@.price > $.expensive)]'
Much more examples can be found in jsonslice_test.go
Benchmarks (Core i5-7500)
$ go test -bench=. -benchmem -benchtime=4s
goos: linux
goarch: amd64
pkg: github.com/bhmj/jsonslice
++ here's a couple of operations usually needed to get an object by jsonpath (for reference):
Benchmark_Unmarshal-4 500000 14712 ns/op 4368 B/op 130 allocs/op
Benchmark_Oliveagle_Jsonpath-4 5000000 1560 ns/op 496 B/op 27 allocs/op
++ and here's a jsonslice.Get:
Benchmark_Jsonslice_Get_Simple-4 2000000 3878 ns/op 128 B/op 4 allocs/op
++ Get() involves parsing a jsonpath, here it is:
Benchmark_JsonSlice_ParsePath-4 10000000 858 ns/op 160 B/op 5 allocs/op
++ in case you aggregate some non-contiguous elements, it may take a bit longer (extra mallocs involved):
Benchmark_Jsonslice_Get_Aggregated-4 1000000 5671 ns/op 417 B/op 13 allocs/op
++ usual unmarshalling a large json:
Benchmark_Unmarshal_10Mb-4 100 50744817 ns/op 248 B/op 5 allocs/op
++ jsonslicing the same json, target element is near the start:
Benchmark_Jsonslice_Get_10Mb_First-4 3000000 1851 ns/op 128 B/op 4 allocs/op
++ jsonslicing the same json, target element is near the end: still beats Unmarshal
Benchmark_Jsonslice_Get_10Mb_Last-4 200 38286509 ns/op 133 B/op 4 allocs/op
PASS
ok github.com/bhmj/jsonslice 83.152s
Changelog
1.0.3 (2019-12-24) -- bugfix: $[0].foo
[{"foo":"\\"}]
generated "unexpected end of input"
1.0.2 (2019-12-07) -- nested aggregation ($[:].['a','b']
) now works as expected. TODO: add option to switch nested aggregation mode at runtime!
1.0.1 (2019-12-01) -- "not equal" regexp operator added (!=~
or !~
).
1.0.0 (2019-11-29) -- deepscan operator (..
) added, slice with step $[1:9:2]
is now supported, syntax extensions added. GetArrayElements()
removed.
0.7.6 (2019-09-11) -- bugfix: escaped backslash at the end of a string value.
0.7.5 (2019-05-21) -- Functions count()
, size()
, length()
work in filters.
$.store.bicycle.equipment[?(@.count() == 2)]
->[["light saber", "apparel"]]
0.7.4 (2019-03-01) -- Mallocs reduced (see Benchmarks section).
0.7.3 (2019-02-27) -- GetArrayElements()
added.
0.7.2 (2018-12-25) -- bugfix: closing square bracket inside a string value.
0.7.1 (2018-10-16) -- bracket notation is now supported.
$.store.book[:]['price','title']
->[[8.95,"Sayings of the Century"],[12.99,"Sword of Honour"],[8.99,"Moby Dick"],[22.99,"The Lord of the Rings"]]
0.7.0 (2018-07-23) -- Wildcard key (*
) added.
$.store.book[-1].*
->["fiction","J. R. R. Tolkien","The Lord of the Rings","0-395-19395-8",22.99]
$.store.*[:].price
->[8.95,12.99,8.99,22.99]
0.6.3 (2018-07-16) -- Boolean/null value error fixed.
0.6.2 (2018-07-03) -- More tests added, error handling clarified.
0.6.1 (2018-06-26) -- Nested array indexing is now supported.
$.store.bicycle.equipment[1][0]
->"peg leg"
0.6.0 (2018-06-25) -- Regular expressions added.
$.store.book[?(@.title =~ /(dick)|(lord)/i)].title
->["Moby Dick","The Lord of the Rings"]
0.5.1 (2018-06-15) -- Logical expressions added.
$.store.book[?(@.price > $.expensive && @.isbn)].title
->["The Lord of the Rings"]
0.5.0 (2018-06-14) -- Expressions (aka filters) added.
$.store.book[?(@.price > $.expensive)].title
->["Sword of Honour","The Lord of the Rings"]
0.4.0 (2018-05-16) -- Aggregating sub-queries added.
$.store.book[1:3].author
->["John","William"]
0.3.0 (2018-05-05) -- MVP.
Roadmap
- length(), count(), size() functions
- filters: simple expressions
- filters: complex expressions (with logical operators)
- nested arrays support
- wildcard operator (
*
) - bracket notation for multiple field queries (
$['a','b']
) - deepscan operator (
..
) - syntax extensions:
$.'keys with spaces'.price
- flexible syntax:
$[0]
works on both[1,2,3]
and{"0":"abc"}
- Optionally unmarshal the result
- Option to select aggregation mode (nested or plain)
Contributing
- Fork it!
- Create your feature branch:
git checkout -b my-new-feature
- Commit your changes:
git commit -am 'Add some feature'
- Push to the branch:
git push origin my-new-feature
- Submit a pull request :)
Licence
Author
Michael Gurov aka BHMJ