Comments (13)
This is a very neat idea! I can imagine doing something similar with XPath, for example. I'm not sure if JQ
is the right name for this—how about JSONQuery
, perhaps?
from script.
I'm fine with renaming it to JSONQuery. However there is a JSONQuery DSL available in the NPM ecosystem: https://www.npmjs.com/package/json-query
Also googling JSONQuery lead to a lot of json query languages not equal to the language implemented by jq
.
Providing filters for XPath oder yq (yaml equivalent of jq) adds value too. Do we need all those together in the same design?
from script.
Yes, the only problem with JQ
is that it doesn't tell you what it does, if you're not already familiar with jq
, for example. This is why we have Column
, not Cut
, and First
, not Head
. But, like you, I can't think of a better name. JQ
is fine for now.
Yes, XPath would be nice, but I'll open a separate issue for that.
Can we write a non-trivial script
program which demonstrates the value of a JQ
method over, say, just piping stuff through jq
using an Exec
?
from script.
I included a non-trivial example in my draft PR #72. As you can see, from a coding point of view they do not differ that much. However from a script life-cycle point of view, the pure go variant has only go-library dependencies. Hence the scripts are easier to distribute since my target system either can compile it from source with a common go tool chain, or I can copy standalone binaries of the compiled script to the target system. If we exec to the jq
binary, we have to manage that non-go dependency somehow.
// Get all the slide titles from the httpsbin JSON demo and count the uniq titles.
func main() {
// Golang only with `JQ` function
script.HTTP("https://httpbin.org/json").
JQ(".slideshow.slides[].title").
Freq().
Stdout()
// Call the jq binary
script.HTTP("https://httpbin.org/json").
Exec("jq -r '.slideshow.slides[].title'").
Freq().
Stdout()
}
from script.
Yes, that's true: not having JQ
means that any Go program that wants to use jq
has to make sure it's in the environment somehow.
On the other hand, adding JQ
means that every Go program depends on the gojq
library whether it actually uses JQ or not! I'm not a big fan of that idea, and you'll notice that right now, the entire script
library only depends on two non-standard packages, and one of those is only used for tests. You're asking me to expand the non-test dependency set by 100 percent, for a fairly niche use case.
I'm not saying no, you understand, but I want to be clear in my mind that the benefit is worth the cost. I think I need to see a little bit more benefit at the moment.
from script.
You argument is perfectly fine. I saw, that I advertised this libraries to my colleagues with the minimal dependency set as a major point :-D
What do you think of the following two filters and sinks:
ExternalFilter(func(p *Pipe) *Pipe) *Pipe
and
ExternalSink(func(p *Pipe) err) err
This allows user to develop there own filters and sinks and still use them in the current fluent interface.
Let's use jq
as example.
func JQ(url string) (func (p *Pipe) *Pipe)
External library without ExternalFilter
p := script.File("test.json")
p = JQ(".foo")(p)
p.Stdout()
External library with ExternalFilter
script.File("test.json").ExternalFilter(JQ(".foo")).Stdout()
from script.
That's a really interesting idea! It sounds like it would be a bit similar to EachLine
, but for the whole pipe. What would an implementation of ExternalFilter/JQ
look like in this example?
from script.
Updated my draft branch #72 with the ExternalFilter/JQ
implementation. The ExternalFilter
method is very simple:
func (p *Pipe) ExternalFilter(external func(in *Pipe) *Pipe) *Pipe {
return external(p)
}
And the corresponding JQ
function in the example looks like this:
func JQ(query string) func(in *script.Pipe) *script.Pipe {
return func(in *script.Pipe) *script.Pipe {
q, err := gojq.Parse(query)
if err != nil {
return in.WithError(err)
}
var input map[string]interface{}
json.NewDecoder(in).Decode(&input)
output := bytes.NewBuffer(nil)
iter := q.Run(input)
for {
v, ok := iter.Next()
if !ok {
return in.WithReader(output)
}
if err, ok := v.(error); ok {
return in.WithError(err).WithReader(output)
}
fmt.Fprintf(output, "%v\n", v)
}
return in.WithReader(output)
}
}
The call chain in the example looks like this:
// Golang only with `JQ` function
script.HTTP("https://httpbin.org/json").
ExternalFilter(JQ(".slideshow.slides[].title")).
Freq().
Stdout()
from script.
Added a PR with the implementation given here.
from script.
Don't call something JQ without it actually being 100% compatible and with JQ. Microsoft learned this when they made their powershell http client curl when it wasn't curl.
from script.
@SleepyBrett I'm not sure if I understood what you mean. Are you saying the https://github.com/itchyny/gojq library isn't 100% compatible with JQ? If so, in what respect?
from script.
@bitfield I'm saying I can't tell if it is and I can't tell what they will do with it in the future. If you were to expose this library in your library i'd call it by what it is, gojq and not jq. If there is ever drift between official jq and gojq it could confuse people.
from script.
OK, I understand. Yes, that's a good point, thank you.
from script.
Related Issues (20)
- gitignore entry for Intellij IDEA HOT 4
- Feature Request: Add FilterFunc HOT 2
- Basename introduces a line feed HOT 1
- How can I check the progress of a script.Exec() command? HOT 2
- Interaction with bubbletea HOT 10
- Propose adding a delimeter to Column()
- Add Base64 encode and decode methods HOT 6
- Suggestion: `./go-script -c 'script.Stdin()...'` tool for running these HOT 5
- Should `ExecForEach` sanitize or escape arguments differently? HOT 9
- Parameterize a script.Pipe with a user data struct ? HOT 5
- Execv function HOT 1
- how to get the stdout of the Exec result? HOT 3
- Race condition in Pipe.Exec HOT 4
- Methods for mkdir and touch HOT 2
- Conditinally execute statements. HOT 2
- How to create folder, delete folder and list folder contents ? HOT 4
- Feature Suggestion: Fields HOT 4
- Makefile replacement HOT 2
- Expand prompt logic for PR #199
- Easy Utility Functions to Create Symbolic/Hard Links HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from script.