Add a jq filter about script

[bitfield]

This is a very neat idea! I can imagine doing something similar with XPath, for example. I'm not sure if JQ is the right name for this—how about JSONQuery, perhaps?

[hikhvar]

I'm fine with renaming it to JSONQuery. However there is a JSONQuery DSL available in the NPM ecosystem: https://www.npmjs.com/package/json-query

Also googling JSONQuery lead to a lot of json query languages not equal to the language implemented by jq.

Providing filters for XPath oder yq (yaml equivalent of jq) adds value too. Do we need all those together in the same design?

[bitfield]

Yes, the only problem with JQ is that it doesn't tell you what it does, if you're not already familiar with jq, for example. This is why we have Column, not Cut, and First, not Head. But, like you, I can't think of a better name. JQ is fine for now.

Yes, XPath would be nice, but I'll open a separate issue for that.

Can we write a non-trivial script program which demonstrates the value of a JQ method over, say, just piping stuff through jq using an Exec?

[hikhvar]

I included a non-trivial example in my draft PR #72. As you can see, from a coding point of view they do not differ that much. However from a script life-cycle point of view, the pure go variant has only go-library dependencies. Hence the scripts are easier to distribute since my target system either can compile it from source with a common go tool chain, or I can copy standalone binaries of the compiled script to the target system. If we exec to the jq binary, we have to manage that non-go dependency somehow.

// Get all the slide titles from the httpsbin JSON demo and count the uniq titles.
func main() {
	// Golang only with `JQ` function
	script.HTTP("https://httpbin.org/json").
		JQ(".slideshow.slides[].title").
		Freq().
		Stdout()

	// Call the jq binary
	script.HTTP("https://httpbin.org/json").
		Exec("jq -r '.slideshow.slides[].title'").
		Freq().
		Stdout()
}

https://github.com/bitfield/script/pull/72/files#diff-249585151f561a1112ae0df322cc3d33e2691853deaae9832e0f447daeee36bc

[bitfield]

Yes, that's true: not having JQ means that any Go program that wants to use jq has to make sure it's in the environment somehow.

On the other hand, adding JQ means that every Go program depends on the gojq library whether it actually uses JQ or not! I'm not a big fan of that idea, and you'll notice that right now, the entire script library only depends on two non-standard packages, and one of those is only used for tests. You're asking me to expand the non-test dependency set by 100 percent, for a fairly niche use case.

I'm not saying no, you understand, but I want to be clear in my mind that the benefit is worth the cost. I think I need to see a little bit more benefit at the moment.

[hikhvar]

You argument is perfectly fine. I saw, that I advertised this libraries to my colleagues with the minimal dependency set as a major point :-D

What do you think of the following two filters and sinks:
ExternalFilter(func(p *Pipe) *Pipe) *Pipe
and
ExternalSink(func(p *Pipe) err) err
This allows user to develop there own filters and sinks and still use them in the current fluent interface.

Let's use jq as example.

func JQ(url string) (func (p *Pipe) *Pipe)

External library without ExternalFilter

p := script.File("test.json")
p = JQ(".foo")(p)
p.Stdout()

External library with ExternalFilter

script.File("test.json").ExternalFilter(JQ(".foo")).Stdout()

[bitfield]

That's a really interesting idea! It sounds like it would be a bit similar to EachLine, but for the whole pipe. What would an implementation of ExternalFilter/JQ look like in this example?

[hikhvar]

Updated my draft branch #72 with the ExternalFilter/JQ implementation. The ExternalFilter method is very simple:

func (p *Pipe) ExternalFilter(external func(in *Pipe) *Pipe) *Pipe {
	return external(p)
}

And the corresponding JQ function in the example looks like this:

func JQ(query string) func(in *script.Pipe) *script.Pipe {
	return func(in *script.Pipe) *script.Pipe {
		q, err := gojq.Parse(query)
		if err != nil {
			return in.WithError(err)
		}
		var input map[string]interface{}
		json.NewDecoder(in).Decode(&input)
		output := bytes.NewBuffer(nil)
		iter := q.Run(input)
		for {
			v, ok := iter.Next()
			if !ok {
				return in.WithReader(output)
			}
			if err, ok := v.(error); ok {
				return in.WithError(err).WithReader(output)
			}
			fmt.Fprintf(output, "%v\n", v)
		}
		return in.WithReader(output)
	}
}

The call chain in the example looks like this:

	// Golang only with `JQ` function
	script.HTTP("https://httpbin.org/json").
		ExternalFilter(JQ(".slideshow.slides[].title")).
		Freq().
		Stdout()

[hikhvar]

Added a PR with the implementation given here.

[SleepyBrett]

Don't call something JQ without it actually being 100% compatible and with JQ. Microsoft learned this when they made their powershell http client curl when it wasn't curl.

[bitfield]

@SleepyBrett I'm not sure if I understood what you mean. Are you saying the https://github.com/itchyny/gojq library isn't 100% compatible with JQ? If so, in what respect?

[SleepyBrett]

@bitfield I'm saying I can't tell if it is and I can't tell what they will do with it in the future. If you were to expose this library in your library i'd call it by what it is, gojq and not jq. If there is ever drift between official jq and gojq it could confuse people.

[bitfield]

OK, I understand. Yes, that's a good point, thank you.