ref function does not resolve recursively eval: when it is referencing an object (?)

Suppose we have

{
  "obj": {
    "c": "eval:string:$(ref $(cur).a)",
    "a": "Howdy"
 }
  "key": "eval:object:$(ref $(cur).obj)"
}

In this situation, since .key.c doesn't appear in the list of attributes to be templated initially created, the node will have a value eval:string:$(..., not Howdy.

Probably we should repeat perform_templating function until all the templating keywords (eval: and template:) go away.

The condition to reproduce this issue is not clear yet, though, jq-front sometimes with following message.

jq: error (at <unknown>): Cannot index string with number
ERROR: Failed to merge object nodes:'{"

This hasn't been seen until v0.16.

Following is the stacktrace that jq-front prints.

  }
}': error: /tmp/tmp.bTr8hOOqjN
  at 36 abort /home/hiroshi/opt/jq-front/lib/shared.sh 
  at 99 _merge_object_nodes /home/hiroshi/bin/jq-front 
  at 568 expand_nodelevel_inheritances /home/hiroshi/bin/jq-front 
  at 632 run_jqfront /home/hiroshi/bin/jq-front 
  at 156 _expand_nodelevel_inheritances /home/hiroshi/bin/jq-front 
  at 565 expand_nodelevel_inheritances /home/hiroshi/bin/jq-front 
  at 632 run_jqfront /home/hiroshi/bin/jq-front 
  at 522 expand_filelevel_inheritances /home/hiroshi/bin/jq-front 
  at 624 run_jqfront /home/hiroshi/bin/jq-front 
  at 648 perform_jqfront /home/hiroshi/bin/jq-front 
  at 704 main /home/hiroshi/bin/jq-front 
  at 739 main /home/hiroshi/bin/jq-front 

Following unnecessary message is written to stderr.

jq: error (at <stdin>:1): null (null) has no keys

Found in v0.28

It is necessary and cumbersome to specify the number of levels to climb for the parent built-in function.
That is,

    "eval:$(parent $(parent $(parent $(cur))))"

Instead, I want to be able to write the same thing as follwos

    "eval:$(parent $(cur) 3)"

The default value for the parameter will be 1 and we can keep the backward compatibility.
If you give 0 to the parameter, it will result in the same result as the given path itself.

• Internal Inheritance -> Node Level Inheritance
• External Inheritance -> File Level Inheritance
• Private Nodes -> Local Nodes

A node that references outside with relative(parent) manner cannot be referenced.

{
  "key": "helloWorld",
  "child": {
    "childKey": "eval:string:$(ref $(parent $(cur)).key)"
  }
}

This renders to following.

{
  "key": "helloWorld",
  "child": {
    "childKey": "helloWorld"
  }
}

However, this doesn't and results in an error.

{
  "key": "helloWorld",
  "child": {
    "childKey": "eval:string:$(ref $(parent $(cur)).key)"
  }
  "a": {
    "layer2" : {
      "layer3": "eval:string:$(ref .child.childKey)"
    }
  }
}

Why because, the attribute layer3 is rendered to eval:string:$(ref $(parent $(cur)).key), first.

{
  "key": "helloWorld",
  "child": {
    "childKey": "eval:string:$(ref $(parent $(cur)).key)"
  }
  "a": {
    "layer2" : {
      "layer3": "eval:string:$(ref $(parent $(cur)).key)"
    }
  }
}

And from .a.layer2.layer3, there is no path specified by $(parent $(cur)).key, which is computed to .a.key.
Since jq-front is designed to render text nodes in alphabetical and path length order you can come up with a work around where the rendering for layer3 happens first.
However it is quite an error-prone way and you would not prefer it.
In this situation, it is better to resort to reftag built-in function as follows.

{
  "key": "helloWorld",
  "child": {
    "childKey": "eval:string:$(ref $(parent $(cur)).key)"
  }
  "a": {
    "layer2" : {
      "layer3": "eval:string:$(reftag key)"
    }
  }
}

reftag tries to find a key which has a name specified by the argument given to it by climbing up a node path from current ($(cur)).
Nevertheless, it is a very expensive built-in as of now and it is not advised to use it unnecessarily.

When templating results in a malformed JSON, error message is not helpful.
jq-front only gives an following message.

parse error: Invalid numeric literal at line 2, column 0

Stack trace on the situation is following

DEBUG: error trapped 
  at 6 _debug_abort /home/hiroshi/opt/jq-front/lib/helpers.sh 
  at 102 _render_text_node /home/hiroshi/opt/jq-front/lib/templating.sh 
  at 37 _perform_templating /home/hiroshi/opt/jq-front/lib/templating.sh 
  at 13 perform_templating /home/hiroshi/opt/jq-front/lib/templating.sh 
  at 40 perform_jqfront /home/hiroshi/bin/jq-front 
  at 118 main /home/hiroshi/bin/jq-front 
  at 161 main /home/hiroshi/bin/jq-front 

• Input

{
  "$local": {
    "database_default": {
      "server": {
        "ip": "192.168.1.5",
        "port": 2000
      },
      "db_name": "test",
      "user": {
        "name": "root",
        "password": "root"
      }
    }
  },
  "foo_database": {
    "$extends": "database_default",
    "server": {
      "port": 2001
    },
    "db_name": "foo",
    "user": {
      "password": "foo_root"
    }
  }
}

• Expectation

{
  "foo_database": {
    "server": {
      "ip": "192.168.1.5",
      "port": 2001
    },
    "db_name": "foo",
    "user": {
      "name": "root",
      "password": "foo_root"
    }
  }
}

• Actual output

jq: error (at <stdin>:1): Cannot iterate over string ("database_d...)
{
  "foo_database": {
    "server": {
      "port": 2001
    },
    "db_name": "foo",
    "user": {
      "password": "foo_root"
    }
  }
}

This object results in an empty object {}.

{
  "key1": {
    "key2": {
      "key3": false
    }
  }
}

Following is rendered correctly.

{
  "key":{
    "booleanValue": true
  }
}

Following will achieve it.

  function refexists() {
    # shellcheck disable=SC2181
    [[ $? == 0 ]] || abort "${_error_prefix}Failure was detected."
    local _path="${1}"
    has_value_at "${_path}" "$(cat "$(self)")"
  }

  function reftag() {
    # shellcheck disable=SC2181
    [[ $? == 0 ]] || abort "${_error_prefix}Failure was detected."
    local _tagname="${1}"
    local _curpath
    _curpath="$(curn)"
    while [[ "${_curpath}" != "" ]]; do
    _curpath="$(parent "${_curpath}")"
    if refexists "${_curpath}.${_tagname}";  then
      ref "${_curpath}.${_tagname}"
      return 0
    fi
    done
    abort "${_error_prefix}The specified tag:'${_tagname}' was not found from the current path:'$(curn)'"
  }

Support 'super' built-in function, which allows a user to access attributes in the "super" JSON node.
"super" node can be defined as a node where an empty object extends all the nodes specified by "$extends" in the current node.

Make it possible to use user custom function by following syntax.

{
  "$extends": [
    "SS.sh;SOURCE"
  ],
  "key": "eval:string:hello_world=$(hello_world d),$(echo HELLO)"
}

If SS.sh has a following content,

function hello_world() {
  echo "Hello, world. My Function!"
}

then, following should be rendered.

{
  "key": "hello_world=Hello, world. My Function!,HELLO"
}

Following should work.

{
"a":[1,2,3]
"v":"eval:array:$(ref $(cur).a)"
}

Version number shown by docker version is not correct.

{
  "releaseVersion": "2.12.0",
  "snapshotVersion": "template:string:$(ref .releaseVersion)-SNAPSHOT"
}

{
  "releaseVersion": "2.12.0",
  "snapshotVersion": "template:$(ref .releaseVersion)-SNAPSHOT"
}

These will be rendered to

{
  "releaseVersion": "2.12.0",
  "snapshotVersion": "2.12.0-SNAPSHOT"
}

And

{
  "objectNode": {"hello": "world"},
  "snapshotVersion": "template:object:$(ref .objectNode)"
}

will become

{
  "objectNode": {
    "hello": "world"
  },
  "snapshotVersion": {
    "hello": "world"
  }
}

Right now, references in a string node is expanded in a following order and logic.

0. Before templating, all the file-level and node-level inheritances are expanded and we have a big JSON file.
1. All the nodes in the big JSON file are soreted by the depth and then the name of the nodes in dictionary order.
2. Those nodes are scanned one by one and templating happens for each.

With this mechanism user can know and to some extent control how templating happens in case a node contains $(ref) command.

However, we ultimately want to make it possible to reference a node from another without noticing such mechanism while in case a cyclic reference is warned as an error.

Improve behavior on an error during templating, where a failing command in a templated string is silently ignored.
It should abort the entire process of jq-front.

For instance, following prints the entire file, which can be very huge.

ERROR: Failed to expand node level inheritance for node:'{...

jq-front version: The latest version as of Nov 5, 2019

Reproduce procedure

1. Prepare a jq-front file like follows.
   test.json:

{
    "$local": {
        "nodeA": {
            "aa": "aa"
        },
        "nodeB": {
        }
    },
    "a": [
        {
            "$extends": ["nodeA"],
            "a": "a"
        }
    ]
}

2. Run jq-front with the above file.

./jq-front test.json

Expected

{
    "a": [
        {
            "aa": "aa",
            "a": "a"
        }
    ]
}

Actual

{
    "a": [
        {
            "a": "a"
        }
    ]
}

As long as a yaml can be translated into a json file, it can be included in inheritance tree of jq-front's processing pipeline.
And it would be useful.

The current logic is awkward.
It can be improved as follows.

def path2pexp(v):
  reduce .[] as $segment ("";  .
    + ($segment
       | if type == "string" then ".\"" + . + "\"" else "[\(.)]" end));

paths(scalars_or_empty)|path2pexp(.)

Make processing program configurable through a .rc file.

If you have a file .jq-front.rc with following content,

function resolve_processor() {
  local _absfile="${1}"
  local _processor=""
  if is_localnode "${_absfile}" || [[ ${_absfile} == *.json ]]; then
    _processor="jq ."
  elif [[ "${_absfile}" == *.yaml || "${_absfile}" == *.yml ]]; then
    _processor="yaml2json"
  elif [[ "${_absfile}" == *.sh ]]; then
    _processor="bash -eu"
  elif [[ "${_absfile}" == *.py ]]; then
    _processor="python"
  fi
  echo "${_processor}"
}

you become able to use python not only JSON, yaml, and bash script files.

string node starts with 'template:' doesn't complain of circular dependency.

• Use this tool: https://www.npmjs.com/package/yamljs to avoid installing python and pip, which consume >400MB
• Maybe time to quit ubuntu for base image.
• ajv -> try using ajv-package to avoid installing npm and entire ajv

In node-level inheritances, overriding priority is not consistent with file-level's and not correct.

{
  "child": {
    "$extends": [
      "A.json",
      "B.json"
    ]
  }
}

Should become

{
  "child": {
    "b": "B",
    "o": "A",
    "a": "A"
  }
}

But it gives

{
  "child": {
    "b": "B",
    "o": "B",
    "a": "A"
  }
}

This issue was found in until v0.37

I have a following key-value.

{
  "hello": "HELLO",
  "world": "WORLD",
  "value": "eval: $(ref .hello) $(echo $(ref .world))"
}

The intention is to render following element.

 "value": "HELLO $(echo WORLD)"

However, jq-front renders this into :

 "value": "HELLO WORLD"

Because it just considers the given data as a nested command substitution.
How can I define the element to get the desired string?

Improve logic to merge objects.
Rather than simple multiplication (*), following logic does a better job with regard to handling elements inside arrays.

def value_at($n; $p):
  $n | getpath($p);

def merge_objects($a; $b):
  $a | [paths(scalars_or_empty)]
     | reduce .[] as $p ($b; setpath($p; value_at($a; $p)));

merge_objects({
  "k":"k-a",
  "k-a": "k-a",
  "arr":[{"arro": "arro-a", "arro-a":"arro-a"}, {}, {}],
  "o":{
    "oo":"oo-a",
    "oo-a":"oo-a"
  }
}; {
  "k":"k-b",
  "k-b": "k-b",
  "arr":[{"arro": "arro-b", "arro-b":"arro-b"}, {"arro1":"arro1-b"}],
  "o":{
    "oo":"oo-b",
    "oo-b":"oo-b"
  }
})

Provide a built-in function that performs multi-line replacement based on provided regex.

function replace_multilines() {
  local string_before_replacement="${1}"
  local begin_regex="${2}"
  local end_regex="${3}"
  local replacement="${4}"
  ...
  echo "${string_after_replacement}"
}

replace_multilines 'function hi_there() {
  echo "hi there"
}' 'hi_[a-z]+\(\)' '\}' 'hi_there() { echo "HI THERE" }'

Results in

function hi_there() { echo "HI THERE"}

I try to run the following json. But, I got error. How can I express for getting expectation?

json source
{ "k":"eval:string:\"'\\\"'\"" }

expectaion
{"k":"'\"'"}

Since the docker mode execution has a significant performance overhead, we need to provide a way to process a number of files in batch.

Followiing doesn't render as expected.

{
  "key": {
   "hello world":"eval:string:$(parent $(curn))"
  }
}

Expectation is following.

{
  "key": {
   "hello world":".\"key\""
  }
}

And results in this.

{
  "key": {
   "hello world":""
  }
}

In order to avoid naming collision, I have renamed this project from 'powerjson' to 'jf'.
However, I found that there are quite a few projects in GitHub named 'jf'....

Let's rename it again to 'jq-front'.

By introducing following syntax, make it possible to inherit an optional file.

"$extends": [ "optionalFile.json?" ]

When a file name ends with ?, extend it if it exists, otherwise simply ignore it without throwing an exception.

JF_PATH is not respected as designed when jf is used without docker