dvershinin / sde

Structured data editor for CLI

sde

sde is not sed. It's a structured data editor for CLI.

Why?

Many people asked for a simple JSON in-place editing and jq was the solution:

• Modify a key-value in a json using jq in-place
• How to modify a key's value in a JSON file from command line

jq '.address = "abcde"' test.json|sponge test.json

Does this seem readable or elegant to you?

How about this instead:

sde address abcde test.json

sde is not a substitute for jq or sed.

It allows simple in-place JSON/YAML value changes, for structured data.

Sample JSON

{
   "name":"John",
   "age":31,
   "city":"New York",
   "extra": {
       "gender": null
   }
}

Sample YAML

database:
  user: example
  password: secret

Modify data

sde name Jack data.json
sde extra.gender male data.json
sde database.user john data.yml

It is possible to modify data in arrays using a dotted notation. Let's take another sample:

{
    "users": [
        {
            "username": "foo", 
            "enabled": true
        },
        {
            "username": "bar", 
            "enabled": true
        }      
    ],
}

We can set the first user's enabled property to false:

sde users.0.enabled false data.json

Installation for CentOS/RHEL 7, 8 or Amazon Linux 2, or Fedora Linux

sudo yum -y install https://extras.getpagespeed.com/release-latest.rpm
sudo yum -y install sde

Installation for other systems

Installing with pip is easiest:

pip install sde

Notes

Quoting in JSON

Quoting is avoided for null, true, false, and numeric values. To ensure that a given value is quoted, use -s (or --string) option:

sde -s key null file.json

Force-fail on missing keys

If you must edit the file, by ensuring to update only the existing key, use -e (--must-exist) option. The program will exit without adding the key which doesn't exist.

sde -e key val file.json

Force-fail on unchanged file

If the data is unchanged after running sde (values already match), you can force a failure exit code 2 by passing the -m option:

sde -m key sameval file.json
# > exit code 0
sde -m key sameval file.json
# > exit code 2

TODO

Work with stdin

echo $json | sde name Jack

Query simple data

sdg name data.json

sde's Issues

INI-style files support

• Detect ini file by contents, since often .cfg or .conf are ini files
• Choose a library that preserves comments,e.g. https://github.com/pyscaffold/configupdater (that means dropping Python 2 support... or make the feature non-op for Python 2)

Example, editing DNF config:

sde main.best True /etc/dnf/dnf.conf

Add option to support JSON or YAML-encoded replacement values

I have a use case in which I want to set a value to some already-encoded structure. I extract the structure I need from some source using jq or jp, and then want to set that as the value under a key in another JSON file using sde. I tried that with the current code, but it encodes my replacement JSON data as a single string value. That's completely safe and reasonable behavior, but not what I need in this case.

May I suggest adding an option, perhaps --value_type=json or --value_type=yaml, that would treat the value data according to the specified type before updating the target file? This seems sort of like a generalization of your existing --string option, but allowing for other structured interpretations of the provided replacement value.

I've not collaborated on public GItHub projects before but I'd be happy to try this myself and submit a pull request if you're amenable.

Thank you for your consideration and for sharing this work!

floats are converted to strings

sde latitude 34 file.json stores 34

sde latitude 34.2 file.json stores "34.2", when I was expecting 34.2 without quotes.

yaml support?

What do you think about yaml support? Seems like it would be straightforward to add without much additional support overhead.

Checking before I do a pull request...

Add -v to report version information

Thanks for fixing my previous bug. As Its not obvious how to use the new version at the command line, I'll be waiting for the pip version to be updated, and a -v option to report the software version would be handy to track that.

John

How to put null value via SDE

This issue is pretty simple, how do you insert null, into json via sde?

Steps:
1 - sde Foo null file.json

Result

{
   Foo = "null" <--- notice string paranticies
}

Expected

{
   Foo = null
}

There should be syntax to add null into the json, preferably string null should be converted to null object.

UnboundLocalError: local variable 'val' referenced before assignment

-s doesn't work.

val is only defined if -s is not true:

  if not args.is_string:
        val = normalize_val(args.val)
        

(virtualenv-3.6.8) <perfdash-web1:/opt/perfdash/perfdash.git/bin>$ sde -s logs foo foo.yaml
Traceback (most recent call last):
  File "/opt/perfdash/perfdash.git/virtualenv-3.6.8/bin/sde", line 8, in <module>
    sys.exit(main())
  File "/opt/perfdash/perfdash.git/virtualenv-3.6.8/lib64/python3.6/site-packages/sde/sde.py", line 114, in main
    edit_file(args.key, val, args.file, fmt, must_exist=args.must_exist)
UnboundLocalError: local variable 'val' referenced before assignment

Replace json object values in large blobs

Hey! I have a JSON file that has about 32000 objects. Is there a way that I could find a particular object based on a Key-value and replace it using sde?

Example:

{
"directDependencies" : [ ],
"hash" : "358620021da9f373601c1d02581b3fac",
"name" : "target1"
}

I would want to find the JSON object by the name and replace the hash in the blob.