spraakbanken / json-arrays-py

json-arrays

Read and write JSON lazy, especially json-arrays.

Handles both the JSON format:

[
  {
    "a": 1
  },
  {
    "a": 2
  }
]

As well as JSON LINES format:

{"a":1}
{"a": 2}

Also supports streaming from gzipped files.

Uses orjson if present, otherwise standard json.

Usage

Installation

# Using standard json
pip install json-arrays

# Using orjson
pip install json-arrays[orjson]

Note

This library prefers files opened in binary mode. Therefore does all dumps-methods return bytes.

All loads methods handles str, bytes and bytesarray arguments.

Examples

Allows you to use json.load and json.dump with both json and json-lines files as well as dumping generators.

import json_arrays

# This command tries to guess format and opens the file
data = json_arrays.load_from_file("data.json") # or data.jsonl

# Write to file, again guessing format
json_arrays.dump_to_file(data, "data.jsonl")

from json_arrays import json_iter, jsonl_iter

# Open and read the file without guessing
data = json_iter.load_from_file("data.json")

# Process file

# Write to file without guessing
jsonl_iter.dump_to_file(data, "data.jsonl")

import json_arrays
def process(data):
    for entry in data:
        # process
        yield entry

def read_process_and_write(filename_in, filename_out):

    json_arrays.dump_to_file(
        process(
            json_arrays.load_from_file(filename_in)
        ),
        filename_out
    )

You can also use json_arrays as a sink, that you can send data to.

import json_arrays

with open("out.json", "bw") as fp:
  # guessing format
  with json_arrays.sink(fp) as sink:
    for data in data_source():
      sink.send(data)

Release Notes

This projects keeps a CHANGELOG.

Development

This project uses pdm. After cloning the repo, just run

make dev
make test

to setup a virtual environment, install dev dependencies and run the unit tests.

Note: If you run the command in a activated virtual environment, that environment is used instead.

Deployment

Push a tag in the format v\d+.\d+.\d+to main-branch, to build & publish package to PyPi.

json-arrays-py's Issues

Make it possible to serialize decimal from json_arrays

@pytest.mark.parametrize(
    "json_format", [json_arrays.JsonFormat.JSON, json_arrays.JsonFormat.JSON_LINES]
)
def test_dump_decimal(
    json_format: json_arrays.JsonFormat, snapshot, array_of_dicts: list[dict]
) -> None:
    with tempfile.TemporaryFile() as fp:
        json_arrays.dump([10.20, "10.20", Decimal("10.20")], fp, json_format=json_format)
        fp.seek(0)
        data_written = fp.read()

    assert data_written == snapshot

Use snapshot tests

Make files.BinaryFile a contextmanager

Add doctests

Add sinks for writing arrays to json and jsonl

Add the following:

• json_iter.array_sink(fp)
• jsonl_iter.array_sink(fp)
• json_iter.array_sink_from_filename(filename)
• jsonl_iter.array_sink_from_filename(filename)
• json_streams.array_sink(fp)
• json_streams.array_sink_from_filename(filename)

Add support for python 3.12

Make it possible to stream bz2 files

• https://docs.python.org/3/library/bz2.html
• https://stackoverflow.com/questions/30142112/decompress-streaming-bz2-from-memory-in-python
• https://stackoverflow.com/questions/27508442/unpack-bz2-url-without-temporary-file-in-python

Strengthen CI security

Add documentation to API

Add support for writing and reading gzipped files

https://stackoverflow.com/questions/39450065/python-3-read-write-compressed-json-objects-from-to-gzip-file

Write documentation for library

Make it possible to stream brotli files

Add benches

Support file extension ndjson

Automatically recognise new-line delimited JSON (NDJSON) .ndjson file names.

Error when parsing JSON with floats

In version 0.12.3, parsing a JSON file adds Decimal to the result. This does not happen for JSONL.

>>> list(json_streams.load_from_file("test.json"))
[{'value': Decimal('1.2')}]
>>> list(json_streams.load_from_file("test.jsonl"))
[{'value': 1.2}]

Contents of test.json:

[{"value": 1.2}]

Contents of test.jsonl:

{"value": 1.2}

Combine readers for JSON and NDJSON/JSONL

Minimize duplicated code.

• Evaluate running time.

Combine Writers for JSON and NDJSON/JSONL

Minimized duplicated code.