torfsen / barely_json

A Python parser for data that only looks like JSON

Barely JSON

A Python package for parsing data that only looks like JSON

from barely_json import parse
print(parse('[what is this?, , {perhaps, json: no}]'))

# Prints ['what is this?', '', {'perhaps': '', 'json': False}]

Quite a bit of data looks like JSON at a first glance but turns out not to comply completely with the JSON specification -- often because the exporting software is broken, but sometimes simply because the format was never intended to be JSON in the first place.

No matter how you ended up with the data, now you want to parse it! However, most JSON parsers are pretty strict, so you're out of luck with your JSON-esque mess.

That's where Barely JSON steps in and tries to parse anything that remotely looks like JSON. In addition to the pure parsing, Barely JSON will also try to post-process your data into suitable Python types even if your data provider uses, for example, on and off as boolean literals.

Installation

The supported Python versions are 3.7 and later.

pip install barely_json

Usage

The main routine is parse:

> from barely_json import parse
> parse("[NaN, , {state: off, where's my value?}, NULL]")

[nan, '', {'state': False, "where's my value?": ''}, None]

As you can see, parse by default tries to convert values that are illegal in JSON into hopefully appropriate Python types, which often works well. But sometimes that's not what you want, so you can disable the auto-conversion:

> parse("[NaN, , {state: off, where's my value?}, NULL]", resolver=None)

[<IllegalValue 'NaN'>,
 <IllegalValue ''>,
 {<IllegalValue 'state'>: <IllegalValue 'off'>,
  <IllegalValue "where's my value?">: <IllegalValue ''>},
 <IllegalValue 'NULL'>]

In that case any value that's illegal or missing is wrapped in an instance of a special IllegalValue class. You can also provide your own resolver for illegal values, which is simply a callback that maps strings to arbitrary values:

> from barely_json import default_resolver
>
> def my_resolver(text):
>     if text.lower() == 'one':
>         return 1
>     return default_resolver(text)
>
> parse('[one, FALSE]', resolver=my_resolver)

[1, False]

When writing your own resolver it's often handy to fall back todefault_resolver after you've handled your special cases.

Change Log

See CHANGELOG.md.

License

Distributed under the MIT license. See the file LICENSE for details.

Contributors

• @torfsen
• @tusharmakkar08
• @gvx

Development

Clone the repository:

git clone https://github.com/torfsen/barely_json.git
cd barely_json

Install the development dependencies

pip install -r requirements-dev.txt

Run the tests:

tox

For pull requests, the tests are run using GitHub actions.

barely_json's Issues

JSON with no delimiter

Hey there,

It's a great library you made :-) And it is a good idea to base it on a grammar processor instead of a simple homemade state machine.

I would like to make the library work on malformed JSON with absolutely no delimiter:

badjson = '''{"ghosting_calculation": {"process_types": ["T1"  "T2"  "FLAIR"  "SWI" ]    "mask_ap": "qc_templates/maskAP.nii.gz"  "scan_type_order": ["T1"  "FLAIR"  "T2" ]    "mask_side": "qc_templates/maskSide.nii.gz"}  "segmentation": {"scan_type_order": ["T1"  "FLAIR"  "T2" ]    "brain_mask": "qc_templates/wbrainmask.nii.gz"}  "dicom2nifti": {"process_types": ["T1"  "T2"  "FLAIR"  "SWI"  "DTI"  "b0"  "rs_fMRI" ]    "deface": true  "deface_types": ["T1"  "T2"  "FLAIR"  "SWI"  "T2*" ]    "overwrite": false}  "tbi_dti_qc": {"check_gradient_table": {"process_types": ["DTI" ]  }  "dti_tensor_residual_calculation": {"process_types": ["DTI" ]  }  "temporal_snr": {"process_types": ["DTI" ]    "brain_mask": "qc_templates/wbrainmask.nii.gz"}  "percentage_signal_change": {"process_types": ["DTI" ]    "brain_mask": "qc_templates/wbrainmask.nii.gz"  "mask_threshold": 0.9}  "input_data": {"process_types": ["DTI" ]    "data_types": ["NIFTI"  "BVAL"  "BVEC" ]  }  "fa_md_feature_estimation": {"process_types": ["DTI" ]  }}  "dicom_qc": {"process_types": ["T2"  "T1"  "FLAIR"  "DTI"  "SWI"  "rs_fMRI" ]    "center": "dicom_qc"  "server_name": "neuro-imaging.center-tbi.eu"}  "smoothed_image_statistics": {"process_types": ["T1"  "T2"  "FLAIR" ]  }  "cnr_calculation": {"process_types": ["T1"  "T2"  "FLAIR"  "DTI" ]    "noise_mask": "qc_templates/maskNoise.nii.gz"  "mask_threshold": 0.9}  "scan_type_translation": {"server_name": "neuro-imaging.center-tbi.eu"}  "input_data": {"proce ss_types": ["T1"  "T2"  "FLAIR"  "SWI"  "DTI"  "b0"  "rs_fMRI" ]    "data_types": ["DICOM"  "NIFTI"  "BVAL"  "BVEC" ]  }  "tbi_report": {}  "tbi_fmri_qc": {"fmri_motion_detection": {"process_types": [  "rs_fMRI" ]    "translation_threshold": 3  "rotation_threshold": 3}  "input_data": {"process_types": ["rs_fMRI" ]    "data_types": ["NIFTI" ]  }  "percentage_signal_change": {"process_types": ["rs_fMRI" ]    "brain_mask": "qc_templates/wbrainmask.nii.gz"  "mask_threshold": 0.9}  "temporal_snr": {"process_types": ["rs_fMRI" ]    "brain_mask": "qc_templates/wbrainmask.nii.gz"}  "fmri_snr_calculation": {"process_types": ["rs_fMRI" ]  }}  "head_coverage": {"process_types": ["T1"  "T2"  "FLAIR"  "SWI"  "DTI"  "b0"  "rs_FMRI" ]    "atlas_mask": "NormalSharpAtlas/mni_icbm152_nlin_asym_09a/mni_icbm152_t1_tal_nlin_asym_09a_mask.nii.gz"}  "registration": {"process_types": ["T1"  "T2"  "FLAIR"  "SWI"  "DTI"  "b0"  "rs_fMRI" ]    "scan_type_order": ["T1"  "FLAIR"  "T2" ]  }  "gm_wm_statistics": {"process_types": ["T1"  "T2"  "FLAIR" ]  }  "snr_calculation": {"process_types": ["T1"  "T2"  "FLAIR"  "DTI" ]    "noise_mask": "qc_templates/maskNoise.nii.gz"  "mask_types": {"SWI": "CSF"  "DTI": "CSF"  "T2": "CSF"  "T1": "WM"  "B0": "CSF"  "FLAIR": "GM"}  "mask_threshold": 0.9}  "upload_results": {}  "protocol_check": {"validate_order": true  "sequence": ["T2"  "T1"  "FLAIR"  "SWI"  "DTI"  "DTI"  "rs_fMRI" ]  }}'''

This peculiar bug might happen when the host software generating this json tries to export this json data as one field of a CSV file export with the default comma delimiter, in this case, all commas will inside the JSON field will be stripped out.

Do you have any idea how to fix that?

Thanks a lot!

Floats are not parsed correctly

Allow parsing "json" which is wrapped in "[ ]", not "{ }"

Would it be possible the library to parse json which is wrapped in [ ], not { }, like this file (from the settings file of Sublime Text, if you're interested)):

[
    {
        "caption": "Word Wrap: Toggle",
        "command": "toggle_setting",
        "args": {"setting": "word_wrap"}
    },
    {
        "caption": "Convert Case: Upper Case",
        "command": "upper_case"
    },
    { "caption": "Indexing Status", "command": "show_progress_window" },
]

String escapes are not parsed correctly

AttributeError: module 'collections' has no attribute 'MutableMapping'

I'm running Python 3.10.6 on Ubuntu, and trying out barely_json for the first time. I just installed barely_json-0.1.1 with pip. Importing barely_json gives the following error:

Traceback (most recent call last):
  File "/home/velle/a/parkscraper/cmdGetJsonToQueue.py", line 1, in <module>
    import barely_json
  File "/home/jdoe/.virtualenvs/testenv/lib/python3.10/site-packages/barely_json/__init__.py", line 15, in <module>
    from pyparsing import *
  File "/home/jdoe/.virtualenvs/testenv/lib/python3.10/site-packages/pyparsing.py", line 943, in <module>
    collections.MutableMapping.register(ParseResults)
AttributeError: module 'collections' has no attribute 'MutableMapping'