queckezz / json-tokenize

Splits a JSON string into an annotated list of tokens

json-tokenize

Splits a JSON string into an annotated list of tokens.

• Keeps track of a tokens position
• Whitespace significant
• Doesn't validate the correctness of the syntax

Screenshot - formatted tokens outputted to stdout

_{View Example}

Installation

npm install --save json-tokenize

Or even better

yarn add json-tokenize

Import and Usage Example

const tokenize = require('json-tokenize')

const obj = { "test": 1.0 }

tokenize(JSON.stringify(obj))
// ->
[
  { type: 'punctuation',
    position: { lineno: 1, column: 1 },
    raw: '{',
    value: '{' },
  { type: 'whitespace',
    position: { start: [Object], end: [Object] },
    raw: '\n  ',
    value: '\n  ' },
  { type: 'string',
    position: { start: [Object], end: [Object] },
    raw: '"test"',
    value: 'test' },
  { type: 'punctuation',
    position: { lineno: 2, column: 9 },
    raw: ':',
    value: ':' },
  { type: 'whitespace',
    position: { lineno: 2, column: 10 },
    raw: ' ',
    value: ' ' },
  { type: 'number',
    position: { lineno: 2, column: 11 },
    raw: '1',
    value: 1 },
  { type: 'whitespace',
    position: { start: [Object], end: [Object] },
    raw: '\n',
    value: '\n' },
  { type: 'punctuation',
    position: { lineno: 3, column: 1 },
    raw: '}',
    value: '}' }
]

Tokens

• whitespace - Allowed white space between the actual relevant tokens.
• punctuator - The characters sorrounding your data: {, }, [, ], : and ,
• string - JSON String
• number - JSON Number
• literal - true, false or null

Author

json-tokenize © Fabian Eichenberger, Released under the MIT License.
Authored and maintained by Fabian Eichenberger with help from contributors (list).

GitHub @queckezz · Twitter @queckezz

json-tokenize's Issues

Double-encoded JSON fails

Basically, if you've got a blob of JSON inside of another JSON string, the tokenizing fails:

  /home/espenh/webdev/json-tokenize/dist.js:65

   64:   }, null);                         
   65:   var createToken = ref.createToken;
   66:   var str = ref.str;                

  Error: Cannot read property 'createToken' of null

  tokenize (dist.js:65:24)
  next (dist.js:95:12)
  tokenize (dist.js:85:10)
  next (dist.js:95:12)
  tokenize (dist.js:85:10)
  next (dist.js:95:12)
  tokenize (dist.js:85:10)
  next (dist.js:95:12)
  tokenize (dist.js:85:10)
  next (dist.js:95:12)
  tokenize (dist.js:85:10)
  next (dist.js:95:12)
  tokenize (dist.js:85:10)
  next (dist.js:95:12)
  tokenize (dist.js:85:10)
  next (dist.js:95:12)
  tokenize (dist.js:85:10)

Test case:

test('double-encoded json', (t) => {
  const inner = JSON.stringify({ x: 1, y: '2' })
  const outer = JSON.stringify({ type: 'blob', data: inner })
  const innerEnc = JSON.stringify(inner)
  validateJson(t, outer, [
    { type: 'punctuation', raw: '{', value: '{' },
    { type: 'string', raw: '"type"', value: 'type' },
    { type: 'punctuation', raw: ':', value: ':' },
    { type: 'string', raw: '"blob"', value: 'blob' },
    { type: 'punctuation', raw: ',', value: ',' },
    { type: 'string', raw: '"data"', value: 'data' },
    { type: 'punctuation', raw: ':', value: ':' },
    { type: 'string', raw: innerEnc, value: inner },
    { type: 'punctuation', raw: '}', value: '}' }
  ])
})

space in string throws

tokenize('"a string"') // -> throws
tokenize('"astring"') // -> doesn't throw

skip whitespace option

dont use state; split into position, tokens