mojotech / json-type-validation

TypeScript JSON type validation

JSON Type Validation

A TypeScript library to perform type checking and validation on untyped JSON data at runtime.

This library owes thanks to:

• JsonDecoder by Daniel van den Eijkel
• Type-safe JSON Decoder by Wesley Merkel
• The Elm Json.Decode API

Installation

npm i @mojotech/json-type-validation

Projects using < [email protected] will need a polyfill for the unknown type, such as unknown-ts.

Motivation

Let's say we're creating a web app for our pet sitting business, and we've picked TypeScript as one of our core technologies. This is a great choice because the extra stability and type safety that TypeScript provides is really going to help us market our business.

We've defined the data we need to track about each client's pet:

interface Pet {
  name: string;
  species: string;
  age?: number;
  isCute?: boolean;
}

And we've got some data about current client's pets which is stored as JSON:

const croc: Pet = JSON.parse('{"name":"Lyle","species":"Crocodile","isCute":true}')
const moose: Pet = JSON.parse('{"name":"Bullwinkle","age":59}')

But that can't be right -- our data for moose is missing information required for the Pet interface, but TypeScript compiles the code just fine!

Of course this isn't an issue with TypeScript, but with our own type annotations. In TypeScript JSON.parse has a return type of any, which pushes the responsibility of verifying the type of data onto the user. By assuming that all of our data is correctly formed, we've left ourselves open to unexpected errors at runtime.

Unfortunately TypeScript doesn't provide a good built-in way to deal with this issue. Providing run-time type information is one of TypeScript's non-goals, and our web app is too important to risk using a forked version of TypeScript with that added functionality. Type guards work, but are limited in that they circumvent type inference instead of working with it, and can be cumbersome to write.

With json-type-validation we can define decoders that validate untyped json input. Decoders are concise, composable, and typecheck against our defined types and interfaces.

import {Decoder, object, string, optional, number, boolean} from '@mojotech/json-type-validation'

const petDecoder: Decoder<Pet> = object({
  name: string(),
  species: string(),
  age: optional(number()),
  isCute: optional(boolean())
})

Finally, we can choose from a number of decoding methods to validate json and report success or failure. When some json input fails validation the decoder clearly shows how the data was malformed.

const lyle: Pet = petDecoder.runWithException(croc)

const bullwinkle: Pet = petDecoder.runWithException(moose)
// Throws the exception:
// `Input: {"name":"Bullwinkle","age":59}
//  Failed at input: the key 'species' is required but was not present`

Documentation

Documentation.

Building

With Nix

There exists some Nix infrastructure that can be used to reproduce a build environment exactly. A helper shell script lives at bin/jtv that you can use to enter environments for multiple uses. You'll need to follow the directions on the Nix website to install and use the Nix package manager.

• To enter a shell suitable for building the library run ./bin/jtv build-shell. This will leave you in the root of the project and automatically install any project and npm dependencies. You can run further yarn commands here.
• To build the library for distribution and exit you can run ./bin/jtv distribute.
• To enter a build shell and run the build process, watching for changes, run ./bin/jtv build-watch.
• To run an arbitrary command in a build environment use ./bin/jtv run COMMAND. For example, ./bin/jtv run yarn test will run the tests and exit.

json-type-validation's Issues

Provide script that can rewrite TS interfaces to object({})

Lodash Dependency

May I suggest removing Lodash as a dependency?

The single call to the Lodash isEqual function could be probably replaced with a simple local implemention:

function isEqual(a: any, b: any) {
  if (a === b) {
    return true;
  }
  if (a === null && b === null) {
    return true;
  }
  if (typeof (a) !== typeof (b)) {
    return false;
  }
  if (typeof (a) === 'object') {
    // Array
    if (Array.isArray(a)) {
      if (!Array.isArray(b)) {
        return false;
      }
      if (a.length !== b.length) {
        return false;
      }
      for (let i = 0; i < a.length; i++) {
        if (!isEqual(a[i], b[i])) {
          return false;
        }
      }
      return true;
    }
    // Hash table
    const keys = Object.keys(a);
    if (keys.length !== Object.keys(b).length) {
      return false;
    }
    for (let i = 0; i < keys.length; i++) {
      if (!b.hasOwnProperty(keys[i])) {
        return false;
      }
      if (!isEqual(a[keys[i]], b[keys[i]])) {
        return false;
      }
    }
    return true;
  }
}

andThen decoder should return intersection type

Hi There! Here is my case

class A {
  propA: string
  
  static decoder(): Decoder<A> {
    return object({
      propA: string()
    })
  }
}

class B extends A {
  propB: string

  static decoder(): Decoder<B> {
    return object({
      propB: string()
    }).andThen(() => A.decoder())
  }
}

This code gives me the following error:

TS2322: Type 'Decoder<A>' is not assignable to type 'Decoder<B>'. 

  Type 'A' is not assignable to type 'B'.     Property 'propB' is missing in type 'A'.

I think that signature of this method should be changed from

andThen: <B>(f: (value: A) => Decoder<B>) => Decoder<B>;

to

andThen: <B>(f: (value: A) => Decoder<B>) => Decoder<A & B>;

I'm going to prepare the PR for this shortly :) Thanks :)

object() compiles to Object() in Babel which breaks the lib

I'm planning on using this API in Vue that uses the Babel compiler. I found that you're using what I suspect are reserved words (object, number, string) as function names that I found are unsafe to compile in Babel. Example: object() is translated to Object() which fails to execute the correct function.

I'd need to fork your project and rename these to something like vObject(), vString() etc or perhaps use JsonDecode.

Possible to represent fixed-length arrays / "tuples"?

What would a decoder for type [number, number, number] look like?

For example,

import { array, Decoder, number } from '@mojotech/json-type-validation'

type customTuple = [number, number, number]

const tupleDecoder: Decoder<customTuple> = array(number())

results in:

Error:(5, 7) TS2322: Type 'Decoder<number[]>' is not assignable to type 'Decoder<[number, number, number]>'.
  Type 'number[]' is not assignable to type '[number, number, number]'.
    Property '0' is missing in type 'number[]'.

Generate OpenAPI schema objects from decoders

Im working on generating OpenAPI for my server. I'm already using mojo tech json type validation so I figured I might as well use this syntax for defining POST body and response definitions.

Support custom field name for validation

Sometimes the convention from the server in json doesn't directly map to the styles on the front end. For example, data from the API may be in snake_case when the javascript is using camelCase. Since this library is already parsing and converting raw json to correctly formed typescript compatible types, it seems this would be a great layer to adapt to these differences.

For example, something like the following syntax would be ideal:

const userDecoder: Decoder<User> = object({
  firstName: string('first_name'),
  lastName: string('last_name'),
  username: string()
)};

where the the string decoder has an optional string parameter.

Decode index type objects?

Is it possible to represent this type with a decoder?

interface Map<T> {
    [key: string]: T;
}

Should decoders match their decoded type exactly, or should there be multiple valid decoders for a type?

Recently I've been exploring ways to require that the optional decoder be used for fields that are optional. From that process I've realized that I need to make a number of subtle design decisions related to how loosely or strictly a decoder needs to match a type. On the one hand, we should leverage typescript to help us write decoders that are as accurate as possible. On the other hand, I don't want the rules/guidelines around writing decoders to get too complicated, and I also don't want to be overbearing and prevent the user from writing the exact decoder they intend to.

With all that in mind, I've got a few examples of situations where I could change the library to more strictly fit decoders to types. Please respond with feedback to the three questions, plus any other concerns or observations you've got.

1. Here are four decoders for the interface AB. All four decoders are valid and will compile without errors. In an ideal world, which of these decoders would you want to be valid, and which ones should produce an error?

interface AB {
  a: string;
  b?: number;
}

const decoderAB1 = object({
  a: string(),
  b: optional(number())
});

const decoderAB2 = object({
  a: string(),
  b: number()
});

const decoderAB3 = object({
  a: string(),
  b: union(number(), constant(undefined))
});

const decoderAB4 = object({
  a: string()
});

2. Ditto for CD. All four decoders are valid, but as a library user which ones would you want to be valid?

interface CD {
  c: string;
  d: number | undefined;
}

const decoderCD1 = object({
  c: string(),
  d: optional(number())
});

const decoderCD2 = object({
  c: string(),
  d: constant(undefined)
});

const decoderCD3 = object({
  c: string(),
  d: union(number(), constant(undefined))
});

const decoderCD4 = object({
  c: string()
});

3. Ditto for E.

interface E {
  e: string | number;
}

const decoderE1 = object({
  e: union(string(), number())
});

const decoderE2 = object({
  e: string()
});

Feature: 'nullToUndefined' decoder type

Many APIs provide missing data fields with a value of null (see issue #29).

As a consumer of such an API using TypeScript and this very library it is currently not possible to decode such values to undefined.

Personally, I prefer to use undefined instead of null for my TypeScript models and I try to avoid null wherever possible. While this might be just my own preference, I believe it would make sense to add a standard decoder to this library that will automatically transform a given null value to undefined.

Here is snippet that does exactly what I need:
const nullToUndefined = <T>(decoder: Decoder<T>): Decoder<T | undefined> => JTV.union(decoder, JTV.constant(null)).map((val) => val === null ? undefined : val);

I am not sure if this is the best solution to the given problem, but it works.
I am happy to file a PR if that is desired 🙂

'runWithException' should throw instance of Error

Decoder.runWithException() throws a plain DecoderError object which is not an Error instance. This means that the error has no stack trace attached and does not play well with generic exception handling mechanisms that expect an Error instance. (In my case, the error object was logged by Sentry which failed to provide useful information like error message and stack trace.) The same issue exists with Decoder.runPromise().

I’d suggest turning DecoderError into a subclass of Error, providing it with a message, and keeping the rest of the attributes. (Maybe it would make sense to drop kind in favor of the more standard name.) This change would be backwards compatible. I’d be happy to implement the change.

object decoder should have an option to throw error if extra properties are present

If an object is run through the decoder at runtime, there should be a way to configure the decoder to throw an error if the decoded object has extra properties not defined on the original decoder.

For example, given the following decoder:

interface MyType {
  myProperty?: string;
}

const MyTypeDecoder: Decoder<MyType> = object({
  myProperty: optional(string());
});

There should be an additional combinator to make the decoder throw an error given the following input:

const myTypeRuntimeValue: MyType = {
  myProperty: "someValue",
  myOtherProperty: "This one isn't defined on the interface"
};

// I want this to throw an error because myTypeRuntimeValue has
// an additional property not defined in the interface
const decoded = MyTypeDecoder.runWithException(myTypeRuntimeValue);

The combinator could be something like exact(value), e.g.

interface MyType {
  myProperty?: string;
}

const MyTypeDecoder: Decoder<MyType> = exact(object({
  myProperty: optional(string());
}));

Recursion not working

Hello, I was testing some recursive JSON, but while the library does not give any error, it does not detect the incorrect fields either.
Here is my example:

const text = `
{
  "val1":1,
  "val2": [
    {
      "WRONG_val1":2,
      "val2":null
    },
    {
      "val1":3,
      "val2":null
    }]
}`;

interface Elem {
  val1: number;
  val2: Elem[];
};

var elemDecoder: Decoder<Elem> = object({
  val1: number(),
  val2: array(elemDecoder)
});

function Decode() {
  try {
    let elem1: Elem = JSON.parse(text);
    console.log("elem1", elem1);
    let elem2: Elem = elemDecoder.runWithException(elem1);
    console.log("elem2", elem2);
  }
  catch (error) {
    console.error(error);
  }
}

As you can see the field name "WRONG_val1" is not defined in the interface, but the decoder does not detect any error and returns a JSON object.
Here is the link to the running code at stackblits. You have to open the console for seeing the results.

So I was wondering if recursion is supported?
Thank you very much.
Regards,
Pablo

TypeScript compilation error

% tsc
decoder.ts:432:55 - error TS2339: Property 'error' does not exist on type 'Result<A, Partial<DecoderError>>'.
  Property 'error' does not exist on type 'Ok<A>'.

432             return Result.err(prependAt(`[${i}]`, nth.error));
                                                          ~~~~~


Found 1 error.

In this code block:

  static tuple<A>(decoders: Decoder<A>[]) {
    return new Decoder((json: unknown) => {
      if (isJsonArray(json)) {
        if (json.length !== decoders.length) {
          return Result.err({
            message: `expected a tuple of length ${decoders.length}, got one of length ${
              json.length
            }`
          });
        }
        const result = [];
        for (let i: number = 0; i < decoders.length; i++) {
          const nth = decoders[i].decode(json[i]);
          if (nth.ok) {
            result[i] = nth.result;
          } else {
            return Result.err(prependAt(`[${i}]`, nth.error));
          }
        }
        return Result.ok(result);
      } else {
        return Result.err({message: expectedGot(`a tuple of length ${decoders.length}`, json)});
      }
    });
  }

Possible to validate hash table of arbitrary length?

Is this library able to validate a hash table with arbitrary keys, yet consistent value types?

For example:

type Example = {
    a: string,
    b: number,
    c: { [name: string]: number }
};

Can c be validated to ensure the keys are strings and the values are numbers, even though the keys are not explicitly declared?

const exampleDecoder: Decoder<Example> = object({
  a: string(),
  b: number(),
  c: object(/* not sure what goes here */)
})

object() decoder needs a 'strict' option to disallow undefined fields

Just discovered this awesome library. Just what I was looking for! Thanks very much.

I had a look at the code for the object decoder and noted that it iterates over the fields of the fields of the object specification and returns as a result only these fields. That great for cleaning JSON input values, however I often need to check that fields in the input json object are only those defined by the object decoder. To support this I think we need a strict argument on the object() decoder that checks and reports an error if any additional fields are found in the json object.

  static object<A>(decoders?: DecoderObject<A>, strict?: boolean = false) {
    return new Decoder((json: unknown) => {
      if (isJsonObject(json) && decoders) {
        let obj: any = {};
        for (const key in decoders) {
          if (decoders.hasOwnProperty(key)) {
            const r = decoders[key].decode(json[key]);
            if (r.ok === true) {
              // tslint:disable-next-line:strict-type-predicates
              if (r.result !== undefined) {
                obj[key] = r.result;
              }
            } else if (json[key] === undefined) {
              return Result.err({message: `the key '${key}' is required but was not present`});
            } else {
              return Result.err(prependAt(`.${key}`, r.error));
            }
          }
        }
        // ADDED
        if (strict) {
          for (const key in json) {
            if (!decoders.hasOwnProperty(key)) {
              return Result.err({message: `an undefined key '${key}' is present in the object`});
            }
          }
        }
        return Result.ok(obj);
      } else if (isJsonObject(json)) {
        return Result.ok(json);
      } else {
        return Result.err({message: expectedGot('an object', json)});
      }
    });
  }

Support `nullable` type.

interface User {
    user_id: number,
    nickname: string | null
}

export const UserDecoder: Decoder<User> = object({
    user_id: number(),
    nickname: nullable(string())
})

Feature: dict with number keys

Currently the dict decoder only supports keys as string types :

json-type-validation/src/decoder.ts

Line 403 in c28d34d

static dict = <A>(decoder: Decoder<A>): Decoder<Record<string, A>> =>

Would you accept a pull request to provide the ability to decode dicts with numbers as keys? Happy to do the work and raise if so

Enum decoders are cumbersome and not intuitive

Given a TS enum

enum Directions { North, South, East, West }

We currently write the decoder as

const directionsDecoder: Decoder<Directions> = oneOf(
  constant(Directions.North), 
  constant(Directions.South), 
  constant(Directions.East), 
  constant(Directions.West)
);

which isn't great.

It's worth noting that the most ideal syntax is not possible, since TS compiles out enums and hardcodes the enum values during compilation

const directionsDecoder = enumDecoder(Directions); // this doesn't work!

Something like this should be possible:

const enumDecoder = <E>(...enumValues: Array<E[keyof E]>): Decoder<E> =>
  oneOf(...enumValues.map((v) => constant(v)));

const directionsDecoder = enumDecoder(
  Directions.North, Directions.South, Directions.East, Directions.West
);

But no matter how I bash at the types I am unable to get the type checker to reconcile the relationship between the elements of the enum and the enum in total.

Release recent changes?

Is it possible to get a release that includes #32 and the other recent changes?

I'd like to grab those updates via npm.

Inference from TypeScript objects

Would it be possible to have this library accept a TypeScript interface as a parameter and infer the Decoder shape from it?

There seems to be quite a lot of duplication involved in declaring a TS interface and then declaring (in most cases) the same exact shape for the Decoder of that object. Could this library provide a function that would accept the TS interface object as an argument and return a fully formed decoder from it?

I haven't look at TS internals enough yet to know, but I'm assuming we may be able to do this?