seandstewart / typical

Typical: Fast, simple, & correct data-validation using Python 3 typing.

typical: Python's Typing Toolkit

Introduction

Typical is a library devoted to runtime analysis, inference, validation, and enforcement of Python types, PEP 484 Type Hints, and custom user-defined data-types.

Typical is fully compliant with the following Python Typing PEPs:

• PEP 484 -- Type Hints
• PEP 563 -- Postponed Evaluation of Annotations
• PEP 585 -- Type Hinting Generics In Standard Collections
• PEP 586 -- Literal Types
• PEP 589 -- TypedDict: Type Hints for Dictionaries with a Fixed Set of Keys
• PEP 604 -- Allow writing union types as X | Y

It provides a high-level Protocol API, Functional API, and Object API to suit most any occasion.

Getting Started

Installation is as simple as pip install -U typical.

Help

The latest documentation is hosted at python-typical.org.

Starting with version 2.0, All documentation is hand-crafted markdown & versioned documentation can be found at typical's Git Repo. (Versioned documentation is still in-the-works directly on our domain.)

A Typical Use-Case

The decorator that started it all:

typic.al(...)

import typic


@typic.al
def hard_math(a: int, b: int, *c: int) -> int:
    return a + b + sum(c)

hard_math(1, "3")
#> 4


@typic.al(strict=True)
def strict_math(a: int, b: int, *c: int) -> int:
    return a + b + sum(c)

strict_math(1, 2, 3, "4")
#> Traceback (most recent call last):
#>  ...
#> typic.constraints.error.ConstraintValueError: Given value <'4'> fails constraints: (type=int, nullable=False, coerce=False)
  

Typical has both a high-level Object API and high-level Functional API. In general, any method registered to one API is also available to the other.

The Protocol API

import dataclasses
from typing import Iterable

import typic


@typic.constrained(ge=1)
class ID(int):
    ...


@typic.constrained(max_length=280)
class Tweet(str):
    ...


@dataclasses.dataclass # or typing.TypedDict or typing.NamedTuple or annotated class...
class Tweeter:
    id: ID
    tweets: Iterable[Tweet]


json = '{"id":1,"tweets":["I don\'t understand Twitter"]}'
protocol = typic.protocol(Tweeter)

t = protocol.transmute(json)
print(t)
#> Tweeter(id=1, tweets=["I don't understand Twitter"])

print(protocol.tojson(t))
#> '{"id":1,"tweets":["I don\'t understand Twitter"]}'

protocol.validate({"id": 0, "tweets": []})
#> Traceback (most recent call last):
#>  ...
#> typic.constraints.error.ConstraintValueError: Tweeter.id: value <0> fails constraints: (type=int, nullable=False, coerce=False, ge=1)

The Functional API

import dataclasses
from typing import Iterable

import typic


@typic.constrained(ge=1)
class ID(int):
    ...


@typic.constrained(max_length=280)
class Tweet(str):
    ...


@dataclasses.dataclass # or typing.TypedDict or typing.NamedTuple or annotated class...
class Tweeter:
    id: ID
    tweets: Iterable[Tweet]


json = '{"id":1,"tweets":["I don\'t understand Twitter"]}'

t = typic.transmute(Tweeter, json)
print(t)
#> Tweeter(id=1, tweets=["I don't understand Twitter"])

print(typic.tojson(t))
#> '{"id":1,"tweets":["I don\'t understand Twitter"]}'

typic.validate(Tweeter, {"id": 0, "tweets": []})
#> Traceback (most recent call last):
#>  ...
#> typic.constraints.error.ConstraintValueError: Tweeter.id: value <0> fails constraints: (type=int, nullable=False, coerce=False, ge=1)

The Object API

from typing import Iterable

import typic


@typic.constrained(ge=1)
class ID(int):
    ...


@typic.constrained(max_length=280)
class Tweet(str):
    ...


@typic.klass
class Tweeter:
    id: ID
    tweets: Iterable[Tweet]
    

json = '{"id":1,"tweets":["I don\'t understand Twitter"]}'
t = Tweeter.transmute(json)

print(t)
#> Tweeter(id=1, tweets=["I don't understand Twitter"])

print(t.tojson())
#> '{"id":1,"tweets":["I don\'t understand Twitter"]}'

Tweeter.validate({"id": 0, "tweets": []})
#> Traceback (most recent call last):
#>  ...
#> typic.constraints.error.ConstraintValueError: Given value <0> fails constraints: (type=int, nullable=False, coerce=False, ge=1)

Changelog

See our Releases.

typical's Issues

Some bugs for Callable: AttributeError for common function and TranslatorTypeError for instance method

• typical version: 2.0.17
• Python version: 3.7.4
• Operating System:

Description

@seandstewart , there's some prolems with Callable in the new version 2.0.17, and it's OK in version 2.0.15.

import typic
import typing


@typic.al
def foo(f: typing.Callable[[typing.Any], None], a: typing.Any):
    f(a)


def func(a: typing.Any):
    print(a)


ls = []

foo(func, 1)  # AttributeError
foo(ls.append, 1)  # TranslatorTypeError
typic.validate(typing.Callable[[typing.Any], None], func)  # TypeError
typic.validate(typing.Callable[[typing.Any], None], ls.append)  # TypeError
typic.validate(typing.Callable, func)  # OK
typic.validate(typing.Callable, ls.append)  # OK
typic.transmute(typing.Callable, ls.append)  # TranslatorTypeError
typic.transmute(typing.Callable, func)  # AttributeError
typic.transmute(typing.Callable[[typing.Any], None], func)  # AttributeError
typic.transmute(typing.Callable[[typing.Any], None], ls.append) # TranslatorTypeError

What I Did

Paste the command(s) you ran and the output.
If there was a crash, please include the traceback here.

Translate from Path to DirectoryPath

• typical version: 2.0.15
• Python version: 3.8.2
• Operating System: macOS

Since DirectoryPath is a subclass of Path, should it be possible to translate from Path to DirectoryPath if Path is a directory?
typic.translate(pathlib.Path.cwd(), typic.DirectoryPath)

I now get a the following error:
TranslatorTypeError: Cannot translate to type <class 'typic.types.path.DirectoryPath'>. Unable to determine target fields.

Add option to coerce return value.

We get the return annotation for free, so we should allow users to do something with it.

JSON schema generated for date field is invalid upon calling `validate`

It seems like typical doesn't convert the StringFormat.DATE enum field into a primitive (e.g. str), which fastjsonschema doesn't like.

>>> import typic
>>> from datetime import date
>>> from dataclasses import dataclass
>>> @typic.al
... @dataclass
... class Foo:
...   foo: date
...
>>> typic.schema(Foo)
ObjectSchemaField(title='Foo', description='Foo(foo: datetime.date)', properties={'foo': StrSchemaField(format=<StringFormat.DATE: 'date'>)}, additionalProperties=False, required=('foo',))
>>> typic.schema(Foo).validate({})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../venv/lib/python3.7/site-packages/typic/schema/field.py", line 194, in validate
    return self.validator(obj)
  File ".../venv/lib/python3.7/site-packages/typic/util.py", line 226, in __get__
    cache[attrname] = self.func(instance)
  File ".../venv/lib/python3.7/site-packages/typic/schema/field.py", line 189, in validator
    return fastjsonschema.compile(self.asdict())
  File ".../venv/lib/python3.7/site-packages/fastjsonschema/__init__.py", line 167, in compile
    exec(code_generator.func_code, global_state)
  File "<string>", line 5
    raise JsonSchemaException("data must be object", value=data, name="data", definition={'type': 'object', 'title': 'Foo', 'description': 'Foo(foo: datetime.date)', 'properties': {'foo': {'type': 'string', 'format': <StringFormat.DATE: 'date'>}}, 'additionalProperties': False, 'required': ['foo'], 'definitions': {}}, rule='type')
                                                                                                                                                                                                                         ^
SyntaxError: invalid syntax

duck example with enum does not works

• typical version: master
• Python version: 3.7.7
• Operating System: ubuntu 18.04

Description

Trying to run example from documentation.

import dataclasses
import datetime
import enum
import uuid

import typic

class DuckType(str, enum.Enum):
    WHT: "white"
    BLK: "black"
    MLD: "mallard"


@typic.al
@dataclasses.dataclass
class Duck:
    name: str
    type: DuckType
    created_on: datetime.datetime = dataclasses.field(
        default_factory=datetime.datetime.utcnow
    )
    id: uuid.UUID = dataclasses.field(
        default_factory=uuid.uuid4
    )

What I Did

Imported module above, and observed followint traceback:

Traceback (most recent call last):
  File "/home/user/dev/test/blya.py", line 15, in <module>
    @dataclasses.dataclass
  File "/home/user/.local/lib/python3.7/site-packages/typic/api.py", line 367, in typed
    return _typed(_cls_or_callable) if _cls_or_callable is not None else _typed
  File "/home/user/.local/lib/python3.7/site-packages/typic/api.py", line 359, in _typed
    return wrap_cls(obj, delay=delay, strict=strict)  # type: ignore
  File "/home/user/.local/lib/python3.7/site-packages/typic/api.py", line 324, in wrap_cls
    wrapped: Type[WrappedObjectT] = cls_wrapper(klass)
  File "/home/user/.local/lib/python3.7/site-packages/typic/api.py", line 322, in cls_wrapper
    return _resolve_class(cls_, strict=strict, jsonschema=jsonschema, serde=serde)
  File "/home/user/.local/lib/python3.7/site-packages/typic/api.py", line 193, in _resolve_class
    protos = protocols(cls, strict=strict)
  File "/home/user/.local/lib/python3.7/site-packages/typic/serde/resolver.py", line 535, in protocols
    annotation, parameter=param, name=name, is_strict=strict
  File "/home/user/.local/lib/python3.7/site-packages/typic/serde/resolver.py", line 448, in resolve
    flags=flags,
  File "/home/user/.local/lib/python3.7/site-packages/typic/serde/resolver.py", line 379, in annotation
    if is_static
  File "/home/user/.local/lib/python3.7/site-packages/typic/serde/resolver.py", line 277, in _get_configuration
    for x, y in util.cached_type_hints(origin).items()
  File "/home/user/.local/lib/python3.7/site-packages/typic/util.py", line 375, in cached_type_hints
    return get_type_hints(obj)
  File "/usr/local/lib/python3.7/typing.py", line 978, in get_type_hints
    value = _eval_type(value, base_globals, localns)
  File "/usr/local/lib/python3.7/typing.py", line 263, in _eval_type
    return t._evaluate(globalns, localns)
  File "/usr/local/lib/python3.7/typing.py", line 467, in _evaluate
    eval(self.__forward_code__, globalns, localns),
  File "<string>", line 1, in <module>
NameError: name 'white' is not defined

Dynamic default value or factory for function argument

• typical version: 2.0.1
• Python version: 3.7.4
• Operating System:

Description

Hey, @seandstewart . I have another feature request.
For data model, there is default_factory which can be used to set dynamic default value. For function argument, if set the default value as a instance of typic.Field, it won't be validated as a default value. datetime.now as a default_factory is quite common. Every time I have to handle it like this:

from datetime import datetime
import typic

@typic.al
def foo(dt: datetime = None):
    dt = datetime.now() if dt is None else dt
    ...

Especilly when we have many functions with dynamic default value, it would be a problem. Currently, beacuse the default value would not be validated, I defined a Helper class to solve this.

from typing import Callable, Optional, Any
from functools import wraps
import inspect
from dataclasses import MISSING
import typic

class DefaultArgument:
    def __init__(self, default: Any = MISSING, *, factory: Optional[Callable] = None):
        assert default is not MISSING or callable(factory)
        self.default = default
        self.factory = factory

    def __call__(self, *args, **kwargs):
        if self.default is not MISSING:
            return self.default
        return self.factory(*args, **kwargs)

    def __repr__(self):
        return f"{self.__class__.__name__}(default={self.default}, factory={self.factory!r})"


def validate(func: Optional[Callable] = None, *, coerce: bool = True, strict: bool = False) -> Callable:
    def _validate(_func: Callable):
        @wraps(func)
        def wrapped(*args, **kwargs):
            bind = inspect.signature(_func).bind(*args, **kwargs)
            bind.apply_defaults()  # apply_defaults is not available in typic.bind
            for k, v in bind.arguments.items():
                if isinstance(v, DefaultArgument):
                    bind.arguments[k] = v()
            return typic.bind(_func, *bind.args, partial=False, coerce=coerce, strict=strict, **bind.kwargs).eval()

        return wrapped

    if callable(func):
        return _validate(func)
    else:
        return _validate

Now we can redefine foo like this:

from datetime import datetime

@validate
def foo(dt: datetime = DefaultArgument(factory=datetime.now)):
    ...

By the way, what if the user want to use the key words such as dict, schema, coerce, strict and so on, as the function argument or data attribute?

Properly validate VarArgs.

• typical version: v2.0.0b22
• Python version: 3.*
• Operating System: Any

Description

varargs are not properly validated - each item should be checked in turn.

What I Did

See #43 for an explanation.

transmute from datetime.datetime -> pendulum.Datetime not handling tzinfo right

• typical version: 2.0.16
• Python version: 3.8.2
• Operating System: macOS

Hi again,
I should probably soon try to make some pr, but take some time to understand the structure of typical and how things is put together. But until then here are an observation of a something that do not behave as expected.

import typic
import pendulum
from datetime import datetime
import pytz

dt = datetime(2020,5,24,tzinfo=pytz.UTC)

I would expect that,

x = typic.transmute(pendulum.DateTime, dt)
x
#> DateTime(2020, 5, 24, 0, 0, 0, tzinfo=UTC)

give same the same as,

y = pendulum.instance(dt)
y
#> DateTime(2020, 5, 24, 0, 0, 0, tzinfo=Timezone('UTC'))

but as you can see timezone type is diffrent

type(x.tzinfo)
#> pytz.UTC
type(y.tzinfo)
#> pendulum.tz.timezone.FixedTimezone

What do you think?

transmuted value is not an instance of the annotation after subclass?

• typical version: 2.0.2
• Python version: 3.7.4
• Operating System:

Description

@seandstewart , is this a bug?

import typic

@typic.klass
class Foo:
    foo: str


@typic.klass
class FooBar(Foo):
    bar: str


isinstance(Foo.transmute({'foo': 'foo'}), Foo)  # True
isinstance(FooBar.transmute({'foo': 'foo', 'bar': 'bar'}), FooBar)  # False

[Bug]: transmute truncate datetime

• typical version: 2.0.0b23
• Python version: 3.8.1
• Operating System: Windows 10

transmute truncate everything after hours:

In: typic.transmute(pendulum.DateTime, pendulum.datetime(2020, 3, 23, 16, 27, 12))
Out:  DateTime(2020, 3, 23, 0, 0, 0)

transmute do not preserve timezone:

In: typic.transmute(pendulum.DateTime, pendulum.datetime(2020, 3, 23, 16, 27, 12, tz='Europe/Oslo'))
Out: DateTime(2020, 3, 23, 0, 0, 0)

support for DefaultDict?

• typical version: 2.0.5
• Python version: 3.7.4
• Operating System:

Description

It seams that typical doesn't support some types ini collections e.g. defaultdict.

import typic


@typic.klass
class Foo:
    s: typing.DefaultDict[str, int]
# ValueError: no signature found for builtin type <class 'collections.defaultdict'>

Subclass of builtin types like set, list, without __init__, will raise the same error:

import typic

class MySet(set):
    ...

@typic.klass
class Foo:
    s: MySet

# ValueError: no signature found for builtin type <class '__main__.MySet'>

What I Did

Paste the command(s) you ran and the output.
If there was a crash, please include the traceback here.

docs: The example which mentions that pydantic will raise an exception is not accurate

According to the docs:

For instance, pydantic will coerce a float to an int, but will raise an error if a str is passed to a field marked as datetime.

This is not true:

import datetime
import pydantic

class Foo(pydantic.BaseModel):
    a_date: datetime.date
    a_datetime: datetime.datetime

a = Foo(a_date="2019-01-01", a_datetime="2019-01-01 17:31:31")
print(type(a.a_date))
print(type(a.a_datetime))

You might want to rephrase that

Benchmarks

We should post definitive benchmarks - perhaps make use of Pydantic's own benchmarking suite.

Add dataclass_factory to benchmarks

Hello! Your project is awesome!

I think we should consider to add dataclass_factory by @Tishka17 to benchmarks because it's fast too.

Schema Gen

We should be able to generate JSON Schema definitions from typed classes.

inspect.Signature is slow.

Currently, the largest bottle-neck for coercing callable inputs is the inspect.Signature object and binding inputs to the function args, which we need to do manually so that we can coerce to the appropriate type.

In my testing, running a simple function with one or two args is up to 100x slower as a result of using that object.

support for pandas basic types such as DataFrame and Series

• typical version: 2.0.10
• Python version: 3.7.4
• Operating System:

Description

pandas is quite a common used lib for data proccessing. Except pd.Timestamp, the most basic types DateFrame and Series is not supported currently.

import typic
import pandas as pd


@typic.klass
class Foo: 
    df: pd.DataFrame
# TypeError: 'NoneType' object is not callable


@typic.klass
class Bar:
    ss: pd.Series
# TypeError: 'NoneType' object is not callable

Is there a simple way that typical can support arbitrary types?

Return multiple validation errors at once

Not sure if it's possible, but it would be useful if you could get a list of e.g. ValueErrors, rather than just the first one that failed when there are multiple issues.
It would make typical much more useful for validation purposes.

Perhaps it could be configured somehow which behavior you want, since there could be a performance hit, I suppose.

Example:

>>> @typic.al
... @dataclass
... class N:
...   a: str
...   b: int
...   c: int
... 
>>> N(a="a", b="a", c="b")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 3, in __init__
  File "../typic/api.py", line 143, in __setattr_coerced__
    value = ann[name](value) if name in ann else value
  File "../typic/coercer.py", line 117, in coerce
    return self.coercer(value)
  File "<string>", line 3, in coerce__class_int___class_inspect__empty__False
ValueError: invalid literal for int() with base 10: 'a'

Generating schema from TypedDict fails with ValueError

Cool library. It's nice to have a competitor to pydantic which seems a bit too loose for my taste. I was hoping I might be able to use it to:

• Coerce JSON data (as dicts) into TypedDicts rather than dataclasses and
• Generate JSON schema from TypedDicts

(not sure if this is one of the goals of the project to make this work, but it seems like it fits in with the philosophy of this library as being unopinionated)

I tried the both, and they both fail. The latter fails with:

>>> from typing_extensions import TypedDict
>>> class F(TypedDict):
...   a: str
...   b: int
... 
>>> typic.schema(F)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/.../venv/lib/python3.7/site-packages/typic/coercer.py", line 868, in schema
    annotation = self.resolve(obj)
  File "/.../venv/lib/python3.7/site-packages/typic/util.py", line 286, in _cached_method_wrapper
    result = func(*args, **kwargs)
  File "/.../venv/lib/python3.7/site-packages/typic/coercer.py", line 500, in resolve
    use, default=parameter.default, is_optional=is_optional
  File "/.../venv/lib/python3.7/site-packages/typic/coercer.py", line 430, in get_coercer
    return self._build_coercer(annotation, default=default, is_optional=is_optional)
  File "/.../venv/lib/python3.7/site-packages/typic/coercer.py", line 402, in _build_coercer
    self._build_mapping_coercer(func, args, anno_name)
  File "/.../venv/lib/python3.7/site-packages/typic/coercer.py", line 316, in _build_mapping_coercer
    key_type, item_type = args
ValueError: not enough values to unpack (expected 2, got 0)


# Same happens when it's wrapped in a dataclass
>>> from dataclasses import dataclass
>>> @typic.al
... @dataclass
... class Wrapper:
...   foo: str
...   bar: F
... 
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/.../venv/lib/python3.7/site-packages/typic/api.py", line 184, in typed
    return _typed(_cls_or_callable) if _cls_or_callable is not None else _typed
  File "/.../venv/lib/python3.7/site-packages/typic/api.py", line 176, in _typed
    return wrap_cls(obj, delay=delay)  # type: ignore
  File "/.../venv/lib/python3.7/site-packages/typic/api.py", line 116, in wrap_cls
    coerce.annotations(klass)
  File "/.../venv/lib/python3.7/site-packages/typic/util.py", line 327, in _fast_cached_method_wrapper
    return memoget(arg)
  File "/.../venv/lib/python3.7/site-packages/typic/util.py", line 315, in __missing__
    self[key] = ret = func(instance, key)
  File "/.../venv/lib/python3.7/site-packages/typic/coercer.py", line 606, in annotations
    annotation, parameter=param, name=name, constraints=constraints
  File "/.../venv/lib/python3.7/site-packages/typic/util.py", line 286, in _cached_method_wrapper
    result = func(*args, **kwargs)
  File "/.../venv/lib/python3.7/site-packages/typic/coercer.py", line 500, in resolve
    use, default=parameter.default, is_optional=is_optional
  File "/.../venv/lib/python3.7/site-packages/typic/coercer.py", line 430, in get_coercer
    return self._build_coercer(annotation, default=default, is_optional=is_optional)
  File "/.../venv/lib/python3.7/site-packages/typic/coercer.py", line 402, in _build_coercer
    self._build_mapping_coercer(func, args, anno_name)
  File "/.../venv/lib/python3.7/site-packages/typic/coercer.py", line 316, in _build_mapping_coercer
    key_type, item_type = args
ValueError: not enough values to unpack (expected 2, got 0)

Coercion to Union does not result in error but returns passed in value -- Union could be expanded

It's stated in the README:

The following annotations are special forms which cannot be resolved:
Union
Any
Because these signal an unclear resolution, Typical will ignore this flavor of annotation ...

However, when I try to coerce some values to Unions, I didn't expect Typical to silently ignore Union annotations: (e.g. it returns a str when a str is passed in, when none of the Union members are str). Since it is ignored, I can't rely on passing my value through coercion to actually get the correct type returned to me. I think this is a major limitation and a source of errors.

from typing import Union
import typic


# Case A. Incorrect. Expected: ValueError
print(typic.coerce("foo", Union[float, int]))  # foo

from enum import Enum

class MyEnum(Enum):
  a = "foo"
  b = "bar"

# Case C1. Correct: Union with None works.
print(typic.coerce("foo", Union[MyEnum, None]))  # MyEnum.a
print(typic.coerce(None, Union[MyEnum, None]))  # None

# Case C1. Incorrect. Expected: MyEnum.a
print(typic.coerce("foo", Union[MyEnum, int]))  # foo
print(typic.coerce("foo", Union[MyEnum, bool]))  # foo

class Sentinel:
  pass

# Case B. Incorrect. Expected: MyEnum.a
print(typic.coerce("foo", Union[MyEnum, Sentinel]))  # foo

# Case C1: Correct: The value is an instance of one of the types
print(typic.coerce(1, Union[bool, int]))  # 1
print(typic.coerce(False, Union[bool, int]))  # False

# Case B. Incorrect. Expected: True (since it can be coerced to bool, but not to int)
print(typic.coerce("asdf", Union[bool, int]))  # asdf

# Case C2. Expected: TypeError, since it could be coerced to both
print(typic.coerce("1", Union[bool, int]))  # 1

Would it really be problematic to handle coercing to Union in some cases?

I can imagine scenarios where it's unambiguous, and one scenario where it is and an error could be raised:

• A. If the value cannot be coerced to any of the types: raise one of the coercion errors (e.g. the last)
• B. If the value can be coerced to only one of the types (the others may raise ValueError/TypeError): return that coerced value
• C. If the value can be coerced to multiple of the types:
  • C1: If the value is an instance of one of the union's types, then "coerce" it to that type -- coerce in this context means just perform any constraint checks that there might be (This coercion is already handled for Union[T, None])
  • C2: Otherwise: Raise a TypeError (this is the ambiguous scenario)

What do you think?

In my mind, it might look something like this:

from typing import Any, Iterable, Type
from typic import coerce

# Sentinel value
class Empty:
    pass


_empty = Empty()


def coerce_union(value: Any, union: Iterable[Type]) -> Any:
    coerced_value = _empty

    for typ in union:
        # If the value is already an instance of one of the types in the Union, coerce using that type
        # Presumably, if the type was constrained, coercing could raise errors here, but that would be expected.
        if isinstance(value, typ):
            return coerce(value, typ)
    error = None

    for typ in union:
        try:
            new_coerced_value = coerce(value, typ)
        except (ValueError, TypeError) as e:
            # Store the error and continue
            error = e
            continue
        if coerced_value is not _empty:
            raise TypeError("Ambiguous coercion: the value could be coerced to multiple types in the Union")
        coerced_value = new_coerced_value
    if coerced_value is not _empty:
        return coerced_value
    # The value couldn't be coerced to any of the types in the Union
    raise error

Custom validation feature request

Description

Would like to see a feature to implement custom validations that could span multiple fields. For example:

Let's say you had a vacation start date and vacation end date field. Vacation start date should be before vacation end date.

Inconsistent Behavior with Object Translation and `.transmute()`

• typical version: 2.0.5
• Python version: 3.7
• Operating System: macos

Description

Translation between higher-order objects isn't behaving properly when signatures don't exactly align.

Additionally, calling .transmute() on data with unknown fields fails, but should succeed.

What I Did

import typic
from typing import Optional


@typic.klass
class Source:
    test: Optional[str] = typic.field(init=False)
    field_to_ignore: str = "Ignore me"
    def __post_init__(self):
        self.test = "Something"


@typic.klass
class Dest:
    test: Optional[str] = None


Dest.transmute(Source())
# => throws "Source is missing fields: ('test',)"

Dest.transmute(Source().primitive())
# => throws "TypeError: __init__() got an unexpected keyword argument 'field_to_ignore'"

Add Discriminator Support for Polymorphic Types (Unions)

Description

A common pattern for polymorphic data in statically typed languages is the use of a discriminator field on the object which can be used to determine which type to deserialize said data into. This is main use-case for users registering custom type deserializers in typical. A better route would be to expose a method at time of declaration by which the user my declare a discriminator.

Potential Implementation

from typing import ClassVar, Union
import typic

@typic.klass
class Something:
    type: ClassVar[str] = "something"


@typic.klass
class Else:
    type: ClassVar[str] = "else"


discriminator = typic.discriminator(field="type", mapping={"something": Something, "else": Else})


@typic.klass
class Poly:
    morphic: Union[Something, Else] = typic.field(discriminator=discriminator)

This relates to #56

Explore Cython Build

Compiling down to Cython at build time could provide some quick wins on performance if the effort isn't too great.

Switch to own implementation from dataclasses

Heyo! I very like your library! But when i saw the docs, i found that typical uses python dataclasses from standart library which, of course, will decrease performance because they are slow. What do you think about reinventing it for more performance? :)

Add mypy Support

This library is poorly supported by mypy, and as a result makes mypy complain quite loudly for some normal operations.

Tasks to complete:

1. Add mypy plugin.
   • Look at the Pydantic/dataclass plugins.
2. Add py.typed file to project.

Support for typing.Iterator and better support for typing.Iterable.

• typical version: 2.0.17
• Python version: 3.7.4
• Operating System:

Description

Iterable can be infinitely.

from typing import Iterable
import typic

def infinite_numbers():
    i = 0
    while 1:
        yield i
        i += 1

@typic.al
def foo(iterable: Iterable[int]):
    print(iterable)

foo(range(5))  # [0, 1, 2, 3, 4]
foo(infinite_numbers())  # never stop

What I Did

Paste the command(s) you ran and the output.
If there was a crash, please include the traceback here.

TypeError if an annotation is annotated by defaultdict

• typical version: 2.0.9
• Python version: 3.7.4
• Operating System:

Description

Hey, @seandstewart , there still some problem with defaultdict.

import typic
from collections import defaultdict
from typing import List, DefaultDict


@typic.klass
class Foo:
    foo: DefaultDict[str, List[int]] = typic.field(
        default_factory=lambda: defaultdict(list), init=False
    )

@typic.klass
class Bar:
    bar: Foo
# TypeError: keywords must be strings

Coercion from None to str

I didn't expect that None would be coerced to a string for a field marked as str. Can that behavior be disabled?

>>> @typic.al
... @dataclass
... class Y:
...   a: str
... 
>>> Y(None)
Y(a='None')

The same behavior happens if you try to coerce other types:

>>> Y(dict)
Y(a="<class 'dict'>")

Is it possible to simplify registering user defined type?

• typical version: 2.0.1
• Python version: 3.7
• Operating System: CentOS 8

Description

Hi, I like typical and I think it's more lightweight than pydantic. But add user defined type by typic.register is something tedious or there's another convenient way I didn't found. If so, is it possible to simplify it like pydantic's __get_validators__ or some else?

Mishandling of strict mode `Mapping` and `Dict` with non-string keys

• typical version: 2.0.12 (and I verified the bug exists in current master.)
• Python version: CPython 3.7.7 (via homebrew)
• Operating System: macOS 10.15.4

Description

I tried using typic.al and typic.klass with the type annotation Mapping[Path, Path].

This failed with the error TypeError: sequence item 1: expected str instance, PosixPath found -- full backtrace at the very end, not that it matters, along with a little other debug output.

The goal was to have strict runtime validation; the decoration reflects the actual type passed. If it matters, I wanted runtime validation to catch type punning errors like passing a str where a List[str] was expected. The code in question mostly runs external applications, and operates on very small datasets, so the performance cost was not a concern.

Minimal reproduction

#!/usr/bin/env python3
from typing import Mapping
import typic

@typic.al(strict=True)
def bad(val: Mapping[int, str]) -> None:
  pass
bad({1: 1})

What is broken

The validation code tries to use the raw key in a call to str.join, which fails. Comes from the code in MappingConstraints._set_item_validator_keys_values_line() in typic/constraints/mapping.py

The code generated for describing the invalid mapping value is:

        field = f"'.'.join(({self.VALTNAME}, {self.X}))"

The correct version is probably:

# too horrible for words, really...
field = f"f'{{{self.VALTNAME}}}[{{repr({self.X})}}]'"
# nice version that someone could actually read...
field = ''.join(("f'{", self.VALTNAME, "}[{repr(", self.X, ")}]'"))

That fixes the bug in two ways, first by not using an almost-Any as an argument to str.join where a str is expected, but second by using repr to get a python-ish version of the value, not a string, which is kind of confusing.

Regardless, you could also fix it in the current model with:

        field = f"'.'.join(({self.VALTNAME}, repr({self.X})))"

FWIW, I'd very strongly argue that, at least for Mapping and Dict, using the . dot separator is wrong. That really belongs to class attributes, and that isn't the right context here. Given:

@dataclass
class MyDataclass:
  MyDictField: Dict[str, str]

MyDataclass(MyDictField={'hello': 12}

The validation output should probably be something like

MyDataclass.MyDictField['hello'] was int, expected str

Rather than:

MyDataclass.MyDictField.hello was int, expected str

Longer demo, showing various failure types

] python3 tmp/demo.py
bad: {1: PosixPath('/Users/slippycheeze')}
sequence item 1: expected str instance, int found
/Users/slippycheeze
ugly: {PosixPath('/Users/slippycheeze'): PosixPath('/Users/slippycheeze')}
Given value <PosixPath('/Users/slippycheeze')> fails constraints: (type='int', nullable=False, coerce=False)
/Users/slippycheeze
worse: {(1, 2): True}
sequence item 1: expected str instance, tuple found
True

Code for the long failure demo

#!/usr/bin/env python3
from typing import Any, Dict, Mapping, Tuple
from pathlib import Path
import typic

@typic.al(strict=True)
def bad(val: Mapping[Path, Path]) -> None:
  pass

@typic.al(strict=True)
def ugly(val: Dict[int, Path]) -> None:
  pass

@typic.al(strict=True)
def worse(val: Dict[Tuple, Any]) -> None:
  pass

# In each case the demo is that the key *is* valid, and is not simply an
# identity comparison.  Not that this is necessary, but y'know, thorough.
try:
  data = {1: Path.home()}
  print("bad:", repr(data))
  bad(data)
except Exception as e:
  print(e)
  print(repr(data[1]))

try:
  data = {Path.home(): Path.home()}
  print("ugly:", repr(data))
  ugly(data)
except Exception as e:
  print(e)
  print((data[Path.home()]))

try:
  data = {(1,2,): True}
  print("worse:", repr(data))
  worse(data)
except Exception as e:
  print(e)
  print(repr(data[(1,2,)]))

Long Backtrace of real world failure, pdb bits

[...elided the uninteresting parts]
  File "/Users/slippycheeze/share/fonts/conform.py", line 83, in task_fonts
    CopyFiles(copies)
  File "<string>", line 3, in __init__
  File "/Users/slippycheeze/homebrew/lib/python3.7/site-packages/typic/api.py", line 220, in __setattr_typed__
    self, name, __trans[name](item) if name in protos else item,
  File "/Users/slippycheeze/homebrew/lib/python3.7/site-packages/typic/constraints/common.py", line 103, in validate
    valid, value = self.validator(value)
  File "<typical generated validator_4371568336>", line 9, in validator_4371568336
    valid, val = validator_4371568336_item_validator(val, addtl)
  File "<typical generated validator_4371568336_item_validator>", line 7, in validator_4371568336_item_validator
    retx, rety = validator_4371568336_item_validator_keys_validator(x), validator_4371568336_item_validator_vals_validator(y, field='.'.join((valtname, x)))

] python3 tmp/demo.py
> <typical generated validator_4460259152_item_validator>(7)validator_4460259152_item_validator()
-> retx, rety = validator_4460259152_item_validator_keys_validator(x), validator_4460259152_item_validator_vals_validator(y, field='.'.join((valtname, x)))
(Pdb) p validator_4460259152_item_validator_vals_validator(y)
PosixPath('/Users/slippycheeze/b')
(Pdb) p '.'.join((valtname, x)
*** SyntaxError: unexpected EOF while parsing
(Pdb) p '.'.join((valtname, x))
*** TypeError: sequence item 1: expected str instance, PosixPath found
(Pdb) p valtname
'Mapping'
(Pdb) p x
PosixPath('/Users/slippycheeze/a')
(Pdb) p validator_4460259152_item_validator_keys_validator(x)
PosixPath('/Users/slippycheeze/a')
(Pdb) p validator_4460259152_item_validator_keys_validator
<bound method __AbstractConstraints.validate of (type='Path', nullable=False)>
(Pdb) p '.'.join((valtname, str(x)))
'Mapping./Users/slippycheeze/a'

Not expected behavior for datetime with strict option.

• typical version: 2.0.0
• Python version: 3.8.2
• Operating System: macOS Catalina

Expected that when using strict with datetime, that it only accept datetime object. And do not do any type coercer. But the following example do not behave as I expected. Expected to get an exception in both cases.

>>> import typic
>>> typic.transmute(typic.Strict[pendulum.DateTime], '2020-04-09') 
DateTime(2020, 4, 9, 0, 0, 0, tzinfo=Timezone('UTC'))

or similary

>>> @typic.klass(strict=True)
... class Test:
... date: pendulum.DateTime
>>> Test('2020-04-09') 
Test(date=DateTime(2020, 4, 9, 0, 0, 0, tzinfo=Timezone('UTC')))

Deepcopy fails for NetAddrInfo

• typical version: current
• Python version: ^3.7
• Operating System: Any

Description

NetAddrInfo is a Frozen dataclass with slots, which is currently broken for deepcopy: PR in cPython

TypeError when father class defined __init_subclass__

• typical version: 2.0.3
• Python version: 3.7.4
• Operating System:

Description

As the title said. If I subclass a base class with __init_subclass__, the method super will raise TypeError.

import typic


@typic.klass
class Base:
    def __init_subclass__(cls, **kwargs):
        print('subclassed!')
        super().__init_subclass__(**kwargs)


@typic.klass
class Foo(Base):
    foo: str

I guess this might be related to #58

Validation of Literal types

Literal is a way of specifying that a variable is one of a set of specific values.

It'd be great if typical would be able to validate data using Literal annotations.

An example where I did a similar validation manually: https://github.com/bluelabsio/records-mover/blob/master/records_mover/records/delimited/hint.py#L43-L71

TypedDict JSON schema generation: Optionals and required fields

So I played around a bit with the new support for TypedDict schema generation and noticed a few things:

>>> class G(TypedDict, total=False):
...   a: str
... 
>>> typic.schema(G)
ObjectSchemaField(title='G', properties={'a': StrSchemaField()}, additionalProperties=False, required=('a',))

Since the TypedDict is not total, the field a is not required to be present, so shouldn't be required in the JSON schema.

Also, somewhat related: Making a field Optional seems to make it not required even though it doesn't have a default value. But Optional just means that the value can be None, not that it could be missing.

>>> class G(TypedDict, total=True):
...   a: Optional[int]
... 
>>> typic.schema(G)
ObjectSchemaField(title='G', properties={'a': IntSchemaField()}, additionalProperties=False)

validation error: `Optional[Mapping[str, str]]`

• typical version: 2.0.12
• Python version:
• Operating System: macOS 15

Description

Looks like optional mapping validation in non-strict mode fails to compile. The data being checked is irrelevant, problem happens during type compilation.

from typing import *
import typic
typic.validate(Optional[Mapping[str, str]], {})

Note: I'd expect the minimal repro to validate, but this is about the type validation compiler, not the validate call itself -- that is just the easy way to trigger it. I ran into it in an annotated function where one argument was Optional[Mapping[Strict[str], str]], but cut it down to the above.

Optional[Mapping] will succeed, so it seems to be an assumption in the key/value validation code for Mapping that didn't expect the Union type above it or something.

] python3 -c 'from typing import *; import typic; typic.validate(Optional[Mapping[str, str]], None)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/slippycheeze/homebrew/lib/python3.7/site-packages/typic/serde/resolver.py", line 120, in validate
    resolved: SerdeProtocol = self.resolve(annotation)
  File "/Users/slippycheeze/homebrew/lib/python3.7/site-packages/typic/serde/resolver.py", line 458, in resolve
    deserializer, validator = self.des.factory(anno, constraints)
  File "/Users/slippycheeze/homebrew/lib/python3.7/site-packages/typic/serde/des.py", line 527, in factory
    deserializer = self._build_des(annotation)
  File "/Users/slippycheeze/homebrew/lib/python3.7/site-packages/typic/serde/des.py", line 438, in _build_des
    self._build_mapping_des(func, anno_name, annotation)
  File "/Users/slippycheeze/homebrew/lib/python3.7/site-packages/typic/serde/des.py", line 324, in _build_mapping_des
    if issubclass(annotation.origin, defaultdict):
TypeError: issubclass() arg 1 must be a class

Consider using lazy stringification for field paths where possible

• typical version: 2.0.12 / master

Right now typical eagerly generates the field path for displaying in error messages. This should ideally be lazy, so that the cost of a potentially complex __str__ or __repr__ is avoided in the common -- non-failure -- case.

This costs an object construction and a couple refcounts, but avoids creating and destroying just as many strings as those objects -- and storing a list is probably cheap enough.

I'm sure there is a module for this, but last time I needed it I just wrote my own. The implementation is pretty trivial, honestly, but this is off the top of my head, and in the github editor, so if it don't compile, don't be surprised. :)

class LazyReprConcat(object):
  __slots__ = ['things']
  def __init__(self, *things):
    self.things = things
  def __str__(self):
    return ''.join(repr(thing) for thing in self.things)
  __repr__ = __str__

The reason it did the concat was to allow easy recursive use: field = LazyReprConcat(parentPath, fieldName), but there are probably a million viable ways to handle all that, including more automagic with + and stuff...

JSON Schema Types

We should add JSON Schema defined types and formats.

typic.validate bug?

• typical version: 2.0.15
• Python version: 3.8.2
• Operating System: macOS

Did not expected this. Have I misunderstood something?

typic.validate(int, 'a')
#> 'a'

Incorrect determination of `total` for Object Constraints

• typical version: 2.0.11
• Python version: 3.8
• Operating System: macOS

Description

Constraints generated for typed classes are incorrectly loose, allowing for extra fields. These should not be allowed when applying validation, even if we do allow them when transmuting inputs.

Investigate orjson as an alternative to ujson

Description

Related: #44

orjson now supports serialization of subclassed.builtins, which potentially allows us to use this library (if available) for .tojson().

pandas.Timestamp as an annotation causes TypeError

• typical version: 2.0.8
• Python version: 3.7.4
• Operating System:

Description

pandas.Timestamp is a sub-class of datetime, but as an annotation will cause TypeError:

import typic
from datetime import datetime
import pandas as pd


issubclass(pd.Timestamp, datetime)  # True


@typic.klass
class Foo:
    dt: pd.Timestamp
# TypeError: __init__() got an unexpected keyword argument 'additionalProperties'

This is OK at version 2.0.5.
Personally, I think pandas.Timestamp is more convenient than builtin datetime in data analysis.

add methods, eg dict, tuple, namedtuple, and attr __typic_fields_tuple__ to the class decorated by typic.klass

• typical version: 2.0.17
• Python version: 3.7.4
• Operating System: Centos8

Description

There's many cases that user wants to transform the data object to a common dict, tuple, or namedtuple. Method dict maybe have some args like exclude or include_private and so on.

The attr named __typic_fields_tuple__, maybe some other name, is a tuple only contains the filed names. Although __typic_fileds__ is also a tuple, but contains detailed field info. Everytime I want to get the field names, I have to do something like:

fields = [f.name for f in self.__typic_fields__]

What I Did

Paste the command(s) you ran and the output.
If there was a crash, please include the traceback here.

Comparison to typeguard

I haven't read through this project thoroughly but it looks like it overlaps a lot with https://github.com/agronholm/typeguard, at least on the validation side. Just thought you might want to know about it and maybe mention it in your docs somewhere. Maybe you two can even help each other.

validate: seems to be confused by `Union[str, Path]` vs `Path` instance

• typical version: 2.0.14
• Python version: 3,8,3
• Operating System: Windows 10 (also macOS)

Description

I suspect this may be a known issue, caused by having more than one plausible conversion applied to the union type, but:

from pathlib import Path
from typing import *
import typic

good = [str, Path, Union[str], Union[Path]]
for types in good:
    typic.validate(this_type, Path.home())
    # success, whatever you homedir is, like:
    WindowsPath('c:/Users/me')

typic.validate(Union[str, Path], Path.home())
# fails:
ConstraintValueError: Given value <WindowsPath('C:/Users/me')> fails constraints: (constraints=((type=str, nullable=False, coerce=False), (type=Path, nullable=False)), nullable=False)
# also fails, identically other than the order of output
typic.validate(Union[Path, str], Path.home())

This confusing type union came from my discovering that I don't actually want strict mode, I want the type coercion to happen, and removing the strict=True from various things. The original type annotation was Union[str, Path, PosixPath] because I found this on macOS, and worked around it blindly thinking it was odd, but strict=True required it.

Now I'm pretty sure it is a ... well, not ideal, but probably an issue picking which type to coerce into or something?

Error with class name collisions

• typical version: 2.0.6
• Python version: 3.8.1
• Operating System: Mac OSX

Description

Ran into some weird bugs when using multiple classes with the same name (defined in different files). There must be some kind of caching-by-class-name going on.

Seems to be related to calling .transmute() with one another as an argument.

What I Did

# index.py
from other import my_function
import typic

@typic.klass
class MyClass:
  field: int
  def __post_init__(self):
    print("index.py: MyClass is being constructed")

# Construct a MyClass in other.py
other = my_function()

# Construct a local MyClass
MyClass(field=1)

# This should only construct a local MyClass, but instead calls __post_init__
# in other.py
MyClass.transmute(other)



# other.py
import typic

@typic.klass
class MyClass:
  field: int
  def __post_init__(self):
    print("other.py: MyClass is being constructed")

def my_function():
  val = MyClass(field=1)
  return val

Running python index.py prints this:

other.py: MyClass is being constructed
index.py: MyClass is being constructed
other.py: MyClass is being constructed

You would expect this:

other.py: MyClass is being constructed
index.py: MyClass is being constructed
index.py: MyClass is being constructed

Streaming primitive() / Faster serialization

Description

Allowing for the real-time streaming of the results of calls to primitive could have huge performance gains when dumping an object directly to JSON. This should be investigated.

RecursionError after subclassed if sub class defined __setattr__

• typical version: 2.0.4
• Python version:
• Operating System:

Description

This might still be related to MRO:

import typic


@typic.klass
class Base:
    ...


@typic.klass
class Foo(Base):
    foo: str

    def __setattr__(self, k, v):
        # print('__setattr__')
        super().__setattr__(k, v)


foo = Foo(foo='foo')  # RecursionError

Support "ORM" mode

Pydantic features an "orm" mode for translating directly from an ORM model.

This should be a relatively easy feature to implement, since we're really just creating a mapping between objects.

https://github.com/samuelcolvin/pydantic/blob/master/pydantic/main.py#L448

We could probably easily support to/from with very little effort.