Error when running is_of_type on defaultdict about pytypes

[lubieowoce]

I just saw this issue browsing the repo, but are you sure that Dict[str, List] is a valid type? [EDIT: turns out it is, see my next comment] I'm pretty sure that typing.List is meant to be used with a type parameter, like List[int]. If you don't want to specify the types inside, you can do List[Any] or list (i.e. the Python built-in).

[mitar]

List should be equivalent to List[Any]. In any case, this should not throw and error.

[lubieowoce]

My bad, it looks like you were right! PEP 484 says:

If a generic type is used without specifying type parameters, they (sic) assumed to be Any:

def use_map(m: Mapping) -> None:  # Same as Mapping[Any, Any]
    ...

So this usage should be supported.

[Stewori]

I think this originates from deep_type lacking support for defaultdict. We'll have to improve deep_type to handle classes from collections properly.
Re missing parameters also see pytypes.check_unbound_types and pytypes.strict_unknown_check.
Finally let me remind you to post stacktraces with having called pytypes.enable_clean_traceback(False). For completeness and convenience, here the full stacktrace:

Traceback (most recent call last):
  File "/data/workspace/linux/pytypes/quick_test/issue_52.py", line 12, in <module>
    print pytypes.is_of_type(collections.defaultdict(list), typing.Dict[str, typing.List])
  File "/data/workspace/linux/pytypes/pytypes/type_util.py", line 1909, in _isinstance
    bound_typevars_readonly, follow_fwd_refs, _recursion_check)
  File "/data/workspace/linux/pytypes/pytypes/type_util.py", line 1774, in _issubclass
    bound_typevars_readonly, follow_fwd_refs, _recursion_check)
  File "/data/workspace/linux/pytypes/pytypes/type_util.py", line 1801, in _issubclass_2
    _recursion_check)
  File "/data/workspace/linux/pytypes/pytypes/type_util.py", line 1145, in _issubclass_Mapping_covariant
    return issubclass(subclass, superclass)
  File "/usr/local/lib/python2.7/dist-packages/typing.py", line 1243, in __subclasscheck__
    raise TypeError("Parameterized generics cannot be used with class "
TypeError: Parameterized generics cannot be used with class or instance checks

[mitar]

Finally let me remind you to post stacktraces with having called pytypes.enable_clean_traceback(False).

Ah, sorry. If forget about this.

[Stewori]

Ah, sorry. If forget about this.

Actually it's my fault that this is still not pointed out in the readme. Will add that hint...

Currently the code in deep_type that resolves dict is:

elif res == dict:
	if len(obj) == 0:
		return Empty[Dict]
	if max_sample == -1 or max_sample >= len(obj)-1 or len(obj) <= 2:
		try:
			# We prefer a view (avoid copy)
			tpl1 = tuple(_deep_type(t, checked, checked_len2, depth-1, None, get_type) \
					for t in obj.viewkeys())
			tpl2 = tuple(_deep_type(t, checked, checked_len2, depth-1, None, get_type) \
					for t in obj.viewvalues())
		except AttributeError:
			# Python 3 gives views like this:
			tpl1 = tuple(_deep_type(t, checked, checked_len2, depth-1, None, get_type) for t in obj.keys())
			tpl2 = tuple(_deep_type(t, checked, checked_len2, depth-1, None, get_type) for t in obj.values())
	else:
		try:
			kitr = iter(obj.viewkeys())
			vitr = iter(obj.viewvalues())
		except AttributeError:
			kitr = iter(obj.keys())
			vitr = iter(obj.values())
		ksmpl = []
		vsmpl = []
		block = (len(obj) // max_sample)-1
		# I know this method has some bias towards beginning of iteration
		# sequence, but it's still more random than just taking the
		# initial sample and better than O(n) random.sample.
		while len(ksmpl) < max_sample:
			if block > 0:
				j = random.randint(0, block)
				k = random.randint(0, block)
				while j > 0:
					next(vitr) # discard
					j -= 1
				while k > 0:
					next(kitr) # discard
					k -= 1
			ksmpl.append(next(kitr))
			vsmpl.append(next(vitr))
		tpl1 = tuple(_deep_type(t, checked, checked_len2, depth-1, None, get_type) for t in ksmpl)
		tpl2 = tuple(_deep_type(t, checked, checked_len2, depth-1, None, get_type) for t in vsmpl)
	res = Dict[Union[tpl1], Union[tpl2]]

where res holds the result of type(obj). We need to refine it to consider defaultdict and ideally also the other dict-like types from collections, see https://docs.python.org/2/library/collections.html.
I think this is mostly a question of improving the entrance elif res == dict:. Would elif issubclass(res, dict): do it? I suspect this might target some unwanted types as well but I don't know right now. Ideas?
Specifically for defaultdict I suppose we'd want to include the default type into the resulting valuetypes union. Can someone contribute a code snippet to achieve this?

[mitar]

It seems also collections.OrderedDict fails with currently released code in the same way.

Sadly I do not have answers to questions you raised above. I guess issubclass(res, dict) would be OK to do.

[Stewori]

Yes, all classes from collections are currently not supported. deep_type does not work with subclasses of builtin types. E.g. if someone defines (can tuple be subclassed? If not, this still illustrates the issue for other types that can)

class mytuple(tuple):
    blah

deep_type(mytuple(1, 2, 3)) # would result in `mytuple`

would we want this to result in Tuple[int, int, int]? I.e. let deep_type use the closest PEP 484 type to declare a custom type? But that would loose the info that it is not an ordinary tuple but actually a mytuple.
Let's say the author of mytuple also created an appropriate generic type MyTuple[...]. I would like to have a way she can register MyTuple in pytypes such that it can utilize it to represent types of mytuple objects. We could then have PEP 484 generics like OrderedDict which can simply subclass Dict. DefaultDict already exists: https://www.python.org/dev/peps/pep-0484/#instantiating-generic-classes-and-type-erasure. Is there actually an official PEP 484 way to describe OrderedDict? I think it's just Dict. What if a function really requires an ordered dict rather than a dict?

Okay, I suggest the following:
Lets add support for datattypes in collections explicitly in the usual fashion.
Sooner or later I would like to modulize deep_type such that we have a set of objects of kind of class typer that contain the logic to find the PEP 484 type for an object of a given ordinary type. We would then have typers for dict, defaultdict, tuple etc and a user can simply inject support for mytuple by adding a corresponding typer to that list. However, that's more the longterm plan.

[Stewori]

typer could also serve to place optionally some custom logic for type checking or subtype checking specific types. This would also somewhat modulize the complex logic in is_subtype.

[mitar]

I found out that this occurs even with a regular dict: #56