[bug] Generated classes reference types they don't import, breaks mypy checks about conjure-python

[iamdanfox]

@tjb9dc could you try 3.11.0? @ferozco merged a fix here: #79

[tjb9dc]

@iamdanfox @ferozco sadly only fixed part of the issue. The following is the same run after generating with 3.11.0 and copying over those stubs.

Imports for sundance_entity_extractor_api (net_owl_api is just missing Dict)

from abc import ABCMeta, abstractmethod
import builtins
from conjure_python_client import ConjureBeanType, ConjureFieldDefinition, ConjureUnionType, ListType, OptionalType
from typing import Optional, Set

Mypy errors

/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/net_owl_api/sundance_netowl_api/__init__.py:7: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/net_owl_api/sundance_netowl_api/__init__.py:48: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/net_owl_api/sundance_netowl_api/__init__.py:131: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/net_owl_api/sundance_netowl_api/__init__.py:193: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/net_owl_api/sundance_netowl_api/__init__.py:227: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/net_owl_api/sundance_netowl_api/__init__.py:234:64: error: Argument 1 to "OptionalType" has incompatible type "ListType"; expected "Union[Type[int], Type[float], Type[bool], Type[str], Type[ConjureType], Type[List[Any]], Type[Dict[Any, Any]]]"
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/net_owl_api/sundance_netowl_api/__init__.py:235:60: error: Argument 1 to "OptionalType" has incompatible type "ListType"; expected "Union[Type[int], Type[float], Type[bool], Type[str], Type[ConjureType], Type[List[Any]], Type[Dict[Any, Any]]]"
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/net_owl_api/sundance_netowl_api/__init__.py:275: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/net_owl_api/sundance_netowl_api/__init__.py:282:60: error: Argument 1 to "OptionalType" has incompatible type "ListType"; expected "Union[Type[int], Type[float], Type[bool], Type[str], Type[ConjureType], Type[List[Any]], Type[Dict[Any, Any]]]"
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/net_owl_api/sundance_netowl_api/__init__.py:316: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/net_owl_api/sundance_netowl_api/__init__.py:350: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/net_owl_api/sundance_netowl_api/__init__.py:398: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/net_owl_api/sundance_netowl_api/__init__.py:418: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/net_owl_api/sundance_netowl_api/__init__.py:445: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/net_owl_api/sundance_netowl_api/__init__.py:472: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:11: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:40: error: Name 'Any' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:50:39: error: Invalid base class
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:52: error: Name 'Any' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:57: error: Name 'Any' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:65: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:113: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:133: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:181: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:193: error: Name 'List' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:210: error: Name 'List' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:222: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:270: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:290: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:301: error: Name 'List' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:307: error: Name 'List' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:312: error: Name 'List' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:317: error: Name 'List' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:324: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:344: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:364: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:384: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:418: error: Name 'Dict' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:432: error: Name 'List' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:456: error: Name 'List' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:461: error: Name 'List' is not defined
/Users/tbaker/Projects/etta/sundance-entity-extractor/service/stubs/sundance_entity_extractor_api/sundance_entityextractor_api/__init__.py:466: error: Name 'List' is not defined

As you can see, ListType is now imported, but Any, Dict, and List are not. Also will need to look into what's wrong with ListType not being allowed to OptionalType and the Invalid base class error.

[tjb9dc]

I'll do some digging, could you all reopen?

[tjb9dc]

As for the Invalid Base class, it didn't like

class AdditionalPropertyFieldsVisitor(ABCMeta('ABC', (object,), {})):

Can that just inherit from ABCMeta? Switching to the following makes mypy happy

class AdditionalPropertyFieldsVisitor(ABCMeta):

[tjb9dc]

As for the ListType not allowed in OptionalType, the generated code it's upset about is

    @builtins.classmethod
    def _fields(cls):
        # type: () -> Dict[str, ConjureFieldDefinition]
        return {
            'ontology': ConjureFieldDefinition('ontology', str),
            'value': ConjureFieldDefinition('value', str),
            'id': ConjureFieldDefinition('id', str),
            'entity_arg': ConjureFieldDefinition('entity-arg', OptionalType(ListType(EntityArg))),
            'property': ConjureFieldDefinition('property', OptionalType(ListType(Property)))
        }

(args to both entity_arg and property), and the conjure yml is here

          entity-arg: optional<list<EntityArg>>
          property: optional<list<Property>>

Probably something weird with wrapping List in Optional

I would expect mypy to be okay with this because ListType is a child of ConjureType which is allowed there...mypy makes me sad sometime

[tjb9dc]

Just for others' SA, to run mypy on files that bring in imports from custom stubs (such as conjure-python generated classes importing stuff from conjure_python_client, or if you're running mypy on a separate file that leverage those conjure-python generated classes), you will have to copy over the packages you want mypy to traverse into a directory somewhere named stubs and set the environment variable MYPYPATH to that stubs directory.

My CircleCI check does something like

            source conda/bin/activate conda/envs/<your-env>/
            pip install mypy
            if [ -d "stubs" ]; then
              # Recreate the stubs directory
              rm -rf stubs
            fi
            mkdir stubs
            cp -r conda/envs/<your-env>/lib/python3.6/site-packages/<conjure-python-package> stubs/
            cp -r conda/envs/<your-env>/lib/python3.6/site-packages/conjure_python_client stubs/
            echo 'Added the following site-packages to stubs folder'
            ls -l stubs/

            mypy *.py <this throws any errors, etc>

[robert3005]

this should be fixed in 4.0.0. The problem you've had is that ListType(x) is an instantiation and not a class and is not allowed as type definition