Module loaded more than once? about pdoc

[KMouratidis]

Fair point. MWE, two files in the same directory.

server.py

print(__name__)
from dash import Dash

app = Dash(__name__)
app.config['suppress_callback_exceptions'] = True

and app.py:

print(__name__)
from server import app
from dash.dependencies import Input, Output
import dash_html_components as html

app.layout = html.Div([
    html.H4("Hello world", id="greeter"),
    html.Button("Hide this", id="sneaky", n_clicks=0)
])

# Show or hide H4 element
@app.callback(Output("greeter", "style"),
              [Input("sneaky", "n_clicks")])
def show_login_input(value):
    """DOCSTRING!"""

    if value % 2 == 0:
        return {"display": "inline"}
    return {"display": "none"}

(Edit: actually running the app or not with app.run_server inside app.py, or debug=True/False didn't matter so I removed them).
(Edit2: adding a print statement in app.py and another in server.py shows that app.py is loaded twice)

Then using the cli: pdoc --html app.py returns the complete traceback:

Click to expand full traceback

  Traceback (most recent call last):
  File "/home/kmourat/venv/lib/python3.6/site-packages/pdoc/__init__.py", line 540, in import_module
    module.__loader__.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/kmourat/Desktop/mew/app.py", line 13, in <module>
    [Input("sneaky", "n_clicks")])
  File "/home/kmourat/venv/lib/python3.6/site-packages/dash/dash.py", line 1018, in callback
    self._validate_callback(output, inputs, state)
  File "/home/kmourat/venv/lib/python3.6/site-packages/dash/dash.py", line 836, in _validate_callback
    raise exceptions.DuplicateCallbackOutput(msg)
dash.exceptions.DuplicateCallbackOutput: 
You have already assigned a callback to the output
with ID "greeter" and property "style". An output can only have
a single callback function. Try combining your inputs and
callback functions together into one function.


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kmourat/venv/bin/pdoc", line 11, in <module>
    sys.exit(main())
  File "/home/kmourat/venv/lib/python3.6/site-packages/pdoc/cli.py", line 338, in main
    for module in args.modules]
  File "/home/kmourat/venv/lib/python3.6/site-packages/pdoc/cli.py", line 338, in <listcomp>
    for module in args.modules]
  File "/home/kmourat/venv/lib/python3.6/site-packages/pdoc/__init__.py", line 542, in import_module
    raise ImportError('Error importing {!r}: {}'.format(filename, e))
ImportError: Error importing 'app.py': 
You have already assigned a callback to the output
with ID "greeter" and property "style". An output can only have
a single callback function. Try combining your inputs and
callback functions together into one function.

Versions (pdoc is 0.5.3 as mentioned above, python 3.6.6, GCC 8.0.1 20180414):

dash_html_components==0.15.0
dash_core_components==0.46.0
dash==0.41.0
dash_callback_chain==0.0.2
dash_table==3.6.0

[kernc]

No, no, no, this should work!

Welcome to test PR #62 as it contains importing improvements that might make a difference for you.

[KMouratidis]

Nice! It works correctly now! Thanks once again :)

[kernc]

Possible to get a full, reproducible mwe (including requirements)?

I'd opine pdoc --html app.py imports app.py only once.

[kernc]

Thanks. Added the prints for you.

I think I understand the problem:

$ python -c 'import app.py'

app
server
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named 'app.py'; 'app' is not a package

pdoc first tries to import the CLI-specified module (app.py) with importlib.import_module(). The python import machinery thinks app.py is a package path, so it imports app from the current directory, then it tries to import module py from it, whereby it fails because app is not a package. But app remains imported and is later imported once again as a file.

I'm not sure what to do about it. 😕

[KMouratidis]

I am not sure about it either, I can only write about a few things I tried but there will probably be more noise rather information. Anyway, here goes nothing:

Of course commenting these out at least creates the html for the single file, but it doesn't work for the whole project. Sadly, even manually trying to use pdoc-cli on every file doesn't succeed for a significant number of files, with more or less errors of the same nature:

Example Traceback

Traceback (most recent call last):
  File "/home/kmourat/venv/lib/python3.6/site-packages/pdoc/__init__.py", line 540, in import_module
    module.__loader__.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/kmourat/Desktop/MWE2/EDA_miner/apps/exploration/Exploration.py", line 114, in <module>
    [State("user_id", "children")])
  File "/home/kmourat/venv/lib/python3.6/site-packages/dash/dash.py", line 1018, in callback
    self._validate_callback(output, inputs, state)
  File "/home/kmourat/venv/lib/python3.6/site-packages/dash/dash.py", line 836, in _validate_callback
    raise exceptions.DuplicateCallbackOutput(msg)
dash.exceptions.DuplicateCallbackOutput: 
                Multi output ..xvars_2d.options...yvars_2d.options...yvars_2d.multi.. contains an `Output` object
                that was already assigned.
                Duplicates:
                {'yvars_2d.multi', 'yvars_2d.options', 'xvars_2d.options'}
                

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kmourat/venv/bin/pdoc", line 11, in <module>
    sys.exit(main())
  File "/home/kmourat/venv/lib/python3.6/site-packages/pdoc/cli.py", line 338, in main
    for module in args.modules]
  File "/home/kmourat/venv/lib/python3.6/site-packages/pdoc/cli.py", line 338, in <listcomp>
    for module in args.modules]
  File "/home/kmourat/venv/lib/python3.6/site-packages/pdoc/__init__.py", line 542, in import_module
    raise ImportError('Error importing {!r}: {}'.format(filename, e))
ImportError: Error importing './apps/exploration/Exploration.py': 
                Multi output ..xvars_2d.options...yvars_2d.options...yvars_2d.multi.. contains an `Output` object
                that was already assigned.
                Duplicates:
                {'yvars_2d.multi', 'yvars_2d.options', 'xvars_2d.options'}

Of course it could be me not handling the project structure correctly, since the MWE now works.

Directory tree of actual project

.
├── app.py
├── apps
│   ├── analyze
│   │   ├── Classification.py
│   │   ├── Clustering.py
│   │   ├── Econometrics.py
│   │   ├── __init__.py
│   │   ├── Model_Builder.py
│   │   ├── models
│   │   │   ├── graph_structures.py
│   │   │   ├── __init__.py
│   │   │   ├── pipeline_classes.py
│   │   │   ├── pipeline_creator.py
│   │   │   └── utils.py
│   │   ├── Pipelines.py
│   │   └── Regression.py
│   ├── analyze_view.py
│   ├── data
│   │   ├── APIs.py
│   │   ├── data_utils
│   │   │   ├── api_connectors.py
│   │   │   ├── api_layouts.py
│   │   │   └── __init__.py
│   │   ├── __init__.py
│   │   ├── Upload.py
│   │   └── View.py
│   ├── data_view.py
│   ├── exploration
│   │   ├── Exploration3D.py
│   │   ├── Exploration.py
│   │   ├── graphs
│   │   │   ├── graphs2d.py
│   │   │   ├── graphs3d.py
│   │   │   ├── __init__.py
│   │   │   ├── kpis.py
│   │   │   └── word_cloud.py
│   │   ├── __init__.py
│   │   ├── KPIs.py
│   │   ├── Networks.py
│   │   ├── PDF_report.py
│   │   └── TextVisualizations.py
│   ├── exploration_view.py
│   └── __init__.py
├── assets
│   ├── 10_plotly_base.css
│   ├── 20_styles.css
│   ├── 30_styles.css
│   ├── favicon.ico
│   └── images
│       ├── graph_abs.jpg
│       ├── icons
│       │   ├── Cleaning.png
│       │   ├── decision_tree.png
│       │   ├── files.png
│       │   ├──  hierarchical_clustering.png
│       │   ├── knn.png
│       │   ├── layers.png
│       │   ├──  linear_regression.png
│       │   ├──  logistic_regression.png
│       │   ├── new_name.png
│       │   ├──  random_forests.png
│       │   ├──  ridge_regression.png
│       │   └── svm.png
│       └── y2d.png
├── Data
│   └── Example_CSVs
│       ├── boston.feather
│       └── Wholesale customers data.xlsx
├── default_wordcloud.png
├── Dockerfile
├── html
│   ├── app.html
│   ├── apps
│   │   ├── analyze
│   │   │   ├── Econometrics.html
│   │   │   ├── __init__.html
│   │   │   ├── models
│   │   │   │   ├── graph_structures.html
│   │   │   │   ├── __init__.html
│   │   │   │   ├── pipeline_classes.html
│   │   │   │   ├── pipeline_creator.html
│   │   │   │   └── utils.html
│   │   │   ├── Pipelines.html
│   │   │   └── Regression.html
│   │   ├── analyze_view.html
│   │   ├── data
│   │   │   ├── data_utils
│   │   │   │   ├── api_connectors.html
│   │   │   │   ├── api_layouts.html
│   │   │   │   └── __init__.html
│   │   │   ├── __init__.html
│   │   │   ├── Upload.html
│   │   │   └── View.html
│   │   ├── data_view.html
│   │   ├── exploration
│   │   │   ├── graphs
│   │   │   │   ├── graphs2d.html
│   │   │   │   ├── graphs3d.html
│   │   │   │   ├── __init__.html
│   │   │   │   ├── kpis.html
│   │   │   │   └── word_cloud.html
│   │   │   ├── __init__.html
│   │   │   ├── Networks.html
│   │   │   └── PDF_report.html
│   │   ├── exploration_view.html
│   │   └── __init__.html
│   ├── __init__.html
│   ├── layouts.html
│   ├── menus.html
│   ├── styles.html
│   └── utils.html
├── images
│   ├── callback_chain.png
│   ├── directory_tree.png
│   └── screenshots
│       ├── API_connect2.png
│       ├── API_connect.png
│       ├── Baseline.png
│       ├── FittingModels.png
│       ├── Landing_page.png
│       ├── Matplotlib.png
│       ├── ModelBuilder.png
│       ├── PDF_Reports.png
│       ├── Preview_Data.png
│       ├── Upload.png
│       └── WordCloud.png
├── __init__.py
├── layouts.py
├── menus.py
├── printable_layout.txt
├── README.md
├── redisData.pkl
├── reportapp.py
├── requirements.txt
├── server.py
├── static
│   └── images
├── styles.py
└── utils.py

However it also fails when I'm trying to do use pdoc-cli on the MWE directory.

Directory tree of MWE

MWE
├── app.py
├── __init__.py
└── server.py

All that said, I did notice something of potential interest: it is imports within the python modules that break pdoc. For example here is a truncated version of a problematic module:

### OTHER IMPORTS

from server import app
import layouts
from utils import create_dropdown, mapping, get_data
from apps.exploration.graphs.graphs2d import scatterplot

### VARIOUS STUFF

@app.callback(
### CALLBACK STUFF
)
def fit(VARIOUS_VARS):
    """A SIMPLE DOCSTRING"""

    ### VARIOUS STUFF

    if len(xvars) == 2:
        trace = scatterplot(df[xvars[0]], df[xvars[1]],
                            marker={'color': labels.astype(np.float)})
    ### OTHER STUFF

It turns out removing this line from apps.exploration.graphs.graphs2d import scatterplot solves the issue. Either I should change how imports are done ( don't know how :/ ) or there might be a problem with how pdoc loads sibling-packages?

[KMouratidis]

Anyhow, don't fret too much as this probably is a Dash-only issue. Monkey-patching the app.callback to do no validation, along with manually calling pdoc on every file (os.walk + os.system("pdoc...")) fixes the issue and all the files are correctly handled. Any idea how to create an index file manually?

[KMouratidis]

Thanks! It works! I've also added a few packages and modules in the MWE and it works correctly as well.

I can't seem to get it to work with my project due to some import error: ImportError: Error importing 'apps.analyze': expected str, bytes or os.PathLike object, not NoneType. I've been trying for hours to understand what's wrong but I can't figure it out. Here's what reproduces the error:

A folder (models) with only a single file (example.py), and using the CLI (e.g. pdoc --html . from within the directory or pdoc --html models from outside):

"""Docstring"""

# version 0.20.3
from sklearn.base import BaseEstimator, ClassifierMixin, TransformerMixin
from sklearn.linear_model import LinearRegression

pass

ImportError: Error importing 'models.example': expected str, bytes or os.PathLike object, not NoneType

Complete traceback

/home/kmourat/venv/lib/python3.6/distutils/__init__.py:4: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
Traceback (most recent call last):
  File "/usr/lib/python3.6/pkgutil.py", line 412, in get_importer
    importer = sys.path_importer_cache[path_item]
KeyError: None

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kmourat/venv/lib/python3.6/site-packages/pdoc/__init__.py", line 180, in import_module
    module = importlib.import_module(module_path)
  File "/home/kmourat/venv/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/kmourat/GitHub/EDA_Miner_docs/testPDOC/EDA_miner/apps/analyze/models/example.py", line 6, in <module>
    from sklearn.linear_model import LinearRegression
  File "/home/kmourat/venv/lib/python3.6/site-packages/sklearn/linear_model/__init__.py", line 12, in <module>
    from .base import LinearRegression
  File "/home/kmourat/venv/lib/python3.6/site-packages/sklearn/linear_model/base.py", line 38, in <module>
    from ..preprocessing.data import normalize as f_normalize
  File "/home/kmourat/venv/lib/python3.6/site-packages/sklearn/preprocessing/__init__.py", line 6, in <module>
    from ._function_transformer import FunctionTransformer
  File "/home/kmourat/venv/lib/python3.6/site-packages/sklearn/preprocessing/_function_transformer.py", line 5, in <module>
    from ..utils.testing import assert_allclose_dense_sparse
  File "/home/kmourat/venv/lib/python3.6/site-packages/sklearn/utils/testing.py", line 63, in <module>
    from nose.tools import raises as _nose_raises
  File "/home/kmourat/venv/lib/python3.6/site-packages/nose/__init__.py", line 1, in <module>
    from nose.core import collector, main, run, run_exit, runmodule
  File "/home/kmourat/venv/lib/python3.6/site-packages/nose/core.py", line 11, in <module>
    from nose.config import Config, all_config_files
  File "/home/kmourat/venv/lib/python3.6/site-packages/nose/config.py", line 9, in <module>
    from nose.plugins.manager import NoPlugins
  File "/home/kmourat/venv/lib/python3.6/site-packages/nose/plugins/__init__.py", line 185, in <module>
    from nose.plugins.manager import *
  File "/home/kmourat/venv/lib/python3.6/site-packages/nose/plugins/manager.py", line 418, in <module>
    import pkg_resources
  File "/home/kmourat/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3086, in <module>
    @_call_aside
  File "/home/kmourat/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3070, in _call_aside
    f(*args, **kwargs)
  File "/home/kmourat/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 3099, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/home/kmourat/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 565, in _build_master
    ws = cls()
  File "/home/kmourat/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 558, in __init__
    self.add_entry(entry)
  File "/home/kmourat/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 614, in add_entry
    for dist in find_distributions(entry, True):
  File "/home/kmourat/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1866, in find_distributions
    importer = get_importer(path_item)
  File "/usr/lib/python3.6/pkgutil.py", line 416, in get_importer
    importer = path_hook(path_item)
TypeError: expected str, bytes or os.PathLike object, not NoneType

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kmourat/venv/bin/pdoc", line 11, in <module>
    sys.exit(main())
  File "/home/kmourat/venv/lib/python3.6/site-packages/pdoc/cli.py", line 421, in main
    for module in args.modules]
  File "/home/kmourat/venv/lib/python3.6/site-packages/pdoc/cli.py", line 421, in <listcomp>
    for module in args.modules]
  File "/home/kmourat/venv/lib/python3.6/site-packages/pdoc/__init__.py", line 595, in __init__
    m = import_module(fullname)
  File "/home/kmourat/venv/lib/python3.6/site-packages/pdoc/__init__.py", line 182, in import_module
    raise ImportError('Error importing {!r}: {}'.format(module, e))
ImportError: Error importing 'models.example': expected str, bytes or os.PathLike object, not NoneType

Is this because of relative imports? It seems some libraries (e.g. networkx) have the same issue, while others don't (e.g. pickle, quandl, dash).

[kernc]

Fixed in the last commit added to the PR (a3d0f25).

[kernc]

Thank you for the help! 🍰