seanfisk / python-project-template Goto Github PK

I think—and I could be wrong, but I’m pretty sure about this—pylint can do everything flake8 does, and more.

As usual for this sort of tool, it’s configurable on a user- and project-basis, and the checks it performs can be further refined on the command line. Multiple output formats exist, including configuration options for tailoring the existing outputs to your liking.

Plus, I just like tools with lint in their name better than tools with flake in their name.

(“Flakes?” Yuck! I’ll admit my code has lint, but I never want to hear that it has flakes! 👅)

Flake8 now offers a much-improved interface and it shouldn't be necessary to subprocess for it anymore.

Currently there is a hard-dependency on git in pavement.py:get_project_files().

There could be reasons to use another version control system (e.g. company policy), or project is probably not yet under version control at all (fresh project).

I would expect that "all" paver commands work without any dependencies on a specific version control system.

In main.py.tpl, rename main to entry_point and _main to main. The justification here is that Python code that calls the program's main will almost always want to pass arguments, in which case it would call the old _main. However, having a leading underscore indicates a private function.

This will also help when main is dependency injected, in which case we can have three functions: A zero-arg entry_point for setuptools, a main which is passed argv, and _main which is passed argv and the dependency-injected functions.

Hi,

I am trying out your package in order to create templates for my own work. I followed the instructions, but when I execute step 11 paver test_all I get the following error:

usage: paver [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: paver --help [cmd1 cmd2 ...]
or: paver --help-commands
or: paver cmd --help

error: invalid command 'test_all'

I tried using paver test, which causes a different error, namely

Traceback (most recent call last):
File "/usr/local/bin/paver", line 11, in <module>
 sys.exit(main())
File "/usr/local/lib/python2.7/site-packages/paver/tasks.py", line 878, in main
  _launch_pavement(args)
File "/usr/local/lib/python2.7/site-packages/paver/tasks.py", line 846, in _launch_pavement
  exec(compile(source, environment.pavement_file, 'exec'), mod.__dict__)
File "pavement.py", line 12, in <module>
  from setup import (
ImportError: cannot import name setup_dict

Any idea what is going on? I see the functions and the required data in the scripts. Does it need to run in order to use the template (I wasn't sure if the pave command would do additional changes or not)? Does my packages code go in main.py or in another file (e.g.__main__.py or my package.py) and the is called through __init__.py?

Thanks for the help!

In setup.py.tpl, only install the argparse module if we are using 2.6.

---> pavement.test_all


Captured Task Output:
---------------------

---> pavement.test_all
Traceback (most recent call last):
  File "/home/flip111/python/myproject/venv/lib64/python3.6/site-packages/paver/tasks.py", line 195, in _run_task
    return do_task()
  File "/home/flip111/python/myproject/venv/lib64/python3.6/site-packages/paver/tasks.py", line 192, in do_task
    return func(**kw)
  File "pavement.py", line 114, in test_all
    retcode = _test_all()
  File "./setup.py", line 196, in _test_all
    return _lint() + _test()
  File "./setup.py", line 169, in _lint
    project_python_files = [filename for filename in get_project_files()
  File "./setup.py", line 170, in <listcomp>
    if filename.endswith(b'.py')]
TypeError: endswith first arg must be str or a tuple of str, not bytes

python 3.6.2

Add an option to Paver to run pytest on multiple processors using xdist.

Check out paver.easy. Specifically, we should wrap our destructive commands in dry() and print using error(), info(), and debug().

Not sure if we should use sh() at this point since it runs it through the shell, which is almost never necessary and often a bad idea.

This will allow us to be more strict in making sure the docs build successfully. This can be done with the -W flag to sphinx-build, but we'll probably have to override paver.doctools.html() to do it.

I write here to document my trials and errors how to run this template in case someone else run to this problem

After changing my_module/metadata.py as suggested and running internal/generate.py, to my surprise both <my_named_module>/main.py and tests/test_main.py complained, that they can not find <my_named_module> (note, "paver test_all" worked).

The solution was to add path/to/project to PYTHONPATH

• Add LICENSE file (says ISC in setup.py.tpl, but I want to change to MIT).
• Add table of licenses of dependencies and check that they're compatible.

I have a number of projects on which work is currently being done that use this template project. It would be nice if I could run a script to update the project to the current version of the template project.

A possible solution would involve the following steps:

• Create a new "fake" project based on the template project.
• Replace the fake project's metadata.py with the old project.
• Run generation on the fake project.
• Diff the output on relevant files from the old project to the fake project and save to a patch file.
• Use git apply or plain patch to apply the patch.

Some conflict resolution would likely be necessary for most projects, but it would be much better than doing a manual merge.

In #3, we decided to take out the runtime requirements.txt file in favor of having libraries in the setup.py file. However, I've been doing some reading on pip requirements files and have a new idea for what they are actually used.

According to the aforementioned docs, the requirements.txt file should be used to create a completely reproducible environment, and these steps should be followed for reproducibility. Ian Bicking (author of pip) mentions in this StackOverflow response that "install_requires is to keep people away from things that you know don't work, while requirements files to lead people towards things you know do work."

What I'm planning to do is create a clean virtual environment and generate a requirements.txt using pip freeze. I will also add a section to the README explaining the difference between requirements files and install_requires. Opinions welcome.

________________________ ERROR collecting tests/scheduler/test_scheduler.py ________________________
.tox/py27/lib/python2.7/site-packages/py/_path/local.py:620: in pyimport
>           __import__(modname)
.tox/py27/lib/python2.7/site-packages/_pytest/assertion/rewrite.py:134: in find_module
>       co = _read_pyc(fn_pypath, pyc)
.tox/py27/lib/python2.7/site-packages/_pytest/assertion/rewrite.py:311: in _read_pyc
>           co = marshal.load(fp)
E           ValueError: bad marshal data (unknown type code)
===================================== 12 error in 0.46 seconds =====================================
  ___ _   ___ _    ___ ___
 | __/_\ |_ _| |  | __|   \
 | _/ _ \ | || |__| _|| |) |
 |_/_/ \_\___|____|___|___/

I found a discussion of this problem, and the corresponding solution.

I did a quick analysis on a fresh generated project using pyqver, a tool for querying the required Python version.

bird:src benjixx$ cp -r python-project-template/ pytemp

bird:src benjixx$ cd pytemp/

bird:pytemp benjixx$ python GENERATE.py
Substituting ./pavement.py.tpl -> ./pavement.py
Substituting ./docs/make.bat.tpl -> ./docs/make.bat
Substituting ./docs/Makefile.tpl -> ./docs/Makefile
Substituting ./docs/source/conf.py.tpl -> ./docs/source/conf.py
Substituting ./docs/source/index.rst.tpl -> ./docs/source/index.rst
Substituting ./docs/source/README.tpl -> ./docs/source/README
Substituting ./my_module/__init__.py.tpl -> ./my_module/__init__.py
Substituting ./my_module/main.py.tpl -> ./my_module/main.py
Substituting ./tests/test_main.py.tpl -> ./tests/test_main.py
Renaming the package: my_module -> my_module
Generation script imploding...

bird:pytemp benjixx$ find . -name "*.py" | xargs python ../pyqver/pyqver2.py  -v
./docs/source/conf.py
./my_module/__init__.py
./my_module/main.py
    2.7 argparse
    2.6 string literal .format(), string literal .format(), string literal .format()
./my_module/metadata.py
./pavement.py
    2.6 string literal .format()
    2.5 with statement
    2.4 subprocess, set, function decorator, function decorator, function decorator, function decorator, function decorator, function decorator, function decorator, function decorator
./tests/test_main.py
    2.6 string literal .format()
    2.5 with statement, with statement, with statement, with statement
    2.4 function decorator, function decorator

The argparse mentioned requiring Python 2.7 is already included in requirements/run.txt as a compatibility module for older Python versions.
So, looks like we currently require at least Python 2.6.

What do you think? Is this sufficient? Or should it be Python>=2.5?

I guess Python 3 is something that should be equally well supported.

I cloned the repo and trying to run the tests for which i'm getting the following error.
can you please help me out here.

Here i'm trying to run it from a virtualenv

 Traceback (most recent call last):
  File "/home/someuser/virtualenvs/python/2.7/bin/tox", line 9, in <module>
    load_entry_point('tox==1.6.1', 'console_scripts', 'tox')()
  File "/home/someuser/virtualenvs/python/2.7/local/lib/python2.7/site-packages/tox/_cmdline.py", line 26, in main
    retcode = Session(config).runcommand()
  File "/home/someuser/virtualenvs/python/2.7/local/lib/python2.7/site-packages/tox/_cmdline.py", line 301, in runcommand
    return self.subcommand_test()
  File "/home/someuser/virtualenvs/python/2.7/local/lib/python2.7/site-packages/tox/_cmdline.py", line 433, in subcommand_test
    if self.setupenv(venv):
  File "/home/someuser/virtualenvs/python/2.7/local/lib/python2.7/site-packages/tox/_cmdline.py", line 358, in setupenv
    status = venv.update(action=action)
  File "/home/someuser/virtualenvs/python/2.7/local/lib/python2.7/site-packages/tox/_venv.py", line 142, in update
    self.install_deps(action)
  File "/home/someuser/virtualenvs/python/2.7/local/lib/python2.7/site-packages/tox/_venv.py", line 272, in install_deps
    self._install(deps, action=action)
  File "/home/someuser/virtualenvs/python/2.7/local/lib/python2.7/site-packages/tox/_venv.py", line 334, in _install
    extraenv=extraenv)
  File "/home/someuser/virtualenvs/python/2.7/local/lib/python2.7/site-packages/tox/_venv.py", line 303, in run_install_command
    extraenv=env, action=action)
  File "/home/someuser/virtualenvs/python/2.7/local/lib/python2.7/site-packages/tox/_venv.py", line 382, in _pcall
    return action.popen(args, cwd=cwd, env=env, redirect=redirect)
  File "/home/someuser/virtualenvs/python/2.7/local/lib/python2.7/site-packages/tox/_cmdline.py", line 96, in popen
    popen = self._popen(args, cwd, env=env, stdout=f, stderr=STDOUT)
  File "/home/someuser/virtualenvs/python/2.7/local/lib/python2.7/site-packages/tox/_cmdline.py", line 153, in _popen
    stdout=stdout, stderr=stderr, env=env)
  File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

I also tried to install tox and run the tests but same error is occurring again.

I have different kind of python projects in mind that require slightly different templates:

• lib (library that is used in other Python projects),
• cli (application that is used -- probably interactively -- from command line),
• daemon (application that is prepared to run as a daemon, including init.d scripts, etc.)

I'm already working on a prototype that uses Jinja (on which we already depend through Sphinx) together with template inheritance to support multiple project templates without (too much) code duplication.

Also it would be nice to get this template generation process integrated with virtualenvwrapper's mkproject command. Ideas and thoughts about this probably should go into a separate ticket.

pipenv[1] is a tool recommended by Python Packaging Authority that deprecates requirements.txt. The biggest added value is separating direct and indirect dependencies and really reproducible builds. Also has very convenient workflow regarding virtual environments. And it was written by the guy who wrote requests if you know...

[1] https://docs.pipenv.org/

When trying to install the dev requirements using pip install, I run into the following error:

$ pip install -r requirements-dev.txt
Collecting pytest==2.5.1 (from -r requirements-dev.txt (line 5))
Collecting py==1.4.19 (from -r requirements-dev.txt (line 6))
Collecting mock==1.0.1 (from -r requirements-dev.txt (line 7))
Collecting flake8==2.1.0 (from -r requirements-dev.txt (line 10))
Collecting mccabe==0.2.1 (from -r requirements-dev.txt (line 11))
Collecting pep8==1.4.6 (from -r requirements-dev.txt (line 12))
Collecting pyflakes==0.7.3 (from -r requirements-dev.txt (line 13))
Collecting Sphinx==1.2 (from -r requirements-dev.txt (line 16))
  Using cached Sphinx-1.2-py33-none-any.whl
Collecting docutils==0.11 (from -r requirements-dev.txt (line 17))
Collecting Jinja2==2.7.1 (from -r requirements-dev.txt (line 18))
Collecting MarkupSafe==0.18 (from -r requirements-dev.txt (line 19))
Collecting Pygments==1.6 (from -r requirements-dev.txt (line 20))
Collecting Paver==1.2.1 (from -r requirements-dev.txt (line 23))
  Using cached Paver-1.2.1.tar.gz
Could not import setuptools which is required to install from a source distribution.
Traceback (most recent call last):
  File "/Users/stew/.envs/rummage/lib/python3.5/site-packages/pip/req/req_install.py", line 387, in setup_py
    import setuptools  # noqa
  File "/Users/stew/.envs/rummage/lib/python3.5/site-packages/setuptools/__init__.py", line 10, in <module>
    from six.moves import filter, map
ImportError: No module named 'six'

This is fixed by changing the Paver version in requirements-dev.txt from 1.2.1 to the current 1.2.4. It's possible I'm doing something wrong--I'm not clear on why this was happening in the first place--but this change resolves the issue. I'm happy to provide further context if you tell me what info would be helpful.

It looks like the requirements-dev.txt on the master branch is 3 years old--should it be updated? I'll make a pull request so you can update it easily if you think it should be.

Distribute and Setuptools have merged into Setuptools, so it looks like Setuptools is the future. We should upgrade.