linkedin / pygradle

Using Gradle to build Python projects

PyGradle

PyGradle is an enterprise Python build system.

PyGradle leverages Gradle to empower Python's existing ecosystem to solve problems like dependency management, polyglot projects, and lots more. LinkedIn has been using PyGradle for several years to successfully manage thousands of Python dependencies.

PyGradle produces artifacts, e.g., source distributions, that are forward and backward compatible with artifacts produced by vanilla setuptools, so there is nothing stopping you from using PyGradle with one, two, or all of your Python projects right away!

As a general philosophy, we strive to enhance Python rather than replace it, thereby keeping open source Python development idiomatic and intuitive.

Usage

PyGradle does not include batteries: there are a few things that you're going to have to do to use PyGradle for things beyond our provided examples.

In particular, you're going to have to:

• Decide how you want to host your artifacts.
• Decide how you want to distribute a custom setuptools distribution class.
• Author your build.gradle file.

These decisions and tasks shouldn't take too long for you finish, but they are required. A deeper dive into what it takes to get started, how you can use our demonstrative Artifactory instance, and a few project examples is available in our detailed getting started guide.

Although not required, if you've never used Gradle before, you might find reading Gradle's user guide helpful.

Plugins

PyGradle comes with several plugins available, for specific details on each plugin check the documentation specific to that plugin.

Plugin	Usage
com.linkedin.python	Extending PyGradle
com.linkedin.python-sdist	Developing Source Distributions
com.linkedin.python-web-app	Developing Flask/Gunicorn Web Applications
com.linkedin.python-cli	Developing Command Line Applications
com.linkedin.python-flyer	Developing Flyer (Flask + Ember) Applications
com.linkedin.python-pex	Developing Pex Applications

Developers

We're actively maintaining PyGradle and accepting pull requests.

If you're interested in contributing code to PyGradle, please see our developers document to get started. This document covers how to build, test, and debug PyGradle, and kindly asks that you follow a few conventions when submitting pull requests.

Compatibility

Software

We support PyGradle on the following software versions.

Software	Version
Java	8.0
Gradle	5.0

We're happy to review and merge pull requests that add additional support to future versions of software.

Platforms

PyGradle works on the following platforms.

Platform	Support
Linux	Full
OS X	Full
Windows	Partial

Additional compatibility notes are availabe here. We're happy to review and merge pull requests that add support for additional platforms.

pygradle's Issues

Exclude dependency from Python3 build?

I am trying to build a project with python 3.4 and during build i get an error at one of the dependencies and it looks like:

Install functools32-3.2.3-2 ......................................... [STARTING]
Starting process 'command '/home/user/project/build/venv/bin/python''. Working directory: /home/user/project Command: /home/user/project/build/venv/bin/python /home/user/project/build/venv/bin/pip install --disable-pip-version-check --no-deps /home/user/.gradle/caches/modules-2/files-2.1/pypi/functools32/3.2.3-2/a520082a56af52c7af8d2d1390856bf1a0d755df/functools32-3.2.3-2.tar.gz
Successfully started process 'command '/home/user/project/build/venv/bin/python''
Processing /home/user/.gradle/caches/modules-2/files-2.1/pypi/functools32/3.2.3-2/a520082a56af52c7af8d2d1390856bf1a0d755df/functools32-3.2.3-2.tar.gz
    Complete output from command python setup.py egg_info:
    This backport is for Python 2.7 only.
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/user/1261914/pip-76419vxe-build
:project:installPythonRequirements FAILED
:project:installPythonRequirements (Thread[Daemon worker Thread 2,5,main]) completed. Took 0.565 secs.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':project:installPythonRequirements'.
> Failed to install functools32-3.2.3-2. Please see above output for reason, or re-run your build using ``gradle -i build`` for additional logging.

* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output.

BUILD FAILED

I should also mention that I am able to build and test the application in Python 2.7.6 using Gradle without any problem and that when creating a virtual environment for Python 3 specific for this project I do not run into any issues installing the dependencies (... even tried matching the version of my pip with forceVersion).

The build.gradle looks like:

plugins {
  id "com.linkedin.python-cli" version "0.3.36"
}

version=0.1

python {
  testDir = file('tests')
  details.prependExecutableDirectory(new File('/usr/bin/python3.4'))
  details.pythonVersion = '3.4'
}
// I did try with and without this option, maybe the order is wrong, trying to identify dependency which causes it.
project.tasks.findByName('installPythonRequirements').sorted = false

dependencies {
    python 'pypi:pathlib:1.0.1'
    python 'pypi:connexion:1.0.129'
    python 'pypi:lxml:3.6.4'
    python 'pypi:beautifulsoup4:4.5.1'
    python 'pypi:rdflib:4.2.1'
    python 'pypi:rdflib-jsonld:0.4.0'
    python 'pypi:PyMySQL:0.7.9'
    python 'pypi:gunicorn:19.6.0'
    python 'pypi:pytest-html:1.12.0'
    test 'pypi:nose:1.3.7'
}

repositories {
    // using a local repository
    ivy{
      name 'pypi-local'
  		url "/home/user/pivy"
  		layout 'pattern' , {
  			artifact '[organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]'
  			ivy '[organisation]/[module]/[revision]/[module]-[revision].ivy'
  		}
  	}
}

task wrapper(type: Wrapper) {
   gradleVersion = '3.2.1'
}

task addConfigs(type: Copy) {
    from '.'
    into 'build/deployable'
    include 'database.conf'
}

packageDeployable.dependsOn addConfigs

Maybe excluding this from the build might help ... however not sure how to do that?

Version number missing from extracted docs folder name

Issue

Once the documentation is generated the version number is correctly part of the tar.gz filename. Once I extract it out, the version number is missing and 'null' is in it's place.

For example, let say we have the project 'Example' and it's version is '1.0'. The resulting tar.gz file for html would look like Example-1.0-docs-html.tar.gz. When I extract the tar.gz file, I end up with a folder named, Example-null-docs-html.

Environment

Pygradle version: 0.3.9

Running pivy-importer fails on Mac

I attempted to run pivy-importer on my MacBook Pro 2015, Siera v. 10.12.3, Java 1.8.0_112 and ended up with this error

MUSRS186002-942:tmp rs186002$ java -jar pivy-importer-0.4.1-all.jar --repo ~/tmp/pypi virtualenv:15.0.1 pip:7.1.2 --replace alabaster:0.7=alabaster:0.7.1,pytz:0a=pytz:2016.4,Babel:0.8=Babel:1.0,sphinx_rtd_theme:0.1=sphinx_rtd_theme:0.1.1
alabaster:0.7=alabaster:0.7.1
pytz:0a=pytz:2016.4
Babel:0.8=Babel:1.0
sphinx_rtd_theme:0.1=sphinx_rtd_theme:0.1.1
12:53:02.579 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in virtualenv:15.0.1
12:53:04.256 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in pip:7.1.2
12:53:04.777 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in pytest:3.0.6
12:53:05.196 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in virtualenv:1.10
12:53:05.872 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in scripttest:1.3
12:53:05.936 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in mock:2.0.0
12:53:06.671 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in py:1.4.29
12:53:06.825 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in setuptools:34.1.1
12:53:06.940 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in argparse:1.4.0
12:53:07.010 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in colorama:0.3.7
12:53:07.076 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in pbr:0.11.0
12:53:07.161 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in six:1.9.0
12:53:07.228 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in funcsigs:1.0.0
12:53:07.354 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in Sphinx:1.5.2
12:53:09.066 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in Jinja2:2.7
12:53:09.386 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in Pygments:2.0
12:53:10.152 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in Sphinx:1.3
12:53:11.369 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in unittest2:1.1.0
12:53:11.506 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in pip:9.0.1
12:53:11.748 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in ordereddict:1.1
12:53:11.811 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in six:1.5.0
12:53:11.883 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in Jinja2:2.3
12:53:12.307 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in docutils:0.11
12:53:12.514 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in snowballstemmer:1.1.0
12:53:12.637 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in Babel:1.3
12:53:13.104 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in alabaster:0.7.1
12:53:13.169 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in imagesize:0.7.1
12:53:13.232 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in requests:2.4.0
12:53:13.403 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in colorama:0.3.5
12:53:13.473 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in nose:1.3.7
12:53:13.568 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in simplejson:3.10.0
12:53:13.638 INFO  c.l.p.i.deps.DependencyDownloader - Pulling in html5lib:1.0b10
Exception in thread "main" java.lang.RuntimeException: Unable to find version 2.2 for chardet
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:80)
	at org.codehaus.groovy.reflection.CachedConstructor.doConstructorInvoke(CachedConstructor.java:74)
	at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrap.callConstructor(ConstructorSite.java:84)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:60)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:235)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:247)
	at com.linkedin.python.importer.pypi.ProjectDetails.maybeFixVersion(ProjectDetails.groovy:55)
	at com.linkedin.python.importer.pypi.ProjectDetails$maybeFixVersion.call(Unknown Source)
	at com.linkedin.python.importer.distribution.SourceDistPackage$_parseRequiresText_closure1.doCall(SourceDistPackage.groovy:81)
	at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019)
	at groovy.lang.Closure.call(Closure.java:426)
	at groovy.lang.Closure.call(Closure.java:442)
	at org.codehaus.groovy.runtime.DefaultGroovyMethods.callClosureForLine(DefaultGroovyMethods.java:5236)
	at org.codehaus.groovy.runtime.StringGroovyMethods.eachLine(StringGroovyMethods.java:615)
	at org.codehaus.groovy.runtime.StringGroovyMethods.eachLine(StringGroovyMethods.java:636)
	at org.codehaus.groovy.runtime.StringGroovyMethods.eachLine(StringGroovyMethods.java:595)
	at org.codehaus.groovy.runtime.dgm$1060.invoke(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274)
	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
	at com.linkedin.python.importer.distribution.SourceDistPackage.parseRequiresText(SourceDistPackage.groovy:50)
	at com.linkedin.python.importer.distribution.SourceDistPackage.getDependencies(SourceDistPackage.groovy:41)
	at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.MethodMetaProperty$GetBeanMethodMetaProperty.getProperty(MethodMetaProperty.java:76)
	at org.codehaus.groovy.runtime.callsite.GetEffectivePogoPropertySite.callGroovyObjectGetProperty(GetEffectivePogoPropertySite.java:68)
	at com.linkedin.python.importer.deps.DependencyDownloader.downloadDependency(DependencyDownloader.groovy:72)
	at com.linkedin.python.importer.deps.DependencyDownloader$downloadDependency.callCurrent(Unknown Source)
	at com.linkedin.python.importer.deps.DependencyDownloader.download(DependencyDownloader.groovy:50)
	at com.linkedin.python.importer.ImporterCLI.main(ImporterCLI.java:73)

Update EntryPointHelpers#writeEntryPointScript to accept template

com.linkedin.gradle.python.util.EntryPointHelpers#writeEntryPointScript should be able to accept a template (probably as a String containing the template vs a resource name) so that users can update it as needed.

Multiple Source Directories?

We have a project that contains both generated sources and hand written sources.
It looks like this plugin only supports one source directory. Is this correct?

https://github.com/linkedin/pygradle/blob/master/pygradle-plugin/src/main/groovy/com/linkedin/gradle/python/PythonExtension.groovy#L51

Multiple Source Directories?

Issue building with pygradle and local pypi repository

I have created a local pypi repository using pyivy-importer and referring to the dependencies accordingly. Here is the completely build.gradle:

plugins {
    id "com.linkedin.python-sdist" version "0.3.9"
}

dependencies {
    python 'pypi:requests:2.11.1'
    python 'pypi:pandas:0.18.1'
    python 'pypi:PyYAML:3.11'
    test 'pypi:mock:2.0.0'
}

repositories {
    // This repository is provided for example purposes only
    ivy {
        name 'pypi-local'   //optional, but nice
        url '<DIR>/pypi'
        layout "pattern", {
          ivy "[organisation]/[module]/[revision]/[module]-[revision].ivy"
          artifact "[organisation]/[module]/[revision]/[artifact]-[revision](-[classifier]).[ext]"
          m2compatible = true
        }
    }
}

I know that the pypi reference is being done correctly. However I am still getting the following errors. Do I need to override the dependency version for each task. The reason I ask is because when I created the local pypi repository using pyivy-importer, as an example, setuptools version 19.1.1 was never installed. All I have are version 11.3, 18.5 and 27.3.0. So how do I tell gradle to look for setuptools version 27.3.0 and not 19.1.1. There are other similar failures.

* What went wrong:
Execution failed for task ':installSetupRequirements'.
> Could not resolve all dependencies for configuration ':setupRequires'.
   > Could not find pypi:setuptools:19.1.1.
     Searched in the following locations:
         file:/<DIR>/pypi/setuptools/19.1.1/setuptools-19.1.1.ivy
         file:/<DIR>/pypi/setuptools/19.1.1/setuptools-19.1.1.jar
     Required by:
         :data-ingestion-scripts:unspecified

ImporterCLI wrapped in a gradle task

I'm using this code for our internal creation of the ivy repo.
Others may find this useful.
This file is in our buildSrc.

import com.linkedin.python.importer.ImporterCLI
import org.gradle.api.DefaultTask
import org.gradle.api.tasks.Input
import org.gradle.api.tasks.OutputDirectory
import org.gradle.api.tasks.TaskAction

import java.util.stream.Collectors

class ImporterTask extends DefaultTask {

    @OutputDirectory
    File repo

    @Input
    Collection<String> dependencies

    @Input
    Map<Object, Object> replace

    @TaskAction
    def runAction() {
        final List<String> args = new ArrayList<>()
        args.add("--repo")
        args.add(repo.absolutePath)
        args.addAll(dependencies)
        if (replace != null) {
            args.add("--replace")
            final String replaceString = replace
                    .entrySet()
                    .stream()
                    .map {e -> "${e.key}=${e.value}".toString()}
                    .collect(Collectors.joining(","))
            args.add(replaceString)
        }
        ImporterCLI.main(args.toArray(new String[args.size()]))
    }
}

Support for software model

In the native world, we are extensively exposed to the new software model (components container) and we did a quick implementation of a Python plugin following this model. The main advantage is to have multiple component in the same project. For example, it allows you to have a component which is a REST server and another one which is the API to talk with the server colocated inside the same project. What's the roadmap for this? As a milestone, managed type could be used for simplifying the current implementation? I could provide some help on this. Thanks for the input.

"setup.py sdist --dist-dir" command converts version numbers with `_` to `-`

This seems to be the opposite of what the setup.py bdist_egg command does.

When setup.py sdist encounters a version with an _ character in it the resulting tar.gz file has the version converted to a -

So a version 'gradleBuild_2016111701.preview'

Creates a tar.gz with the name: $project.name-gradleBuild-2016111701.preview.tar.gz

The getSdistOutput method in SourceDistTask should handle this.

How to point to internal PIP repo for installations and how to publish artifacts to Artifactory and/or PIP ?

When i try to run build it tries to reach out to default pip repo. I have my .pip/pip.conf configured to point to internal repo.

Please can you let me know how can i configure it to use an internal repo ?

12:01:05     Complete output from command python setup.py egg_info:
12:01:05     Download error on https://pypi.python.org/simple/setuptools_scm/: [Errno 101] Network is unreachable -- Some packages may not be found!

As a second question, How do i publish tar.gz to artifactory or to pip repo ? I tried publications closure does not seem to work . Thoughts ?

use /usr/bin/env interpreter for pex

While using the python-pex plugin version 0.3.9, generated .pex files use the Python interpreter in venv. This inhibits their portability. Per the pex documentation they ought to use #!/usr/bin/env python.

Convert codebase to Java 7

In an effort to make sure that the code is stable and doesn't get small bugs, lets try to move most (not all) of the code into Java. We should also adapt the code to match java8 style.

Some things to consider:
[ ] - Using Files and Paths vs File
[ ] - Using try with resource
[ ] - Doing multi catches

:installProject Exception when running example-project

I ran gradlew -i build in the example-project directory and got the following exception.
(Replaced the actual directory with <DIR>). This only happens when I copy the example-project directory out of the bigger pygradle folder to <DIR>. If I run the build command when the directory is not moved out of the bigger folder, it works. What am I missing?

Executing task ':installProject' (up-to-date check took 0.0 secs) due to:
  Task has not declared any outputs.
Install example-project ............................................. [STARTING]
Starting process 'command '<DIR>/example-project/build/venv/bin/python''. Working directory: <DIR>/example-project Command: <DIR>/example-project/build/venv/bin/python <DIR>/example-project/build/venv/bin/pip install --disable-pip-version-check --no-deps --editable <DIR>/example-project
Successfully started process 'command '<DIR>/example-project/build/venv/bin/python''
Obtaining file://<DIR>/example-project
No files/directories in <DIR>/example-project (from PKG-INFO)
:installProject FAILED
:installProject (Thread[Daemon worker Thread 2,5,main]) completed. Took 0.561 secs.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':installProject'.
> Failed to install example-project. Please see above output for reason, or re-run your build using ``ligradle -i build`` for additional logging.

* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output.

BUILD FAILED

Total time: 3.286 secs

The gradle file is

plugins {
  id "com.linkedin.python-sdist" version "0.3.9"
}

dependencies {
    python 'pypi:requests:2.9.1'
    test 'pypi:mock:1.3.0'
}

repositories {
   pyGradlePyPi()
}

Repository format

I see not really a problem but a serious limiting factor. The repository format is a fixed ivy format. I don't understand the need for a custom layout on the ivy repo, but there it is. Here's where its going to bite the project

• Sonatype Nexus (the biggest repo manager out there) has no direct support for Ivy. All its Ivy support is done thru Maven2 compatibility.
• Nexus does not support custom repo layouts. In fact, in Nexus, the whole concept of a layout is kinda obsolete. You may be able to make it work, but im not sure
• Artifactory OSS does support Ivy, but not custom layouts.
• Of course Artifactory pro supports everything.

I don't understand why you chose a custom Ivy layout, but you may want to consider porting to a standard Maven 2 layout to maximize your compatibility. And when you are in an organization like mine where you can get $500k to write a useless report at the drop of a hat, but $3k for Artifactory Pro takes a year and a half (and counting) to be approved, its an issue.

update code docs

Just a couple spots that have outdated comments:

pygradle/pygradle-plugin/src/main/groovy/com/linkedin/gradle/python/plugin/PythonPlugin.groovy

Lines 156 to 161 in 01d079e

	/**
	* Create a symlink to product-spec.json and config directory.
	*
	* This maintains compatibility with many of the assumptions our tooling
	* makes around Python projects and mppy.
	*/

pygradle/pygradle-plugin/src/main/groovy/com/linkedin/gradle/python/plugin/PythonPlugin.groovy

Line 202 in 01d079e

* A products Python requirements are those listed in the product-spec.json in the external or product sections.

✌️

example cannot build success

11:41:59.041 [DEBUG] [org.gradle.process.internal.DefaultExecHandle] Changing state to: FAILED
11:41:59.041 [DEBUG] [org.gradle.process.internal.DefaultExecHandle] Process 'command '/Users/fox/workspace/python/pygradle/examples/example-project/build/venv/bin/python'' finished with exit value 1 (state: FAILED)
11:41:59.041 [LIFECYCLE] [org.gradle.api.Task] Obtaining file:///Users/fox/workspace/python/pygradle/examples/example-project
No files/directories in /Users/fox/workspace/python/pygradle/examples/example-project (from PKG-INFO)
11:41:59.042 [DEBUG] [org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter] Finished executing task ':installProject'
11:41:59.042 [LIFECYCLE] [class org.gradle.internal.buildevents.TaskExecutionLogger] :installProject FAILED
11:41:59.042 [INFO] [org.gradle.execution.taskgraph.AbstractTaskPlanExecutor] :installProject (Thread[Daemon worker Thread 5,5,main]) completed. Took 0.44 secs.
11:41:59.042 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationWorkerRegistry] Worker root.8 completed (0 in use)
11:41:59.042 [DEBUG] [org.gradle.execution.taskgraph.AbstractTaskPlanExecutor] Task worker [Thread[Daemon worker Thread 5,5,main]] finished, busy: 2.718 secs, idle: 0.0 secs
11:41:59.043 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]
11:41:59.043 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] FAILURE: Build failed with an exception.

it seems to install self as denpendence,so can not find
gradle version:3.1
java: 1.8.0_77

Split Sphinx into its own plugin

I intend to experiment with this but I wanted your opinion. I have a huge number of java projects i want to add pygradle to because I want to position Sphinx as our language independent code documentation system and i love pygradles dependency management system. The issue is that every time you do a sphinx compile it executes the project install task. Before i waste any time trying to rip things apart and play, off the cuff, what would happen if i pulled the sphinx tasks off into their own plugin, referenced them back into the sdist plugin like before, and tried to execute them independently without the venv and install tasks running? My gut feeling is im going to screw something up in the dependency resolution system but i wanted to ask first before wasting time on it.

no main manifest attribute, in pivy-importer jar

I am trying to run the following command from the directory /pygradle/build/pivy-importer/libs after running gradlew build inside pivy-importer:

java -jar pivy-importer-0.3.16-SNAPSHOT.jar --repo /tmp/libs/ virtualenv:15.0.1 pip:7.1.2

along the lines of the example command in the docs java -jar pivy-importer.jar --repo /path/to/destination virtualenv:15.0.1 pip:7.1.2

and it gives the error:

no main manifest attribute, in pivy-importer-0.3.16-SNAPSHOT.jar

Is there any other way to generate the jar file that has the main manifest attribute?

pivy-importer: how to include a package from a URL/wheel file

E.g. the library tensorflow is not in pypi. Instead the instructions to install the library are

$ export TF_BINARY_URL=https://storage.googleapis.com/tensorflow/mac/gpu/tensorflow-0.10.0-py2-none-any.whl
sudo pip install --upgrade $TF_BINARY_URL

How do I include this package in the ivy-repo generated by pivy-importer?

Upgrading from 0.4.1 to 0.4.5 pex issue no module named ...

First of all thanks for PyGradle, been using it successfully for the past few months. I also kept a close eye on new releases, which brings me to the issue I encountered.

While trying to build a CLI python app using the pygradle com.linkedin.python-cli plugin 0.4.5 version, I noticed that it stopped working. I mean stopped working because building with version 0.4.1 it still works.

To replicate this issue one can use the https://github.com/linkedin/pygradle/tree/master/examples/iris-classification to do this.

The only lines I modified in the example are the ones related to the version and obviously the ivy repository. The version bump:

plugins {
  id "com.linkedin.python-cli" version "0.4.5"
}

The result:

$ ./build/deployable/bin/classify_iris 
Traceback (most recent call last):
  File ".bootstrap/_pex/pex.py", line 324, in execute
  File ".bootstrap/_pex/pex.py", line 257, in _wrap_coverage
  File ".bootstrap/_pex/pex.py", line 289, in _wrap_profiling
  File ".bootstrap/_pex/pex.py", line 363, in _execute
  File ".bootstrap/_pex/pex.py", line 425, in execute_entry
  File ".bootstrap/_pex/pex.py", line 439, in execute_pkg_resources
  File ".bootstrap/pkg_resources/__init__.py", line 2386, in resolve
ImportError: No module named classify_iris.run

It seems like a temporary solution after much fiddling around with dependencies is to revert back to pip 7.1.2 (pygradle 0.4.5 version uses 9.0.1):

python {
    forceVersion('pypi', 'pip', '7.1.2')
}

While I tried identifying the issue it seems to be pex related (maybe to be fixed with this: #110). Any other suggestions would be welcomed or if there is another workaround.

Thanks.

Could not create an instance of type com.linkedin.gradle.python.PythonExtension_Decorated.

When I run the example of PyGradle, I came across this exception.

my environments:
win7 64-bit
jdk1.8.0_51
gradle-3.1

thanks

Update PythonDetails to users can programmatically add directories to be added to the path

As a user, I want to be able to programmatically change how pygradle will resolve Python versions. For this feature to be successful, a user should be able to prepend to the search path and append to the search path.

Remove the need for the "replace" functionality in pivy

Seems like we shouldn't require end-users to use the replace functionality. Being explicit can be good in some cases but just automatically picking the newest version that satisfies the requirement would be nice.

It's really bad UX to constantly have to tell pivy "yeah there is no version 1.0.0, just use version 1.1.0".

Build Wheels

It'd be nice if pygradle could produce wheels. Similar to #78

Would you accept PRs to accomplish this?

main project's wheel not being added to distribution or pex

I have a pygradle project that builds/tests/runs successfully on linux but when I build it on windows the build/wheel-cache contains my main module's .whl and other dependent .whl's but the build/pex-cache never gets my module despite getting the others. I think that's why the deployable reports an import error when calling them.

C:\Projects\workflow>git workflow.py -h
Traceback (most recent call last):
  File ".bootstrap\_pex\pex.py", line 324, in execute
  File ".bootstrap\_pex\pex.py", line 257, in _wrap_coverage
  File ".bootstrap\_pex\pex.py", line 289, in _wrap_profiling
  File ".bootstrap\_pex\pex.py", line 367, in _execute
  File ".bootstrap\_pex\pex.py", line 425, in execute_entry
  File ".bootstrap\_pex\pex.py", line 439, in execute_pkg_resources
  File ".bootstrap\pkg_resources\__init__.py", line 2386, in resolve
ImportError: No module named git_extend

If I open the distribution I can see that my git_extend .whl isn't there compared to the linux build where it is there.
Windows git_extend missing from pex-cache etc

Linux git_extend present everywhere

support for shared lib dependencies?

I have dependencies that in turn depend on an .so. Is this supported currently? What is the best pygradle way to accomplish this?

It's failing for me during :installPythonRequirements, during setup.py of the module that depends, that's when it notices the library's missing.

Do I need to create an extension module and publish it to a local pypi repo? If so how would I ensure that the dependency is included in the virtualenv in time?

Thanks

:createVirtualEnvironment UP-TO-DATE
:installLinks UP-TO-DATE
:installSetupRequirements
... snip ...
Install Cython-0.25.2 ............................................... [SKIPPING]
Install h5py-2.6.0 .................................................. [STARTING]
... snip ...
  Autodetection skipped [libhdf5.so: cannot open shared object file: No such file or directory]
... more errors ...

how to include custom ivy-repo generated by pivy-importer in build.gradle

Using the ivy repo in my project's gradle file in the following way gives an error

apply plugin: "com.linkedin.python-sdist"
    repositories {
        ivy {url '<DIR>/pygradle/build/ivy-repo/'}
    }

> Could not find pypi:requests:2.9.1.
  Searched in the following locations:
      file:<DIR>/pygradle/build/ivy-repo/pypi/requests/2.9.1/ivy-2.9.1.xml
      file:<DIR>/pygradle/build/ivy-repo/pypi/requests/2.9.1/requests-2.9.1.jar
  Required by:
 <user>
> Could not find pypi:numpy:1.9.0.
  Searched in the following locations:
      file:<DIR>/pygradle/build/ivy-repo/pypi/numpy/1.9.0/ivy-1.9.0.xml
      file:<DIR>/pygradle/build/ivy-repo/pypi/numpy/1.9.0/numpy-1.9.0.jar
  Required by:
 <user>

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Why is it looking for .jar and .xml files? The ivy-repo has .ivy and .tar.gz files.

(Sorry for so many questions. If you would rather have me post on stackoverflow, please let me know the tag I should use.)

Using the official PyPi repo

I'm trying to define dependency that is not in the demo repo pyGradlePyPi(). I can't find anywhere in the documentation how to reference the official pypi.python.org repo and also if it's possible to reference Artifactory-hosted PyPi repo? Can anyone provide a hint?

Windows compatibility

Am i missing something or is pygradle not compatible with Windows? I added it to a Java project so I can enable sphinx and i get the error

Error:(16, 0) Unable to find or execute python
<a href="openFile:C:\code\SDA\build.gradle">Open File</a>

now this worked perfectly on my Mac, I was actually very excited. so i did some digging and i found this in com.linkedin.gradle.python.extension.PythonDetails

    private void findPythonWhenAbsent() {
        if (pythonInterpreter == null) {
            File python = ExecutablePathUtils.getExecutable(searchPath, "python");
            if (python == null) {
                python = new File("/usr/bin/python");
            }
            setSystemPythonInterpreter(python.getAbsolutePath());
        }
    }

and this in com.linkedin.gradle.python.util.internal.ExecutablePathUtils

    public static File getExecutable(List<File> pathList, String exeName) {
        for (File dir : pathList) {
            File candidate = new File(dir, exeName);
            if (candidate.isFile()) {
                return candidate;
            }
        }
        return null;
    }

Neither method takes into account that on a Winblows machine the executable name is python.exe and not python. And in my situation, this code is run on Winblows, Mac, and Linux so hard coding the thing in the build.gradle isn't practical.

Am I right or am I missing something?

Version should be configurable in plugin

The plugin currently uses the hard coded project.version which may, in development enviroments, not be compliant with PEP-0440.

Additionally, in the java/maven world, -SNAPSHOT is used for versions while python uses .preview.

This problem exposes itself when you try to create an egg as I have done in #78.
The egg for the version master_2016111701-SNAPSHOT has its version converted to master_2016111701_SNAPSHOT (notice the - character has become a _).

The result is that the egg has the name $project.name-$project.version.toString().replace("-", "_") which is very non-obvious.

The fix to this would be to put the project.version in the python plugin's properties but allow the user to override this value.

Example of location where version is hard coded

pivy-importer: local repository could not find artifact ... under different name

I am having a issue with creating a specific artifact in the local repository, namely swagger-spec-validator.
Create the artifact using java -jar /var/lib/pivy-importer-0.3.39-all.jar --repo /data swagger-spec-validator:2.0.2, it adds it however under a different artifact name than the one specified in the .ivy file below.

<ivy-module version="2.0" xmlns:e="http://ant.apache.org/ivy/extra" xmlns:m="http://ant.apache.org/ivy/maven">
  <info organisation="pypi" module="swagger-spec-validator" revision="2.0.2" />
  <configurations>
    <conf name="default" description="auto generated configuration for default" />
    <conf name="source" description="auto generated configuration for source" extends="default" />
  </configurations>
  <publications>
    <artifact name="swagger-spec-validator" ext="tar.gz" conf="default" type="tar.gz" />
  </publications>
  <dependencies defaultconfmapping="*-&gt;default">
    <dependency org="pypi" name="jsonschema" rev="2.5.1" conf="default" />
    <dependency org="pypi" name="setuptools" rev="33.1.1" conf="default" />
    <dependency org="pypi" name="six" rev="1.10.0" conf="default" />
  </dependencies>
</ivy-module>

It is under the name: swagger_spec_validator-2.0.2.tar.gz. Changing the name helps however ... is it a bug? Or is there a way to have the proper artifact name from the start? Or I am missing something?

Example of using PyGradle in Intellij Idea

I am experimenting with the example-project and trying to have both Java and Python code in one repo. In Intellij Idea, if I set the venv directory as the SDK then it doesn't understand Java and vice-versa. I am new to Gradle, and thus, PyGradle. Can someone please tell me or add an example of a project that uses both Java and Python, and the directions of importing it to an IDE (preferably, Intellij Idea)?

Plugin works on command line but fails with python not found in eclipse and idea

Hi,
I'm using Mac El Cap - Latest Idea or latest eclipse and gradlew using 3.1 with plugin id "com.linkedin.python-sdist" version "0.3.20". Config is a copy of the example more or less.
Most strange can't find /usr/bin/python.
systemPythonInterpreter override made no difference.
Works perfectly in terminal window.

Any ideas?
Other than that a great plugin.
Peter
Unable to find or execute python .... Caused by: java.lang.RuntimeException: Unable to find or execute python at com.linkedin.gradle.python.extension.PythonDetails.updateFromPythonInterpreter(PythonDetails.java:48) at com.linkedin.gradle.python.extension.PythonDetails.setPythonVersion(PythonDetails.java:111) at build_5kainnwc1tl7krk6rw33j2pjw$_run_closure2$_closure5.doCall(linkedin/build.gradle:31) at build_5kainnwc1tl7krk6rw33j2pjw$_run_closure2$_closure5.doCall(linkedin/build.gradle) at com.linkedin.gradle.python.PythonExtension.details(PythonExtension.groovy:140) at org.gradle.internal.metaobject.BeanDynamicObject$MetaClassAdapter.invokeMethod(BeanDynamicObject.java:382)

Creating an egg as a distribution?

I've configured my build to create the tar.gz using the sdist plugin.

Is there any easy way to also create an .egg artifact as well?

Looking at the source it seems like the arguments passed are hard coded so I'm going to assume no but I figured I should ask.

https://github.com/linkedin/pygradle/blob/master/pygradle-plugin/src/main/groovy/com/linkedin/gradle/python/tasks/SourceDistTask.java#L42

Update EntryPointHelpers#writeEntryPointScript to use python wrapper

com.linkedin.gradle.python.util.EntryPointHelpers#writeEntryPointScript should be updated so that the bash script can be replaced with a python wrapper.

Definition of an explicit installation sequence of dependencies?

Hi all,

this question might be more related to Gradle than to pygradle, so please redirect me if necessary. Anyhow, I think its worth discussing in the context of pygradle here.

Currently, I am trying to build a minimal working example in the setting of Machine Learning (say, a getting started project on the Iris dataset) using:

• Docker as a build container
• pygradle to build my python project
• pivy-importer to have a locally cached pypi repository (I consider moving that to Artifactory in the future, but since we don't own a Pro instance here, I am stuck to the open source version)

It is my aim to make a PR for an other example project once everything runs smoothly. Currently, I am facing one single last problem...

Besides others, I am defining python project dependencies for:

• scipy (0.18.1) and
• scikit-learn (0.18).

scikit-learn depends on scipy, but does not mention this dependency in any metadata explicitly.
As a result, If one intends to install scikit-learn (even in a fresh environment), the installation procedure obviously fails due to the missing required scipy library (see the attached log1.txt file for this scenario from within pygradle).

From the same log1.txt file, I deduce that dependencies are installed using alphabetical order ( I have tested some permutations in the build.gradle file without succedd). Since "scikit-learn" < "scipy" (on ASCII level), the build step "installPythonRequirements" will always fail when resolving dependencies in alphabetical order.

As a work around, one can

1. remove the dependency for scikit-learn for the first build run,
2. wait for the import error in the source code,
3. re-add the dependency for scikit-learn, and
4. run the build step again.

Since the same virtualenv as for the first run is used, scipy is correctly detected. This, in turn, results in a successful installation of scikit-learn s.t. the build finally succeeds.

Therefore my question (in this scenario so one can image multiple others): How to define or configure that scikit-learn is installed after scipy?

Thanks!
André

"Unable to locate finder for 'pip._vendor.distlib'" in Python 3.6

I'm on Windows, using Python 3.6 and PyGradle 0.4.7. When I run gradlew build I get the following error:

Processing c:\users\administrator\.gradle\caches\modules-2\files-2.1\pypi\pbr\1.8.0\cc7bfd10c2031a87881bf1c92b2b03f565c76f08\pbr-1.8.0.tar.gz
Building wheels for collected packages: pbr
  Running setup.py bdist_wheel for pbr
  Stored in directory: C:\Users\Administrator\AppData\Local\pip\Cache\wheels\13\f0\a9\2ef1f08c899f898b99ec457a8d36d78e3bac314758f87c3bcc
Successfully built pbr
Installing collected packages: pbr
Exception:
Traceback (most recent call last):
  File "C:\Users\Administrator\gitstuff\xim\build\venv\lib\site-packages\pip\basecommand.py", line 211, in main
    status = self.run(options, args)
  File "C:\Users\Administrator\gitstuff\xim\build\venv\lib\site-packages\pip\commands\install.py", line 311, in run
    root=options.root_path,
  File "C:\Users\Administrator\gitstuff\xim\build\venv\lib\site-packages\pip\req\req_set.py", line 646, in install
    **kwargs
  File "C:\Users\Administrator\gitstuff\xim\build\venv\lib\site-packages\pip\req\req_install.py", line 803, in install
    self.move_wheel_files(self.source_dir, root=root)
  File "C:\Users\Administrator\gitstuff\xim\build\venv\lib\site-packages\pip\req\req_install.py", line 998, in move_wheel_files
    isolated=self.isolated,
  File "C:\Users\Administrator\gitstuff\xim\build\venv\lib\site-packages\pip\wheel.py", line 479, in move_wheel_files
    maker.make_multiple(['%s = %s' % kv for kv in console.items()])
  File "C:\Users\Administrator\gitstuff\xim\build\venv\lib\site-packages\pip\_vendor\distlib\scripts.py", line 364, in make_multiple
    filenames.extend(self.make(specification, options))
  File "C:\Users\Administrator\gitstuff\xim\build\venv\lib\site-packages\pip\_vendor\distlib\scripts.py", line 353, in make
    self._make_script(entry, filenames, options=options)
  File "C:\Users\Administrator\gitstuff\xim\build\venv\lib\site-packages\pip\_vendor\distlib\scripts.py", line 257, in _make_script
    self._write_script(scriptnames, shebang, script, filenames, ext)
  File "C:\Users\Administrator\gitstuff\xim\build\venv\lib\site-packages\pip\_vendor\distlib\scripts.py", line 193, in _write_script
    launcher = self._get_launcher('t')
  File "C:\Users\Administrator\gitstuff\xim\build\venv\lib\site-packages\pip\_vendor\distlib\scripts.py", line 332, in _get_launcher
    result = finder(distlib_package).find(name).bytes
  File "C:\Users\Administrator\gitstuff\xim\build\venv\lib\site-packages\pip\_vendor\distlib\resources.py", line 324, in finder
    raise DistlibException('Unable to locate finder for %r' % package)
pip._vendor.distlib.DistlibException: Unable to locate finder for 'pip._vendor.distlib'
:installSetupRequirements FAILED

According to pypa/pip#3964 this was fixed in pip 9.0.1, so it may be worth updating the version from 7.1.2.

support for conda packages

Example of a more complicated dependency?

Hello, is there an existing example of build.gradle where may be I want to run another external program before I build my Python package?

Create a 'fall back' version of python if the requested one doesn't exist.

Some users may want to not fail builds if the requested version of python isn't available so they should be able to set a 'fallback' version that would be picked up if needed.

improve documentation, provide more examples

n/t

sdist does not get rebuilt when source changes

The sdist task(SourceDistTask.java) is not properly configured with inputs/outputs to detect changes in source, for example. Deriving from AbstractPythonMainSourceDefaultTask instead of DefaultTask(present case) fixes this problem.

We should also exclude **/*.egg-info/ from source Inputfiles.

Install Tasks runs every time

It seems like this plugin doesn't take advantage of gradle's incremental build system very well.

Even when there have been no changes to the list of dependencies the plugin seems to consistently install dependencies that were loaded by the last build.

:PlexxiCorePythonAPI:pinRequirements UP-TO-DATE
:PlexxiCorePythonAPI:createVirtualEnvironment UP-TO-DATE
:PlexxiShell:processResources UP-TO-DATE
:PlexxiCorePythonAPI:installLinks UP-TO-DATE
:PlexxiCorePythonAPI:installSetupRequirements
Install setuptools-19.1.1 ........................................... [SKIPPING]
Install wheel-0.26.0 ................................................ [SKIPPING]
Install pip-7.1.2 ................................................... [SKIPPING]
Install setuptools-git-1.1 .......................................... [SKIPPING]
Install pbr-1.8.0 ................................................... [SKIPPING]
Install argparse-1.4.0 .............................................. [SKIPPING]
:PlexxiCorePythonAPI:installBuildRequirements
Install Babel-1.3 ................................................... [SKIPPING]
Install Jinja2-2.3 .................................................. [SKIPPING]
Install Pygments-2.0 ................................................ [SKIPPING]
Install Sphinx-1.4.1 ................................................ [SKIPPING]
Install alabaster-0.7.1 ............................................. [SKIPPING]
Install colorama-0.3.5 .............................................. [STARTING]
.... Some task output from a parallel task here
Install (0:01.567 s) ................................................ [FINISHED]
Install docutils-0.11 ............................................... [SKIPPING]
Install flake8-2.5.4 ................................................ [SKIPPING]
Install imagesize-0.7.1 ............................................. [SKIPPING]
Install mccabe-0.2.1 ................................................ [SKIPPING]
Install pep8-1.5.7 .................................................. [SKIPPING]
Install pyflakes-0.8.1 .............................................. [SKIPPING]
Install pytz-2016.4 ................................................. [STARTING]
Install (0:03.192 s) ................................................ [FINISHED]
Install six-1.10.0 .................................................. [SKIPPING]
Install snowballstemmer-1.1.0 ....................................... [SKIPPING]
:PlexxiCorePythonAPI:copyPythonSources UP-TO-DATE
:PlexxiCorePythonAPI:installPythonRequirements
Install jsonpickle-0.6.1 ............................................ [SKIPPING]
Install pytz-2013b .................................................. [STARTING]
Install (0:03.631 s) ................................................ [FINISHED]
Install requests-2.4.3 .............................................. [SKIPPING]
:PlexxiCorePythonAPI:installTestRequirements
Install argparse-1.4.0 .............................................. [SKIPPING]
Install colorama-0.3.7 .............................................. [STARTING]
Install (0:01.275 s) ................................................ [FINISHED]
Install coverage-3.7.1 .............................................. [SKIPPING]
Install nose-1.3.7 .................................................. [SKIPPING]
Install py-1.4.29 ................................................... [SKIPPING]
Install pytest-cov-2.2.1 ............................................ [SKIPPING]
Install pytest-xdist-1.14 ........................................... [SKIPPING]
Install pytest-2.9.1 ................................................ [SKIPPING]
Install jsonpickle-0.6.1 ............................................ [SKIPPING]
Install pytz-2013b .................................................. [STARTING]
Install (0:02.861 s) ................................................ [FINISHED]
Install requests-2.4.3 .............................................. [SKIPPING]

This ends up slowing down my build significantly.

How do I generate test and code coverage reports?

Actually giving more examples as to how do I generate unit-test (nose), pylint and code coverage (coverage) reports will be helpful.

Refactoring uncovered conditional task operations

I began the code intended to split the sphinx functionality off into its own plugin. The spaghetti code in this thing was driving me insane so I decided to just go ahead and convert the plugin classes to a far easier to read and manage style. I wanted to make sure that i didn't change the functions of a task in the process of my refactoring so i setup a big array of tests that check the task graph of each task, so i have a baseline of before refactoring and then after refactoring. During the test run, i found something very troubling. The operations of a task change depending on what plugin applies it. meaning the buildPex task that is in the python-pex plugin is very different from the buildPex thats in the python-cli plugin.

Now my personal style says this is a huge no-no. The buildPex task should have the same function no matter what. Thats how we ensure our builds are repeatable. So there should be a buildPex and a buildPex-cli.

What do you want me to do?

Here's what i mean.

Problems I have found
the 'buildPex' task changes based on what plugin is installed. Meaning if you include the 'com.linkedin.python-pex' plugin, it has this trait

:buildPex
\--- :buildWheels
5\--- :installProject
55\--- :installTestRequirements
555\--- :installPythonRequirements
5555\--- :installBuildRequirements
55555\--- :installSetupRequirements
555555\--- :installLinks
5555555\--- :createVirtualEnvironment
55555555\--- :pinRequirements

But, if you add the com.linkedin.python-cli plugin, then the 'buildPex' task looks like this

:buildPex
+--- :buildWheels
|4\--- :installProject
|54\--- :installTestRequirements
|554\--- :installPythonRequirements
|5554\--- :installBuildRequirements
|55554\--- :installSetupRequirements
|555554\--- :installLinks
|5555554\--- :createVirtualEnvironment
|55555554\--- :pinRequirements
\--- :generateCompletions
5+--- :installBuildRequirements
5|4\--- :installSetupRequirements
5|54\--- :installLinks
5|554\--- :createVirtualEnvironment
5|5554\--- :pinRequirements
5\--- :installProject
55\--- :installTestRequirements
555\--- :installPythonRequirements
5555\--- :installBuildRequirements
55555\--- :installSetupRequirements
555555\--- :installLinks
5555555\--- :createVirtualEnvironment
55555555\--- :pinRequirements

This happens because the python-cli plugin adds an additional dependency to the task

This issue extends to the package deployable task from pex as well where if you just add the pex plugin, the task looks like

:packageDeployable
\--- :buildPex
5\--- :buildWheels
55\--- :installProject
555\--- :installTestRequirements
5555\--- :installPythonRequirements
55555\--- :installBuildRequirements
555555\--- :installSetupRequirements
5555555\--- :installLinks
55555555\--- :createVirtualEnvironment
555555555\--- :pinRequirements

but after applying the cli task, the task looks like this

:packageDeployable
\--- :buildPex
5+--- :buildWheels
5|4\--- :installProject
5|54\--- :installTestRequirements
5|554\--- :installPythonRequirements
5|5554\--- :installBuildRequirements
5|55554\--- :installSetupRequirements
5|555554\--- :installLinks
5|5555554\--- :createVirtualEnvironment
5|55555554\--- :pinRequirements
5\--- :generateCompletions
55+--- :installBuildRequirements
55|4\--- :installSetupRequirements
55|54\--- :installLinks
55|554\--- :createVirtualEnvironment
55|5554\--- :pinRequirements
55\--- :installProject
555\--- :installTestRequirements
5555\--- :installPythonRequirements
55555\--- :installBuildRequirements
555555\--- :installSetupRequirements
5555555\--- :installLinks
55555555\--- :createVirtualEnvironment
555555555\--- :pinRequirements

the flyer plugin does the same, by adding a depends on the installproject task of setupResourceLink, it changes the way the task operates for the whole stack

build warnings when compiling pygradle

these warnings come up every time you compile pygradle

:pygradle-plugin:validateTaskProperties
Task property validation finished with warnings:
  - Warning: Task type 'com.linkedin.gradle.python.tasks.BuildPexTask' declares property that is not annotated: 'reason'.
  - Warning: Task type 'com.linkedin.gradle.python.tasks.InstallVirtualEnvironmentTask' declares property that is not annotated: 'container'.
  - Warning: Task type 'com.linkedin.gradle.python.tasks.InstallVirtualEnvironmentTask' declares property that is not annotated: 'pythonDetails'.
  - Warning: Task type 'com.linkedin.gradle.python.tasks.InstallVirtualEnvironmentTask' declares property that is not annotated: 'reason'.
  - Warning: Task type 'com.linkedin.gradle.python.tasks.PipInstallTask' declares property that is not annotated: 'configurationFiles'.
  - Warning: Task type 'com.linkedin.gradle.python.tasks.PipInstallTask' declares property that is not annotated: 'reason'.
  - Warning: Task type 'com.linkedin.gradle.python.tasks.PyCoverageTask' declares property that is not annotated: 'extraArgs'.
  - Warning: Task type 'com.linkedin.gradle.python.tasks.PyCoverageTask' declares property that is not annotated: 'outputStream'.
  - Warning: Task type 'com.linkedin.gradle.python.tasks.PyCoverageTask' declares property that is not annotated: 'specificFileGiven'.
  - Warning: Task type 'com.linkedin.gradle.python.tasks.PyTestTask' declares property that is not annotated: 'extraArgs'.
  - Warning: Task type 'com.linkedin.gradle.python.tasks.PyTestTask' declares property that is not annotated: 'specificFileGiven'.
  - Warning: Task type 'com.linkedin.gradle.python.tasks.SourceDistTask' declares property that is not annotated: 'distDir'.

PythonDetails is not serializable

PythonDetails implements Serializable but it takes a Project that is not serializable.

If you try to make it so that the PipInstallTask gets cached you get an exception that it can't be

Failed to execute org.gradle.cache.internal.AsyncCacheAccessDecoratedCache$2@613b3d80.
org.gradle.api.UncheckedIOException: Could not add entry ':PlexxiCorePythonAPI:installPythonRequirements' to cache taskArtifacts.bin (/Users/jonathanleitschuh/work/git/plexxicontrol/.gradle/3.4-20161216120313+0000/taskArtifacts/taskArtifacts.bin).
        at org.gradle.cache.internal.btree.BTreePersistentIndexedCache.put(BTreePersistentIndexedCache.java:158)
        at org.gradle.cache.internal.DefaultMultiProcessSafePersistentIndexedCache$2.run(DefaultMultiProcessSafePersistentIndexedCache.java:53)
        at org.gradle.cache.internal.DefaultFileLockManager$DefaultFileLock.doWriteAction(DefaultFileLockManager.java:184)
        at org.gradle.cache.internal.DefaultFileLockManager$DefaultFileLock.writeFile(DefaultFileLockManager.java:174)
        at org.gradle.cache.internal.DefaultCacheAccess$UnitOfWorkFileAccess.writeFile(DefaultCacheAccess.java:457)
        at org.gradle.cache.internal.DefaultMultiProcessSafePersistentIndexedCache.put(DefaultMultiProcessSafePersistentIndexedCache.java:51)
        at org.gradle.cache.internal.AsyncCacheAccessDecoratedCache$2.run(AsyncCacheAccessDecoratedCache.java:48)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
        at org.gradle.cache.internal.CacheAccessWorker$2.run(CacheAccessWorker.java:187)
        at org.gradle.internal.Factories$1.create(Factories.java:25)
        at org.gradle.cache.internal.DefaultCacheAccess.useCache(DefaultCacheAccess.java:179)
        at org.gradle.cache.internal.DefaultCacheAccess.useCache(DefaultCacheAccess.java:162)
        at org.gradle.cache.internal.CacheAccessWorker.flushOperations(CacheAccessWorker.java:177)
        at org.gradle.cache.internal.CacheAccessWorker.run(CacheAccessWorker.java:147)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
        at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.gradle.api.GradleException: Unable to store task input properties. Property 'pythonDetails' with value 'com.linkedin.gradle.python.extension.PythonDetails@79695a29' cannot be serialized.
        at org.gradle.api.internal.changedetection.state.InputPropertiesSerializer.write(InputPropertiesSerializer.java:42)
        at org.gradle.api.internal.changedetection.state.CacheBackedTaskHistoryRepository$LazyTaskExecution$TaskExecutionSnapshotSerializer.write(CacheBackedTaskHistoryRepository.java:441)
        at org.gradle.api.internal.changedetection.state.CacheBackedTaskHistoryRepository$TaskExecutionListSerializer.write(CacheBackedTaskHistoryRepository.java:224)
        at org.gradle.api.internal.changedetection.state.CacheBackedTaskHistoryRepository$TaskExecutionListSerializer.write(CacheBackedTaskHistoryRepository.java:200)
        at org.gradle.cache.internal.btree.BTreePersistentIndexedCache$DataBlock.setValue(BTreePersistentIndexedCache.java:659)
        at org.gradle.cache.internal.btree.BTreePersistentIndexedCache$DataBlock.<init>(BTreePersistentIndexedCache.java:646)
        at org.gradle.cache.internal.btree.BTreePersistentIndexedCache.put(BTreePersistentIndexedCache.java:150)
        ... 18 more
Caused by: org.gradle.internal.serialize.MapSerializer$EntrySerializationException: Unable to write entry with key: 'pythonDetails' and value: 'com.linkedin.gradle.python.extension.PythonDetails@79695a29'.
        at org.gradle.internal.serialize.MapSerializer.write(MapSerializer.java:50)
        at org.gradle.api.internal.changedetection.state.InputPropertiesSerializer.write(InputPropertiesSerializer.java:40)
        ... 24 more
Caused by: java.io.NotSerializableException: org.gradle.api.internal.project.DefaultProject_Decorated
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
        at org.gradle.internal.serialize.DefaultSerializer.write(DefaultSerializer.java:53)
        at org.gradle.internal.serialize.MapSerializer.write(MapSerializer.java:48)
        ... 25 more

The solution would be to mark the Project field in PipInstallTask as transient.

Example project failing at build

This appears to be a similar issue to #42, but I checked the setup.py file and the bottom section was commented out as suggested in that post.

Install example-project ............................................. [STARTING]
Obtaining file:///pygradle-master/examples/example-project
No files/directories in /pygradle-master/examples/example-project (from PKG-INFO)
:installProject FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':installProject'.
> Failed to install example-project. Please see above output for reason, or re-run your build using ``ligradle -i build`` for additional logging.

Docs don't get rebuilt when changes are made to docs/*.rst files

SphinxDocumentationTask (buildDocs) stays up-to-date even after edits to docs/*.rst files. This this because docs directory is not part of the inputs.