dropbox / affectedmoduledetector Goto Github PK

View Code? Open in Web Editor NEW

583.0 583.0 37.0 511 KB

A Gradle Plugin to determine which modules were affected by a set of files in a commit.

License: Apache License 2.0

Kotlin 100.00%

affectedmoduledetector's People

Contributors

Stargazers

Watchers

affectedmoduledetector's Issues

excludedModules is not working with custom task.

Environment

macOS 12.5
Gradle 7.5
AMD 0.1.6
AGP 7.2.1
Detekt 1.21.0
Java 17
Kotlin 1.6.21

Description

I'm integrating AMD into my project and I created two custom tasks. When I run `./gradlew runAffectedDetekt', I'm getting this error even though linter is specified in the excludedModules.

* What went wrong:
Could not determine the dependencies of task ':runAffectedDetekt'.
> Task with path ':linter:detekt' not found in root project 'Rappi-Android-User'.

This error makes sense since Detekt is not applied in the linter module; however, it should be ignored by AMD if it's specified in the excluded modules.

Configuration

    import com.dropbox.affectedmoduledetector.AffectedModuleConfiguration

    apply plugin: "com.dropbox.affectedmoduledetector"

    def branch = System.getenv('DESTINATION_BRANCH')
    if (!branch) {
        branch = "origin/develop"
    }

    logger.lifecycle("Destination branch: $branch")

    affectedModuleDetector {
        baseDir = "${project.rootDir}"
        pathsAffectingAllModules = [
                "tools/global-dependencies.gradle"
        ]
        logFilename = "output.log"
        logFolder = "${project.rootDir}"
        specifiedBranch = "$branch"
        compareFrom = "SpecifiedBranchCommit"
        excludedModules = ["linter", ":linter"]
        customTasks = [
                new AffectedModuleConfiguration.CustomTask(
                        "runAffectedDetekt,
                        "detekt",
                        "Run detekt"
                ),
                new AffectedModuleConfiguration.CustomTask(
                        "runAffectedAndroidLint",
                        "lintDebug",
                        "Run lint"
                ),
        ]

Add an option to exclude jvm test paths

Hey,

Could it be possible to add support to exclude certain path(s) from jvm test execution?

There is a exclude(Iterable<String> excludes) function in org.gradle.api.tasks.testing.Test class for it.

Use case:
We need it to avoid running e.g. paparazzi screenshot tests as part of main unit test task launched by runAffectedUnitTests task.

Up until now we use command line parameter:

-PexcludeTests="**/screenshots/**, **/another_excluded_path/**" That is being read and applied in testOptions.unitTest block

Fix link in README to AndroidX project

runAffectedUnitTests -Paffected_module_detector.enable failed for onlyIf

When run ./gradlew runAffectedUnitTests -Paffected_module_detector.enable it fails in my project with error:

* What went wrong:
Could not evaluate onlyIf predicate for task ':somemodule:testDebugUnitTest'.
> Nonzero exit value running git command.

Stacktrace

* Exception is:
org.gradle.api.GradleException: Could not evaluate onlyIf predicate for task ':api:somemodule:testDebugUnitTest'.
        at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:46)
        at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:200)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:195)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$3.execute(DefaultBuildOperationRunner.java:75)
        at org.gradle.internal.operations.DefaultBuildOperationRunner$3.execute(DefaultBuildOperationRunner.java:68)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:153)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:68)
        at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:62)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.lambda$call$2(DefaultBuildOperationExecutor.java:79)
        at org.gradle.internal.operations.UnmanagedBuildOperationWrapper.callWithUnmanagedSupport(UnmanagedBuildOperationWrapper.java:54)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:79)
        at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
        at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:74)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:408)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:395)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:388)
        at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:374)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.lambda$run$0(DefaultPlanExecutor.java:127)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:191)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.executeNextNode(DefaultPlanExecutor.java:182)
        at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:124)
        at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
        at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalStateException: Nonzero exit value running git command.
        at com.dropbox.affectedmoduledetector.GitClientImpl$RealCommandRunner.execute(GitClient.kt:162)
        at com.dropbox.affectedmoduledetector.GitClientImpl$RealCommandRunner.executeAndParse(GitClient.kt:166)
        at com.dropbox.affectedmoduledetector.GitClientImpl$RealCommandRunner.executeAndParseFirst(GitClient.kt:176)
        at com.dropbox.affectedmoduledetector.commitshaproviders.ForkCommit.get(ForkCommit.kt:8)
        at com.dropbox.affectedmoduledetector.GitClientImpl.findChangedFiles(GitClient.kt:80)
        at com.dropbox.affectedmoduledetector.GitClient$DefaultImpls.findChangedFiles$default(GitClient.kt:31)
        at com.dropbox.affectedmoduledetector.AffectedModuleDetectorImpl.findChangedProjects(AffectedModuleDetector.kt:403)
        at com.dropbox.affectedmoduledetector.AffectedModuleDetectorImpl.access$findChangedProjects(AffectedModuleDetector.kt:314)
        at com.dropbox.affectedmoduledetector.AffectedModuleDetectorImpl$changedProjects$2.invoke(AffectedModuleDetector.kt:354)
        at com.dropbox.affectedmoduledetector.AffectedModuleDetectorImpl$changedProjects$2.invoke(AffectedModuleDetector.kt:314)
        at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:74)
        at com.dropbox.affectedmoduledetector.AffectedModuleDetectorImpl.getChangedProjects(AffectedModuleDetector.kt)
        at com.dropbox.affectedmoduledetector.AffectedModuleDetectorImpl.findAffectedProjects(AffectedModuleDetector.kt:462)
        at com.dropbox.affectedmoduledetector.AffectedModuleDetectorImpl.access$findAffectedProjects(AffectedModuleDetector.kt:314)
        at com.dropbox.affectedmoduledetector.AffectedModuleDetectorImpl$affectedProjects$2.invoke(AffectedModuleDetector.kt:350)
        at com.dropbox.affectedmoduledetector.AffectedModuleDetectorImpl$affectedProjects$2.invoke(AffectedModuleDetector.kt:314)
        at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:74)
        at com.dropbox.affectedmoduledetector.AffectedModuleDetectorImpl.getAffectedProjects(AffectedModuleDetector.kt)
        at com.dropbox.affectedmoduledetector.AffectedModuleDetectorImpl.shouldInclude(AffectedModuleDetector.kt:366)
        at com.dropbox.affectedmoduledetector.AffectedModuleDetector$Companion.isProjectAffected(AffectedModuleDetector.kt:254)
        at com.dropbox.affectedmoduledetector.AffectedModuleDetectorPlugin$withPlugin$1$$special$$inlined$let$lambda$1$1.isSatisfiedBy(AffectedModuleDetectorPlugin.kt:98)
        at com.dropbox.affectedmoduledetector.AffectedModuleDetectorPlugin$withPlugin$1$$special$$inlined$let$lambda$1$1.isSatisfiedBy(AffectedModuleDetectorPlugin.kt:36)
        at org.gradle.api.specs.AndSpec.isSatisfiedBy(AndSpec.java:50)
        at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:44)
        ... 30 more

Remove jcenter

jCenter support should be removed as it is being sunset

https://jfrog.com/blog/into-the-sunset-bintray-jcenter-gocenter-and-chartcenter/

Allow modules to specify which tasks to run

Using an extension, allow each module to specify which task it would like to run when executing tests.

At the moment, the plugin may not find tests associated with java only modules, and subsequently, not run any tests in those modules.

The latest versions of the plugin is not available

Hey Team,

I am wondering when the latest version 0.1.0.2 will be publicly available? Also, any plans on releasing the 0.1.3 version?

set affectedTestConfiguration globally

I cannot set like this in project root's build.gradle.kts file.

subprojects {
   affectedTestConfiguration { jvmTestTask = "test" }
}

all my subprojects are java, it would be nice if we can set this at one place instead of forced to add affectedTestConfiguration { jvmTestTask = "test" } to every subprojects

my multi-module project : https://github.com/xmlking/micro-apps

Add ability to compare between two branches

Currently, the git client will only query for HEAD~1 and determine the files in the last commit.

We should add the ability to let a developer specify a branch to compare against and determine all the changed files between HEAD and that branch.

We should configure this in the extension and let the user decide which option to use and the branch to compare against.

variantToTest is not implemented

I am playing around with this plugin and it seems like the variantToTest that is mentioned in the README isn't actually implemented.

affectedTestConfiguration{
   variantToTest = "debug" //default is debug
}

You can see variantToTest isn't defined in AffectedTestConfiguration. However, you can see an example of this in the sample app AffectedTasksPlugin

To reproduce, you can try adding variantToTest in one of the sample/build.gradle files such as https://github.com/dropbox/AffectedModuleDetector/blob/main/sample/sample-app/build.gradle#L8-L10. I actually run into a compile error when I try to build after adding variantToTest.

I am working on making changes so this works but so far, still only seeing the tests for debug variant running.

I changed the AffectedTestConfiguration to:

open class AffectedTestConfiguration {

    var variantToTest : String = "debug"
    var assembleAndroidTestTask : String? = "assemble${variantToTest.capitalize()}AndroidTest"
    var runAndroidTestTask : String?  = "connected${variantToTest.capitalize()}AndroidTest"
    var jvmTestTask : String? = "test${variantToTest.capitalize()}UnitTest"


    companion object {
        const val name = "affectedTestConfiguration"
    }
}

Any pointers? Would love to make a PR with the fix.

`ProjectSubset.CHANGED_PROJECTS` should respect `pathsAffectingAllModules`

Currently, if a commit changes only a path listed in pathsAffectingAllModules and you specify Paffected_module_detector.changedProjects when running AMD, no modules will be found as affected.

The relevant code is in the findAffectedProjects method of AffectedModuleDetector.kt, where we return early if projectSubset == ProjectSubset.CHANGED_PROJECTS. At that point, if the only file change is a path that affects all modules, changedProjects will be an empty set.

This goes against our documentation where ProjectSubset.ALL_AFFECTED_PROJECTS is the union of CHANGED_PROJECTS and DEPENDENT_PROJECTS. In the example, ALL_AFFECTED_PROJECTS would be every project whereas CHANGED_PROJECTS and DEPENDENT_PROJECTS would be empty sets.

We should update findChangedProjects to include the check for whether any of the changed files affect all modules. In the case that it does, every module should be added to the set of changed projects. The logic in findAffectedProjects around detecting if a path changed that affects all projects should be removed and we should define the affectedProjects field to be the union of changedProjects and dependentProjects

Moving files occasionally results in missed modules

In changes with file moves, occasionally the detector doesn't catch that the move occurred and that affected modules should be run. This can be seen in the log file when the git diff command is run.

pathsAffectingAllModules is only observed if the changed files are also unknown files

pathsAffectingAllModules will only be observed if the files are also unknown.

In our case, we have files which are associated with projects, but should also trigger all the builds.

See https://github.com/dropbox/AffectedModuleDetector/blob/main/affectedmoduledetector/src/main/kotlin/com/dropbox/affectedmoduledetector/AffectedModuleDetector.kt#L465

Missing excludedModules

According to the docs, we can specify excludedModules while in the sample code its using version 0.1.2-SNAPSHOT and the version available on maven central is 0.1.1, any plans to release this new version ?? This is kinda blocking us from adopting this plugin, thx in advance.

DependencyTracker written in Groovy

I had to re-write the DependencyTracker in Groovy, seems a waste to just delete it now I've finished.
Here is it for posterity.

class DependencyTracker {
    private final Project rootProject
    private final Logger logger

    DependencyTracker(Project rootProject, Logger logger) {
        this.rootProject = rootProject
        this.logger = logger
    }

    @Lazy Map<Project, Set<Project>> dependentList = {
        Map<Project, Set<Project>> result = new HashMap<>()
        rootProject.subprojects.forEach { project ->
            logger?.debug("checking ${project.path} for dependencies")
            project.configurations.forEach { config ->
                logger?.debug("checking config ${project.path}/$config for dependencies")
                config
                        .dependencies
                        .findAll { it instanceof ProjectDependency }
                        .forEach {
                            logger?.debug("there is a dependency from ${project.path} to " + it.dependencyProject.path)
                            def value = result.get(it.dependencyProject)
                            if (value == null) {
                                value = new HashSet<Project>()
                            }
                            value.add(project)
                            result.put(it.dependencyProject, value)
                        }
            }
        }
        result
    }()

    private def addAllDependents(Set<Project> result, Project project) {
        if (result.add(project)) {
            dependentList[project]?.forEach {
                addAllDependents(result, it)
            }
        }
    }

    Set<Project> findAllDependents(Project project) {
        println("finding dependents of ${project.path}")
        Set<Project> result = new HashSet<>()
        addAllDependents(result, project)
        logger?.debug("dependents of ${project.path} is ${result.collect { it.path }}")
        // the project isn't a dependent of itself
        return result - project
    }
}

Question about AMD use

Hello 👋

I have one question : what happens when a Pull Request has more than one commits ? For example a Pull Request might have 3 commits: Commit A changes Module_A, Commit B changes Module_B and Commit C changes Module_C. Assuming the last one was Commit C then trying to run AMD would only run tests for Module_C. From the tests I made all three options: ForkCommit, PreviousCommit and specifiedBranch will only run Module_C's tests. But ideally what we want here is to run tests for all affected modules, cause that Pull Request would affect all 3 modules I mentioned.

Is that the case or maybe I don't understand something ? If yes then how do you deal with it ? What's the "proper" way to utilise this cool library to save me some time?

Thank you!!

Tests being run for all projects

I set up the plugin by adding this to my top-level build.gradle:

apply plugin: "com.dropbox.affectedmoduledetector"

affectedModuleDetector {
    baseDir = "${project.rootDir}"
    specifiedBranch = "tmp2"
    compareFrom = "SpecifiedBranchCommit"
    includeUncommitted = false
    logFilename = "output.log"
    logFolder = "${project.rootDir}/output"
}
subprojects {
    affectedTestConfiguration {
        jvmTestTask = "foo"
    }
    task foo() {
        println "${project.name} running tests"
    }
}

Then made a code change to the gradle module called :monographs, so that HEAD diffs with tmp2 to show only this code change.

Then I run: ./gradlew runAffectedUnitTests -Paffected_module_detector.enable -Paffected_module_detector.changedProjects

which gives an output of:

> Configure project :
app running tests
articles running tests
authlib running tests
monographs running tests
pharma running tests
platform running tests
rteditor running tests
search running tests
shared-api running tests
shared-contract running tests
shared-test running tests
shared-ui running tests
snippets running tests
SwipeLayout running tests

> Configure project :app
WARNING:DSL element 'dexOptions' is obsolete and should be removed.
It will be removed in version 8.0 of the Android Gradle plugin.
Using it has no effect, and the AndroidGradle plugin optimizes dexing automatically.
The Hilt configuration option 'enableTransformForLocalTests' is no longer necessary when com.android.tools.build:gradle:4.2.0+ is used.
...

So, tests are being run for every module in my project! I would expect to see only monographs running tests, and no others in the output.

Here's my log file from AMD:

Details

[INFO] [amd] Modules provided: null
[INFO] [amd] Using real detector with CHANGED_PROJECTS
[INFO] [amd] running command git rev-parse tmp2 in /home/hin/code/android-avocado
[INFO] [amd] Response: 47a71a0bfe4891534a49826a9f25726dddb7de47

[INFO] [amd] running command git --no-pager diff --name-only -M95 HEAD..47a71a0bfe4891534a49826a9f25726dddb7de47 in /home/hin/code/android-avocado
[INFO] [amd] Response: monographs/src/main/java/de/miamed/amboss/monograph/feedback/MonographFeedbackConfig.kt

[INFO] [amd] Paths affecting all modules: []
[INFO] [amd] initializing ProjectGraph
[INFO] [amd] creating node for :app
[INFO] [amd] relative path: app , sections: [app]
[INFO] [amd] creating node for :articles
[INFO] [amd] relative path: articles , sections: [articles]
[INFO] [amd] creating node for :authlib
[INFO] [amd] relative path: authlib/lib , sections: [authlib, lib]
[INFO] [amd] creating node for :monographs
[INFO] [amd] relative path: monographs , sections: [monographs]
[INFO] [amd] creating node for :pharma
[INFO] [amd] relative path: pharma , sections: [pharma]
[INFO] [amd] creating node for :platform
[INFO] [amd] relative path: platform , sections: [platform]
[INFO] [amd] creating node for :rteditor
[INFO] [amd] relative path: rteditor/RTEditor , sections: [rteditor, RTEditor]
[INFO] [amd] creating node for :search
[INFO] [amd] relative path: search , sections: [search]
[INFO] [amd] creating node for :shared-api
[INFO] [amd] relative path: shared-api , sections: [shared-api]
[INFO] [amd] creating node for :shared-contract
[INFO] [amd] relative path: shared-contract , sections: [shared-contract]
[INFO] [amd] creating node for :shared-test
[INFO] [amd] relative path: shared-test , sections: [shared-test]
[INFO] [amd] creating node for :shared-ui
[INFO] [amd] relative path: shared-ui , sections: [shared-ui]
[INFO] [amd] creating node for :snippets
[INFO] [amd] relative path: snippets , sections: [snippets]
[INFO] [amd] creating node for :SwipeLayout
[INFO] [amd] relative path: AndroidSwipeLayout/library , sections: [AndroidSwipeLayout, library]
[INFO] [amd] finished creating ProjectGraph com.dropbox.affectedmoduledetector.ProjectGraph$Node@106a7c49
[INFO] [amd] finding containing project for monographs/src/main/java/de/miamed/amboss/monograph/feedback/MonographFeedbackConfig.kt , sections: [monographs, src, main, java, de, miamed, amboss, monograph, feedback, MonographFeedbackConfig.kt]
[INFO] [amd] finding [monographs, src, main, java, de, miamed, amboss, monograph, feedback, MonographFeedbackConfig.kt] with index 0 in root
[INFO] [amd] finding [monographs, src, main, java, de, miamed, amboss, monograph, feedback, MonographFeedbackConfig.kt] with index 1 in :monographs
[INFO] [amd] no child found, returning :monographs
[INFO] [amd] search result for monographs/src/main/java/de/miamed/amboss/monograph/feedback/MonographFeedbackConfig.kt resulted in :monographs
[INFO] [amd] For file monographs/src/main/java/de/miamed/amboss/monograph/feedback/MonographFeedbackConfig.kt containing project is project ':monographs'. Adding to changedProjects.
[INFO] [amd] checking whether I should include :app and my answer is false
[INFO] [amd] checking whether I should include :authlib and my answer is false
[INFO] [amd] checking whether I should include :articles and my answer is false
[INFO] [amd] checking whether I should include :monographs and my answer is true
[INFO] [amd] checking whether I should include :search and my answer is falsealse
[INFO] [amd] checking whether I should include :rteditor and my answer is false
[INFO] [amd] checking whether I should include :pharma and my answer is false
[INFO] [amd] checking whether I should include :shared-contract and my answer is false
[INFO] [amd] checking whether I should include :shared-test and my answer is false
[INFO] [amd] checking whether I should include :shared-ui and my answer is false
[INFO] [amd] checking whether I should include :snippets and my answer is false
[INFO] [amd] checking whether I should include :SwipeLayout and my answer is false

Is there something I am missing here?

Publish plugin marker artifact

Could you publish a plugin marker artifact to maven central to make it possible to use the plugin like this without needing any extra config?

plugins {
    id "com.dropbox.affectedmoduledetector" version "..."
}

When you use the java-gradle-plugin and maven-publish together it should be created automatically. Here are the Gradle docs https://docs.gradle.org/current/userguide/plugins.html#sec:plugin_markers

Include uncommitted code

Currently the command used when including uncommitted code to be compared to a commit is (seen here):

git --no-pager diff --name-only HEAD..$sha

this is incorrect as HEAD points to the last commit on the branch and therefore does not include uncommitted changes.

The command you are looking for is:

git --no-pager diff --name-only $sha

I can put a fix in after my PR is merged

Support exotic non standard Gradle project structures

Carrying on the discussion from here. I plan to raise a PR to support exotic / non standard Gradle projects. These are defined by Gradle projects whose sub projects are not descendants of the root project (in the File system sense). This is a non standard, but valid, layout in the Gradle sense.

Yes it is an edge case but there are no special instructions in AMD to state compatibility of Gradle project layouts so I feel we should support these, still valid, set ups.

A "traditional" Gradle project will look like this:

my-app <-- projectRootDir & gitRootDir
|
-- .git
|
-- module-1
|
-- module-2
|
-- module-3
|
-- settings.gradle.kts

The .git folder is a child of the my-app folder on the file system and thus the my-app folder is the gitRootDir. There are no customisations in settings.gradle.kts other than to include the modules module-1, module-2, and module-3. Gradle will also see my-app as the projectRootDir. This is what most Android projects look like and is a fine and sensible layout to assume.

However I came across this setup at work:

android-project <-- gitRootDir
|
-- .git
|
-- diagrams
|
-- ci
|
-- proposals
|
-- my-app <-- projectRootDir
    |
    -- settings.gradle.kts
|
-- common
    |
    -- module-1
    |
    -- module-2
    |
    -- module-3

By instructing settings.gradle.kts where to look on the File system for the sub projects this type of layout is valid. It is certainly non standard, I will admit 😆 .

I think I have tracked the issue down to the init block in ProjectGraph where we build up a relationship between the Gradle project graph and the File system graph (later to be used as the Git graph). If we are in a non standard Gradle project we will start to see ".." entries in the sections that we need to remove otherwise we start seeing false negatives in the detection. An example of the changes look like this.

Obviously I will add tests in a proper PR but this highlights the issue and opens it up for discussion.

See if we can make tasks cacheable

This will be a tracking ticket. In 0.1.6 we will be marking those tasks as not cacheable but would like to investigate if it's possible to somehow do this. Our current problem is this stack trace

Caused by: org.gradle.api.UnknownDomainObjectException: Extension with name 'AffectedModuleDetectorPlugin' does not exist. Currently registered extension names: [ext]
        at org.gradle.internal.extensibility.ExtensionsStorage.unknownExtensionException(ExtensionsStorage.java:144)
        at org.gradle.internal.extensibility.ExtensionsStorage.getByName(ExtensionsStorage.java:123)
        at org.gradle.internal.extensibility.DefaultConvention.getByName(DefaultConvention.java:174)
        at com.dropbox.affectedmoduledetector.AffectedModuleDetector$Companion.getInstance(AffectedModuleDetector.kt:195)
        at com.dropbox.affectedmoduledetector.AffectedModuleDetector$Companion.getOrThrow(AffectedModuleDetector.kt:200)
        at com.dropbox.affectedmoduledetector.AffectedModuleDetector$Companion.configureTaskGuard$lambda-3(AffectedModuleDetector.kt:233)
        at org.gradle.api.specs.AndSpec.isSatisfiedBy(AndSpec.java:50)
        at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:44)
        ... 24 more

As noted in this slack message on the gradle slack, the apply method is not called on plugins which cached so the call in our onlyIf block to check if a project is enabled and affected no longer works. If we could move that check into the task possibly rather then the plugin maybe this is doable

Task 'runAffectedUnitTests' not found in root project

Whenever I apply the plugin to a project and run the ./gradlew runAffectedUnitTests -Paffected_module_detector.enable command, I get the following error:

FAILURE: Build failed with an exception.

* What went wrong:
Task 'runAffectedUnitTests' not found in root project 'my-project'.

Running ./gradle tasks or ./gradlew tasks -Paffected_module_detector.enable results in no tasks added by this plugin either.

I have tried with multiple Android/Java only projects which all have different configurations, but all come up with the same results. I even copy pasted this entire block into the root build.gradle (thus having two buildscript blocks) to no success.

buildscript {
  repositories {
    maven {
      url "https://plugins.gradle.org/m2/"
    }
  }
  dependencies {
    classpath "com.dropbox.affectedmoduledetector:affectedmoduledetector:0.1.0"
  }
}
apply plugin: "com.dropbox.affectedmoduledetector"

Use auto-plugin for generating gradle-plugins property files

Consider using https://github.com/ansman/auto-plugin for generation of the META-INF properties files.

Support ForkCommit with rebasing

"ForkCommit" as I understand it currently requires a merge commit in order to determine the parent branch. However if a project is practicing rebasing as a way of working and merging work, then there will be no merge commits. Not sure what the solution to this would be off the top of my head, just pointing out this is an issue.

Allow dispatch various flows based on the type of changes?

Hi,
Thanks for sharing this to the world.

I would like to run different flows based on the package changed (for example)
And have a default set running no matter what was the change (in addition).

Thanks.

AffectedModuleDetector.isProjectAffected doss not work correctly on windows

Noob question, how do I use the api? I want to build (not test) affected modules only.

`assembleAffectedAndroidTests` assembles APKs for all modules

Hello!

I am running into an issue where calling

gradle assembleAffectedAndroidTests -Paffected_module_detector.enabled

assembles the APKs for all modules.

Based on the documentation, I am under the assumption that this task should generate test apks for only the affected module(s).

Here are the configurations I have set up:

affectedModuleDetector {
    baseDir = "${project.rootDir}"
    logFolder = "${project.rootDir}/artifacts"
    specifiedBranch = "${base_branch}"
    compareFrom = 'SpecifiedBranchCommit'
}

affectedTestConfiguration {
    assembleAndroidTestTask = 'assembleStandardDebugAndroidTest'
    runAndroidTestTask = 'assembleStandardDebugAndroidTest'
    jvmTestTask = 'testStandardDebugUnitTest'
}

Cross-referencing the logs and the apk output, I see that

[INFO] [amd] checking whether I should include :lib:featuretoggle and my answer is false

but

./lib/featuretoggle/build/outputs/apk/androidTest/standard/debug/featuretoggle-standard-debug-androidTest.apk

is still being generated.

Please let me know if this is the expected behavior or if I am doing something wrong. Thank you!

Consider renaming project

AffectedModuleDetector is a lot of syllables and could be something more concise while still descriptive.

DropTask has been a suggestion.

Open to other suggestions as well.

Cannot deploy using configuration cache

Blocked by vanniktech/gradle-maven-publish-plugin#259

AffectedModuleDetectorPlugin is eagerly creating tasks at configuration time

This plugin still needs to be updated to support the Gradle task configuration avoidance APIs documented here:

https://docs.gradle.org/current/userguide/task_configuration_avoidance.html

Some examples of the use of the old/wrong APIs can be found in filterAndroidTests:

    private fun filterAndroidTests(project: Project) {
        val tracker = DependencyTracker(project, null)
        project.tasks.all { task ->
        ...

...and filterJvmTests:

        project.tasks.withType(Test::class.java) { task ->
            AffectedModuleDetector.configureTaskGuard(task)
        }

As a result of the above, our large project is eagerly creating thousands of testing tasks at configuration time.

The migration guide may be useful as a resource.

Unable to find method affectedTestConfiguration()

I am running the sample app and changed the version to 0.1.0 since I do not able to access the snapshot version form all mavens. When trying to build the sample, I get the following build error:

I get the same error when I try to integrate it to another project, as shown below:

How can I get a Test Coverage report with this plugin ?

As a part of the effort to reduce feedback time on PRs we are considering applying this plugin, but we would still like to get code coverage reports after. Thanks in advance

Exclude defined modules from being considered

Proposal: Parse a list of module names into a configuration which wont be considered by the dependency decision making process, as well as the test apk building and running.

Please let me know of any objections, otherwise I will start the work for this in the upcoming days.

Assemble only affected Apps in multimodle project

On our gradle project we have several apps and lots of modules. Is there any way to only assemble the APKs of affected apps?

runAffectedUnitTests -Paffected_module_detector.enable fails on CI only

Hey,
this was mentioned before but in my case it's only failing on CI
When I run ./gradlew runAffectedUnitTests -Paffected_module_detector.enable on Bitrise it fails in my project with error:

* What went wrong:
Could not evaluate onlyIf predicate for task ':module:testDebugUnitTest'.
> Nonzero exit value running git command.

the task is working fine locally
My git project is setup on CI and i do git clone before invoking this command

Change default value of `jvmTestTask` for non Android modules

It would be cool if the plugin would use test as default value for jvmTestTask when it's being applied to a pure Java/Kotlin module. This would remove the need to manually configure this.

Task '.enable' not found in root project 'my-multi-module-project'.

I'm not able to use this plugin like described as I'm running into the issue from the title as soon as the parameters are attached to my gradle command.

this is my root build.gradle

plugins {
    id "com.dropbox.affectedmoduledetector" version "0.1.5"
}

apply plugin: 'java'
apply plugin: 'idea'

affectedModuleDetector {
    baseDir = "${project.rootDir}"
    pathsAffectingAllModules = []
    logFolder = "${project.rootDir}/tools/impact-analysis/output"
    specifiedBranch = "reactive"
    compareFrom = "SpecifiedBranchCommit"
    includeUncommitted = false
}

settings.gradle

pluginManagement {
    repositories {
        mavenLocal()
        mavenCentral()
        gradlePluginPortal()
        maven {
            url "https://plugins.gradle.org/m2/"
        }
    }
    plugins {
        id "${quarkusPluginId}" version "${quarkusPluginVersion}"
        id "com.nike.pdm.localstack" version "1.0.0"
    }
}

The logs

PS C:\Users\My-PC\Workspace\my-multi-module-project> ./gradlew build -Paffected_module_detector.enable                 

Welcome to Gradle 7.4.2!

Here are the highlights of this release:
 - Aggregated test and JaCoCo reports
 - Marking additional test source directories as tests in IntelliJ
 - Support for Adoptium JDKs in Java toolchains

For more details see https://docs.gradle.org/7.4.2/release-notes.html


FAILURE: Build failed with an exception.

* What went wrong:
Task '.enable' not found in root project 'my-multi-module-project'.

* Try:
> Run gradlew tasks to get a list of available tasks.
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 2s
PS C:\Users\My-PC\Workspace\my-multi-module-project>

What am I missing?

Open Source AffectedTaskPlugin

Internally we build up tasks to run rather than skipping unaffected. We should push that code public too

Publish snapshot builds using CI with new commits on main branch

Add publishing snapshots to sonatype and gradle plugin portal whenever a new commit on main.

Config property 'customTasks' present In README but not in 0.1.5 version

Feature request: Need support for includeBuild

More details about includeBuild here

Include build can be used as dependency, but for gradle that kind of dependency is external dependency.
ProjectGraph can't work correct with includeBuild dependencies. For ProjectGraph includeBuild dependencies is external dependencies, not project dependencies. In that case not all modules can be found, and marked as unknownFiles.

Build failure for task `createDebugAndroidTestApkListingFileRedirect` upon upgrading to `gradle plugin 7.1.2`

Hello!

I am noticing failures when running ./gradlew assembleAffectedAndroidTests ... when updating gradle plugin to version 7.1.2. It looks like the task createDebugAndroidTestApkListingFileRedirect fails for modules that are marked to skip apk generation.

Here's the failure output:

A problem was found with the configuration of task ':lib:core:createStandardDebugAndroidTestApkListingFileRedirect' (type 'ListingFileRedirectTask').
  - In plugin 'com.android.internal.version-check' type 'com.android.build.gradle.internal.tasks.ListingFileRedirectTask' property 'listingFile' specifies file '/Users/runner05/actions-runner/_work/android/android/lib/core/build/outputs/apk/androidTest/standard/debug/output-metadata.json' which doesn't exist.

Take dependency resolution as a task input

https://docs.gradle.org/7.5-rc-1/release-notes.html

runAffectedUnitTests still runs all tests!

git status     
On branch develop
Your branch is up to date with 'origin/develop'.

nothing to commit, working tree clean

my configuration

affectedModuleDetector {
    baseDir = "${project.rootDir}"
    pathsAffectingAllModules = setOf("gradle/libs.versions.toml")
    logFilename = "output.log"
    logFolder = "${rootProject.buildDir}/affectedModuleDetector"
    specifiedBranch = "develop"
    compareFrom = "SpecifiedBranchCommit" // default is PreviousCommit
}

./gradlew runAffectedUnitTests -Paffected_module_detector.enable

runs all tests
I am using org.gradle.parallel=true. is this a problem?

How to reproduce: please clone repo: https://github.com/xmlking/micro-apps and run above command.

Plugin doesn't work for java services.

Hey Team,

I am trying to apply the plugin for one of our repos which mainly consists of java libraries and applications. The libraries have java-library plugin applied. However, the applications have only java plugin applied.

So the affecttedModule detector won't run the tests of applications as part of runAffectedUnitTests .

I propose to change the implementation to support java applications as well.

This can be achieved by changing this piece of code from :

`                val pluginIds = setOf("com.android.application", "com.android.library", "java-library", "kotlin")
                
                pluginIds.forEach { pluginId ->

                    if (pluginId == "java-library" || pluginId == "kotlin")`
                    .
                    .
                    .
                    }

                val pluginIds = setOf("com.android.application", "com.android.library", "java", "kotlin")

                pluginIds.forEach { pluginId ->

                    if (pluginId == "java" || pluginId == "kotlin")
                    .
                    .
                    .
                    
                    }

in AffectedModuleDetectorPlugin class.

Also, what is the process for raising a PR? Thank you so much for this lovely library/plugin.

CustomTask should use Gradle's NamedDomainObjectContainer or NamedDomainObjectSet

See https://github.com/dropbox/AffectedModuleDetector/pull/130/files#diff-6b5be434aa24824fb2973ce77a9dcda484db988e76842f367fb289129851c318R44

var customTasks = emptySet<AffectedModuleConfiguration.CustomTask>() would be better migrated to DomainObject APIs.
Here's a relevant article: https://blog.mrhaki.com/2016/02/gradle-goodness-create-objects-with-dsl.html

Error when adding to plugins {} block.

Hi,

I am looking to set this up on our project and also leverage it for what modules to run lint for via custom plugin.

Ideally, I would include this as part of our existing plugins block in root/build.gradle:

plugins {
    ...
    ...
    id "com.dropbox.affectedmoduledetector" version "0.1.0.2"
}

The above results in an error (see below) but if I define the version as 0.1.0 it works. However, that's an older version.

Build file '/Users/michael.carrano/Development/android-wwmobile/build.gradle' line: 45

Plugin [id: 'com.dropbox.affectedmoduledetector', version: '0.1.0.2'] was not found in any of the following sources:

* Try:
Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Exception is:
org.gradle.api.plugins.UnknownPluginException: Plugin [id: 'com.dropbox.affectedmoduledetector', version: '0.1.0.2'] was not found in any of the following sources:

- Gradle Core Plugins (plugin is not in 'org.gradle' namespace)
- Plugin Repositories (could not resolve plugin artifact 'com.dropbox.affectedmoduledetector:com.dropbox.affectedmoduledetector.gradle.plugin:0.1.0.2')
  Searched in the following repositories:
    Gradle Central Plugin Repository
	...

It looks like there are a few versions missing from https://plugins.gradle.org/plugin/com.dropbox.affectedmoduledetector and I'm not familiar with how to publish them there.

Compare that to the versions listed at maven central: https://search.maven.org/artifact/com.dropbox.affectedmoduledetector/affectedmoduledetector

For the time being, using the apply plugin method defined in the README works for me.

Not checking modules correctly

Apologies in advance if i'm using the plugin wrong.

When running the plugin on my experimentation project it doesn't seem to be working correctly. If i make a change in a module that is an immediate dependency of the app module (either core or core-navigation) then the command ./gradlew runAffectedUnitTests -Paffected_module_detector.enable works correctly.

However, if I make a change to a module that is a transitive dependency of core (e.g. feature-pokemondetail or feature-pokemonlist) then the output of the logs does not work correctly.

Feel free to clone my repo and try for yourself. Any help appreciated 😄

Different behaviour between CI and local machine

Hi! Please, help me with setup AMD on the CI.

According to a AMD log file a lot of modules on the CI marked as true, but on the local machine the modules marked as false.
command: ./gradlew runAffectedUnitTests -Paffected_module_detector.enable -Pci=true --scan

Environment:
Doker + GitHub actions

Changed files:
Only 1 changed file for test with minor syntax change in feature module.

Expected result: Only few modules was affected.
Actual result: 89 modules was affected on the CI.

Example:
CI machine - 89 modules affected (true) 23 modules is not affected (false)
Local machine - only 3 modules affected (true) 109 modules is not affected (false)

Hence, build time on the CI is not efficent.

build.gradle

affectedModuleDetector {
    baseDir = "${project.rootDir}"
    pathsAffectingAllModules = []
    logFolder = "${project.rootDir}/tools/impact-analysis/output"
    excludedModules = [
        "appLint"
    ]
    specifiedBranch = "origin/dev"
    compareFrom = "SpecifiedBranchCommit"
    includeUncommitted = false
}

CI: github workflow

  unit-test_by_impact:
    runs-on: [ some-build ]
    if: ${{ !startsWith(github.head_ref, 'release') }}
    steps:
      - uses: actions/checkout@v2
        with:
          ref: ${{ github.event.pull_request.head.sha }}
      - name: 'Fetch dev'
        run: git fetch origin dev:dev
      - name: 'Run unit tests by impact'
        run: |
          source ./ci/_builder_docker.sh
          runInContainer "./ci/unit_tests_by_impact.sh"
      - name: 'Upload test_by_impact log file'
        uses: actions/upload-artifact@v2
        with:
          name: test_by_impact-log-${{ github.run_number }}
          path: tools/impact-analysis/output
          retention-days: 1

Run arbitrary tasks

Hi there!

If I understand correctly this plugin figures out where a change came from and then run all the unit tests, all the instrumentation tests, or simply build everything that falls under the scope depending on the configuration and where the change was made.

However, what if I want to run additional tasks? Say, for example, Sonarqube or other static analysis? Does this plugin support that? Or could it in the future?

Many thanks.

dropbox / affectedmoduledetector Goto Github PK

affectedmoduledetector's People

Contributors

Stargazers

Watchers

Forkers

affectedmoduledetector's Issues

Environment

Description

Configuration

Issue

Reason

Solution

Recommend Projects

Recommend Topics

Recommend Org