dropbox / affectedmoduledetector Goto Github PK
View Code? Open in Web Editor NEWA Gradle Plugin to determine which modules were affected by a set of files in a commit.
License: Apache License 2.0
A Gradle Plugin to determine which modules were affected by a set of files in a commit.
License: Apache License 2.0
jCenter
support should be removed as it is being sunset
https://jfrog.com/blog/into-the-sunset-bintray-jcenter-gocenter-and-chartcenter/
Hello!
I am running into an issue where calling
gradle assembleAffectedAndroidTests -Paffected_module_detector.enabled
assembles the APKs for all modules.
Based on the documentation, I am under the assumption that this task should generate test apks for only the affected module(s).
Here are the configurations I have set up:
affectedModuleDetector {
baseDir = "${project.rootDir}"
logFolder = "${project.rootDir}/artifacts"
specifiedBranch = "${base_branch}"
compareFrom = 'SpecifiedBranchCommit'
}
affectedTestConfiguration {
assembleAndroidTestTask = 'assembleStandardDebugAndroidTest'
runAndroidTestTask = 'assembleStandardDebugAndroidTest'
jvmTestTask = 'testStandardDebugUnitTest'
}
Cross-referencing the logs and the apk output, I see that
[INFO] [amd] checking whether I should include :lib:featuretoggle and my answer is false
but
./lib/featuretoggle/build/outputs/apk/androidTest/standard/debug/featuretoggle-standard-debug-androidTest.apk
is still being generated.
Please let me know if this is the expected behavior or if I am doing something wrong. Thank you!
When run ./gradlew runAffectedUnitTests -Paffected_module_detector.enable
it fails in my project with error:
* What went wrong:
Could not evaluate onlyIf predicate for task ':somemodule:testDebugUnitTest'.
> Nonzero exit value running git command.
Stacktrace
* Exception is:
org.gradle.api.GradleException: Could not evaluate onlyIf predicate for task ':api:somemodule:testDebugUnitTest'.
at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:46)
at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:200)
at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:195)
at org.gradle.internal.operations.DefaultBuildOperationRunner$3.execute(DefaultBuildOperationRunner.java:75)
at org.gradle.internal.operations.DefaultBuildOperationRunner$3.execute(DefaultBuildOperationRunner.java:68)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:153)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:68)
at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:62)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.lambda$call$2(DefaultBuildOperationExecutor.java:79)
at org.gradle.internal.operations.UnmanagedBuildOperationWrapper.callWithUnmanagedSupport(UnmanagedBuildOperationWrapper.java:54)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:79)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:74)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:408)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:395)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:388)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:374)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.lambda$run$0(DefaultPlanExecutor.java:127)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:191)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.executeNextNode(DefaultPlanExecutor.java:182)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:124)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalStateException: Nonzero exit value running git command.
at com.dropbox.affectedmoduledetector.GitClientImpl$RealCommandRunner.execute(GitClient.kt:162)
at com.dropbox.affectedmoduledetector.GitClientImpl$RealCommandRunner.executeAndParse(GitClient.kt:166)
at com.dropbox.affectedmoduledetector.GitClientImpl$RealCommandRunner.executeAndParseFirst(GitClient.kt:176)
at com.dropbox.affectedmoduledetector.commitshaproviders.ForkCommit.get(ForkCommit.kt:8)
at com.dropbox.affectedmoduledetector.GitClientImpl.findChangedFiles(GitClient.kt:80)
at com.dropbox.affectedmoduledetector.GitClient$DefaultImpls.findChangedFiles$default(GitClient.kt:31)
at com.dropbox.affectedmoduledetector.AffectedModuleDetectorImpl.findChangedProjects(AffectedModuleDetector.kt:403)
at com.dropbox.affectedmoduledetector.AffectedModuleDetectorImpl.access$findChangedProjects(AffectedModuleDetector.kt:314)
at com.dropbox.affectedmoduledetector.AffectedModuleDetectorImpl$changedProjects$2.invoke(AffectedModuleDetector.kt:354)
at com.dropbox.affectedmoduledetector.AffectedModuleDetectorImpl$changedProjects$2.invoke(AffectedModuleDetector.kt:314)
at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:74)
at com.dropbox.affectedmoduledetector.AffectedModuleDetectorImpl.getChangedProjects(AffectedModuleDetector.kt)
at com.dropbox.affectedmoduledetector.AffectedModuleDetectorImpl.findAffectedProjects(AffectedModuleDetector.kt:462)
at com.dropbox.affectedmoduledetector.AffectedModuleDetectorImpl.access$findAffectedProjects(AffectedModuleDetector.kt:314)
at com.dropbox.affectedmoduledetector.AffectedModuleDetectorImpl$affectedProjects$2.invoke(AffectedModuleDetector.kt:350)
at com.dropbox.affectedmoduledetector.AffectedModuleDetectorImpl$affectedProjects$2.invoke(AffectedModuleDetector.kt:314)
at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:74)
at com.dropbox.affectedmoduledetector.AffectedModuleDetectorImpl.getAffectedProjects(AffectedModuleDetector.kt)
at com.dropbox.affectedmoduledetector.AffectedModuleDetectorImpl.shouldInclude(AffectedModuleDetector.kt:366)
at com.dropbox.affectedmoduledetector.AffectedModuleDetector$Companion.isProjectAffected(AffectedModuleDetector.kt:254)
at com.dropbox.affectedmoduledetector.AffectedModuleDetectorPlugin$withPlugin$1$$special$$inlined$let$lambda$1$1.isSatisfiedBy(AffectedModuleDetectorPlugin.kt:98)
at com.dropbox.affectedmoduledetector.AffectedModuleDetectorPlugin$withPlugin$1$$special$$inlined$let$lambda$1$1.isSatisfiedBy(AffectedModuleDetectorPlugin.kt:36)
at org.gradle.api.specs.AndSpec.isSatisfiedBy(AndSpec.java:50)
at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:44)
... 30 more
This plugin still needs to be updated to support the Gradle task configuration avoidance APIs documented here:
https://docs.gradle.org/current/userguide/task_configuration_avoidance.html
Some examples of the use of the old/wrong APIs can be found in filterAndroidTests
:
private fun filterAndroidTests(project: Project) {
val tracker = DependencyTracker(project, null)
project.tasks.all { task ->
...
...and filterJvmTests
:
project.tasks.withType(Test::class.java) { task ->
AffectedModuleDetector.configureTaskGuard(task)
}
As a result of the above, our large project is eagerly creating thousands of testing tasks at configuration time.
The migration guide may be useful as a resource.
More details about includeBuild here
Include build can be used as dependency, but for gradle that kind of dependency is external dependency.
ProjectGraph can't work correct with includeBuild dependencies. For ProjectGraph includeBuild dependencies is external dependencies, not project dependencies. In that case not all modules can be found, and marked as unknownFiles.
Hi,
Thanks for sharing this to the world.
I would like to run different flows based on the package changed (for example)
And have a default set running no matter what was the change (in addition).
Thanks.
Using an extension, allow each module to specify which task it would like to run when executing tests.
At the moment, the plugin may not find tests associated with java only modules, and subsequently, not run any tests in those modules.
Hello!
I am noticing failures when running ./gradlew assembleAffectedAndroidTests ...
when updating gradle plugin to version 7.1.2. It looks like the task createDebugAndroidTestApkListingFileRedirect
fails for modules that are marked to skip apk generation.
Here's the failure output:
A problem was found with the configuration of task ':lib:core:createStandardDebugAndroidTestApkListingFileRedirect' (type 'ListingFileRedirectTask').
- In plugin 'com.android.internal.version-check' type 'com.android.build.gradle.internal.tasks.ListingFileRedirectTask' property 'listingFile' specifies file '/Users/runner05/actions-runner/_work/android/android/lib/core/build/outputs/apk/androidTest/standard/debug/output-metadata.json' which doesn't exist.
pathsAffectingAllModules
will only be observed if the files are also unknown.
In our case, we have files which are associated with projects, but should also trigger all the builds.
Hey Team,
I am wondering when the latest version 0.1.0.2
will be publicly available? Also, any plans on releasing the 0.1.3
version?
Currently the command used when including uncommitted code to be compared to a commit is (seen here):
git --no-pager diff --name-only HEAD..$sha
this is incorrect as HEAD points to the last commit on the branch and therefore does not include uncommitted changes.
The command you are looking for is:
git --no-pager diff --name-only $sha
I can put a fix in after my PR is merged
The link to the original AndroidX project in the README leads to a 404 Not Found error.
https://android.googlesource.com/platform/frameworks/support/+/androidx-master-dev/buildSrc/src/main/kotlin/androidx/build/dependencyTracker
This is because the androidx-master-dev
branch was replaced by androidx-main
:
https://android.googlesource.com/platform/frameworks/support/+/c19a44147515bcc4404409e27db5de4e6e9aa73c
The link should be updated to:
https://android.googlesource.com/platform/frameworks/support/+/androidx-main/buildSrc/src/main/kotlin/androidx/build/dependencyTracker
Side note: I found this repo via this Dropbox blog which also has the old link. ๐
Hello ๐
I have one question : what happens when a Pull Request has more than one commits ? For example a Pull Request might have 3 commits: Commit A
changes Module_A
, Commit B
changes Module_B
and Commit C
changes Module_C
. Assuming the last one was Commit C
then trying to run AMD would only run tests for Module_C
. From the tests I made all three options: ForkCommit
, PreviousCommit
and specifiedBranch
will only run Module_C
's tests. But ideally what we want here is to run tests for all affected modules, cause that Pull Request would affect all 3 modules I mentioned.
Is that the case or maybe I don't understand something ? If yes then how do you deal with it ? What's the "proper" way to utilise this cool library to save me some time?
Thank you!!
This will be a tracking ticket. In 0.1.6 we will be marking those tasks as not cacheable but would like to investigate if it's possible to somehow do this. Our current problem is this stack trace
Caused by: org.gradle.api.UnknownDomainObjectException: Extension with name 'AffectedModuleDetectorPlugin' does not exist. Currently registered extension names: [ext]
at org.gradle.internal.extensibility.ExtensionsStorage.unknownExtensionException(ExtensionsStorage.java:144)
at org.gradle.internal.extensibility.ExtensionsStorage.getByName(ExtensionsStorage.java:123)
at org.gradle.internal.extensibility.DefaultConvention.getByName(DefaultConvention.java:174)
at com.dropbox.affectedmoduledetector.AffectedModuleDetector$Companion.getInstance(AffectedModuleDetector.kt:195)
at com.dropbox.affectedmoduledetector.AffectedModuleDetector$Companion.getOrThrow(AffectedModuleDetector.kt:200)
at com.dropbox.affectedmoduledetector.AffectedModuleDetector$Companion.configureTaskGuard$lambda-3(AffectedModuleDetector.kt:233)
at org.gradle.api.specs.AndSpec.isSatisfiedBy(AndSpec.java:50)
at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:44)
... 24 more
As noted in this slack message on the gradle slack, the apply
method is not called on plugins which cached so the call in our onlyIf
block to check if a project is enabled and affected no longer works. If we could move that check into the task possibly rather then the plugin maybe this is doable
Hey Team,
I am trying to apply the plugin for one of our repos which mainly consists of java
libraries and applications. The libraries have java-library
plugin applied. However, the applications have only java
plugin applied.
So the affecttedModule
detector won't run the tests
of applications as part of runAffectedUnitTests
.
I propose to change the implementation to support java
applications as well.
This can be achieved by changing this piece of code from :
` val pluginIds = setOf("com.android.application", "com.android.library", "java-library", "kotlin")
pluginIds.forEach { pluginId ->
if (pluginId == "java-library" || pluginId == "kotlin")`
.
.
.
}
to
val pluginIds = setOf("com.android.application", "com.android.library", "java", "kotlin")
pluginIds.forEach { pluginId ->
if (pluginId == "java" || pluginId == "kotlin")
.
.
.
}
in AffectedModuleDetectorPlugin class.
Also, what is the process for raising a PR? Thank you so much for this lovely library/plugin.
AffectedModuleDetector is a lot of syllables and could be something more concise while still descriptive.
DropTask has been a suggestion.
Open to other suggestions as well.
Carrying on the discussion from here. I plan to raise a PR to support exotic / non standard Gradle projects. These are defined by Gradle projects whose sub projects are not descendants of the root project (in the File system sense). This is a non standard, but valid, layout in the Gradle sense.
Yes it is an edge case but there are no special instructions in AMD to state compatibility of Gradle project layouts so I feel we should support these, still valid, set ups.
A "traditional" Gradle project will look like this:
my-app <-- projectRootDir & gitRootDir
|
-- .git
|
-- module-1
|
-- module-2
|
-- module-3
|
-- settings.gradle.kts
The .git
folder is a child of the my-app
folder on the file system and thus the my-app
folder is the gitRootDir
. There are no customisations in settings.gradle.kts
other than to include
the modules module-1
, module-2
, and module-3
. Gradle will also see my-app
as the projectRootDir
. This is what most Android projects look like and is a fine and sensible layout to assume.
However I came across this setup at work:
android-project <-- gitRootDir
|
-- .git
|
-- diagrams
|
-- ci
|
-- proposals
|
-- my-app <-- projectRootDir
|
-- settings.gradle.kts
|
-- common
|
-- module-1
|
-- module-2
|
-- module-3
By instructing settings.gradle.kts
where to look on the File system for the sub projects this type of layout is valid. It is certainly non standard, I will admit ๐ .
I think I have tracked the issue down to the init
block in ProjectGraph
where we build up a relationship between the Gradle project graph and the File system graph (later to be used as the Git graph). If we are in a non standard Gradle project we will start to see ".."
entries in the sections
that we need to remove otherwise we start seeing false negatives in the detection. An example of the changes look like this.
Obviously I will add tests in a proper PR but this highlights the issue and opens it up for discussion.
Consider using https://github.com/ansman/auto-plugin for generation of the META-INF properties files.
On our gradle project we have several apps and lots of modules. Is there any way to only assemble the APKs of affected apps?
I'm integrating AMD into my project and I created two custom tasks. When I run `./gradlew runAffectedDetekt', I'm getting this error even though linter is specified in the excludedModules.
* What went wrong:
Could not determine the dependencies of task ':runAffectedDetekt'.
> Task with path ':linter:detekt' not found in root project 'Rappi-Android-User'.
This error makes sense since Detekt is not applied in the linter module; however, it should be ignored by AMD if it's specified in the excluded modules.
import com.dropbox.affectedmoduledetector.AffectedModuleConfiguration
apply plugin: "com.dropbox.affectedmoduledetector"
def branch = System.getenv('DESTINATION_BRANCH')
if (!branch) {
branch = "origin/develop"
}
logger.lifecycle("Destination branch: $branch")
affectedModuleDetector {
baseDir = "${project.rootDir}"
pathsAffectingAllModules = [
"tools/global-dependencies.gradle"
]
logFilename = "output.log"
logFolder = "${project.rootDir}"
specifiedBranch = "$branch"
compareFrom = "SpecifiedBranchCommit"
excludedModules = ["linter", ":linter"]
customTasks = [
new AffectedModuleConfiguration.CustomTask(
"runAffectedDetekt,
"detekt",
"Run detekt"
),
new AffectedModuleConfiguration.CustomTask(
"runAffectedAndroidLint",
"lintDebug",
"Run lint"
),
]
Hey,
Could it be possible to add support to exclude certain path(s) from jvm test execution?
There is a exclude(Iterable<String> excludes)
function in org.gradle.api.tasks.testing.Test
class for it.
Use case:
We need it to avoid running e.g. paparazzi screenshot tests as part of main unit test task launched by runAffectedUnitTests
task.
Up until now we use command line parameter:
-PexcludeTests="**/screenshots/**, **/another_excluded_path/**"
That is being read and applied in testOptions.unitTest
block
Noob question, how do I use the api? I want to build (not test) affected modules only.
var customTasks = emptySet<AffectedModuleConfiguration.CustomTask>()
would be better migrated to DomainObject APIs.
Here's a relevant article: https://blog.mrhaki.com/2016/02/gradle-goodness-create-objects-with-dsl.html
Hi,
I am looking to set this up on our project and also leverage it for what modules to run lint for via custom plugin.
Ideally, I would include this as part of our existing plugins block in root/build.gradle
:
plugins {
...
...
id "com.dropbox.affectedmoduledetector" version "0.1.0.2"
}
The above results in an error (see below) but if I define the version as 0.1.0
it works. However, that's an older version.
Build file '/Users/michael.carrano/Development/android-wwmobile/build.gradle' line: 45
Plugin [id: 'com.dropbox.affectedmoduledetector', version: '0.1.0.2'] was not found in any of the following sources:
* Try:
Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Exception is:
org.gradle.api.plugins.UnknownPluginException: Plugin [id: 'com.dropbox.affectedmoduledetector', version: '0.1.0.2'] was not found in any of the following sources:
- Gradle Core Plugins (plugin is not in 'org.gradle' namespace)
- Plugin Repositories (could not resolve plugin artifact 'com.dropbox.affectedmoduledetector:com.dropbox.affectedmoduledetector.gradle.plugin:0.1.0.2')
Searched in the following repositories:
Gradle Central Plugin Repository
...
It looks like there are a few versions missing from https://plugins.gradle.org/plugin/com.dropbox.affectedmoduledetector and I'm not familiar with how to publish them there.
Compare that to the versions listed at maven central: https://search.maven.org/artifact/com.dropbox.affectedmoduledetector/affectedmoduledetector
For the time being, using the apply plugin
method defined in the README works for me.
As a part of the effort to reduce feedback time on PRs we are considering applying this plugin, but we would still like to get code coverage reports after. Thanks in advance
I had to re-write the DependencyTracker in Groovy, seems a waste to just delete it now I've finished.
Here is it for posterity.
class DependencyTracker {
private final Project rootProject
private final Logger logger
DependencyTracker(Project rootProject, Logger logger) {
this.rootProject = rootProject
this.logger = logger
}
@Lazy Map<Project, Set<Project>> dependentList = {
Map<Project, Set<Project>> result = new HashMap<>()
rootProject.subprojects.forEach { project ->
logger?.debug("checking ${project.path} for dependencies")
project.configurations.forEach { config ->
logger?.debug("checking config ${project.path}/$config for dependencies")
config
.dependencies
.findAll { it instanceof ProjectDependency }
.forEach {
logger?.debug("there is a dependency from ${project.path} to " + it.dependencyProject.path)
def value = result.get(it.dependencyProject)
if (value == null) {
value = new HashSet<Project>()
}
value.add(project)
result.put(it.dependencyProject, value)
}
}
}
result
}()
private def addAllDependents(Set<Project> result, Project project) {
if (result.add(project)) {
dependentList[project]?.forEach {
addAllDependents(result, it)
}
}
}
Set<Project> findAllDependents(Project project) {
println("finding dependents of ${project.path}")
Set<Project> result = new HashSet<>()
addAllDependents(result, project)
logger?.debug("dependents of ${project.path} is ${result.collect { it.path }}")
// the project isn't a dependent of itself
return result - project
}
}
Hi! Please, help me with setup AMD on the CI.
According to a AMD log file a lot of modules on the CI marked as true, but on the local machine the modules marked as false.
command: ./gradlew runAffectedUnitTests -Paffected_module_detector.enable -Pci=true --scan
Environment:
Doker + GitHub actions
Changed files:
Only 1 changed file for test with minor syntax change in feature module.
Expected result: Only few modules was affected.
Actual result: 89 modules was affected on the CI.
Example:
CI machine - 89 modules affected (true) 23 modules is not affected (false)
Local machine - only 3 modules affected (true) 109 modules is not affected (false)
Hence, build time on the CI is not efficent.
build.gradle
affectedModuleDetector {
baseDir = "${project.rootDir}"
pathsAffectingAllModules = []
logFolder = "${project.rootDir}/tools/impact-analysis/output"
excludedModules = [
"appLint"
]
specifiedBranch = "origin/dev"
compareFrom = "SpecifiedBranchCommit"
includeUncommitted = false
}
CI: github workflow
unit-test_by_impact:
runs-on: [ some-build ]
if: ${{ !startsWith(github.head_ref, 'release') }}
steps:
- uses: actions/checkout@v2
with:
ref: ${{ github.event.pull_request.head.sha }}
- name: 'Fetch dev'
run: git fetch origin dev:dev
- name: 'Run unit tests by impact'
run: |
source ./ci/_builder_docker.sh
runInContainer "./ci/unit_tests_by_impact.sh"
- name: 'Upload test_by_impact log file'
uses: actions/upload-artifact@v2
with:
name: test_by_impact-log-${{ github.run_number }}
path: tools/impact-analysis/output
retention-days: 1
Blocked by vanniktech/gradle-maven-publish-plugin#259
Whenever I apply the plugin to a project and run the ./gradlew runAffectedUnitTests -Paffected_module_detector.enable
command, I get the following error:
FAILURE: Build failed with an exception.
* What went wrong:
Task 'runAffectedUnitTests' not found in root project 'my-project'.
Running ./gradle tasks
or ./gradlew tasks -Paffected_module_detector.enable
results in no tasks added by this plugin either.
I have tried with multiple Android/Java only projects which all have different configurations, but all come up with the same results. I even copy pasted this entire block into the root build.gradle (thus having two buildscript
blocks) to no success.
buildscript {
repositories {
maven {
url "https://plugins.gradle.org/m2/"
}
}
dependencies {
classpath "com.dropbox.affectedmoduledetector:affectedmoduledetector:0.1.0"
}
}
apply plugin: "com.dropbox.affectedmoduledetector"
Apologies in advance if i'm using the plugin wrong.
When running the plugin on my experimentation project it doesn't seem to be working correctly. If i make a change in a module that is an immediate dependency of the app module (either core or core-navigation) then the command ./gradlew runAffectedUnitTests -Paffected_module_detector.enable
works correctly.
However, if I make a change to a module that is a transitive dependency of core (e.g. feature-pokemondetail or feature-pokemonlist) then the output of the logs does not work correctly.
Feel free to clone my repo and try for yourself. Any help appreciated ๐
I cannot set like this in project root's build.gradle.kts
file.
subprojects {
affectedTestConfiguration { jvmTestTask = "test" }
}
all my subprojects are java, it would be nice if we can set this at one place instead of forced to add affectedTestConfiguration { jvmTestTask = "test" }
to every subprojects
my multi-module project : https://github.com/xmlking/micro-apps
Add publishing snapshots to sonatype and gradle plugin portal whenever a new commit on main
.
Currently, the git client will only query for HEAD~1
and determine the files in the last commit.
We should add the ability to let a developer specify a branch to compare against and determine all the changed files between HEAD
and that branch.
We should configure this in the extension and let the user decide which option to use and the branch to compare against.
"ForkCommit" as I understand it currently requires a merge commit in order to determine the parent branch. However if a project is practicing rebasing as a way of working and merging work, then there will be no merge commits. Not sure what the solution to this would be off the top of my head, just pointing out this is an issue.
I set up the plugin by adding this to my top-level build.gradle
:
apply plugin: "com.dropbox.affectedmoduledetector"
affectedModuleDetector {
baseDir = "${project.rootDir}"
specifiedBranch = "tmp2"
compareFrom = "SpecifiedBranchCommit"
includeUncommitted = false
logFilename = "output.log"
logFolder = "${project.rootDir}/output"
}
subprojects {
affectedTestConfiguration {
jvmTestTask = "foo"
}
task foo() {
println "${project.name} running tests"
}
}
Then made a code change to the gradle module called :monographs
, so that HEAD
diffs with tmp2
to show only this code change.
Then I run: ./gradlew runAffectedUnitTests -Paffected_module_detector.enable -Paffected_module_detector.changedProjects
which gives an output of:
> Configure project :
app running tests
articles running tests
authlib running tests
monographs running tests
pharma running tests
platform running tests
rteditor running tests
search running tests
shared-api running tests
shared-contract running tests
shared-test running tests
shared-ui running tests
snippets running tests
SwipeLayout running tests
> Configure project :app
WARNING:DSL element 'dexOptions' is obsolete and should be removed.
It will be removed in version 8.0 of the Android Gradle plugin.
Using it has no effect, and the AndroidGradle plugin optimizes dexing automatically.
The Hilt configuration option 'enableTransformForLocalTests' is no longer necessary when com.android.tools.build:gradle:4.2.0+ is used.
...
So, tests are being run for every module in my project! I would expect to see only monographs running tests
, and no others in the output.
Here's my log file from AMD:
[INFO] [amd] Modules provided: null
[INFO] [amd] Using real detector with CHANGED_PROJECTS
[INFO] [amd] running command git rev-parse tmp2 in /home/hin/code/android-avocado
[INFO] [amd] Response: 47a71a0bfe4891534a49826a9f25726dddb7de47
[INFO] [amd] running command git --no-pager diff --name-only -M95 HEAD..47a71a0bfe4891534a49826a9f25726dddb7de47 in /home/hin/code/android-avocado
[INFO] [amd] Response: monographs/src/main/java/de/miamed/amboss/monograph/feedback/MonographFeedbackConfig.kt
[INFO] [amd] Paths affecting all modules: []
[INFO] [amd] initializing ProjectGraph
[INFO] [amd] creating node for :app
[INFO] [amd] relative path: app , sections: [app]
[INFO] [amd] creating node for :articles
[INFO] [amd] relative path: articles , sections: [articles]
[INFO] [amd] creating node for :authlib
[INFO] [amd] relative path: authlib/lib , sections: [authlib, lib]
[INFO] [amd] creating node for :monographs
[INFO] [amd] relative path: monographs , sections: [monographs]
[INFO] [amd] creating node for :pharma
[INFO] [amd] relative path: pharma , sections: [pharma]
[INFO] [amd] creating node for :platform
[INFO] [amd] relative path: platform , sections: [platform]
[INFO] [amd] creating node for :rteditor
[INFO] [amd] relative path: rteditor/RTEditor , sections: [rteditor, RTEditor]
[INFO] [amd] creating node for :search
[INFO] [amd] relative path: search , sections: [search]
[INFO] [amd] creating node for :shared-api
[INFO] [amd] relative path: shared-api , sections: [shared-api]
[INFO] [amd] creating node for :shared-contract
[INFO] [amd] relative path: shared-contract , sections: [shared-contract]
[INFO] [amd] creating node for :shared-test
[INFO] [amd] relative path: shared-test , sections: [shared-test]
[INFO] [amd] creating node for :shared-ui
[INFO] [amd] relative path: shared-ui , sections: [shared-ui]
[INFO] [amd] creating node for :snippets
[INFO] [amd] relative path: snippets , sections: [snippets]
[INFO] [amd] creating node for :SwipeLayout
[INFO] [amd] relative path: AndroidSwipeLayout/library , sections: [AndroidSwipeLayout, library]
[INFO] [amd] finished creating ProjectGraph com.dropbox.affectedmoduledetector.ProjectGraph$Node@106a7c49
[INFO] [amd] finding containing project for monographs/src/main/java/de/miamed/amboss/monograph/feedback/MonographFeedbackConfig.kt , sections: [monographs, src, main, java, de, miamed, amboss, monograph, feedback, MonographFeedbackConfig.kt]
[INFO] [amd] finding [monographs, src, main, java, de, miamed, amboss, monograph, feedback, MonographFeedbackConfig.kt] with index 0 in root
[INFO] [amd] finding [monographs, src, main, java, de, miamed, amboss, monograph, feedback, MonographFeedbackConfig.kt] with index 1 in :monographs
[INFO] [amd] no child found, returning :monographs
[INFO] [amd] search result for monographs/src/main/java/de/miamed/amboss/monograph/feedback/MonographFeedbackConfig.kt resulted in :monographs
[INFO] [amd] For file monographs/src/main/java/de/miamed/amboss/monograph/feedback/MonographFeedbackConfig.kt containing project is project ':monographs'. Adding to changedProjects.
[INFO] [amd] checking whether I should include :app and my answer is false
[INFO] [amd] checking whether I should include :authlib and my answer is false
[INFO] [amd] checking whether I should include :articles and my answer is false
[INFO] [amd] checking whether I should include :monographs and my answer is true
[INFO] [amd] checking whether I should include :search and my answer is falsealse
[INFO] [amd] checking whether I should include :rteditor and my answer is false
[INFO] [amd] checking whether I should include :pharma and my answer is false
[INFO] [amd] checking whether I should include :shared-contract and my answer is false
[INFO] [amd] checking whether I should include :shared-test and my answer is false
[INFO] [amd] checking whether I should include :shared-ui and my answer is false
[INFO] [amd] checking whether I should include :snippets and my answer is false
[INFO] [amd] checking whether I should include :SwipeLayout and my answer is false
Is there something I am missing here?
I am playing around with this plugin and it seems like the variantToTest
that is mentioned in the README isn't actually implemented.
affectedTestConfiguration{
variantToTest = "debug" //default is debug
}
You can see variantToTest
isn't defined in AffectedTestConfiguration. However, you can see an example of this in the sample app AffectedTasksPlugin
To reproduce, you can try adding variantToTest
in one of the sample/build.gradle files such as https://github.com/dropbox/AffectedModuleDetector/blob/main/sample/sample-app/build.gradle#L8-L10. I actually run into a compile error when I try to build after adding variantToTest
.
I am working on making changes so this works but so far, still only seeing the tests for debug variant running.
I changed the AffectedTestConfiguration
to:
open class AffectedTestConfiguration {
var variantToTest : String = "debug"
var assembleAndroidTestTask : String? = "assemble${variantToTest.capitalize()}AndroidTest"
var runAndroidTestTask : String? = "connected${variantToTest.capitalize()}AndroidTest"
var jvmTestTask : String? = "test${variantToTest.capitalize()}UnitTest"
companion object {
const val name = "affectedTestConfiguration"
}
}
Any pointers? Would love to make a PR with the fix.
Proposal: Parse a list of module names into a configuration which wont be considered by the dependency decision making process, as well as the test apk building and running.
Please let me know of any objections, otherwise I will start the work for this in the upcoming days.
git status
On branch develop
Your branch is up to date with 'origin/develop'.
nothing to commit, working tree clean
my configuration
affectedModuleDetector {
baseDir = "${project.rootDir}"
pathsAffectingAllModules = setOf("gradle/libs.versions.toml")
logFilename = "output.log"
logFolder = "${rootProject.buildDir}/affectedModuleDetector"
specifiedBranch = "develop"
compareFrom = "SpecifiedBranchCommit" // default is PreviousCommit
}
./gradlew runAffectedUnitTests -Paffected_module_detector.enable
runs all tests
I am using org.gradle.parallel=true
. is this a problem?
How to reproduce: please clone repo: https://github.com/xmlking/micro-apps and run above command.
Internally we build up tasks to run rather than skipping unaffected. We should push that code public too
According to the docs, we can specify excludedModules
while in the sample code its using version 0.1.2-SNAPSHOT
and the version available on maven central is 0.1.1
, any plans to release this new version ?? This is kinda blocking us from adopting this plugin, thx in advance.
In changes with file moves, occasionally the detector doesn't catch that the move occurred and that affected modules should be run. This can be seen in the log file when the git diff
command is run.
It would be cool if the plugin would use test
as default value for jvmTestTask
when it's being applied to a pure Java/Kotlin module. This would remove the need to manually configure this.
Currently, if a commit changes only a path listed in pathsAffectingAllModules
and you specify Paffected_module_detector.changedProjects
when running AMD, no modules will be found as affected.
The relevant code is in the findAffectedProjects
method of AffectedModuleDetector.kt
, where we return early if projectSubset == ProjectSubset.CHANGED_PROJECTS
. At that point, if the only file change is a path that affects all modules, changedProjects
will be an empty set.
This goes against our documentation where ProjectSubset.ALL_AFFECTED_PROJECTS
is the union of CHANGED_PROJECTS
and DEPENDENT_PROJECTS
. In the example, ALL_AFFECTED_PROJECTS
would be every project whereas CHANGED_PROJECTS
and DEPENDENT_PROJECTS
would be empty sets.
We should update findChangedProjects
to include the check for whether any of the changed files affect all modules. In the case that it does, every module should be added to the set of changed projects. The logic in findAffectedProjects
around detecting if a path changed that affects all projects should be removed and we should define the affectedProjects
field to be the union of changedProjects
and dependentProjects
I'm not able to use this plugin like described as I'm running into the issue from the title as soon as the parameters are attached to my gradle command.
this is my root build.gradle
plugins {
id "com.dropbox.affectedmoduledetector" version "0.1.5"
}
apply plugin: 'java'
apply plugin: 'idea'
affectedModuleDetector {
baseDir = "${project.rootDir}"
pathsAffectingAllModules = []
logFolder = "${project.rootDir}/tools/impact-analysis/output"
specifiedBranch = "reactive"
compareFrom = "SpecifiedBranchCommit"
includeUncommitted = false
}
settings.gradle
pluginManagement {
repositories {
mavenLocal()
mavenCentral()
gradlePluginPortal()
maven {
url "https://plugins.gradle.org/m2/"
}
}
plugins {
id "${quarkusPluginId}" version "${quarkusPluginVersion}"
id "com.nike.pdm.localstack" version "1.0.0"
}
}
The logs
PS C:\Users\My-PC\Workspace\my-multi-module-project> ./gradlew build -Paffected_module_detector.enable
Welcome to Gradle 7.4.2!
Here are the highlights of this release:
- Aggregated test and JaCoCo reports
- Marking additional test source directories as tests in IntelliJ
- Support for Adoptium JDKs in Java toolchains
For more details see https://docs.gradle.org/7.4.2/release-notes.html
FAILURE: Build failed with an exception.
* What went wrong:
Task '.enable' not found in root project 'my-multi-module-project'.
* Try:
> Run gradlew tasks to get a list of available tasks.
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 2s
PS C:\Users\My-PC\Workspace\my-multi-module-project>
What am I missing?
Hi there!
If I understand correctly this plugin figures out where a change came from and then run all the unit tests, all the instrumentation tests, or simply build everything that falls under the scope depending on the configuration and where the change was made.
However, what if I want to run additional tasks? Say, for example, Sonarqube or other static analysis? Does this plugin support that? Or could it in the future?
Many thanks.
Could you publish a plugin marker artifact to maven central to make it possible to use the plugin like this without needing any extra config?
plugins {
id "com.dropbox.affectedmoduledetector" version "..."
}
When you use the java-gradle-plugin
and maven-publish
together it should be created automatically. Here are the Gradle docs https://docs.gradle.org/current/userguide/plugins.html#sec:plugin_markers
Hey,
this was mentioned before but in my case it's only failing on CI
When I run ./gradlew runAffectedUnitTests -Paffected_module_detector.enable on Bitrise it fails in my project with error:
* What went wrong:
Could not evaluate onlyIf predicate for task ':module:testDebugUnitTest'.
> Nonzero exit value running git command.
the task is working fine locally
My git project is setup on CI and i do git clone before invoking this command
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.