Develop Reference

Gradle Daemon stopped working in parallel due to file lock timeouts

Yesterday all of the sudden my projects on a Windows 10 machine stopped running in parallel due to file lock timeouts.
All my projects are using gradle-wrapper and provide a run task
When I start the 1st run-task, it work normally, but any following run-tasks break with the error like this:
> .\gradlew run
Starting a Gradle Daemon, 1 busy and 4 stopped Daemons could not be reused, use --status for details
FAILURE: Build failed with an exception.
* What went wrong:
Gradle could not start your build.
> Could not create service of type FileAccessTimeJournal using GradleUserHomeScopeServices.createFileAccessTimeJournal().
> Timeout waiting to lock journal cache (C:\Users\injec\.gradle\caches\journal-1). It is currently in use by another Gradle instance.
Owner PID: 16440
Our PID: 12216
Owner Operation:
Our operation:
Lock file: C:\Users\injec\.gradle\caches\journal-1\journal-1.lock
the --status option shows:
> .\gradlew --status
PID STATUS INFO
12216 IDLE 6.9.1
16440 BUSY 6.9.1
14992 STOPPED (stop command received)
7856 STOPPED (other compatible daemons were started and after being idle for 0 minutes and not recently used)
26680 STOPPED (by user or operating system)
18556 STOPPED (by user or operating system)
I tried different tricks, like switching the Gradle verison 5.6.1 - 6.8.3 - 6.9.1 and using the --stop option, but the error remains.
Adding the --stacktrace to the run command reveals that not only journal-1 cache is involved, but also some others dirs like modules-2.
I didn't do any changes to my system, apart from regular Win10 updates.
How can the problem be fixed?
TIA

It's likely that gradle process was exited abnormally and left the lock file behind. Check in the task manager if process with id 16440 exists, and if not just remove the orphan lock file C:\Users\injec\.gradle\caches\journal-1\journal-1.lock

This may be the file-system permissions of C:\Users\injec\.gradle... while you may have overseen one detail: you're calling .\gradlew instead of ./gradlew or gradlew.bat ...which means that you are not running on CMD, but on PS or WSL. gradlew.bat run would run directly on CMD.

Check .gradle file system. Gradle not works well on non-native file system. https://github.com/gradle/gradle/issues/15881
File system watching supports the following file system types:
APFS
btrfs
ext3
ext4
XFS
HFS+
NTFS
Gradle also supports VirtualBox’s shared folders.
Network file systems like Samba and NFS are not supported.
Symlinks
File system watching is not compatible with symlinks. If your project files include symlinks, symlinked files do not benefit from file system watching optimizations.
or you can disable file system watch for the build https://docs.gradle.org/current/userguide/file_system_watching.html#disable
Gradle maintains a Virtual File System (VFS) to calculate what needs to be rebuilt on repeat builds of a project. By watching the file system, Gradle keeps the VFS current between builds.

Make "gradle javadoc" task work with Java 9

I have a multi-module Gradle Java project using source/target = 1.9/1.9. There are two modules, my.base and my.dependsOnBase. The my.base module has no other dependencies:
module my.base {
exports my.base.foo;
exports my.base.bar;
}
The my.dependsOnBase module has only a single dependency, which is my.base:
module my.dependsOnBase {
requires my.base;
exports my.dependsOnBase.baz;
}
When I run $ gradle javadoc it works fine on my.base. But when it gets to my.dependsOnBase I get the following error output:
/path/to/my $ gradle javadoc
> Task :dependsOnBase:javadoc FAILED
/path/to/my/dependsOnBase/src/main/java/module-info.java:26: error: module not found: my.base
requires my.base;
^
1 error
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':dependsOnBase:javadoc'.
> Javadoc generation failed. Generated Javadoc options file (useful for troubleshooting): '/path/to/my/dependsOnBase/build/tmp/javadoc/javadoc.options'
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
* Get more help at https://help.gradle.org
BUILD FAILED in 2s
7 actionable tasks: 3 executed, 4 up-to-date
Earlier in the project I was able to get Java compilation, which suffered a similar problem, working using:
compileJava {
inputs.property("moduleName", moduleName)
doFirst {
options.compilerArgs = [
'--module-path', classpath.asPath,
]
classpath = files()
}
}
But those properties aren't directly applicable to the Gradle javadoc task.
How can I get my Javadoc working?

This worked for me
javadoc {
inputs.property("moduleName", moduleName)
doFirst {
options.addStringOption('-module-path', classpath.asPath)
}
}

I tried this on Gradle 7.1 and couldn't get it working, but after a little tinkering found adding this snippet made things work:
tasks.withType(Javadoc) {
doFirst {
options.modulePath = [] + classpath.files
options.classpath = []
}
}
The [] + ... seems to be a necessary workaround to make a list from the org.gradle.api.internal.file.UnionFileCollection returned by Gradle. Alternatively you could use new ArrayList(classpath.files) to achieve the same.

Gradle Task Not Running After Test

I have integration tests setup in my build.gradle file as such:
task integrationSetup(dependsOn: jar, type: Exec) {
workingDir "$projectDir/resources/integration"
commandLine 'sh', './start_service.sh'
}
task testIntegration(dependsOn: integrationSetup, type: Test) {
testClassesDirs = sourceSets.testIntegration.output.classesDirs
classpath = sourceSets.testIntegration.runtimeClasspath
ignoreFailures = true
}
task integrationTearDown(dependsOn: testIntegration, type: Exec) {
workingDir "$projectDir/resources/integration"
commandLine 'sh', './stop_service.sh'
}
testIntegration.mustRunAfter integrationSetup
testIntegration.finalizedBy integrationTearDown
integrationTearDown.mustRunAfter testIntegration
However since upgrading the Gradle Wrapper to version 4+ the tasks no longer execute correctly. The final tear down never runs and the service continues. What has changed between version 3 and 4 to change this behaviour. Pretty upsetting Gradle did this without warning or deprecation notices.
One dumb option is to downgrade the Gradle wrapper version (can confirm this setup still works on 3.1). But that shouldn't be necessary IMO.
UPDATE: Made some changes per user #Opal. However still have issue where if any errors occur during integration tests the final tear down does not run.
> Task :compileTestIntegrationJava
Putting task artifact state for task ':compileTestIntegrationJava' into context took 0.0 secs.
file or directory '/home/project/cleaner/src/testIntegration/java', not found
file or directory '/home/project/cleaner/src/testIntegration/java', not found
Executing task ':compileTestIntegrationJava' (up-to-date check took 0.072 secs) due to:
Output property 'destinationDir' file /home/project/cleaner/build/classes/java/testIntegration has changed.
Output property 'destinationDir' file /home/project/cleaner/build/classes/java/testIntegration/com has been removed.
Output property 'destinationDir' file /home/project/cleaner/build/classes/java/testIntegration/com/project has been removed.
All input files are considered out-of-date for incremental task ':compileTestIntegrationJava'.
file or directory '/home/project/cleaner/src/testIntegration/java', not found
Compiling with JDK Java compiler API.
/home/project/cleaner/src/integration/java/com/project/cleaner/CleansRequestsTests.java:415: error: reached end of file while parsing
}
^
1 error
:compileTestIntegrationJava (Thread[Daemon worker Thread 8,5,main]) completed. Took 0.162 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':compileTestIntegrationJava'.
> Compilation failed; see the compiler error output for details.
* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output.
BUILD FAILED in 8s
8 actionable tasks: 8 executed
Stopped 0 worker daemon(s).

In discussion it turned out that OP wants to stop the service started before running test no matter what e.g. compilation errors. It can be done with the following script:
ext.integrationTearDown = {
workingDir "$projectDir/resources/integration"
commandLine 'sh', './stop_service.sh'
}
task(type: Exec, 'stop_service', integrationTearDown)
gradle.buildFinished {
exec integrationTearDown
}
testIntegration.dependsOn integrationSetup
testIntegration.finalizedBy stop_service
With this piece of code the service will be stopped after every build - event if it succeeds. To avoid this behaviour BuildResult which is passed to buildFinished may be used to determine the required behaviour.

Parallel Test Execution with Gradle maxParallelForks property

I have been searching for the answers last few days and unable to find one. The closest answer I could find is this which does not exactly answer the questions I have.
By the way, I have a Selenium Test Project which is based on Gradle. We build the project on Jenkins run the tests in 20 concurrent threads. Total number of unique test classes I have is 87. So, I expect gradle to exeute at least 5 batches The test project is build using Cucumber JVM, build and triggers tests by Jenkins to Selenium Hub. I tried to increase the parallelism of the tests by utilizing the grid as much as possible. But, the problem started when the number of tests started growing.
When I started the tests from Jenkins, I observed at first shot the test executed all 20 test processes and I see the second batch also started with same amount of processes. After the second batch the processes went back to single mode and the entire job took 14 hours to complete which defeats the purpose of having parallel test execution.
Gradle properties:
jvmArgs '-Xms128m', '-Xmx1024m', '-XX:MaxPermSize=128m'
Runtime.runtime.availableProcessors().toString()) as int
maxParallelForks = PropertyUtils.getProperty('test.parallel', '15') as int
forkEvery = PropertyUtils.getProperty('test.forkEvery', '0') as int
CLI:
gradle clean test -Dtest.single=*TestRun --info
I have read all the documents I can possibly find but failed to get answer. It would be greatly appreciated if someone can help me with these questions
1. How Gradle batch the test runner internally? For example if 20 executors starts and test 1,2,3 done executing faster than the others, do the three executors gets three more test classes or waits for the entire batch to finish executing?
2. Can forkEvery impact how the execution works during parallel testing?
Jenkins log
Successfully started process 'Gradle Test Executor 6'
Successfully started process 'Gradle Test Executor 13'
Successfully started process 'Gradle Test Executor 14'
Successfully started process 'Gradle Test Executor 5'
Successfully started process 'Gradle Test Executor 16'
Successfully started process 'Gradle Test Executor 8'
Successfully started process 'Gradle Test Executor 19'
Successfully started process 'Gradle Test Executor 4'
Successfully started process 'Gradle Test Executor 2'
Successfully started process 'Gradle Test Executor 11'
Successfully started process 'Gradle Test Executor 10'
Successfully started process 'Gradle Test Executor 18'
Successfully started process 'Gradle Test Executor 1'
Successfully started process 'Gradle Test Executor 20'
Successfully started process 'Gradle Test Executor 7'
Successfully started process 'Gradle Test Executor 9'
Successfully started process 'Gradle Test Executor 3'
Successfully started process 'Gradle Test Executor 15'
Successfully started process 'Gradle Test Executor 17'
Successfully started process 'Gradle Test Executor 12'
Gradle Test Executor 13 started executing tests.
Gradle Test Executor 14 started executing tests.
Gradle Test Executor 6 started executing tests.
Gradle Test Executor 5 started executing tests.
Gradle Test Executor 16 started executing tests.
Gradle Test Executor 19 started executing tests.
Gradle Test Executor 8 started executing tests.
Gradle Test Executor 4 started executing tests.
Gradle Test Executor 2 started executing tests.
Gradle Test Executor 10 started executing tests.
Gradle Test Executor 11 started executing tests.
Gradle Test Executor 18 started executing tests.
Gradle Test Executor 1 started executing tests.
Gradle Test Executor 20 started executing tests.
Gradle Test Executor 7 started executing tests.
Gradle Test Executor 3 started executing tests.
Gradle Test Executor 9 started executing tests.
Gradle Test Executor 17 started executing tests.
Gradle Test Executor 15 started executing tests.
Gradle Test Executor 12 started executing tests.

The default of forkEvery is 0
According to the documentation forkEvery is
The maximum number of test classes to execute in a forked test process. The forked test process will be restarted when this limit is reached. The default value is 0 (no maximum).
So gradle (and probably junit) will fork by classes not tests within the class. It sounds like a few of the 87 test classes have long running tests or a large number of tests and they end up in one forked test process. I would consider setting forkEvery to 1. This will ensure that each test class is sent to a new fork. If there is still an issue you may need to find which test classes are taking the most time. Consider splitting these classes up into smaller groups of tests so the tests get spread over each jvm. If it is one test that takes forever consider redesigning it and possibly creating smaller tests from it.
I do not believe that gradle runs tests in batches. As a worker becomes available it takes a test class from the queue of remaining tests. You would really have to look at how JUnit works as I'm sure gradle is simply passing these configurations to JUnit.

How to analyze ant build output for further decision making?

I'm writing an ant build script to run regression tests for an application. I need to run the test cases sequentially and only if the previous test run was successful. Is there a way I can look a the output of the build to decide if the next target can be called?
[exec] [revBuild] RC = 1
[exec] -------------------------------------------------
[exec] Result: 1
BUILD SUCCESSFUL
Total time: 3 minutes 23 seconds
In the above output, the called application has failed. Is there a way I can search for the application return code in the build output, based on which the next ant target(to run the next test case) can be called?

You probably just want to set the failonerror attribute of the exec task to true. If you do this and the executable's return status code is anything other than 0, then the buildwill fail.
Youi could also store this status code in a property using the resultproperty attribute, and execute some task only if this property is set (or is not set).