Hey,
How many parallel builds do you invoke when this problem shows up?
At the moment, timeouts cannot be updated. We want solve the problem, but not really by updating the timeouts.
In my case six parallel builds, but I've coded my wrapper to use max(2, # of cores - 2)
I have a gradle script that scans the subfolders to find a bunch of loosely related projects that need to build in a certain order (determined by custom config files in the project folders) and launch a new gradle command for each one. With 6 worker threads churning through my list of projects.
Some of the projects have large zips as dependencies (C/C++ SDKs) that could take a while to download if they need to be updated. That's the "nativeSDK" configuration you see in the message above. It seems to always be these native SDKs that trip it up.
Thanks for all the information. Can you submit the output of gradle -v and tell us the exact windows version you use?
During the download we release the artifact lock so large downloads should not be an issue.
Windows 7 Professional 64-bit, running Java 7u25
gradle -v
------------------------------------------------------------
Gradle 1.7-rc-2
------------------------------------------------------------
Build time: 2013-07-29 12:04:21 UTC
Build number: none
Revision: 9a7199efaf72c620b33f9767874f0ebced135d83
Groovy: 1.8.6
Ant: Apache Ant(TM) version 1.8.4 compiled on May 22 2012
Ivy: 2.2.0
JVM: 1.7.0_25 (Oracle Corporation 23.25-b01)
OS: Windows 7 6.1 x86
I have the same problem using 1.7 with Gradle wrapper from my Jenkins server.
The setup is one master job that pulls new code from the Git repo and makes a build. Following that job, there are 15 parallel jobs running integration tests with different setup towards external systems/servers.
- Exception is:
org.gradle.cache.CacheOpenException: Could not open task artifact state cache (/scm/hudson/jobs/2216/workspace/repo/.gradle/1.7/taskArtifacts).
at org.gradle.cache.internal.DefaultPersistentDirectoryStore.open(DefaultPersistentDirectoryStore.java:55)
at org.gradle.cache.internal.DefaultPersistentDirectoryStore.open(DefaultPersistentDirectoryStore.java:27)
at org.gradle.cache.internal.DefaultCacheFactory$CacheFactoryImpl.doOpenDir(DefaultCacheFactory.java:72)
at org.gradle.cache.internal.DefaultCacheFactory$CacheFactoryImpl.open(DefaultCacheFactory.java:111)
at org.gradle.cache.internal.DefaultCacheRepository$PersistentCacheBuilder.doOpen(DefaultCacheRepository.java:184)
at org.gradle.cache.internal.DefaultCacheRepository$PersistentCacheBuilder.doOpen(DefaultCacheRepository.java:134)
at org.gradle.cache.internal.DefaultCacheRepository$AbstractCacheBuilder.open(DefaultCacheRepository.java:121)
at org.gradle.api.internal.changedetection.state.DefaultTaskArtifactStateCacheAccess.getCache(DefaultTaskArtifactStateCacheAccess.java:44)
at org.gradle.api.internal.changedetection.state.DefaultTaskArtifactStateCacheAccess.longRunningOperation(DefaultTaskArtifactStateCacheAccess.java:78)
at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor.process(DefaultTaskPlanExecutor.java:31)
at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter.execute(DefaultTaskGraphExecuter.java:89)
at org.gradle.execution.SelectedTaskExecutionAction.execute(SelectedTaskExecutionAction.java:29)
at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:61)
at org.gradle.execution.DefaultBuildExecuter.access$200(DefaultBuildExecuter.java:23)
at org.gradle.execution.DefaultBuildExecuter$2.proceed(DefaultBuildExecuter.java:67)
at org.gradle.execution.DryRunBuildExecutionAction.execute(DryRunBuildExecutionAction.java:32)
at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:61)
at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:54)
at org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:166)
at org.gradle.initialization.DefaultGradleLauncher.doBuild(DefaultGradleLauncher.java:113)
at org.gradle.initialization.DefaultGradleLauncher.run(DefaultGradleLauncher.java:81)
at org.gradle.launcher.exec.InProcessBuildActionExecuter$DefaultBuildController.run(InProcessBuildActionExecuter.java:64)
at org.gradle.launcher.cli.ExecuteBuildAction.run(ExecuteBuildAction.java:33)
at org.gradle.launcher.cli.ExecuteBuildAction.run(ExecuteBuildAction.java:24)
at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:35)
at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:26)
at org.gradle.launcher.cli.RunBuildAction.run(RunBuildAction.java:50)
at org.gradle.api.internal.Actions$RunnableActionAdapter.execute(Actions.java:171)
at org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:201)
at org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:174)
at org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:170)
at org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:139)
at org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:33)
at org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:22)
at org.gradle.launcher.Main.doAction(Main.java:48)
at org.gradle.launcher.bootstrap.EntryPoint.run(EntryPoint.java:45)
at org.gradle.launcher.Main.main(Main.java:39)
at org.gradle.launcher.bootstrap.ProcessBootstrap.runNoExit(ProcessBootstrap.java:50)
at org.gradle.launcher.bootstrap.ProcessBootstrap.run(ProcessBootstrap.java:32)
at org.gradle.launcher.GradleMain.main(GradleMain.java:26)
at org.gradle.wrapper.BootstrapMainStarter.start(BootstrapMainStarter.java:33)
at org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:130)
at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:48)
Caused by: org.gradle.cache.internal.LockTimeoutException: Timeout waiting to lock task artifact state cache (/scm/hudson/jobs/2216/workspace/repo/.gradle/1.7/taskArtifacts). It is currently in use by another Gradle instance.
Owner PID: 10367
Our PID: 24580
Owner Operation:
Our operation:
Lock file: /scm/hudson/jobs/2216/workspace/repo/.gradle/1.7/taskArtifacts/cache.properties.lock
at org.gradle.cache.internal.DefaultFileLockManager$DefaultFileLock.lock(DefaultFileLockManager.java:303)
at org.gradle.cache.internal.DefaultFileLockManager$DefaultFileLock.(DefaultFileLockManager.java:143)
at org.gradle.cache.internal.DefaultFileLockManager.lock(DefaultFileLockManager.java:91)
at org.gradle.cache.internal.DefaultFileLockManager.lock(DefaultFileLockManager.java:78)
at org.gradle.cache.internal.DefaultCacheAccess.open(DefaultCacheAccess.java:88)
at org.gradle.cache.internal.DefaultPersistentDirectoryStore.open(DefaultPersistentDirectoryStore.java:45)
... 42 more
--project-cache-dir seems to solve it but I think that this should work without that fix.
@Joachim: we encounterd the same problem in our Jenkins setup. Just found out that recent gradle plugin for jenkins adds a new option to your jenkins job: "Force GRADLE_USER_HOME to use workspace".
That seems to make a difference.
Not fixing the actual problem, yet making sure it's not blocking you in Jenkins...
@roomsg: Thanks. That is a good alternative, but we use the same workspace so we can reuse artifacts and collect reports easier. We instead added the switch --project-cache-dir to set individual cache directory for each job. That gives us the same result.
We have the same issue using Jenkins: one main project that builds for 30 minutes and doesn't release the artifacts lock; during that time, no other (smaller) project can build, they all time out on the lock.
We tried the --project-cache-dir solution, but that didn't work for us; it looks like that solution forces the cached scripts to be in a different folder, but the artifacts one is still shared and locked during the entire build.
The lock file is located in USER_HOME/.gradle/caches/artifacts-26/artifacts-26.lock
We have the same behavior on Linux using Jenkins and on Windows building a small project manually: that file is locked for the entire build.
I don't see many people complaining about this behavior, so I am starting to think we are doing something wrong 
Here is my Windows environment:
------------------------------------------------------------
Gradle 1.8
------------------------------------------------------------
Build time: 2013-09-24 07:32:33 UTC
Build number: none
Revision: 7970ec3503b4f5767ee1c1c69f8b4186c4763e3d
Groovy: 1.8.6
Ant: Apache Ant(TM) version 1.9.2 compiled on July 8 2013
Ivy: 2.2.0
JVM: 1.6.0_45 (Sun Microsystems Inc. 20.45-b01)
OS: Windows 7 6.1 amd64
We're seeing this with gradle 1.10 in a heavily loaded system with 5-10 builds running concurrently.
Log: [1]https://gist.github.com/bedge/9403128
------------------------------------------------------------
Gradle 1.10
------------------------------------------------------------
Build time: 2013-12-17 09:28:15 UTC
Build number: none
Revision: 36ced393628875ff15575fa03d16c1349ffe8bb6
Groovy: 1.8.6
Ant: Apache Ant(TM) version 1.9.2 compiled on July 8 2013
Ivy: 2.2.0
JVM: 1.7.0_51 (Oracle Corporation 24.45-b08)
OS: Linux 3.8.0-19-generic amd64
----------------------------------------------------------------------------------------
[1] https://gist.github.com/bedge/9403128
We're encountering this fairly regularly lately. We probably have more concurrent builds than most because we're using gradle to process digital media content rather than as a build system for code. we typically run between 8 to 16 jobs at a time and this happens every few days.
Is there anything we can do to gather more relevant information ?
Hi,
This is most likely a symptom of the current algorithm not really attempting any fairness when managing the mutex, therefore requesters stack up.
The only real workaround is to reduce the number of contenders by sharing the load across a few different cache instances. This would entail using a custom GRADLE_USER_HOME and managing the allocation to jobs somehow. The downside will be increased disk usage and network traffic potentially.
If anyone's interested in improving the lock sharing mechanism, that would be great
Thanks Luke, I'll give that a shot and report back. Any thoughts on the "right" number of jobs per HOME?
Here's the stats on the number of parallel jobs vs failures:
Threads: failures
16: 25
24: 85
64: 158
Before we ramped up to scale we were running 4 jobs in parallel and we saw this once every few days.
This implies that we really need a separate HOME for every couple of jobs to eliminate the potential for failures.
I'll take a look at the gradle lock sharing. Any thoughts you have on this would be helpful.
What do you mean exactly by "fairness"?
Can we increase the lock timeouts? I don't really care how long a thread has to wait, just that it gets the lock eventually.
It's difficult to predict what the right number is because it depends on what your build is doing. It's going to take trial and error unfortunately.
“fairness” with regards to locking relates to giving the lock to the contender that has been waiting the longest.
We've been reluctant to allow specification of the timeout because it just shifts the problem.
Another shot in the dark on this.
Could the "was closed 2 times on the line below imply a resource counting bug as that's the same lock (taskArtifacts) that it eventually fails on.
Full log posted here: [1]https://gist.github.com/bedge/9658161
17:31:55.284 [DEBUG] [org.gradle.cache.internal.DefaultCacheAccess] Cache task history cache (/opt/nim/parserV4/.gradle/1.11/taskArtifacts) was closed 2 times.
17:31:55.286 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on buildscript class cache for build file '/opt/nim/parserV4/build.gradle' (/opt/jetty/.gradle/caches/1.11/scripts/build_34lps1mvh2h1ts080k6rh29h7e/ProjectScript/buildscript).
17:31:55.287 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on no_buildscript class cache for build file '/opt/nim/parserV4/build.gradle' (/opt/jetty/.gradle/caches/1.11/scripts/build_34lps1mvh2h1ts080k6rh29h7e/ProjectScript/no_buildscript).
17:31:55.287 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on buildscript class cache for script '/opt/nim/parserV4/lib/common.gradle' (/opt/jetty/.gradle/caches/1.11/scripts/common_4dhd8ikasabtg7ji7hire5sgrn/DefaultScript/buildscript).
17:31:55.287 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on no_buildscript class cache for script '/opt/nim/parserV4/lib/common.gradle' (/opt/jetty/.gradle/caches/1.11/scripts/common_4dhd8ikasabtg7ji7hire5sgrn/DefaultScript/no_buildscript).
17:31:55.287 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on buildscript class cache for script '/opt/nim/parserV4/lib/nimDependencies.gradle' (/opt/jetty/.gradle/caches/1.11/scripts/nimDependencies_adrmigpbgjk9vd2cfok9cq0pi/DefaultScript/buildscript).
17:31:55.287 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on no_buildscript class cache for script '/opt/nim/parserV4/lib/nimDependencies.gradle' (/opt/jetty/.gradle/caches/1.11/scripts/nimDependencies_adrmigpbgjk9vd2cfok9cq0pi/DefaultScript/no_buildscript).
17:31:55.288 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on buildscript class cache for script '/opt/nim/parserV4/lib/readCfg.gradle' (/opt/jetty/.gradle/caches/1.11/scripts/readCfg_5l936ovrqa20iv0bdk22t15mb3/DefaultScript/buildscript).
17:31:55.288 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on no_buildscript class cache for script '/opt/nim/parserV4/lib/readCfg.gradle' (/opt/jetty/.gradle/caches/1.11/scripts/readCfg_5l936ovrqa20iv0bdk22t15mb3/DefaultScript/no_buildscript).
17:31:55.288 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on buildscript class cache for script '/opt/nim/parserV4/lib/testCfg.gradle' (/opt/jetty/.gradle/caches/1.11/scripts/testCfg_6t1e8jito6r13b19dp6o23un1l/DefaultScript/buildscript).
17:31:55.288 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on no_buildscript class cache for script '/opt/nim/parserV4/lib/testCfg.gradle' (/opt/jetty/.gradle/caches/1.11/scripts/testCfg_6t1e8jito6r13b19dp6o23un1l/DefaultScript/no_buildscript).
17:31:55.288 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on buildscript class cache for script '/opt/nim/parserV4/buildSrc/src/main/groovy/com/nim/tools/common/Apt.groovy' (/opt/jetty/.gradle/caches/1.11/scripts/Apt_67gj9bu0f8psu64oenp420pgmn/DefaultScript/buildscript).
17:31:55.288 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on no_buildscript class cache for script '/opt/nim/parserV4/buildSrc/src/main/groovy/com/nim/tools/common/Apt.groovy' (/opt/jetty/.gradle/caches/1.11/scripts/Apt_67gj9bu0f8psu64oenp420pgmn/DefaultScript/no_buildscript).
17:31:55.289 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on buildscript class cache for script '/opt/nim/parserV4/lib/dataset.gradle' (/opt/jetty/.gradle/caches/1.11/scripts/dataset_2ncahp2cnmljllkuiomqjme84i/DefaultScript/buildscript).
17:31:55.289 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on no_buildscript class cache for script '/opt/nim/parserV4/lib/dataset.gradle' (/opt/jetty/.gradle/caches/1.11/scripts/dataset_2ncahp2cnmljllkuiomqjme84i/DefaultScript/no_buildscript).
17:31:55.289 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on buildscript class cache for script '/opt/nim/parserV4/workflows/replica.gradle' (/opt/jetty/.gradle/caches/1.11/scripts/replica_efc9ig0rgoflja64gd4aadv96/DefaultScript/buildscript).
17:31:55.289 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on no_buildscript class cache for script '/opt/nim/parserV4/workflows/replica.gradle' (/opt/jetty/.gradle/caches/1.11/scripts/replica_efc9ig0rgoflja64gd4aadv96/DefaultScript/no_buildscript).
17:31:55.289 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on buildscript class cache for script '/opt/nim/parserV4/lib/generate_bmi.gradle' (/opt/jetty/.gradle/caches/1.11/scripts/generate_bmi_78mme0hehvuls8lv7m4cm0svb3/DefaultScript/buildscript).
17:31:55.290 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on no_buildscript class cache for script '/opt/nim/parserV4/lib/generate_bmi.gradle' (/opt/jetty/.gradle/caches/1.11/scripts/generate_bmi_78mme0hehvuls8lv7m4cm0svb3/DefaultScript/no_buildscript).
17:31:55.290 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on buildscript class cache for script '/opt/nim/parserV4/lib/toc.gradle' (/opt/jetty/.gradle/caches/1.11/scripts/toc_6rn27nsfhsq292du5oashe646h/DefaultScript/buildscript).
17:31:55.290 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on no_buildscript class cache for script '/opt/nim/parserV4/lib/toc.gradle' (/opt/jetty/.gradle/caches/1.11/scripts/toc_6rn27nsfhsq292du5oashe646h/DefaultScript/no_buildscript).
17:31:55.290 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on buildscript class cache for script '/opt/nim/parserV4/lib/assembleContent.gradle' (/opt/jetty/.gradle/caches/1.11/scripts/assembleContent_1hpm8rrs5o5m7soesl3csgpqmq/DefaultScript/buildscript).
17:31:55.290 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on no_buildscript class cache for script '/opt/nim/parserV4/lib/assembleContent.gradle' (/opt/jetty/.gradle/caches/1.11/scripts/assembleContent_1hpm8rrs5o5m7soesl3csgpqmq/DefaultScript/no_buildscript).
17:31:55.290 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on buildscript class cache for script '/opt/nim/parserV4/lib/buildReport.gradle' (/opt/jetty/.gradle/caches/1.11/scripts/buildReport_41buduc82cm4tmbovfdndknir6/DefaultScript/buildscript).
17:31:55.291 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on no_buildscript class cache for script '/opt/nim/parserV4/lib/buildReport.gradle' (/opt/jetty/.gradle/caches/1.11/scripts/buildReport_41buduc82cm4tmbovfdndknir6/DefaultScript/no_buildscript).
17:31:55.292 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.store.CachedStoreFactory] Resolution result cache closed. Cache reads: 0, disk reads: 0 (avg: 0.0 secs, total: 0.0 secs)
17:31:55.292 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.store.CachedStoreFactory] Resolved configuration cache closed. Cache reads: 0, disk reads: 0 (avg: 0.0 secs, total: 0.0 secs)
17:31:55.292 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.resolveengine.store.ResolutionResultsStoreFactory] Deleted 2 resolution results binary files in 0.0 secs
17:31:55.293 [DEBUG] [org.gradle.api.internal.artifacts.ivyservice.ivyresolve.memcache.InMemoryDependencyMetadataCache] In-memory dependency metadata cache closed. Repos cached: 1, cache instances: 1, modules served from cache: 0, artifacts: 0
17:31:55.318 [ERROR] [org.gradle.api.Project] ERROR:class org.gradle.cache.internal.LockTimeoutException : Timeout waiting to lock task history cache (/opt/nim/parserV4/.gradle/1.11/taskArtifacts). It is currently in use by another Gradle instance.
Owner PID: 1850
Our PID: 1849
Owner Operation: Create file snapshot
Our operation: create dir snapshots
Lock file: /opt/nim/parserV4/.gradle/1.11/taskArtifacts/cache.properties.lock
17:31:55.319 [ERROR] [system.err] org.gradle.cache.internal.LockTimeoutException: Timeout waiting to lock task history cache (/opt/nim/parserV4/.gradle/1.11/taskArtifacts). It is currently in use by another Gradle instance.
17:31:55.320 [ERROR] [system.err] Owner PID: 1850
17:31:55.320 [ERROR] [system.err] Our PID: 1849
17:31:55.320 [ERROR] [system.err] Owner Operation: Create file snapshot
17:31:55.320 [ERROR] [system.err] Our operation: create dir snapshots
17:31:55.320 [ERROR] [system.err] Lock file: /opt/nim/parserV4/.gradle/1.11/taskArtifacts/cache.properties.lock
17:31:55.320 [ERROR] [system.err] at org.gradle.cache.internal.DefaultFileLockManager$DefaultFileLock.lock(DefaultFileLockManager.java:258)
17:31:55.320 [ERROR] [system.err] at org.gradle.cache.internal.DefaultFileLockManager$DefaultFileLock.<init>(DefaultFileLockManager.java:127)
17:31:55.321 [ERROR] [system.err] at org.gradle.cache.internal.DefaultFileLockManager.lock(DefaultFileLockManager.java:80)
17:31:55.321 [ERROR] [system.err] at org.gradle.cache.internal.DefaultCacheAccess.onStartWork(DefaultCacheAccess.java:343)
17:31:55.321 [ERROR] [system.err] at org.gradle.cache.internal.DefaultCacheAccess.useCache(DefaultCacheAccess.java:186)
17:31:55.321 [ERROR] [system.err] at org.gradle.cache.internal.DefaultCacheAccess.useCache(DefaultCacheAccess.java:175)
17:31:55.321 [ERROR] [system.err] at org.gradle.cache.internal.DefaultPersistentDirectoryStore.useCache(DefaultPersistentDirectoryStore.java:106)
17:31:55.321 [ERROR] [system.err] at org.gradle.cache.internal.DefaultCacheFactory$ReferenceTrackingCache.useCache(DefaultCacheFactory.java:193)
17:31:55.322 [ERROR] [system.err] at org.gradle.api.internal.changedetection.state.DefaultTaskArtifactStateCacheAccess.useCache(DefaultTaskArtifactStateCacheAccess.java:58)
17:31:55.322 [ERROR] [system.err] at org.gradle.api.internal.changedetection.state.OutputFilesSnapshotter.snapshot(OutputFilesSnapshotter.java:62)
17:31:55.322 [ERROR] [system.err] at org.gradle.api.internal.changedetection.state.OutputFilesSnapshotter.snapshot(OutputFilesSnapshotter.java:41)
17:31:55.322 [ERROR] [system.err] at org.gradle.api.internal.changedetection.rules.OutputFilesStateChangeRule$1.snapshotAfterTask(OutputFilesStateChangeRule.java:79)
17:31:55.322 [ERROR] [system.err] at org.gradle.api.internal.changedetection.rules.CachingTaskStateChanges.snapshotAfterTask(CachingTaskStateChanges.java:84)
17:31:55.322 [ERROR] [system.err] at org.gradle.api.internal.changedetection.rules.SummaryTaskStateChanges.snapshotAfterTask(SummaryTaskStateChanges.java:73)
17:31:55.323 [ERROR] [system.err] at org.gradle.api.internal.changedetection.changes.DefaultTaskArtifactStateRepository$TaskArtifactStateImpl.afterTask(DefaultTaskArtifactStateRepository.java:117)
17:31:55.323 [ERROR] [system.err] at org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:66)
17:31:55.323 [ERROR] [system.err] at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:58)
17:31:55.323 [ERROR] [system.err] at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:42)
17:31:55.323 [ERROR] [system.err] at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:52)
17:31:55.323 [ERROR] [system.err] at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:53)
17:31:55.324 [ERROR] [system.err] at org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
17:31:55.324 [ERROR] [system.err] at org.gradle.api.internal.AbstractTask.executeWithoutThrowingTaskFailure(AbstractTask.java:289)
17:31:55.324 [ERROR] [system.err] at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.executeTask(AbstractTaskPlanExecutor.java:79)
17:31:55.324 [ERROR] [system.err] at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.processTask(AbstractTaskPlanExecutor.java:63)
17:31:55.324 [ERROR] [system.err] at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.run(AbstractTaskPlanExecutor.java:51)
17:31:55.324 [ERROR] [system.err] at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor.process(DefaultTaskPlanExecutor.java:23)
17:31:55.325 [ERROR] [system.err] at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter.execute(DefaultTaskGraphExecuter.java:86)
17:31:55.325 [ERROR] [system.err] at org.gradle.execution.SelectedTaskExecutionAction.execute(SelectedTaskExecutionAction.java:29)
17:31:55.325 [ERROR] [system.err] at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:61)
17:31:55.325 [ERROR] [system.err] at org.gradle.execution.DefaultBuildExecuter.access$200(DefaultBuildExecuter.java:23)
17:31:55.325 [ERROR] [system.err] at org.gradle.execution.DefaultBuildExecuter$2.proceed(DefaultBuildExecuter.java:67)
17:31:55.325 [ERROR] [system.err] at org.gradle.execution.DryRunBuildExecutionAction.execute(DryRunBuildExecutionAction.java:32)
17:31:55.325 [ERROR] [system.err] at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:61)
17:31:55.326 [ERROR] [system.err] at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:54)
17:31:55.326 [ERROR] [system.err] at org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:166)
17:31:55.326 [ERROR] [system.err] at org.gradle.initialization.DefaultGradleLauncher.doBuild(DefaultGradleLauncher.java:113)
17:31:55.326 [ERROR] [system.err] at org.gradle.initialization.DefaultGradleLauncher.run(DefaultGradleLauncher.java:81)
17:31:55.326 [ERROR] [system.err] at org.gradle.launcher.exec.InProcessBuildActionExecuter$DefaultBuildController.run(InProcessBuildActionExecuter.java:64)
17:31:55.326 [ERROR] [system.err] at org.gradle.launcher.cli.ExecuteBuildAction.run(ExecuteBuildAction.java:33)
17:31:55.327 [ERROR] [system.err] at org.gradle.launcher.cli.ExecuteBuildAction.run(ExecuteBuildAction.java:24)
17:31:55.327 [ERROR] [system.err] at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:35)
17:31:55.327 [ERROR] [system.err] at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:26)
17:31:55.327 [ERROR] [system.err] at org.gradle.launcher.cli.RunBuildAction.run(RunBuildAction.java:50)
17:31:55.327 [ERROR] [system.err] at org.gradle.internal.Actions$RunnableActionAdapter.execute(Actions.java:171)
17:31:55.327 [ERROR] [system.err] at org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:201)
17:31:55.327 [ERROR] [system.err] at org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:174)
17:31:55.328 [ERROR] [system.err] at org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:170)
17:31:55.328 [ERROR] [system.err] at org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:139)
17:31:55.328 [ERROR] [system.err] at org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:33)
17:31:55.328 [ERROR] [system.err] at org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:22)
17:31:55.328 [ERROR] [system.err] at org.gradle.launcher.Main.doAction(Main.java:46)
17:31:55.328 [ERROR] [system.err] at org.gradle.launcher.bootstrap.EntryPoint.run(EntryPoint.java:45)
17:31:55.329 [ERROR] [system.err] at org.gradle.launcher.Main.main(Main.java:37)
17:31:55.329 [ERROR] [system.err] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
17:31:55.329 [ERROR] [system.err] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
17:31:55.329 [ERROR] [system.err] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
17:31:55.329 [ERROR] [system.err] at java.lang.reflect.Method.invoke(Method.java:606)
17:31:55.329 [ERROR] [system.err] at org.gradle.launcher.bootstrap.ProcessBootstrap.runNoExit(ProcessBootstrap.java:50)
17:31:55.329 [ERROR] [system.err] at org.gradle.launcher.bootstrap.ProcessBootstrap.run(ProcessBootstrap.java:32)
17:31:55.330 [ERROR] [system.err] at org.gradle.launcher.GradleMain.main(GradleMain.java:23)
----------------------------------------------------------------------------------------
[1] https://gist.github.com/bedge/9658161
After playing around with options here, we've decided to just go with a separate GRADLE_HOME for every job. It's not ideal as it's requires a lot more storage and will spend more time thrashing dependencies, but it seems like the easiest solution. We just don't have the BW to dive into gradle's internals.
Is there anything we can do to mitigate the overhead in transitioning to separate GRADLE_HOME dirs for every job? Specifically in terms of pre-seeding new home dirs with data from a master copy or something of that nature.
Open to other suggestions on how to minimize the performance hit of this change.
Using separate GRADLE_USER_HOME for every build defeats the purpose of having a cache. This is a deal breaker for moving forward with Gradle as an enterprise solution. Gradleware, any idea when this will be addressed?
I am running in the same issue, my situation is a bit different I suppose.
I am running a few gradle docker containers, using https://github.com/metahelicase/docker-gradle (about 10~)
These are spawn up by docker-compose, which runs the containers in parallel.
Some of the projects I run are multi-module.
Now, the gradle image will mount a volume, and that will be shared within all the containers.
Therefore I run in the same issue, multiple gradle processes accessing the same lock file...
Is there anyway to disable it? Or to configure a random lock name at run time?
Sounds like a pretty bad design decision to me to have a global lock like this.
I keep getting locking errors in our Jenkins based build farm, even with latest Gradle 2.13:
Could not resolve all dependencies for configuration 'classpath'.
> Timeout waiting to lock artifact cache (<somewhere>/.gradle/caches/modules-2). It is currently in use by another Gradle instance.
Owner PID: 20286
Our PID: 20434
Owner Operation: resolve configuration ':ejbDeps'
Our operation: resolve configuration 'classpath'
Lock file: <somewhere>/.gradle/caches/modules-2/modules-2.lock
Some core projects trigger a couple of Gradle based downstream jobs. None of them create artifacts in the shared repository, i.e. the cache is used read-only, artifacts are produced in job local build/ directories.
The number of parallel jobs is around 6. All jobs share the same .gradle directory, which is a valid setup according to https://gradle.org/maven_vs_gradle/ and https://docs.gradle.org/current/userguide/dependency_management.html#sub:cache_locking
I am aware of the job local GRADLE_USER_HOME workaround, but then we're back at Maven level.
"There are only two hard things in computer science: cache invalidation, naming things, and off-by-one errors."
– Phil Karlton
Seeing this as well in 2.14:
00:01:22.617 00:01:22.617 FAILURE: Build failed with an exception. 00:01:22.617 00:01:22.617 * What went wrong: 00:01:22.617 A problem occurred configuring root project 'cats'. 00:01:22.618 > Could not resolve all dependencies for configuration ':classpath'. 00:01:22.618 > Timeout waiting to lock artifact cache (/root/.gradle/caches/modules-2). It is currently in use by another Gradle instance. 00:01:22.618 Owner PID: 125 00:01:22.618 Our PID: 137 00:01:22.618 Owner Operation: resolve configuration ':classpath' 00:01:22.618 Our operation: resolve configuration ':classpath' 00:01:22.618 Lock file: /root/.gradle/caches/modules-2/modules-2.lock 00:01:22.619
We started having the problem when we moved to docker based CI slaves. From observing debug logs, Gradle doesn't seem to release its artifact lock aggressively. Instead, it waits for another gradle process to ask it to release it before doing so. It seems to do this by using the pid in the lock file to find the process that is holding the lock, then sends a request to release it over a UDP port bound by that pid. In our case, the gradle processes are not able to communicate with each because they are in separate docker containers with isolated networks (sharing a common cache filesystem). Since they can't communicate over network, the artifact lock is never released and our builds timeout. I'm not sure how to work around this yet - maybe putting the containers on host networking would solve it.
Fundamentally, it seems wrong to me for gradle to rely on network access between gradle processes for dealing with file system locks.
>Fundamentally, it seems wrong to me for gradle to rely on network access between gradle processes for dealing with file system locks.
Yeah…
I'm currently trying to implement next solution:
1. Zip ~/.gradle
2. Start Docker container and pass zip file to it
3. After successful build zip ~/.gradle INSIDE container
4. Get zip from container
5. Unpack zip from container to host ~/.gradle
But it's not so trivial because it requires some locking to avoid similar problems that Gradle gives us…
99.9% of builds are read-only. Very occasionally a build needs to update something in the cache. If only read-only builds are happening, then the locks should not be exclusive. When a build does require changes to the cache, it should acquire write locks only for the things it's actually modifying so that irrelevant builds don't get held up at all. Even other dependent builds should not be blocked for the entire build.
What we're seeing here is basically the exact opposite. If someone commits 4 things in rapid succession, or 4 people happen to commit at around the same time, Jenkins spins up 4 jobs. The jobs share the same Gradle cache. Even though no dependencies have been changed, the first build locks the cache. Subsequent builds now wait, and ultimately time out. This causes spurious build failures. Basically the kind of thing we shouldn't have to suffer when using a tool which alleges to make development life better.
I also get the feeling that it shouldn't time out at all. Suppose it acquires the lock and then starts downloading an artifact. How could Gradle possibly know how fast our Internet is, in order to come up with a meaningful timeout? I would be interested in seeing the reasoning that was used to come to the idea of having a timeout, and particularly the means that were used to determine the current value which is being used for this timeout. Because if it's happening in production, we can be fairly sure it's wrong, and exposing the flawed reasoning which went into coming up with that value might prevent other stupid decisions being made in future versions.
As announced on the Gradle blog we are planning to completely migrate issues from JIRA to GitHub.
We intend to prioritize issues that are actionable and impactful while working more closely with the community. Many of our JIRA issues are inactionable or irrelevant. We would like to request your help to ensure we can appropriately prioritize JIRA issues you’ve contributed to.
Please confirm that you still advocate for your JIRA issue before December 10th, 2016 by:
- Checking that your issues contain requisite context, impact, behaviors, and examples as described in our published guidelines.
- Leave a comment on the JIRA issue or open a new GitHub issue confirming that the above is complete.
We look forward to collaborating with you more closely on GitHub. Thank you for your contribution to Gradle!
Thanks again for reporting this issue. We haven't heard back from you after our inquiry from November 15th. We are closing this issue now. Please create an issue on GitHub if you still feel passionate about getting it resolved.