[GRADLE-1968] Dependency resolution became awfully slow with milestone 6 Created: 28/Nov/11  Updated: 04/Jan/13  Resolved: 19/Dec/11

Status: Resolved
Project: Gradle
Affects Version/s: 1.0-milestone-6
Fix Version/s: 1.0-milestone-7

Type: Bug
Reporter: Bernhard Messerer Assignee: Adam Murdoch
Resolution: Fixed Votes: 0


 Description   

When switching from milestone 5 to milestone 6, the build time jumps from something like 3 minutes to 30 minutes (doing dependency resolution). We have projects (~60) with a few library dependencies (~60-70 "root" references, i.e. excluding their transitives) and project dependencies (3-10 per project) among them in a mixed Java/Scala environment but nothing really special regarding dependencies (all library versions are fixed, i.e. non wildcard).
I know this is not a lot of info but don't have time to create an SSCCE at the moment (may not be possible, otherwise this would probably have occurred to quite a few people), so is there a way I may help to diagnose this (without giving you the whole project - commercial software, sorry)?



 Comments   
Comment by Adam Murdoch [ 28/Nov/11 ]

Could you run with --profile against both milestone-5 and milestone-6, and verify that the increased time is due to dependency resolution? Also, the report might have some clues, for example a particular configuration or project has gotten much slower than it used to be.

Also, what do your repository definitions look like? eg Are you using Maven or Ivy repositories? Which protocols? etc.

Comment by Bernhard Messerer [ 04/Dec/11 ]

Tested this with --profile against m5 and m6. m6 was in fact ten times slower but after some analysis, I separated the idea task from the assemble task and ran both with --profile again.
The assemble task got faster with m6, mostly the when resolving dependencies.
So the culprit is definitely the idea plugin - execution got a bit slower with m6 but dependency resolution ("all dependencies") jumped from 1m40.47s to 24m56.99s, across all projects and configurations (of course, runtime needs longer but this was also the case with m5).

We use maven repositories exclusively (plus a local flat directory), currently maven central, Apache, java.net, JBoss public (http://repository.jboss.org/nexus/content/groups/public-jboss/), guice-maven (http://guice-maven.googlecode.com/svn/trunk) and Scala tools release (http://scala-tools.org/repo-releases), all except apache via http. However, this doesn't seem to be a problem since dependency resolution is fast in 'assemble'.

Comment by Bernhard Messerer [ 04/Dec/11 ]

Up-front explanation: Our current model of compile time dependencies (in fact IDEA's) is that library dependencies are transitive (i.e. if a project P depends on library A which in turn depends on library B, B is also added as dependency to P) while project dependencies are not (i.e. if P depends on X which depends on Y, P does not auto-depend on Y). Since we cannot set compile to transitive=false (and still get transitive library dependencies, see GRADLE-900) we add project dependencies with transitive=false. This has the problem that they're also non-transitive in runtime scope, which should not be the case (for us), so I wrote a "hack" to add all compile/testCompile project dependencies transitively to the runtime/testRuntime scope.
When deleting this piece of code, the idea task is still slower than with m5 but not that much (>2x: from 2m34.82s to 5m55.031s), re-enabling it lets the build time of the idea task jump to 26m29.46s.
Am I doing something completely wrong? If so, why did it work (and perform) with m5? Or is it just the time the idea task needs to resolve these transitives?
Maybe re-classify Component to idea but OTOH it's the dependency resolution that needs so long.

the "add transitively to (test)runtime scope" code looks like (called in aJavaProject.afterEvaluate with Configuration c, Project p, String targetCfgName):

pDeps=c.dependencies.withType(org.gradle.api.artifacts.ProjectDependency.class)
pDeps.each { d->
  p.dependencies.add(targetCfgName, project(d.dependencyProject.path), { transitive=true })
}
Comment by Bernhard Messerer [ 07/Jan/12 ]

Confirming this works, IDEA project files build faster than ever! Thanks for the good work.

Generated at Wed Jun 30 12:09:13 CDT 2021 using Jira 8.4.2#804003-sha1:d21414fc212e3af190e92c2d2ac41299b89402cf.