[GRADLE-114] Improve the speed of the build script evaluation Created: 11/Jun/08  Updated: 04/Jan/13  Resolved: 19/Aug/08

Status: Resolved
Project: Gradle
Affects Version/s: 0.1.4
Fix Version/s: 0.4

Type: Improvement
Reporter: Hans Dockter Assignee: Hans Dockter
Resolution: Fixed Votes: 0

Issue Links:
Supercedes
Supercedes GRADLE-117 Cache the compiled build scripts. Resolved
Supercedes GRADLE-118 Cache the DAG Resolved

 Comments   
Comment by Ittay Dror [ 12/Jun/08 ]

Maybe not improve the speed of evaluation, but save the result so it can be quickly reused.

Build logic doesn't change that much. How about this: if gradle sees a file (or set of files) withe extension .gradlec, it loads them instead of the .gradle files. Otherwise, it loads the regular files, does the evaluation and works on that. Say there's a 'gradlec' target to create this file (or files). Then, if a user wants to, he can run this target, put the file in his version control and all users use it. When the build logic changes, run 'gradlec' again etc. Of course, there can be a timestamp (or md5) check and warnings if the .gradle files have changed after .gradlec and a way to tell gradle to ignore gradlec.

In fact, maybe the .gradlec file(s) can be compiled already to .class file(s)?

I have reasons for wishing drastic measures to improve speed:
1. I work in a project which puts great emphasis on speed. If Gradle is fast enough, I may push it there. This is a huge project.
2. I think it makes the build tool much more usable. Today many people prefer to use the IDE's build because it doesn't get in the way.
3. I think the evaluation phase may grow as more plugins and modules are used.

Comment by Hans Dockter [ 12/Jun/08 ]

I'm completely dedicated to improving the performance as much as possible.

The important thing, at least as I understand it right now, is that the groovy compilation of the gradlefile is only one part of the story. But running this compiled gradlefile also takes surprisingly long when all what it does is executing a createTask method (and the closure is not even evaluated then). For example:

createTask('helloWorld') {
   println 'hello'
}

This takes Gradle on my machine something like 0.4 seconds (just the running of the compiled gradlefile, not the execution of the build). I think this is not a Groovy problem but a Gradle problem.

I gonna have a look with a profiler about what is going on here.

But I'm also completely positive about caching the compiled gradlefiles. Although in Groovy 1.6 the compile speed has improved a lot. I try to get more numbers and then we can discuss what measures are worth doing.

Comment by Ittay Dror [ 12/Jun/08 ]

I'm not talking about just the compilation from groovy, but the whole phase of reading all gradle files, creating project objects for them, creating the dependencies between targets. for a large project, this may take some time.

Comment by Ittay Dror [ 12/Jun/08 ]

In case I wasn't clear, my point was that after creating the DAG, gradle should save the result (DAG and closures of code) to a file (.gradlec), probably next to the root project file. That way, the process is not repeated every build. Compiling the file to .class is another step in optimization.

Comment by Hans Dockter [ 12/Jun/08 ]

I see.

But one has to keep in mind that a Gradle build is more dynamic than XML.

For example:

if (today = 'monday') {
   createTask('x') {
      // do something
   }
} else {
   createTask('x') {
      // do something else
   }
   createTask('y') {
      // do something
   }
}

Depending on the day the dag is different. And I'm sure as people will get acquainted with the dynamicity of Gradle they will use such features more and more. I'm not opposed to your idea, but it is not a silver bullet.

Here a summary of strategies (or outlooks) I see for improving the performance:

  • Caching the compiled gradlefile. With a MD5 or timestamp check it is easy to check invalidation of the cache.
  • Caching the DAG. For very dynamic buildscripts there is no good invalidation strategy for the cache. See above.
  • Trying to improve the performance of Gradle when creating the DAG. After my first profiling attempts I could not find any obvious bottleneck, but profiling Groovy code is much harder than Java code because of the dynamic method dispatch.
  • Right now when doing for example gradle clean install we compile and execute the script two times.
  • Groovy 1.6 will offer significantly better performance but no performance revolution.
  • People who know seem to know what they are talking about predict that with the future JVM support for dynamic languages and further performance tuning Groovy or JRuby will come within the range of Java performance.
Comment by Ittay Dror [ 12/Jun/08 ]

about the second strategy, i think that for those dynamic buildscripts, just offer a flag that can be turned on in the script to prevent saving the dag. i think that a user that uses dynamic creation of targets will be able to figure out that he needs to turn on this flag (and pay the performance penalty).

also, what happens for large projects, with many modules, where i want to build only a specific module? assuming the build uses gradle's configuration injection and inter project dependencies, so that the module cannot be built without realization of the whole multi module graph.

Comment by Hans Dockter [ 13/Jun/08 ]

Or we do it the other way round. You have to set the flag if you want DAG caching. That way we prevent faulty builds.

Right. We always need to build the whole DAG.

First I would implement caching the compiled script and see whether we can improve the performance of the DAG generation (including giving a try to Groovy 1.6-beta2). The next step would be DAG caching.

Comment by Ittay Dror [ 18/Jun/08 ]

Another suggestion is to have gradle always running, with the user triggering targets from the command line. The benefit is that the initial bootstrap is done only once in a while (whenever the user decides to relaunch gradle). This can also be good for CI - the targets can trigger themselves periodically, or when changes have been made to the code. It will also be good for continuing a build after errors have been fixed by the user.

Generated at Wed Jun 30 11:22:13 CDT 2021 using Jira 8.4.2#804003-sha1:d21414fc212e3af190e92c2d2ac41299b89402cf.