[GRADLE-66] Source file dependency analysis for incremental builds Created: 24/Apr/08  Updated: 10/Feb/17  Resolved: 10/Feb/17

Status: Resolved
Project: Gradle
Affects Version/s: None
Fix Version/s: None

Type: New Feature
Reporter: Dave Copeland Assignee: Unassigned
Resolution: Won't Fix Votes: 6

Issue Links:
Related
Related to GRADLE-680 Skip tasks whose inputs have not changed Resolved

 Description   

I would love to see a build system do what make and gcc -M did, which is to perform an analysis on the codebase periodically and generate a dependency graph. This allows incremental builds. For example, if I change one .java file and recompile, both ANT and javac recompile only that file. If other classes use the classes defined in that .java file, there will likely be runtime breakage; they should've been recompiled.

Most solutions involve just doing a clean/recompile, but for a quick debug/code/test cycle, this hampers productivity.

At any rate, this feature would be significant "killer feature" for any build management system.



 Comments   
Comment by Hans Dockter [ 29/Apr/08 ]

On theserverside someone posted

Try Javamake from Sun experimental labs: http://www.experimentalstuff.com/Technologies/JavaMake/index.html
When I used it a several years ago it seemed to deal with most class compilation dependencies, except for multiple top-level classes within a single source file (nested classes were ok).

It provides an ant task that is similar to javac so it is simple to try in an existing ant build.

Have you tried this one?

Comment by Dave Copeland [ 05/May/08 ]

It's currently not downloadable, and the project appears possibly dead. Website looks circa '99, though the copyright notice is 2007. Seems it's not working in jdk1.5, either.

Comment by Ross Black [ 17/Sep/08 ]

I had a quick look at JavaMake... it is currently downloadable via a link from the News page
http://www.experimentalstuff.com/Technologies/downloads.html,id=77a1gsjqa807713hnt8idv
(seems like their download site is continually out of action).

Source code is available, and it is GPL with copyright by Sun. The last release appears to have been Mar2008, so I have no idea whether anyone is looking after it. With my simple tests it works with JDK1.5 but not with 1.6, although this looks like a simple check just needs be added for the classfile version.

If it is being seriously considered we could try contacting the author to find out their plans, or possibility of moving it to a more available site.

Comment by Hans Dockter [ 17/Sep/08 ]

That's funny. I have just today asked Adam, another Gradle developer, about this issue. I would love to have such a functionality implemented. The problem is, how much resources this would consume. If we have to do this all by our own I guess this will consume quite a bit of time. And we have many issue with a higher priority. But if we can integrate an existing tool, that would definitely change the priorities. So yes, we consider this seriously.

Comment by Mattthew Fudge [ 06/Feb/09 ]

In addition I would like like to propose a mechanism that entirely skips processing the dependencies ( for those situations were you know nothing has changed)

A command line option perhaps to say "I know your dependencies have them selves not changed so don't build them again" just use their pre built classpath.

This can be used once you have built your source code once but are having problems with a single subsystem and perhaps which to debug it or just modify its code a bit. (I't would still have to analyse all the settings.gradle but would warn of changes in those and would not attempt to process the build.gradle files.

This would make for very fast builds while you are working on one subproject (debugging test cases, debugging the build it's self (such as code generators which arn't working as expected))

Comment by Ross Black [ 06/Feb/09 ]

Javamake has been reborn at http://kenai.com/projects/jmake.
It seems to be in the very early stages of setup, and the licence is still GPL.

Comment by Hans Dockter [ 27/Jun/09 ]

We would like to get this feature to be implemented soon. The question is what is the best approach. I'm wondering about a couple of issues:

Is jmake doing a better job than the Eclipse JDT compiler?

What does it mean that jmake is GPL?

What are the shortcomings of Ant Depend? Dave Copeland said the following:

Currently, there is no tool I am aware of that properly determines the dependencies between java class files and conditionally builds only (and all of!) what's needed. Neither ANT nor javac does this, and ANT's <depend> task routinely is unable to.

What about the Ant javac's dependency tracking option? It is said to work with the classic compiler and jikes. If used with the classic compiler, is it as good as the alternatives.

Eclipse JDT seems to be very easy to integrate. Either programmatically or via and adapter to Ant javac's. See for example: http://publib.boulder.ibm.com/infocenter/radhelp/v6r0m1/index.jsp?topic=/org.eclipse.jdt.doc.isv/guide/jdt_api_compile.htm

Comment by Steve Appling [ 24/Jul/09 ]

I would love to see support for better incremental builds, but at the very least we should support ant's depend.

Its main shortcoming is that it only works off of the class files, so any time the compiler gets "smart" and inlines something, the dependency relationship is lost. This can happen with a public final static field. It does handle many other cases, though, so we have tended to use it with Ant.

I don't have time to pursue JDT or Javamake right now, but I will add basic support for ant's depend to the current Compile task. The current compile.options.dependencyTracking option will not work for JDK 1.3 or later, only for Jikes, JDK1.1, or JDK1.2. This must be done differently for more modern compilers.

Comment by Hans Dockter [ 27/Jul/09 ]

After looking into this, I would assume that the depends option of the Javac task only means the very basic dependency tracking javac does anyway. Nothing sophisticated. After playing around with the Eclipse JDT I have come to the conclusion that it also does not offer incremental compile. The Eclipse IDE has functionality on top of this to make it work. Our plan is the following:

The JDK 6 javac provides an AST to be analyzed. So Adam plans to write an incremental compile enhancement based on this. If people don't use the JDK 6 javac we will fallback on Ant depend's task. So it would be cool if you if you could add basic support for ant's depend to the current Compile task.

Comment by Steve Appling [ 27/Jul/09 ]

Ant's javac depend only works with JDK1.1, JDK1.2, or Jikes. If you are using another compiler, it will give an error.

Javac by itself doesn't do any dependency tracking - it only compares the timestamp of source files to the timestamps of the class files it would generate. It does nothing with inter-class dependencies.

Do we need to still support the old depend property (called dependencyTracking in current Gradle)? I would like to just remove support for the old one and only support a depend task. I have implemented this and it looks like:

compile.options.depend() // to just turn on dependency tracking with defaults (closure=false, cache=true)

or you can get access to more params:
compile {
options.depend( closure:true, useCache:false, dump:true, classpath:'more stuff in classpath')
}

If you want to keep the old property, I would like to call it something other than depend or dependencyTracking. I think it would need to indicate it was only for older compilers (perhaps 'oldCompilerDepend') but I would prefer to just not support it.

Comment by Hans Dockter [ 28/Jul/09 ]

I could imagine that the depend property for JDK1.1 means, that if you have the classes Car->Engine and you compile Car, that Engine is compiled as well. This is the standard behavior of the current Javac and is also some form of dependency tracking. I don't think we need to support this property.

Comment by Yair Halevi [ 03/Jun/12 ]

Has there been any progress on this issue? The last update is 3 years ago, was anything added since perhaps?

Comment by Benjamin Muschko [ 15/Nov/16 ]

As announced on the Gradle blog we are planning to completely migrate issues from JIRA to GitHub.

We intend to prioritize issues that are actionable and impactful while working more closely with the community. Many of our JIRA issues are inactionable or irrelevant. We would like to request your help to ensure we can appropriately prioritize JIRA issues you’ve contributed to.

Please confirm that you still advocate for your JIRA issue before December 10th, 2016 by:

  • Checking that your issues contain requisite context, impact, behaviors, and examples as described in our published guidelines.
  • Leave a comment on the JIRA issue or open a new GitHub issue confirming that the above is complete.

We look forward to collaborating with you more closely on GitHub. Thank you for your contribution to Gradle!

Comment by Benjamin Muschko [ 10/Feb/17 ]

Thanks again for reporting this issue. We haven't heard back from you after our inquiry from November 15th. We are closing this issue now. Please create an issue on GitHub if you still feel passionate about getting it resolved.

Generated at Wed Jun 30 11:21:07 CDT 2021 using Jira 8.4.2#804003-sha1:d21414fc212e3af190e92c2d2ac41299b89402cf.