[GRADLE-2055] Tooling API: Expose dependencies that belong to custom configurations Created: 20/Jan/12  Updated: 10/Feb/17  Resolved: 10/Feb/17

Status: Resolved
Project: Gradle
Affects Version/s: 1.0-milestone-7
Fix Version/s: None

Type: Bug
Reporter: Denis Zhdanov Assignee: Unassigned
Resolution: Won't Fix Votes: 9


 Description   

Expose information about java dependencies that belong to the non-standard configuration via the tooling api.

Consider the following build file:

build.gradle
apply plugin: 'java'

sourceCompatibility = 1.5
version = '1.0'
jar {
    manifest {
        attributes 'Implementation-Title': 'Gradle Quickstart', 'Implementation-Version': version
    }
}

configurations {
    provided
}

repositories {
    mavenCentral()
}

dependencies {
    compile group: 'commons-collections', name: 'commons-collections', version: '3.2'
    testCompile group: 'junit', name: 'junit', version: '4.+'
    provided group: 'commons-io', name: 'commons-io', version: '1.4'
}

sourceSets {
    main {
        compileClasspath += configurations.provided
    }
}

The tooling api doesn't provide any information about the 'commons-io' dependency here. The following program illustrates that:

package org.jetbrains.plugins.gradle;

import org.gradle.tooling.GradleConnector;
import org.gradle.tooling.ModelBuilder;
import org.gradle.tooling.ProjectConnection;
import org.gradle.tooling.model.idea.BasicIdeaProject;
import org.gradle.tooling.model.idea.IdeaDependency;
import org.gradle.tooling.model.idea.IdeaModule;
import org.gradle.tooling.model.idea.IdeaProject;

import java.io.File;

/**
 * @author Denis Zhdanov
 * @since 12/8/11 5:07 PM
 */
public class GradleStartClass {

  private static final String GRADLE_TO_USE = "/home/denis/dev/gradle/gradle-1.0-milestone-7";
  private static final String GRADLE_PROJECT_PATH = "/home/denis/dev/gradle/gradle-1.0-milestone-7/samples/java/quickstart";

  public static void main(String[] args) throws InterruptedException {
    showDependencies();
  }

  
  private static void showDependencies() {
    GradleConnector connector = GradleConnector.newConnector();
    connector.useInstallation(new File(GRADLE_TO_USE));
    connector.forProjectDirectory(new File(GRADLE_PROJECT_PATH));
    ProjectConnection connection = connector.connect();
    ModelBuilder<? extends IdeaProject> modelBuilder = connection.model(IdeaProject.class);
    IdeaProject project = modelBuilder.get();
    System.out.println("-----------------> Listing modules <----------------");
    for (IdeaModule module : project.getModules()) {
      System.out.printf("    -----------------> Module %s <----------------%n", module.getName());
      for (IdeaDependency dependency : module.getDependencies()) {
        System.out.printf("Dependency: %s, scope: %s%n", dependency, dependency.getScope());
      }
    }
    System.out.println("-----------------> Finishing <----------------");
  }
}
Output
-----------------> Listing modules <----------------
    -----------------> Module quickstart <----------------
Dependency: IdeaLibraryDependency{file=/home/denis/.gradle/caches/artifacts-7/artifacts/56865c09425cc6ecda8606d7e6035ceb/commons-collections/commons-collections/3.2/jar/commons-collections-3.2.jar, source=/home/denis/.gradle/caches/artifacts-7/artifacts/56865c09425cc6ecda8606d7e6035ceb/commons-collections/commons-collections/3.2/source/commons-collections-3.2-sources.jar, javadoc=null, exported=true, scope='IdeaDependencyScope{scope='COMPILE'}'}, scope: IdeaDependencyScope{scope='COMPILE'}
Dependency: IdeaLibraryDependency{file=/home/denis/.gradle/caches/artifacts-7/artifacts/56865c09425cc6ecda8606d7e6035ceb/junit/junit/4.10/jar/junit-4.10.jar, source=/home/denis/.gradle/caches/artifacts-7/artifacts/56865c09425cc6ecda8606d7e6035ceb/junit/junit/4.10/source/junit-4.10-sources.jar, javadoc=null, exported=false, scope='IdeaDependencyScope{scope='TEST'}'}, scope: IdeaDependencyScope{scope='TEST'}
Dependency: IdeaLibraryDependency{file=/home/denis/.gradle/caches/artifacts-7/artifacts/56865c09425cc6ecda8606d7e6035ceb/org.hamcrest/hamcrest-core/1.1/jar/hamcrest-core-1.1.jar, source=/home/denis/.gradle/caches/artifacts-7/artifacts/56865c09425cc6ecda8606d7e6035ceb/org.hamcrest/hamcrest-core/1.1/source/hamcrest-core-1.1-sources.jar, javadoc=null, exported=false, scope='IdeaDependencyScope{scope='TEST'}'}, scope: IdeaDependencyScope{scope='TEST'}
-----------------> Finishing <----------------

We expect to get information about the dependency here. Also note that here the custom configuration name ('provided') is mapped to the IntelliJ IDEA's 'Provided' scope. However, it's possible that there are other configurations which names do not match as well and that tweak compileClasspath/runtimeClasspath. It would be cool to get that information from the tooling api as well

This is based on the following ticket - IDEA-80118.

 Comments   
Comment by Szczepan Faber [ 24/Apr/12 ]

Gradle IDE plugins do not include custom configurations by default because we don't know what is the meaning of the configuration.

There's a idea DSL that one can use to inform idea plugin about the custom configurations: http://gradle.org/docs/current/dsl/org.gradle.plugins.ide.idea.model.IdeaModule.html#org.gradle.plugins.ide.idea.model.IdeaModule:scopes

Basically, if you use this DSL to configure idea in the build.gradle the tooling api will take advantage of this information.

Comment by Denis Zhdanov [ 02/May/12 ]

Hi Szczepan,

I see the rationale but it seems insufficient from my point of view. Couple of notes:

  • tooling api doesn't restrict us from doing that - IdeaDependency.getScope() -> IdeaDependencyScope.getScope() which is a raw String, i.e. 'provided' may be returned from there as well as any other configuration name;
  • tooling api doesn't provide the same feature set as the CLI (dependencies for the custom configurations are not visible to the api clients at all);

From the other hand we already match raw string 'dependency scope' to the IJ-supported one. Hence, there is no problem to support one more matching rule. Also we can report a nice error message to the end-user if we can't perform the match. It's better than just skipping the dependency silently.

I suggest to reopen the ticket.

Comment by Denis Zhdanov [ 02/May/12 ]

Btw, I don't have enough rights at the gradle tracker to change a ticket's name. Can I ask you to rename it to something like 'Tooling API: Expose dependencies that belong to custom configurations'

Comment by Szczepan Faber [ 07/May/12 ]

Hey Denis,

The request start making sense to me. Basically, what you are asking for is that you would like a view of all project's dependencies similar to the one that is shown when one runs 'gradle dependencies'?

Why do you need this information? Don't worry - I'm not opposing the feature - I just want to find out more

Comment by Szczepan Faber [ 07/May/12 ]

Hey Denis,

The request start making sense to me. Basically, what you are asking for is that you would like a view of all project's dependencies similar to the one that is shown when one runs 'gradle dependencies'?

Why do you need this information? Don't worry - I'm not opposing the feature - I just want to find out more

Comment by Denis Zhdanov [ 09/May/12 ]

Hi,

We have a number of reports from our customers with complains that 'provided' dependencies are not picked up by the IJ gradle integration. It seems that it's worth to address that because either IJ or maven have 'provided' scope and nothing prevents us from correctly configuring such dependencies.

Another reason is that I would like to see the tooling api at least as feature-rich as the CLI interface.

Comment by Szczepan Faber [ 18/May/12 ]

Ok, so it seems the use case is that you would like to know if a certain dependency should be deployed/packaged or not, for example for web server deployment. Can you provide links to the youtrack with the user reports?

Comment by Bob Glamm [ 27/Jul/12 ]

Just wanted to add my 2 cents on this issue:

Anyone that develops a WAR using the Servlet API immediately runs into this problem. Containers commonly contain an implementation of "javax.servlet:javax.servlet-api:XXX", so it is unnecessary to bundle that dependency with the WAR, but the WAR still needs to be able to compile against it. Maven's "provided" scope takes care of this quite conveniently, and as much as I've used Gradle configurations to give me the same functionality in the last projected I converted, I wonder if it would be useful to have "provided" added as a first-level configuration type in Gradle (in addition to "compile" and "runtime")?

The fix is definitely worth adding. FWIW, this doesn't hit Eclipse as hard due to the way the Eclipse plugin for Gradle handles the .classpath file - it is possible to inject these "provided" dependencies into the task that builds the .classpath file.

Comment by wujek srujek [ 02/Aug/12 ]

From the first post:
'Gradle IDE plugins do not include custom configurations by default because we don't know what is the meaning of the configuration.'
What does it mean you don't know? Why do you even care? Why can't you provide the information? There are honest questions, please clue me in a bit.

My case is not the provided scope (I will get there soon enough, though...), it is the following: I defined custom sourceSets, and have dependencies for these source sets - in my case, I use a neo4j test jar in the integrationTestCompile scope. It works well on the command line. In IDEA, I just can't get it to work, because they don't know anything about this scope, as you don't tell them the info (the issue is here: http://youtrack.jetbrains.com/issue/IDEA-89638), and so the project is oblivious to the jar, and doesn't compile.

To reiterate what I said in that issue: this prevents me from using IDEs, as now parts of my code (src/integrationTest/groovy) doesn't compile, as it is not possible to import the dependency. I can force-add it manually and assign to, say, test scope, but this is no good - now, src/test/groovy can use the classes as well and by doing so it will compile in the IDE but not on the command line). This has always a cause of a lot of issues in my teams when using eclipse and maven (not the scopes, but rather the facts that IDEs and the CLI behave differently). If there is any other solution to the problem, please tell me what it is.

Comment by Eduard Dudar [ 09/Aug/12 ]

Bob, providedCompile/providedRuntime will work for you in case of 'war' plugin as far as you've mentioned servlets-api. 26.4. Dependency management. The War plugin adds two dependency configurations: providedCompile and providedRuntime.

Comment by Szczepan Faber [ 28/Aug/12 ]

'Gradle IDE plugins do not include custom configurations by default because we don't know what is the meaning of the configuration.'
What does it mean you don't know?

We don't know if custom configuration contains libraries that should be on the classpath. Therefore we cannot just lump all the configurations by default onto the classpath. Does it make sense? If you have some custom configuration and you know that it should be on the classpath, you can very easily tell Gradle about it. This will make the libraries of this configuration available in the IDEA classpath, etc.

To reiterate what I said in that issue: this prevents me from using IDEs

Take a look at the documentation: http://gradle.org/docs/current/dsl/org.gradle.plugins.ide.idea.model.IdeaModule.html#org.gradle.plugins.ide.idea.model.IdeaModule:scopes

You can tell Gradle IDE plugins about your custom configurations and hence work in the IDE. The JetBrains IDEA plugin will also use this information. This should let you compile your project in IDE easily.

Comment by wujek srujek [ 28/Aug/12 ]

I don't think it helps. As far as I understand, this just provides a mapping from 'custom' Gradle scopes to 'standard' IDEA scopes. Fine, it will sometimes help.
My problem is different - I must use one of the standard IDEA scopes, like map 'integration-test' to 'test', and suddenly, in IDEA, test compilation / runtime classpath has all the jars from 'integration-test', which I don't want for various reasons, like the one I mentioned earlier - it can happen that people write test code and by accident use classes from jars that were mapped from 'integration-test' to 'test', and it compiles and runs in IDEA, but suddenly not on Jenkins or the CLI. It is a cause of many spurious build failures. (I honestly don't know how I can explain this point more easily, and many times I have to repeat the same thing.)
Agreed, you might say we are doing things wrong, but not everybody has a team of solely senior developers who always know what they are doing (do they?) and know what classes they can use (because they are really there) and which they can't (because they are there by pure mapping magic); I work in teams with juniors as well, and there are problems - sometimes they don't know, sometimes such code slips through our review process, whatever. People make mistakes, and in this particular case the tools are of no help, they make things more confusing. Just explain to a junior that the classpath is different in IDEA than on the CLI and that there is some mapping somewhere because of something (here: Gradle not wanting to publish the custom scopes) - you would loose me on the 'classpath' word when I was a junior.

The configuration that you mention seems to be for the gradle-idea-plugin. As far as I know, there are two ways to import a Gradle project in IDEA: you use the gradle-idea-plugin that generates the *.iml files (btw, last time I checked Gradle used the deprecated IDEA project structure) and open that, or you import an 'external model' in IDEA, choose Gradle, and it creates it's files. I sincerely am not sure if it uses anything that you configure for the gradle-idea-plugin.

Comment by Szczepan Faber [ 28/Aug/12 ]

I sincerely am not sure if it uses anything that you configure for the gradle-idea-plugin.

The IDEA JetBrains plugin uses the tooling api which behind the hood uses the gradle idea plugin. So whatever you configure in the idea plugin in your build.gradle should be reflected in JetBrains IDEA plugin.

My problem is different - I must use one of the standard IDEA scopes, like map 'integration-test' to 'test', and suddenly, in IDEA

Right, the classpaths / scopes in IDEs are not robust enough to accommodate flexible model of custom classpath scopes. I don't like this just as much as you do. I don't think it is a Tooling API problem. This is not even a Gradle problem - there are other build tools out there that also allow custom classpaths (for example: ant).

Thing is that this is not that painful for most teams. They either accept the fact that classpaths gets flattened in IDE - it is a low-risk problem because it gets detected when you run the build from the command line. If the team does not like the classpath inaccuracy they can avoid custom configurations by using separate projects to some extent. So there's no big push to the IDEs space to start providing custom scopes (at least this is my feeling).

I do appreciate your feedback and that you want to push us to improve the tooling API.

Comment by wujek srujek [ 22/Sep/12 ]

Hi Szczepan. I eventually got the time to try the mapping of custom scopes out - it works fine, thanks for the tip. The only thing that doesn't work for me so far is that IDEA doesn't recognize the languageLevel in the idea.project that I configure. I will try ask them about it.

A comment to one of your previous statements:

"We don't know if custom configuration contains libraries that should be on the classpath. Therefore we cannot just lump all the configurations by default onto the classpath. Does it make sense? If you have some custom configuration and you know that it should be on the classpath, you can very easily tell Gradle about it. This will make the libraries of this configuration available in the IDEA classpath, etc."

I know you don't know, but nobody tells you to do any guessing - Gradle should just be able to tell there are some custom configurations, and that they have jars as compile / runtime deps. Some configurations are configured in the build to extend others (like provided is used to extend runtime, or whatever). If the build doesn't use the configurations, their fault, IMHO. Let's call them abstract. There are other configurations, like integrationTest, that don't extend others, but rather are 'concrete' - they are used on their own. Such configurations also have source code, like, again, integrationTest. That sources get compiled, get used in classpaths for tests or any other tasks. Whatever. The person who writes the build decides what is done with such custom scopes, Gradle doesn't do any guessing as to whether put them on the classpath or not, it just doesn't. I don't think Gradle should care how they are used, it just needs to announce them.
Please explain where I am wrong in my thinking.

Comment by Benjamin Muschko [ 15/Nov/16 ]

As announced on the Gradle blog we are planning to completely migrate issues from JIRA to GitHub.

We intend to prioritize issues that are actionable and impactful while working more closely with the community. Many of our JIRA issues are inactionable or irrelevant. We would like to request your help to ensure we can appropriately prioritize JIRA issues you’ve contributed to.

Please confirm that you still advocate for your JIRA issue before December 10th, 2016 by:

  • Checking that your issues contain requisite context, impact, behaviors, and examples as described in our published guidelines.
  • Leave a comment on the JIRA issue or open a new GitHub issue confirming that the above is complete.

We look forward to collaborating with you more closely on GitHub. Thank you for your contribution to Gradle!

Comment by Benjamin Muschko [ 10/Feb/17 ]

Thanks again for reporting this issue. We haven't heard back from you after our inquiry from November 15th. We are closing this issue now. Please create an issue on GitHub if you still feel passionate about getting it resolved.

Generated at Wed Jun 30 12:11:35 CDT 2021 using Jira 8.4.2#804003-sha1:d21414fc212e3af190e92c2d2ac41299b89402cf.