[GRADLE-2887] Handle case where jar is not available in repository due to licensing constraints Created: 15/Sep/13  Updated: 10/Feb/17  Resolved: 10/Feb/17

Status: Resolved
Project: Gradle
Affects Version/s: None
Fix Version/s: None

Type: Bug
Reporter: Gradle Forums Assignee: Unassigned
Resolution: Won't Fix Votes: 3


 Description   

I have my projects that depend (directly and transitively) on Log4j 1.2.15.

Because of the problem described here:
[1]http://onemanwenttomow.wordpress.com/...
and here:
[2]http://unitstep.net/blog/2009/05/18/r...

adding the depedency to log4j:log4j:1.2.15 in my build files make Gradle 1.0-milestone-7 fail because of the changes made in the resolution of the missing JMX reference implementation JARs (milestone-5 and 6 simply skipped the missing JARs).

So, what I would like to do is to add those missing JARs (jmxtools-1.2.1.jar and jmxri-1.2.1.jar) to my flatDir repository provided with the Gradle project. However, even if my build.gradle says:

repositories {
flatDir dirs: "${rootProject.projectDir}/repo"
mavenRepo url: "${main_repo}"
mavenLocal ()
}

and I put those jars in "${rootProject.projectDir}/repo", Gradle ignores them and fails with (for instance):

Could not resolve all dependencies for configuration ':MyProject:compile'.
Cause: Artifact 'com.sun.jdmk:jmxtools:1.2.1@jar' not found

Changing my dependency declaration towards log4j adding (for instance) "exclude: module 'jmxtools'" is not a viable solution, because log4j 1.2.15 is also required transitively by other dependencies, so I would need to add that exclusion to ALL the dependencies that I know that require Log4j... It's ugly and unmaintainable.

So, my question is: is there a way to tell Gradle to pick up those missing JARs from the "repo" directory?

Actually, this is not the only case where I would need Gradle to look in that directory when it does not find something in the Maven repository. For instance, there are dependencies for which the source/javadoc archives are not in the Maven repository, but I added them in the "repo" dir... However, that is another case in which Gradle just ignores them and says it can't find source/javadoc archives
----------------------------------------------------------------------------------------
[1] http://onemanwenttomow.wordpress.com/2007/12/31/maven2-log4j-and-jmx-dependencies/
[2] http://unitstep.net/blog/2009/05/18/resolving-log4j-1215-dependency-problems-in-maven-using-exclusions/



 Comments   
Comment by Gradle Forums [ 15/Sep/13 ]

Here are a few options:

You can exclude the jmxtools and jmxri globally (this works for both direct and transitive dependencies):

configurations.all {
exclude group: 'com.sun.jdmk', module: 'jmxtools'
exclude group: 'com.sun.jmx', module: 'jmxri'
}

Another option would be to include the jars in a repo directory, but rather than using a flat directory repository, arrange them as a maven repository. You can then define a composite maven repo, that searches in multiple locations:

repositories {
mavenRepo url: "${main_repo}", artifactUrls: ["${rootProject.projectDir}/repo"]
mavenLocal ()
}

Comment by Gradle Forums [ 15/Sep/13 ]

Hi Adam, both your solutions are very interesting.
However, the second one sounds like it's the best one for me.
I have a problem, though. After changing the repositories definition like you said, I now get the following error:

Could not resolve group:javax.mail, module:mail, version:1.4.3.
Required by:
com.cardinis.cardinis.cmod:COMMON:Trunk
com.cardinis.cardinis.cmod:COMMON:Trunk > log4j:log4j:1.2.15
Cause: unsupported protocol: 'file'

The stacktrace is (I'm reporting just the cause and some lines):

Caused by: java.lang.IllegalStateException: unsupported protocol: 'file'
at org.apache.commons.httpclient.protocol.Protocol.lazyRegisterProtocol(Protocol.java:149)
at org.apache.commons.httpclient.protocol.Protocol.getProtocol(Protocol.java:117)
at org.apache.commons.httpclient.HttpHost.<init>(HttpHost.java:107)
at org.apache.commons.httpclient.HttpMethodBase.setURI(HttpMethodBase.java:280)
at org.apache.commons.httpclient.HttpMethodBase.<init>(HttpMethodBase.java:220)
at org.apache.commons.httpclient.methods.GetMethod.<init>(GetMethod.java:89)
at org.gradle.api.internal.artifacts.repositories.transport.http.HttpResourceCollection.getResource(HttpResourceCollection.java:84)
at org.gradle.api.internal.artifacts.repositories.transport.http.HttpResourceCollection.getResource(HttpResourceCollection.java:58)
at org.gradle.api.internal.artifacts.repositories.ResourceCollectionResolver.getResource(ResourceCollectionResolver.java:121)
at org.gradle.api.internal.artifacts.repositories.ResourceCollectionResolver.findStaticResourceUsingPattern(ResourceCollectionResolver.java:91)
at org.gradle.api.internal.artifacts.repositories.ResourceCollectionResolver.findResourceUsingPattern(ResourceCollectionResolver.java:76)
at org.apache.ivy.plugins.resolver.AbstractPatternsBasedResolver.findResourceUsingPatterns(AbstractPatternsBasedResolver.java:93)
at org.gradle.api.internal.artifacts.repositories.MavenResolver.findArtifactRef(MavenResolver.java:157)

What sounds strange for me is that javax.mail:mail:1.4.3 should be found in the remote repository, not in the repo directory.

Any suggestion?

Comment by Gradle Forums [ 15/Sep/13 ]

Should I open an issue in JIRA for the problem I'm encountering now?

Comment by Gradle Forums [ 15/Sep/13 ]

I did some experiments and I'm convinced there's a bug here. With 1.0-milestone-6 and below that error is not produced, although the pick up of the JAR from the repo dir still doesn't work.
I opened GRADLE-2046

Comment by Gradle Forums [ 15/Sep/13 ]

The actual bug is that you can't mix http and file urls in the same maven repository. Unfortunately we are not catching this early and reporting it (like we do with ivy repositories).

I've updated GRADLE-2046 to reflect this.

Comment by Gradle Forums [ 15/Sep/13 ]

Thank you Daz. But what will you do? Fix it in order to report early or make the mixed case work? It would be extremely usefule if the mixed scenario worked (as explained in my original question here).

Comment by Gradle Forums [ 15/Sep/13 ]

Yes, we'd like you to be able to mix file and http urls in a single repository. Originally we didn't think this would be a common use case, but yours is a valid example.

Please vote for this issue, and we'll get to it as priorities dictate. We may well do the trivial thing first (report early) and address the underlying issue a little later.

Comment by Gradle Forums [ 15/Sep/13 ]

Unfortunately I can't vote, since the issue is mine :-P
Anyway, thanks for the info.
Mauro.

Comment by Gradle Forums [ 15/Sep/13 ]

As an additional example, please consider the case when in the remote repository there is the jar but not the javadoc/source bundle and you want to provide one through the use of a local directory.

Comment by Gradle Forums [ 15/Sep/13 ]

Just to clarify what's going on here:

1. If Gradle finds a jar file in a repository, but no meta-data file (ivy/pom), then Gradle will generate a module descriptor based on the presence of this jar. (This descriptor contains no dependency information, and a single 'jar' artifact). This is exactly how the "flatdir" repository works.
2. Gradle prefers to use a module with a real meta-data file than one with a generated module descriptor. So if Gradle finds only `jmxri-1.2.1.jar` in one repository, but `jmxri-1.2.1.pom` in a second, it will use the second repository to provide the module. Gradle considers the presence of a meta-data file to be more authoritative.
3. MavenCentral contains the POM, but not the jarfile for JMX. So that screws up our logic, since the jar-only repository is actually the better one to use. This is not great behaviour: we have an issue to remind us to improve this (GRADLE-2034)

A solution that should currently work would be very similar to your original design, but use a local `maven` repository instead of a `flatdir` repository. This local repository would contain pom files as well as artifacts, so there would be no need to search in subsequent repositories for the JMX module.

Something like:

repositories {
maven { url "${rootProject.projectDir}/repo" }
maven { url "${main_repo}" }
mavenLocal ()
}

PS - do you really need mavenLocal() here?

Comment by Gradle Forums [ 15/Sep/13 ]

To be honest, including source/javadoc artifacts is the "valid example" I am referring to.

I think in the case of Log4j including JMX unnecessarily, you have 2 better (imo) alternatives:

1. Use global excludes if you don't require the JMX functionality of Log4j
2. Create a fully-fledged local Maven repository to provide the JMX modules (including the pom files) if you do require JMX functionality

Comment by Gradle Forums [ 15/Sep/13 ]

I see. However, please consider that if you want to be more "agile" and you don't know in advance whether you'll need JMX functionality or not, having to put up a whole local repository (including POMs) is not as straightforward as it could be having a local repository with just the missing JARs... After all, the mavenRepo() artifactUrls parameter aim sounds like to be exactly this...

Comment by Gradle Forums [ 15/Sep/13 ]

I don't know if I really need maveLocal()... I thought it could be useful to speed up things when I have artifacts in the local Maven cache... however the user guide is not very exhaustive on this topic...

Anyway, thank you for clarifying how things work. However, from what you write it seems like the typical case is that Gradle finds both POMs and JARs in one repository OR that it finds JARs first and then looks for POMs on other repositories... if it finds them, then the secondary repositories are used. However the documentation of mavenRepo() seems to suggest that the typical case is that Gradle find both JARs and POMs in one repository OR that it finds POMs in the first repository, than looks for JARs in other repositories. So the case of jmxri or jmxtools seems to fit perfectly in this second case. I know it is something "unusual", however it's real and since it rises when you require log4j (which is very common) I think it should be taken care of.
Thank you very much for your support!

Comment by Gradle Forums [ 15/Sep/13 ]

I think you are confusing the term "repository" with the url and artifactUrls properties of a single repository.

A single maven repository to Gradle consists of a single `url` that will be searched for pom files and jar files, and a possible set of additional `artifactUrls` that will be searched for jar files only. Together these urls are treated as a single, self-contained unit by Gradle, called a repository.

My description above refers to multiple, separate repositories. The maven repository documentation you are referring to is discussing multiple urls for the same repository. I hope this clarifies things a bit.

Comment by Gradle Forums [ 15/Sep/13 ]

PS: You should remove mavenLocal() unless you are actually consuming artifacts that you publish locally using maven. Any speedup you gain from using it will be negligable, but it can serve to make your build less portable.

Gradle knows about the artifacts in the local maven repository, and will attempt to use them where possible. You can read about artifact reuse and other cache concepts in [1]this new section of the user guide.
----------------------------------------------------------------------------------------
[1] http://gradle.org/docs/nightly/userguide/dependency_management.html#sec:dependency_cache

Comment by Gradle Forums [ 15/Sep/13 ]

Please note GRADLE-2034 was closed without covering this use case. Could you please open a new issue?

Comment by Benjamin Muschko [ 15/Nov/16 ]

As announced on the Gradle blog we are planning to completely migrate issues from JIRA to GitHub.

We intend to prioritize issues that are actionable and impactful while working more closely with the community. Many of our JIRA issues are inactionable or irrelevant. We would like to request your help to ensure we can appropriately prioritize JIRA issues you’ve contributed to.

Please confirm that you still advocate for your JIRA issue before December 10th, 2016 by:

  • Checking that your issues contain requisite context, impact, behaviors, and examples as described in our published guidelines.
  • Leave a comment on the JIRA issue or open a new GitHub issue confirming that the above is complete.

We look forward to collaborating with you more closely on GitHub. Thank you for your contribution to Gradle!

Comment by Benjamin Muschko [ 10/Feb/17 ]

Thanks again for reporting this issue. We haven't heard back from you after our inquiry from November 15th. We are closing this issue now. Please create an issue on GitHub if you still feel passionate about getting it resolved.

Generated at Wed Jun 30 12:34:08 CDT 2021 using Jira 8.4.2#804003-sha1:d21414fc212e3af190e92c2d2ac41299b89402cf.